# Physics-based Shadow Image Decomposition for Shadow Removal

Hieu Le and Dimitris Samaras H. Le is with Amazon Robotics, North Reading, MA
01864. E-mail<EMAIL_ADDRESS>This work is done prior to joining
Amazon. D. Samaras is with the Department of Computer Science, Stony Brook
University, Stony Brook, NY 11794. E-mail<EMAIL_ADDRESS>
###### Abstract

We propose a novel deep learning method for shadow removal. Inspired by
physical models of shadow formation, we use a linear illumination
transformation to model the shadow effects in the image that allows the shadow
image to be expressed as a combination of the shadow-free image, the shadow
parameters, and a matte layer. We use two deep networks, namely SP-Net and
M-Net, to predict the shadow parameters and the shadow matte respectively.
This system allows us to remove the shadow effects from images. We then employ
an inpainting network, I-Net, to further refine the results. We train and test
our framework on the most challenging shadow removal dataset (ISTD). Our
method improves the state-of-the-art in terms of mean absolute error (MAE) for
the shadow area by 20%. Furthermore, this decomposition allows us to formulate
a patch-based weakly-supervised shadow removal method. This model can be
trained without any shadow- free images (that are cumbersome to acquire) and
achieves competitive shadow removal results compared to state-of-the-art
methods that are trained with fully paired shadow and shadow-free images.
Last, we introduce SBU-Timelapse, a video shadow removal dataset for
evaluating shadow removal methods.

###### Index Terms:

shadow removal, physical illumination model, matting, deep neural network

##  1 Introduction

Shadows are present in most natural images. They form as a result of complex
physical interactions between light sources, geometry, and materials of the
objects in the scene. Analyzing shadows [1, 2] gives cues about the physical
properties of objects [3], illumination conditions [4, 5, 6], scene geometry
[7, 8], and object motions[9, 10]. Realistic shadow manipulation is an
integral part of media editing [11, 12] and can greatly improve performance on
various computer vision tasks [13, 14, 15, 16, 17, 18, 19, 20, 21, 22].
Therefore, methods for detecting and removing shadows have been extensively
studied in computer vision. They are getting increased attention in recent
years, following the rapid development of deep learning and increasing
interest in image generation [23, 7] and augmented virtual reality [24].

Figure 1: Shadow Removal via Shadow Image Decomposition. A shadow-free image
$I_{\textrm{shadow-free}}$ can be expressed in terms of a shadow image
$I_{\textrm{shadow}}$, a relit image $I_{\textrm{relit}}$ and a shadow matte
$\alpha$. The relit image is a linear transformation of the shadow image. The
two unknown factors of this system are the shadow parameters $(w,b)$ and the
shadow matte layer $\alpha$. We use two deep networks to estimate these two
unknown factors.

Early shadow removal work was based on physical shadow models [25]. A common
approach is to formulate the shadow removal problem using an image formation
model, in which the image is expressed in terms of material properties and a
light source-occluder system that casts shadows. Hence, a shadow-free image
can be obtained by estimating the parameters of the source-occluder system and
then reversing the shadow effects on the image [26, 27, 28, 29]. These methods
relight the shadows in a physically plausible manner. However, estimating the
correct solution for such illumination models is non-trivial and requires
considerable processing time or user assistance[30, 11].

Alternatively, recently published large-scale datasets [31, 32, 33] allow the
use of deep learning methods for shadow removal. In these cases, a network is
trained in an end-to-end fashion to map the input shadow image to a shadow-
free image. The success of these approaches shows that deep networks can
effectively learn transformations that relight shadowed pixels. However, the
actual physical properties of shadows are ignored although they contain strong
priors to localize and modify shadowed pixels. Without any shadow priors,
there is no guarantee that the networks would learn physically plausible
transformations. Moreover, there are still well known issues with images
generated by deep networks: results tend to be blurry [34, 35] and/or contain
artifacts [36]. How to improve the quality of generated images is an active
research topic [7, 37].

In this work, we propose a novel method for shadow removal that takes
advantage of both shadow illumination modelling and deep learning. Following
early shadow removal works, we propose to use a simplified physical
illumination model to define the mapping between shadow pixels and their
shadow-free counterparts. Our main idea is to use deep networks to estimate
the parameters for such a physical illumination model. We show that this
approach has multiple advantages. The framework removes shadows in a
physically plausible manner, avoiding the blurring and undesired artifacts
commonly introduced by deep-networks. Moreover, the shadow removal mapping is
simpler and easier to constrain since it is strictly defined by the simplified
illumination model. The proposed framework achieves the state-of-the-art
shadow removal performance via training on paired data, and also can be
trained effectively using only unpaired shadow and shadow-free patches,
obviating the need of shadow-free image supervision.

Specifically, our proposed illumination model is a linear transformation
consisting of a scaling factor and an additive constant - per color channel -
for the whole umbra area of the shadow. These scaling factors and additive
constants are the parameters of the model. The illumination model plays a key
role in our method: with correct parameter estimates, we can use the model to
remove shadows from images. Since these parameters are used in a single common
linear model to relight the shadows for all shadowed pixels in the input
image, this approach ensures global consistency for our shadow removal method.
We propose training a deep network (SP-Net) to automatically estimate the
parameters of the shadow model. Through training, SP-Net learns a mapping
function from input shadow images to illumination model parameters.
Furthermore, we use a shadow matting technique [11, 28, 30] to handle the
penumbra area of the shadows. We incorporate our illumination model into an
image decomposition formulation [38, 11], where the shadow-free image is
expressed as a combination of the shadow image, the parameters of the shadow
model, and a shadow density matte. This image decomposition formulation allows
us to reconstruct the shadow-free image, as illustrated in Fig. 1. The shadow
parameters $(w,b)$ represent the transformation from the shadowed pixels to
the illuminated pixels. The shadow matte represents the per-pixel linear
combination of the relit image and the shadow image, which results to the
shadow-free image. Previous work often requires user assistance[39] or solving
an optimization system [40] to obtain the shadow mattes. In contrast, we
propose to train a second network (M-Net) to accurately predict shadow mattes
in a fully automated manner. Lastly, we employ an inpainting network, I-Net,
to handle shadow pixels that might not follow our simplified linear
illumination model (see Sec. 4.3). For example, these cases are due to pixels
with saturated colors or shadows with inconsistent intensities across the
umbra areas.

In essence, we propose a method that removes shadows by regressing a set of
shadow parameters and a matte layer from the input shadow image. This mapping
is simpler than a pixel-wise image-to-image translation and is easier to
constrain via regularization of the shadow parameters and matte layer. Thus we
can train our proposed shadow removal method even without paired shadow and
shadow-free images. In this weakly-supervised setting, we use only shadow and
non-shadow patches cropped from the shadow images, which eliminates the need
for any shadow-free images. This is an unpaired cross-domain image-to-image
translation problem and we approximate this mapping via an adversarial
framework.

We trained and tested our model on the ISTD dataset [32], which is the largest
and most challenging available dataset for shadow removal. Our framework
achieves state-of-the-art shadow removal performance on both the fully-
supervised and weakly-supervised settings. On the fully-supervised setting,
the combination of SP-Net and M-Net alone outperforms the state-of-the-art in
shadow removal [41] by 14% error reduction in terms of MAE on the shadow
areas, from 7.6 to 6.5 MAE. The full framework with SP-Net, M-Net, and the
inpainting network I-Net further improves the results by another 8%, which
yields a MAE of 6.0. On the weakly-supervised setting, our weakly-supervised
method outperforms the state-of-the-art weakly-supervised shadow removal
method[42] by 22%, reducing the MAE on the shadow areas from 12.4 to 9.7. Note
that our method is the first deep-learning approach that can learn a shadow
removal transformation without any shadow-free images.

Last, we introduce a novel challenging testing set to evaluate shadow removal
methods based on time-lapse videos. Note that all current shadow removal
datasets[31, 32] are limited to simple shadows, simple scenes, and without the
oclluders in the images due to the data acquisition scheme[31, 32]. With time-
lapse videos where the only motions are due to illumination effects, we can
obtain paired shadow and non-shadow pixels that move in and out of the shadows
when they travel across the scene. This method allows us to collect paired
shadow data that is impossible to obtain via the standard data acquisition
method such as shadows of immobile occluders, self-cast shadows, and shadows
on various types of backgrounds and materials. We introduce SBU-Timelapse, a
video dataset of 50 clips. Our methods perform better than other methods on
this challenging test.

In summary, the contributions of this work are:

  * •

We propose a new deep learning approach for shadow removal, grounded by a
simplified physical illumination model and an image decomposition formulation.

  * •

We propose a weakly-supervised training scheme for our method that can be
trained without any shadow-free images.

  * •

Our proposed approaches achieve state-of-the-art shadow removal results on the
ISTD dataset, on both fully-supervised and weakly-supervised settings.

  * •

We propose a method to obtain paired shadow data of complex shadows in complex
scenes based on time-lapse videos. We introduced SBU-Timelapse, a video shadow
removal dataset for evaluating shadow removal methods.

The pre-trained model, shadow removal results, and more details can be found
at: https://github.com/cvlab-stonybrook/SID

Figure 2: Shadow Removal Framework. The shadow parameter estimator network SP-
Net takes as input the shadow image and the shadow mask to predict the shadow
parameters $(w,b)$. The relit image $I^{\textrm{relit}}$ is then computed via
Eq. 6 using the estimated parameters from SP-Net. The relit image, together
with the input shadow image and the shadow mask are then input into the shadow
matte prediction network M-Net to get the shadow matte layer $\alpha$. The
system outputs the shadow-free image via Eq. 5, using the shadow image, the
relit image, and the shadow matte. SP-Net learns to predict the shadow
parameters $(w,b)$, denoted as the regression loss. M-Net learns to minimize
the $L_{1}$ distance between the output of the system and the shadow-free
image (reconstruction loss). I-Net takes as input the shadow image, shadow
mask, and the shadow-free image computed from SP-Net and M-Net to output a
residual image that is used to obtain the final shadow-free image.

##  2 Related Work

Shadow Illumination Models: Early research on shadow removal is motivated by
physical modelling of illumination and color [26, 43, 44, 45]. Barrow &
Tenenbaum [25] define an intrinsic image algorithm that separates images into
the intrinsic components of reflectance and shading. Guo _et al_. [28]
simplify this model to represent the relationship between the shadow pixels
and shadow-free pixels via a linear system. They estimate the unknown factors
via pairing shadow and shadow-free regions. Similarly, Shor & Lischinki [29]
propose an illumination model for shadows in which there is an affine
relationship between the lit and shadow intensities at a pixel, including 4
unknown parameters. They define two strips of pixels: one in the shadowed area
and one in the lit area to estimate their parameters. Finlayson _et al_.[46]
create an illuminant-invariant image for shadow detection and removal. Their
work is based on an insight that the shadowed pixels differ from their lit
pixels by a scaling factor. Vicente _et al_. [47, 48] propose a method for
shadow removal where they suggest that the color of the lit region can be
transferred to the shadowed region via histogram equalization.

Shadow Matting: Matting, introduced by Porter & Duff [38], is an effective
tool to handle soft shadows. However, it is non-trivial to compute the shadow
matte from a single image. Chuang _et al_. [11] use image matting for shadow
editing to transfer the shadows between different scenes. They compute the
shadow matte from a sequence of frames in a video captured from a static
camera. Guo et al. [28] and Zhang _et al_. [30] both use a shadow matte for
their shadow removal frameworks, where they estimate the shadow matte via the
closed-form solution of Levin _et al_. [40].

Deep-Learning Based Shadow Removal: Recently published large-scale datasets
[33, 32, 31] enable training deep-learning networks for shadow removal. The
Deshadow-Net of Qu _et al_. [31] is trained to remove shadows in an end-to-end
manner. Their network extracts multi-context features across different layers
of a deep network to predict a shadow matte. This shadow matte is different
from ours as it contains both the density and color offset of the shadows. The
ST-CGAN proposed by Wang _et al_. [32] for both shadow detection and removal
is a conditional GAN-based framework [34] for shadow detection and removal.
Their framework is trained to predict the shadow mask and shadow-free image in
a unified manner, they use GAN losses to improve performance. Mask-ShadowGAN
[49] is the only deep-learning method that learns shadow removal from just
unpaired training data. Their framework is based on the CycleGAN [23], which
approximates a mapping from a source domain, i.e., shadow-images, to a target
domain - shadow-free images. By contrast, our weakly-supervised method can be
trained using only shadow-images.

Inspired by early work, our framework outputs the shadow-free image based on a
physically inspired shadow illumination model and a shadow matte. We, however,
estimate the parameters of our model and the shadow matte via two deep
networks in a fully automated manner.

##  3 Shadow and Image Decomposition Model

###  3.1 Shadow Illumination Model

Let us begin by describing our shadow illumination model. We aim to find a
mapping function $T$ to transform a shadow pixel $I^{\textrm{shadow}}_{x}$ to
its non-shadow counterpart: $I_{x}^{\textrm{shadow-
free}}=T(I^{\textrm{shadow}}_{x},w)$ where $w$ are the parameters of the
model. The form of $T$ has been studied in depth in previous work as discussed
in Sec. 2.

In this paper, similar to the model of Shor & Lischinski [29], we use a linear
function to model the relationship between the lit and shadowed pixels. The
intensity of a lit pixel is formulated as:

$I^{\textrm{shadow-
free}}_{x}(\lambda)=L^{d}_{x}(\lambda)R_{x}(\lambda)+L^{a}_{x}(\lambda)R_{x}(\lambda)$
(1)

where $I^{\textrm{shadow-free}}_{x}(\lambda)$ is the intensity reflected from
point $x$ in the scene at wavelength $\lambda$, $L$ and $R$ are the
illumination and reflectance respectively, $L^{d}$ is the direct illumination
and $L^{a}$ is the ambient illumination.

To cast a shadow on point $x$, an occluder blocks the direct illumination and
a portion of the ambient illumination that would otherwise arrive at $x$. The
shadowed intensity at $x$ is:

$I^{\textrm{shadow}}_{x}(\lambda)=a_{x}(\lambda)L^{a}_{x}(\lambda)R_{x}(\lambda)$
(2)

where $a_{x}(\lambda)$ is the attenuation factor indicating the remaining
fraction of the ambient illumination that arrives at point $x$ at wavelength
$\lambda$. Note that Shor & Lischinski further assume that $a_{x}(\lambda)$ is
the same for all wavelengths $\lambda$ to simplify their model. This
assumption implies that the environment light has the same color from all
directions.

From Eqs.1 and 2, we can express the shadow-free pixel as a linear function of
the shadowed pixel:

$I^{\textrm{shadow-
free}}_{x}(\lambda)=L^{d}_{x}(\lambda)R_{x}(\lambda)+a_{x}(\lambda)^{-1}I^{\textrm{shadow}}_{x}(\lambda)$
(3)

We assume that this linear relation is preserved throughout the color
acquisition process of the camera [50]. Therefore, we can express the color
intensity of the lit pixel $x$ as a linear function of its shadowed value for
all pixels in the shadow area:

$I_{x}^{\textrm{shadow-free}}(k)=w_{k}\times I_{x}^{\textrm{shadow}}(k)+b_{k}$
(4)

where $I_{x}(k)$ represents the value of the pixel $x$ on the image $I$ in
color channel $k$ ($k\in{\textrm{R,G,B color channel}}$), $b_{k}$ is the
response of the camera to direct illumination, and $w_{k}$ is responsible for
the attenuation factor of the ambient illumination at this pixel in this color
channel. We model each color channel independently to account for possibly
different spectral characteristics of the material in shadow as well as the
sensor.

We further assume that the two vectors $w=[w_{R},w_{G},w_{B}]$ and
$b=[b_{R},b_{G},b_{B}]$ are constant across all pixels $x$ in the umbra area
of the shadow. Under this assumption, we can easily estimate the values of $w$
and $b$ given the shadow and shadow-free image using linear regression. We
refer to $(w,b)$ as the shadow parameters in the rest of the paper.

In Sec. 4, we show that we can train a deep-network to estimate these vectors
from a single image.

###  3.2 Shadow Image Decomposition System

We incorporate our proposed shadow illumination model into the following well-
known image decomposition system [11, 38, 51, 52]. The system models the
shadow-free image using the shadow image, the shadow parameter, and the shadow
matte. The shadow-free image can be expressed as:

$I^{\textrm{shadow-
free}}=I^{\textrm{shadow}}\cdot(1-\alpha)+I^{\textrm{relit}}\cdot\alpha$ (5)

where $I^{\textrm{shadow}}$ and $I^{\textrm{shadow-free}}$ are the shadow and
shadow-free image respectively, $\alpha$ is the matting layer, and
$I^{\textrm{relit}}$ is the relit image. We define $\alpha$ and
$I^{\textrm{relit}}$ below.

Each pixel $i$ of the relit image $I^{\textrm{relit}}$ is computed by:

$I_{i}^{\textrm{relit}}=w\cdot I_{i}^{\textrm{shadow}}+b$ (6)

which is the shadow image transformed by the illumination model of Eq. 4. This
transformation maps the shadowed pixels to their shadow-free values.

The matting layer $\alpha$ represents the per-pixel coefficients of the linear
combination of the relit image and the input shadow image that results into
the shadow-free image. Ideally, the value of $\alpha$ should be 1 at the non-
shadow area and 0 at the umbra of the shadow area. For the pixels in the
penumbra of the shadow, the value of $\alpha$ gradually changes near the
shadow boundary.

The value of $\alpha$ at pixel $i$ based on the shadow image, shadow-free
image, and relit image, follows from Eq. 5 :

$\alpha_{i}=\frac{{I_{i}}^{\textrm{shadow-
free}}-{I_{i}}^{\textrm{shadow}}}{{I_{i}}^{\textrm{relit}}-{I_{i}}^{\textrm{shadow}}}$
(7)

We use the image decomposition of Eq. 5 for our shadow removal framework. The
unknown factors are the shadow parameters $(w,b)$ and the shadow matte
$\alpha$. We present our method that uses two deep networks, SP-Net and M-Net,
to predict these two factors in the following section. Sec. 4 presents our
fully-supervised framework trained using paired shadow and shadow-free images.
In Sec. 5, we present our weakly-supervised framework trained using only
shadow images.

##  4 Fully-Supervised Shadow Removal Framework

Our shadow removal framework is based on the shadow image decomposition model
in Eq. 5. The unknowns are the set of shadow parameters $(w,b)$ and the shadow
matte $\alpha$. We train two deep networks, namely, SP-Net, and M-Net to
predict the shadow parameters and the shadow matte respectively. We then train
an inpainting network, I-Net, to handle cases that might not follow the linear
illumination model.

Fig. 2 summarizes our fully-supervised shadow removal framework. The shadow
parameter estimator network SP-Net takes as input the shadow image and the
shadow mask to predict the shadow parameters $(w,b)$. The relit image
$I^{\textrm{relit}}$ is then computed via Eq. 6 with the estimated parameters
from SP-Net. The relit image, together with the input shadow image and the
shadow mask is then input into the shadow matte prediction network M-Net to
get the shadow matte $\alpha$. The system outputs a shadow-free image via Eq.
5. I-Net takes as input the shadow image, shadow mask, and the shadow-free
image computed from SP-Net and M-Net to output a residual image that is used
to obtain the final shadow-free image.

###  4.1 Shadow Parameter Estimator Network

In order to recover the illuminated intensity at the shadowed pixel, we need
to estimate the parameters of the linear model in Eq. 4. Previous work has
proposed different methods to estimate the parameters of a shadow illumination
model [29, 39, 28, 44, 46, 45]. In this paper, we train SP-Net, a deep network
model, to directly predict the shadow parameters from the input shadow image.

To train SP-Net, we first generate training data. Given a training pair of a
shadow image and a shadow-free image, we estimate the parameters of our linear
illumination model using a least squares method [53, 54]. For each shadow
image, we first erode the shadow mask by 5 pixels in order to define a region
that does not contain the partially shadowed (penumbra) pixels. Mapping these
shadow pixel values to the corresponding values in the shadow-free image,
gives us a linear regression system, from which we calculate $w$ and $b$. We
further bound the search space of $w$ to [1,3]. The minimum value of $w=1$
ensures that the transformation always scales up the values of the shadowed
pixels while the upper bound is determined based on the training set. We use
the Trust Region Algorithm [54] to solve this linear system, which is
particularly suitable for solving optimization problems with bounds. The Trust
Region Method defines a region around the current best solution such that a
model could approximate the original function within this region. The trusted
region is then updated based on the new best solution.

We compute parameters for each of the three RGB color channels and then
combine the learned coefficients to form a 6-element vector. This vector is
used as the targeted output to train SP-Net. The input for SP-Net is the input
shadow image and the associated shadow mask. We train SP-Net to minimize the
two losses: 1) the $L_{1}$ distance between the output of the network and
these computed shadow parameters, marked as
“$\mathcal{L}_{\textrm{regression}}$” in Fig. 2, and 2) the final $L_{1}$
image reconstruction loss, marked as “$\mathcal{L}_{\textrm{rec-final}}$”.
Notice that while we use the ground truth shadow mask for training, during
testing we estimate shadow masks using the shadow detection network proposed
by Zhu _et al_.[55].

###  4.2 Shadow Matte Prediction Network

Input Relit Shad. Mask Using S.Mask Shad. Matte Using S.Matte

Figure 3: A comparison of the ground truth shadow mask and our shadow matte.
From left to right: The input image, the relit image computed from the
parameters estimated via SP-Net, the ground truth shadow mask, the final
results when we use the shadow mask, the shadow matte computed using our
M-Net, and the final shadow-free image when we use the shadow matte to combine
the input and relit image. The matting layer handles the soft shadow and does
not generate visible boundaries in the final result. (Please view in
magnification on a digital device to see the difference more clearly.)

Our linear illumination model (Eq. 4) can relight the pixels in the umbra area
(fully shadowed). The shadowed pixels in the penumbra (partially shadowed)
region are more challenging as the illumination changes gradually across the
shadow boundary [27]. A binary shadow mask cannot model this gradual change.
Thus, using a binary mask within the decomposition model in Eq. 5 will
generate an image with visible boundary artifacts. A solution for this is
shadow matting where the soft shadow effects are expressed via the values of a
blending layer.

In this paper, we train a deep network, M-Net, to predict this matting layer.
In order to train M-Net, we use Eq. 5 to compute the output of our framework,
where the shadow matte is the output of M-Net. Then the first loss function
that drives the training of M-Net is an $L_{1}$ distance between the output
image at this stage, i.e., obtained using only SP-Net and M-Net, and the
ground truth training shadow-free image, marked as “$\mathcal{L}_{\textrm{rec-
mat}}$” in Fig. 2. To simply force M-Net to focus on penumbra areas, we use an
additional $L_{1}$ reconstruction loss function applied for these areas only,
denoted as “$\mathcal{L}_{\textrm{penumbra}}$”. We define the penumbra areas
of a shadow as the combination of two areas alongside the shadow boundary,
denoted as $\mathcal{M}_{in}$ and $\mathcal{M}_{out}$ \- see Fig.4.
$\mathcal{M}_{out}$ is the area right outside the boundary, computed by
subtracting the shadow mask, $\mathcal{M}$, from its dilated version
$\mathcal{M}_{dilated}$. The inside area $\mathcal{M}_{in}$ is computed
similarly by subtracting an eroded shadow mask from the shadow mask.

Input Image Shadow Mask $\mathcal{M}_{in}$ & $\mathcal{M}_{out}$

Figure 4: The penumbra area of the shadow. We define two areas alongside the
shadow boundary, denoted as $\mathcal{M}_{in}$ (shown in green) and
$\mathcal{M}_{out}$ (shown in red). These two areas roughly define a small
region surrounding the shadow boundary, which can be considered as the
penumbra area of the shadow.

Moreover, since the shadow effects are assumed to vary smoothly across the
shadow boundaries, we enforce an $L_{1}$ smoothness loss on the spatial
gradients of the matte layer, $\alpha$. This smoothness loss
$\mathcal{L}_{sm-\alpha}$ also prevents M-Net from producing undesired
artifacts since it enforces local uniformity. This loss is:

$\mathcal{L}_{sm-\alpha}=|\nabla{\alpha}|$ (8)

Fig. 3 illustrates the effectiveness of our shadow matting technique. We show
in the figure two shadow removal results which are computed using a ground-
truth shadow mask and a shadow matte respectively. This shadow matte is
computed by our model. One can see that using the binary shadow mask to form
the shadow-free image creates visible boundary artifacts as it ignores the
penumbra. The shadow matte from our model captures well the soft shadow and
generates an image without shadow boundary artifacts.

###  4.3 Inpainting Network

Our model removes shadows in umbra areas via the simplified physical
illumination model. However, there are shadow pixels that might not follow our
proposed illumination model due to various reasons:

  * •

Pixels with low or high intensities might not follow our illumination model
due to color saturation, i.e., when shadows are cast on very “dark” pixels or
relighting very “bright” pixels.

  * •

Our assumptions are strict. A single set of shadow parameters might not
represent all shadow pixels. For example, shadow intensity can be inconsistent
across the shadowed region. Shadow pixels that are closer to the occluding
object often have darker shadows due to more ambient illumination being
blocked.

  * •

There are errors in data acquisition which might lead to incorrect estimates
of the shadow parameters, for example, the inconsistent ambient lighting
between the shadow and shadow-free images.

To mitigate these issues, we train an inpainting network, I-Net, to output a
residual layer to better reconstruct shadow-free images. Previous deep-
learning works also remove shadows via pixel-wise residual layers [31, 41].
However, they typically estimate a residual layer to remove the shadows
directly from the input shadow image while we estimate a residual layer to
further refine the shadow removal result of the proposed SP-Net and M-Net.
Fig.5 visualizes the effect of our inpainting network. The first row shows an
example where the shadow intensity is inconsistent across the shadowed region.
The second row shows an example where pixels with dark colors might not follow
our illumination model. The third row shows an example where our SP-Net does
not estimate correctly the shadow parameters. I-Net mitigates these issues, as
visualized in the last two columns. The loss function that drives the training
of I-Net is a reconstruction loss penalizing an $L_{1}$ distance between
output image and ground truth training shadow-free image, denoted as
$\mathcal{L}_{rec-final}$ in Fig.2.

Input SP+M-Net SP+M Error Map SP+M+I-Net SP+M+I Error Map

Figure 5: The effect of the inpainting network, I-Net. From left to right: The
input images, the shadow-free images computed from our SP-Net and M-Net
(SP+M-Net), the error heat maps of the SP+M-Net’s results, the shadow-free
images computed with the addition of the inpainting network I-Net
(SP+M+I-Net), and the error heat maps of SP+M+I-Net’s results. The inpainting
network I-Net partially corrects the colors of the pixels in the umbra areas.

###  4.4  Objective Function

Fig.2 illustrates the loss functions that drive the training of our framework.
The final objective function for training our system aims to minimize a
weighted sum of all the losses:

$\displaystyle\mathcal{L}_{final}=$
$\displaystyle\lambda_{reg}\mathcal{L}_{regression}+\lambda_{sm}\mathcal{L}_{sm-\alpha}+\lambda_{pen}\mathcal{L}_{penumbra}$
$\displaystyle+\lambda_{rec-mat}\mathcal{L}_{rec-mat}+\lambda_{rec-
final}\mathcal{L}_{rec-final}$

##  5 Weakly-supervised Shadow Removal

Figure 6: Weakly-supervised shadow decomposition. We train our SP-Net and
M-Net together with an adversarial network, D-Net, using only two sets of
unpaired shadow and shadow free patches. These patches are cropped directly
from the shadow images (left panel). SP-Net and M-Net predict the shadow
parameters $(w,b)$ and the matte layer $\alpha$ respectively to jointly remove
the shadow. D-Net is the critic function distinguishing between the generated
image patches and the real shadow-free patches. The only supervision signal is
the set of shadow-free patches, which can be easily obtained using the shadow
masks. The four losses guiding this training are the matting loss, smoothness
loss, boundary loss, and adversarial loss.

$I^{sd}$ $\alpha$ $I^{relit}$ $I^{relit}*\alpha$ $I^{sd}*(1-\alpha)$
$I^{output}$

Figure 7: Weakly-supervised shadow image decomposition. With only shadow mask
supervision, our method automatically learns to decompose the shadow effect in
the input image patch $I^{sd}$ into a matte layer $\alpha$ and a relit image
$I^{relit}$. The matte layer $\alpha$ combines $I^{sd}$ and $I^{relit}$ to
obtain a shadow-free image patch $I^{output}$ via Eq. (5).

The principle of our shadow removal method is to decompose a shadow image into
its intrinsic components including a set of shadow parameters which explain
the shadow effects in the umbra areas, and a matte layer that models the soft
shadows. In this section, we show that such an image decomposition method is
easy to constrain and models closely a shadow removal transformation. Thus, it
can even be trained with a much weaker form of supervision signal rather than
paired shadow and shadow-free images. In this section, we use only two sets of
unpaired patches, shadow and shadow-free, to train our shadow decomposition
framework. Since these patches are directly cropped from the shadow images,
this method can be trained without any shadow-free images.

To obtain shadow and shadow-free patches, we crop the shadow images into small
overlapping patches of size $n\times n$ with a step size of $m$. Based on the
shadow masks, we group these patches into three sets: a non-shadow set
($\mathcal{N}$) containing patches having no shadow pixels, a shadow-boundary
set ($\mathcal{B}$) containing patches lying on the shadow boundaries, and a
full-shadow set ($\mathcal{F}$) containing patches where all pixels are in
shadow. With small enough patch size $n$ and step size $m$, we can obtain
enough training patches in each set. With this training set, we train a shadow
removal system to learn a mapping from patches in the shadow-boundary set
$\mathcal{B}$ to patches in the non-shadow set $\mathcal{N}$.

Fig. 6 summarizes our weakly-supervised training scheme, which consists of
three networks: SP-Net, M-Net, and D-Net. SP-Net and M-Net predict the shadow
parameters $(w,b)$ and the matte layer $\alpha$ respectively to jointly remove
shadows. D-Net is the critic distinguishing between the generated image
patches and the real shadow-free patches. With SP-Net and M-Net being the
generators and D-Net being the discriminator, the three networks form an
adversarial training framework where the main source of training signal is the
set of shadow-free patches.

In theory, as D-Net is trained to distinguish patches containing shadow
boundaries from patches without any shadows, a natural solution to fool D-Net
is to remove the shadows in the input shadow patches to make them
indistinguishable from shadow-free patches. However, such an adversarial
signal from D-Net alone often cannot guide the generators, (SP-Net and M-Net)
to actually remove shadows. The parameter search space is very large and the
mapping is extremely under-constrained. In practice, we observe that without
any constraints, SP-Net tends to output consistently high values of $(w,b)$ as
they would directly increase the overall brightness of the image patches, and
Matte-Net tends to introduce artifacts similar to visual patterns frequently
appearing in the non-shadow areas. Thus, our main intuition is to constrain
this framework with physical shadow properties to force the networks to only
transform the input images in a manner consistent with shadow removal.

First, we bound the search space of SP-Net and M-Net to the appropriate value
ranges that correspond to shadow removal. The search space for the shadow
parameters is defined similarly to Sec. 4.1. M-Net estimates a blending layer
$\alpha$ that combines the shadow image patch and the relit image patch into a
shadow-free image patch via Eq. 5. Thus, we map the output of M-Net to [0,1]
as $\alpha$ is being used as a matting layer and constrain the value of
$\alpha_{i}$ as follows:

  * •

If i indicates a non-shadow pixel, we enforce $\alpha_{i}=0$ so that the value
of the output pixel $I_{i}^{\textrm{output}}$ equals its value in the input
image $I_{i}^{\textrm{shadow}}$.

  * •

If i indicates a pixel in the umbra areas of the shadows, we enforce
$\alpha_{i}=1$ so that the value of the output pixel $I_{i}^{\textrm{output}}$
equals its relit value $I_{i}^{\textrm{relit}}$.

  * •

We do not control the value of $\alpha$ in the penumbra areas of the shadows;
they are learnt though training.

where the umbra, non-shadow or penumbra areas can be roughly specified using
the shadow mask. The above constraints are implemented as the matting loss
$\mathcal{L}_{mat-\alpha}$ computed by the following formula for every pixel
$i$:

$\mathcal{L}_{mat-\alpha}=\sum_{i\in(\mathcal{M}-\mathcal{M}_{in})}|\alpha_{i}-1|+\sum_{i\notin\mathcal{M}_{dilated}}|\alpha_{i}|$
(9)

Moreover, we enforce an $L1$ smoothness loss on the spatial gradients of the
matte layer, $\alpha$:

$\mathcal{L}_{sm-\alpha}=|\nabla{\alpha}|$ (10)

These constraints play a vital part in our weakly-supervised framework, as we
show in the ablation study in Sec.6.4.

Then, given a set of estimated parameters $(w,b)$ and a matte layer $\alpha$,
we obtain an output image $I^{\textrm{output}}$ via the image decomposition
formula (Eq.5). We penalize the $L1$ difference between the average intensity
of pixels lying right outside and inside the shadow boundary,
$\mathcal{M}_{in}$ and $\mathcal{M}_{out}$ respectively. This shadow boundary
loss $\mathcal{L}_{bd}$ is computed by:

$\mathcal{L}_{bd}=\left|\frac{\sum_{i\in\mathcal{M}_{in}}I_{i}^{output}}{\sum_{i\in\mathcal{M}_{in}}}-\frac{\sum_{i\in\mathcal{M}_{out}}I_{i}^{output}}{\sum_{i\in\mathcal{M}_{out}}}\right|$
(11)

Last, we compute the adversarial loss with the feedback from D-Net:

$\mathcal{L}_{GAN}=\log(1-D(I^{output}))$ (12)

where $D(\cdot)$ denotes the output of D-Net.

The final objective function to train Param-Net and Matte-Net is to minimize a
weighted sum of the above losses:

$\mathcal{L}_{final}=\lambda_{sm}\mathcal{L}_{sm-\alpha}+\lambda_{mat}\mathcal{L}_{mat-\alpha}+\lambda_{bd}\mathcal{L}_{bd}+\lambda_{adv}\mathcal{L}_{GAN}$
(13)

By using all the proposed losses together, our method is able to automatically
extract a set of shadow parameters and an $\alpha$ layer from an input image
patch. Fig. 7 visualizes the components extracted from our framework for an
input patch.

Based on our patch-based model, we can estimate a set of shadow parameters and
a matte layer for the input image to remove shadows via Eq. (5). First, we
obtain a shadow mask using the shadow detector of Zhu _et al_. [55]. We crop
the input shadow image into overlapping patches. All patches containing the
shadow boundaries are then input into the three networks. We approximate the
whole image shadow parameters from the patch shadow parameters. We simply
compute the image shadow parameters as a linear combination of the patch
shadow parameters. Similarly, we compute the values of each pixel in the matte
layer by combining the overlapping matte patches. We set the matte layer
pixels in the non-shadow area to $0$ and those in the umbra area to $1$.

##  6 Experiments

###  6.1 Dataset and Evaluation Metric

We train and evaluate on the ISTD dataset [32]. ISTD consists of image
triplets: shadow image, shadow mask, and shadow-free image, captured from
different scenes. The training split has 1870 image triplets from 135 scenes,
whereas the testing split has 540 triplets from 45 scenes.

We notice that the testing set of the ISTD dataset needs to be adjusted since
the shadow images and the shadow-free images have inconsistent colors. This is
a well known issue mentioned in the original paper [32]. The reason is that
the shadow and shadow-free image pairs were captured at different times of the
day which resulted in slightly different environment lighting for each image.
For example, Fig. 8 shows a shadow and shadow-free image pair. The mean-
absolute difference between these two images in the non-shadow area is 12.9.
This color inconsistency appears frequently in the testing set of the ISTD
dataset. On the whole testing set, the mean-absolute distance between the
shadow images and shadow-free images in the non-shadow area is 6.83, as
computed by Wang _et al_.[32].

In order to mitigate this color inconsistency, we use linear regression to
transform the pixel values in the non-shadow area of each shadow-free image to
map into their counterpart values in the shadow image. We use a linear
regression for each color-channel, similar to our method for relighting the
shadow pixels in Sec. 4.1. This simple transformation transfers the color tone
and brightness of the shadow image to its shadow-free counterpart. The third
column of Fig. 8 illustrates the effect of our color-correction method. Our
proposed method reduces the mean-absolute distance between the shadow-free
image and the shadow image from 12.9 to 2.9. The error reduction for the whole
testing set of ISTD goes from 6.83 to 2.6. We evaluate all methods on this
adjusted ISTD testing set in the rest of the paper. We provide the code for
our color-correction method in the project web page.

Shad. Image Original GT Corrected GT

Figure 8: An example of our color correction method. From left to right: input
shadow image, provided shadow-free ground truth image (GT) from ISTD dataset,
and the GT image corrected by our method. Comparing to the input shadow image
on the non-shadow area only, the mean-absolute distance of the original GT is
12.9. This value on our corrected GT becomes 2.9.

###  6.2 Network Architectures and Implementation Details.

We develop SP-Net by customizing a ResNeXt [56] model. We use a U-Net
architecture for M-Net and I-Net. D-Net is a simple 5-layer convolutional
network. To map the outputs of the networks to their proper ranges, we use
Tanh functions together with scaling and additive constants. We use stochastic
gradient descent with the Adam solver [57] to train our model. We choose a
starting learning rate of $\lambda=0.0002$, which is then gradually decreased
after 500 epochs. For each training iteration, the training images are resized
to $320\times 320$ pixels, then randomly cropped to $256\times 256$ pixels,
randomly rotated by $\alpha$ degrees where $\alpha\in[-10:10]$, and randomly
flipped for data augmentation. All networks were trained from scratch. We
experimentally set our training parameters
($\lambda_{reg},\lambda_{sm},\lambda_{pen},\lambda_{rec-met},\lambda_{rec-
final}$) to $(1,1,10,1,1)$ for the fully-supervised framework and
($\lambda_{bd},\;\lambda_{mat-\alpha},\;\lambda_{sm-\alpha},\;\lambda_{adv}$)
to $(0.5,\;100,\;10,\;0.5)$ for the weakly-supervised framework. To choose
these training parameters, we randomly split the original training set of 1330
images into a training set of 1064 images and a validation set of 266 images.
Once the parameters are determined, we re-train the model with the whole
training set.

We use the ISTD dataset [32] for training. For our weakly-supervised training,
each original training image of size $640\times 480$ is cropped into patches
of size $128\times 128$ with a step size of 32. This creates 311,220 image
patches from 1,330 training shadow images. This training set includes 151,327
non-shadow patches, 147,312 shadow-boundary patches, and 12,581 full-shadow
patches.

###  6.3 Shadow Removal Evaluation

We evaluate our method on the adjusted testing set of the ISTD dataset. For
metric evaluation, we follow [32] and compute the MAE in the Lab color space
on the shadow area, non-shadow area, and the whole image, where all shadow
removal results are re-sized into $256\times 256$ to compare with the ground
truth images at this size. Note that in contrast to other methods that only
output shadow free images at that resolution, our shadow removal system works
for input images of any size. Since our method requires shadow masks, we use
the model proposed by Zhu _et al_.[55] pre-trained on the SBU dataset [33] for
detecting shadows. We take the model provided by the author and fine-tune it
on the ISTD dataset for 3000 epochs. This model achieves 2.2 Balanced Error
Rate on the ISTD testing set.

####  6.3.1 Fully-supervised Shadow Removal Evaluation

In Tab. I, we compare the performance of our fully-supervised method with the
fully-supervised shadow removal methods of Qu _et al_.[31], Wang _et al_.[32],
Cun _et al_.[58], and Hu _et al_.[41]. All numbers are computed on the
adjusted testing images so that they are directly comparable. The first row
shows the numbers for the input shadow images, i.e. no shadow removal
performed.

We first evaluate our shadow removal performance using only SP-Net, i.e. we
use the binary shadow mask computed by the shadow detector to form the shadow-
free image from the shadow image and the relit image. The binary shadow mask
is obtained by simply thresholding the output of the shadow detector with a
threshold of 0.95. As shown in column “SP-Net” (second from the right) in Fig.
9, SP-Net correctly estimates the shadow parameters to relight the shadow
area. Even with visible shadow boundaries, SP-Net alone achieves competitive
shadow removal results compared to the state-of-the-art [41].

Input SP-Net SP+M-Net SP+M+I-Net

Figure 9: Shadow removal results of our model at different stages. From left
to right: input shadow images, shadow-free images obtained using SP-Net with
shadow masks from [55], shadow-free images generated by SP-Net and M-Net, and
shadow-free image generated by our full system with SP-Net, M-Net, and I-Net.
The bottom row visualizes the differences between each image and the ground
truth shadow-free image.

Input Wang _et al_. Cun _et al_. Hu _et al_. SP+M+I-Net SP+M+I-Net Ground
[32] [58] [41] No Shd. Mask (Ours) (Ours) Truth

Figure 10: Comparison of fully-supervised shadow removal methods on the ISTD dataset. Qualitative comparison between our method and previous state-of-the-art methods: ST-CGAN [32], Cun _et al_.[58], and DSC [41]. “SP+M+I-Net” are the shadow removal results using the parameters computed from SP-Net, the shadow matte computed from M-Net, and the residual layer computed from I-Net.  TABLE I: Shadow removal results of our fully-supervised method compared to state-of-the-art shadow removal methods on the adjusted ground truth. The metric is MAE (the lower, the better). All methods are trained fully-supervised on the ISTD dataset. Best results are in bold. Methods | Shadow | Non-Shadow | All
---|---|---|---
Input Image | 40.2 | 2.6 | 8.5
DeshadowNet [31] | 15.9 | 6.0 | 7.6
Wang _et al_.[32] | 13.4 | 7.7 | 8.7
Cun _et al_. [58] | 11.4 | 7.2 | 7.9
Hu _et al_. [41] | 7.6 | 3.2 | 3.9
SP-Net | 8.4 | 3.3 | 4.1
SP+M-Net | 6.5 | 3.1 | 3.7
SP+M+I-Net | 6.0 | 3.1 | 3.6
SP+M+I-Net (GT-Mask) | 5.3 | 2.5 | 2.9
Our method with no input shadow-mask
SP-Net | 9.1 | 3.1 | 4.1
SP+M-Net | 8.2 | 3.3 | 4.0
SP+M+I-Net | 6.6 | 3.5 | 4.0
TABLE II: Shadow removal results of our weakly-supervised method compared to weakly-supervised and priors-based shadow removal methods on the adjusted ISTD testing set [32].  Methods | Training Data | Shadow | Non-Shadow | All
---|---|---|---|---
Input Image | - | 40.2 | 2.6 | 8.5
Yang _et al_. [59] | - | 24.7 | 14.4 | 16.0
Gong _et al_. [39] | - | 13.3 | - | -
Hu _et al_.[42] | Shd. Free (U) | 12.5 | 3.8 | 5.2
wSP+M-Net | Shd. Mask | 9.7 | 3.0 | 4.0
wSP+M-Net (GT-Mask) | Shd. Mask | 9.1 | 2.6 | 3.6

We then evaluate the shadow removal results using both SP-Net and M-Net,
denoted as “SP+M-Net” in Tab. I. As shown in Fig. 9, the results of SP+M-Net
do not contain boundary artifacts. Moreover, M-Net also tends to correct some
pixels that are over-lit by SP-Net. As M-Net is trained to blend the relit and
shadow images to create the shadow-free image, it learns to output a smaller
weight for a pixel that is over-lit by SP-Net. Using the matte layer of M-Net
further reduces the MAE on the shadow area by 30%, from 8.4 to 6.5,
outperforming the method of Hu _et al_.[49] by 14%. Our inpainting network,
I-Net, brings another 8% MAE error reduction, helping our full system to
achieve a 6.0 MAE on the shadow areas of the ISTD dataset. Lastly, our method
achieves a 5.3 MAE on the shadow areas when using the ground-truth shadow
masks.

Input Ours GT

Figure 11: Failure cases of our method. The first row shows a case where our
method fails to accurately localize the shadow area while in the second row,
the color of the lit area is incorrect.

Input Cun _et al_. SP+M+I-Net Input Cun _et al_. SP+M+I-Net

Figure 12: Comparison between our shadow removal method and the method of Cun
_et al_.[58] on the SBU dataset[33]

Fig. 10 visualizes the shadow removal results of our method in comparison with
previous state-of-the-art methods including ST-CGAN [32], the method of Cun
_et al_.[58], and DSC [41]. Note that all methods are based on deep-learning
and are trained with paired data. Overall, our method generates better results
without image artifacts. The methods of Wang _et al_.[32] and Cun _et al_.[58]
tend to darken the overall brightness of the images. There are visual
artifacts in all other methods, especially in areas with dark materials. There
are visible boundary artifacts in the outputs of all methods even though they
are milder in our case.

Our method has some failure cases, visualized in Fig. 11. The first row shows
a case where our method fails to accurately localize the shadow area due to
the inaccurate predicted shadow mask while in the second row, the color of the
lit area is incorrect due to the inaccurate estimated shadow parameters.

Additionally, we trained and evaluated an alternative design that does not
input the shadow masks into SP-Net, M-Net, and I-Net. Hence, SP-Net, M-Net,
and I-Net need to learn to localize the shadow areas implicitly. As can be
seen in the three bottom rows of Tab. I, this design performs slightly worse
than our main setting but still outperforms previous state-of-the-art methods
by a large margin. Some examples are visualized in Fig. 10.

Fig.12 visualizes shadow removal results of our method in comparison to the
shadow removal method of Cun _et al_.[58] on the SBU dataset[33], which
includes images with complex shadows on various scenes. Both our method,
SP+M+I-Net, and the method of Cun _et al_. are trained on the ISTD dataset
that only includes images with simple scenes and shadows. Thus, this
comparison evaluates how the methods generalize to unseen and complex shadows
that are present in natural images. As can be seen, the method of Cun _et al_.
only attenuates the shadows in most cases and tends to modify the overall
brightness of the images. Our method removes shadows in a more consistent
manner albeit with the visible boundary artifacts in some cases.

####  6.3.2 Weakly-supervised Shadow Removal Evaluation

We compare our weakly-supervised approach with earlier prior-based and weakly-
supervised shadow removal methods in Tab. II. The methods of Yang _et al_.[59]
and Gong  _et al_.[39] are prior-based while the method of Hu _et al_.[42] is
a weakly-supervised deep-learning method based on CycleGAN [23]. While our
method does not require any shadow-free images to train, Mask-ShadowGAN
requires unpaired shadow and shadow-free images. Our method outperforms Mask-
ShadowGAN by 22%, reducing the MAE in the shadow area from 12.4 to 9.7 while
also achieving lower MAE on the non-shadow area. Note that we even outperform
some fully-supervised methods whose numbers are reported in Tab.I. Our method
achieves slightly better results when using the ground-truth shadow masks,
shown in the last row of Tab. II.

Fig. 13 visualizes the shadow removal results of our weakly-supervised method,
denoted as ”wSP+M-Net”, in comparison with Mask-ShadowGAN. Mask-ShadowGAN
outputs images with visible undesired artifacts even for relatively simple
cases. The shadowed pixels are usually over-lit and the textures beneath the
shadows are not reconstructed properly. Our method handles most cases
relatively well. Compared to our fully-supervised method, our weakly-
supervised method produces more boundary artifacts.

Input MaskShad. wSP+M-Net Ground
-GAN [42] (Ours) Truth

Figure 13: Qualitative comparison between our method and the previous state-
of-the-art weakly-supervised method Mask-ShadowGAN [42]. “wSP+M-Net” are the
shadow removal results of our weakly-supervised method trained using only
shadow images and shadow masks.

###  6.4 Ablation Studies

TABLE III: Ablation Studies. We train our networks with different settings and report the shadow removal performances on the ISTD dataset [32]. The metric is MAE (the lower, the better).  Methods | Shadow | Non-Shadow | All
---|---|---|---
Input Image | 40.2 | 2.6 | 8.5
Fully-Supervised Framework
SP+M-Net (Baseline) | 7.9 | 3.1 | 3.9
SP+M-Net + L.S.S | 7.1 | 3.0 | 3.7
SP+M-Net + L.S.S + $\mathcal{L}_{pen}$ | 6.7 | 3.1 | 3.6
SP+M+I-Net + L.S.S + $\mathcal{L}_{pen}$ | 6.2 | 3.1 | 3.6
SP+M+I-Net + L.S.S + $\mathcal{L}_{pen}$ \+ $\mathcal{L}_{sm-\alpha}$ | 6.0 | 3.1 | 3.6
Weakly-Supervised Framework
wSP+M-Net w/o L.S.S | 47.5 | 2.9 | 9.9
wSP+M-Net w/o $\mathcal{L}_{bd}$ | 41.7 | 3.9 | 9.8
wSP+M-Net w/o $\mathcal{L}_{mat-\alpha}$ | 38.7 | 3.1 | 9.0
wSP+M-Net w/o $\mathcal{L}_{sm-\alpha}$ | 10.2 | 2.8 | 4.0
wSP+M-Net w/o $\mathcal{L}_{GAN}$ | 26.9 | 2.9 | 6.8
wSP+M-Net | 9.7 | 3.0 | 4.0

We conduct ablation studies to better understand the effects of each proposed
component in our framework in both fully-supervised and weakly-supervised
settings. The results are summarized in Tab.III.

For the fully-supervised setting, we start from a baseline model [60] that is
trained without any proposed components, i.e., limiting the search space of
SP-Net and M-Net (denoted as “L.S.S”), the penumbra reconstruction loss
($L_{pen}$), the inpainting network I-Net, and the smoothness loss
($L_{sm-\alpha}$). As can be seen in the second row, simply constraining the
values of the shadow parameters and shadow mattes to their proper ranges
significantly improves the shadow removal performance by a 10% error reduction
in terms of MAE on the shadow areas, from 7.9 to 7.1. The penumbra
reconstruction loss guides M-Net to focus more on the penumbra areas of
shadows. Incorporating this loss into our framework improves the shadow
removal performance by 6%. The addition of the inpainting network, I-Net,
brings another 7% error reduction. The last component is a smoothness loss
that regularizes the matte layer, which reduces the MAE to 6.0.

For the weakly-supervised method, we show that all proposed features and loss
functions are crucial in training our framework. Starting from the original
model, we train new models removing the proposed components one at a time. The
first row shows the results of our model when we set the search space of the
scaling factor $w$ to $[-10,10]$ and the search space of the additive constant
$b$ to $[-255,255]$. In this case, the model collapses and consistently
outputs uniformly dark images. Similarly, the model collapses when we omit the
boundary loss $\mathcal{L}_{bd}$. We observe that this loss is essential in
stabilizing the training as it prevents the Param-Net from outputting
consistently high values.

The matting loss $\mathcal{L}_{mat-\alpha}$ and the $\mathcal{L}_{GAN}$ loss
are critical for learning proper shadow removal. We observe that without the
matting loss $\mathcal{L}_{mat-\alpha}$, the model behaves similarly to an
image inpainting model where it tends to modify all parts of the images to
fool the discriminator. Last, dropping the smoothness loss $\mathcal{L}_{sm}$
only results in a slight drop in shadow removal performance, from 9.7 to 10.2
MAE on the shadow areas. However, we observe more visible boundary artifacts
on the output images without this loss.

###  6.5 SBU-Timelapse Dataset

Paired shadow data for complex scenes and complex shadows are difficult to
collect. Current shadow removal datasets only contain images with simple
scenes, simple shadows, and without the oclluders in the images due to the
data acquisition scheme [31, 32] for paired shadow and non-shadow images.

We propose a method to obtain paired shadow data based on time-lapse videos.
We collect SBU-Timelapse, a video dataset of 50 videos, each contains a static
scene without visible moving objects. We cropped those videos to obtain clips
with the only dominant motions caused by the shadows (either by direct light
motion or motion of the unseen occluders). Instead of collecting paired shadow
and shadow-free images, we aim to collect paired shadow and non-shadow pixels.
As the shadows travel across the scene, we can obtain paired shadow and non-
shadow values for pixels that go in or out of the shadows. As this method does
not require interacting with the occluders or capturing the shadow-free
images, it allows collecting images with complex shadows, complex backgrounds,
or self-cast shadows, as can be seen from Fig. 14.

Inspired by [11], we propose a “max-min” technique to obtain a single pseudo
shadow-free frame for each video: since the camera is static and there is no
visible moving object in the frames, the changes in the video are caused by
the moving shadows. We first obtain two images $V_{max}$ and $V_{min}$ by
taking the maximum and minimum intensity values at each pixel location across
the whole video. $V_{max}$ is then the image that contains the shadow-free
values of pixels if they ever go out of the shadows. Similarly, their shadowed
values, if they ever go into the shadows, are captured in $V_{min}$. Fig. 15
shows these two images for an example video. From these two images, we can
trivially obtain a mask, namely moving-shadow $\mathcal{M}$, marking the
pixels appearing in both the shadow and non-shadow areas in the video:

$\displaystyle\mathcal{M}_{i}=\left\\{\begin{array}[]{ll}1&\hskip
12.91663pt\textrm{if }V_{max,i}>V_{min,i}+\epsilon\\\ 0&\hskip
12.91663pt\textrm{otherwise},\end{array}\right.$ (16)

where we set a threshold of $\epsilon=80$. This method allows us to obtain
pairs of shadow and non-shadow pixel values in the moving-shadow mask,
$\mathcal{M}$, for free. Last, we manually cleaned the obtained moving-shadow
mask to eliminate the areas of changes that are not associated to shadows,
e.g., changes due to subtle motions or color shifts in the videos. All the
original video clips, extracted frames, adjusted moving-shadow masks, and
evaluation code are available as the SBU-Timelapse dataset.

Figure 14: Examples of the SBU-Timelapse dataset. We collected a time-lapse
video dataset where both the scene and the visible objects remain static. The
dataset includes videos containing shadows cast by close-up occluders, far
distance occluders, videos with simple-to-complex shadows, and shadows on
various types of backgrounds and materials.

Frame 0 Frame 100 $V_{min}$ $V_{max}$ Moving Shadow

Figure 15: Our method to obtain pseudo shadow-free frames and moving-shadow
masks for shadow removal evaluation. We first obtain $V_{max}$ and $V_{min}$
by taking the maximum and minimum values at each pixel location across the
whole video. We use $V_{max}$ as a single pseudo shadow-free ground truth
frame for the whole video. The moving-shadow masks can be easily obtained by
comparing the $V_{max}$ and $V_{min}$ images. We measure the shadow removal
performance only on the moving shadow mask areas to exclude the pixels that
are constantly in shadow for the whole video.

To measure shadow removal performance, we input the frames of these videos
into the shadow removal algorithm and measure the MAE in the Lab color space
between the output frame and the image $V_{max}$ on the moving-shadow area
$\mathcal{M}$. We compute MAE on each video and take their average to measure
the shadow removal performance on the whole dataset.

Tab. IV summarizes the performance of our methods compared to Mask-
ShadowGAN[42] and the method of Cun _et al_.[58] on these videos. Our methods
generalize better than other methods on the time-lapse video dataset. As can
be seen, the shadow removal method of Cun _et al_.[58] does not generalize
well to this test where it only reduces the MAE on the moving-shadow areas by
8%, compared to the input frames. Our weakly-supervised method outperforms the
method of Cun _et al_.[58] and Mask-ShadowGAN, reducing the MAE by 25% and 11%
respectively. Our method trained with fully-supervised data achieves the
lowest error rate among all methods. However, it does not show significant
improvement over its weakly-supervised counterpart and Mask-ShadowGAN. These
results suggest that data-driven methods trained on a dataset with limited
diversity fail to generalize to other scenes with complex shadows. Note that
while the ISTD dataset mostly contains images with only outdoor scenes with a
single shadow instance per image, shadows presented in our video shadow
dataset are significantly more diverse, as shown in Fig.14.

Some shadow removal examples for a single frame per video are shown in Fig.16.
There are challenging cases with complex shadows where all methods fail to
remove shadows properly. The method of Cun _et al_. and Mask-ShadowGAN tend to
introduce artifacts and are not able to localize the shadow areas correctly.
Our method certainly has limitations: it inherits the error from the shadow
detector (3rd row) and cannot deal with multiple shadow instances in the same
image (4th row).

TABLE IV: Shadow removal results on our proposed SBU-Timelapse dataset. The metric is MAE (the lower, the better), compared to the pseudo shadow-free frame on the moving shadow mask. Methods | Training Data (ISTD Dataset) | MAE
---|---|---
Input Frames | - | 33.8
Hu _et al_.[42] | Unpaired Shadow/Shadow-Free | 27.3
Cun _et al_.[58] | Paired Shadow/Shadow-Free | 31.2
wSP+M-Net (Weakly-trained) | Shadow Images | 23.4
SP+M+I-Net (Fully-trained) | Paired Shadow/Shadow-Free | 20.1

Input Hu _et al_. Cun _et al_. wSP+M-Net SP+M+I-Net $V_{max}$ Moving-sd.
Image [42] [58] (Ours) (Ours) Image Mask

Figure 16: Comparison of shadow removal methods on the SBU-Vid dataset.
Qualitative comparison between our methods and other shadow removal methods:
Hu _et al_.[42] and Cun _et al_.[58]. “wSP+M-Net” are the shadow removal
results of our patch-based weakly-supervised method. SP+M+I-Net are the shadow
removal results of our fully-supervised framework. The last two columns are
the $V_{max}$ images and the moving-shadow masks, respectively.

##  7 Conclusions

In this work, we have presented a novel approach for shadow removal in single
images. Our main contribution is to use deep networks as the parameters
estimators for an illumination model. Our approach has advantages over
previous approaches. Compared to early methods using an illumination model for
removing shadows, our deep networks can estimate the parameters for the model
from a single image accurately and automatically. Compared to deep learning
methods that perform shadow removal via an end-to-end mapping, our shadow
removal framework outputs images with high quality and much fewer artifacts
since we do not use the deep network to output the per-pixel values. Our model
clearly achieves state-of-the-art shadow removal results on the ISTD dataset
and the proposed video dataset, for both fully-supervised [60] and weakly-
supervised [61] settings. Our current approach can be extended in a number of
ways. A more physically sophisticated illumination model would help the
framework to output more realistic images. It would also be useful to develop
a deep-learning based framework for shadow editing via a physical illumination
model.

## Acknowledgments

This work was partially supported by the NSF EarthCube program (Award
1740595), the National Neographic/Microsoft AI for Earth program, the Partner
University Fund, the SUNY2020 Infrastructure Transportation Security Center,
and a gift from Adobe. Computational support provided by the Institute for
Advanced Computational Science and a GPU donation from NVIDIA. We thank Tomas
Vicente, Kumara Kahatapitiya, and Cristina Mata for assistance with the prior
ICCV 2019 and ECCV 2020 manuscripts.

## References

  * [1] H. Le, T. F. Y. Vicente, V. Nguyen, M. Hoai, and D. Samaras, “A+D Net: Training a shadow detector with adversarial shadow attenuation,” in _Proc. ECCV_ , 2018.
  * [2] S. Hu, H. Le, and D. Samaras, “Temporal feature warping for video shadow detection,” 2021.
  * [3] I. Okabe T Sato and Y. Sato, “Attached shadow coding: estimating surface normals from shadows under unknown reflectance and lighting conditions,” in _Proc. ECCV_ , 2009.
  * [4] J.-F. Lalonde, A. A. Efros, and S. G. Narasimhan, “Detecting ground shadows in outdoor consumer photographs,” in _Proc. ECCV_ , 2010.
  * [5] A. Panagopoulos, D. Samaras, and N. Paragios, “Robust shadow and illumination estimation using a mixture model,” in _Proc. CVPR_ , 2009.
  * [6] A. Panagopoulos, C. Wang, D. Samaras, and N. Paragios, “Simultaneous cast shadows, illumination and geometry inference using hypergraphs,” _IEEE PAMI_ , 2013.
  * [7] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of GANs for improved quality, stability, and variation,” in _International Conference on Learning Representations_ , 2018.
  * [8] K. Sunkavalli, T. E. Zickler, and H. Pfister, “Visibility subspaces: Uncalibrated photometric stereo with shadows,” in _Proc. ECCV_ , 2010.
  * [9] Z. Zhu and C. E. Woodcock, “Object-based cloud and cloud shadow detection in landsat imagery,” in _Remote Sensing of Environment_ , 2012.
  * [10] P. KaewTraKulPong and R. Bowden, _An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection_. Springer US, 2002, pp. 135–144.
  * [11] Y.-Y. Chuang, D. B. Goldman, B. Curless, D. H. Salesin, and R. Szeliski, “Shadow matting and compositing,” _ACM Transactions on Graphics_ , vol. 22, no. 3, pp. 494–500, July 2003.
  * [12] Y. Wang, B. Curless, and S. Seitz, “People as scene probes,” in _Proc. ECCV_ , 2020.
  * [13] T. Müller and B. Erdnüeß, “Brightness correction and shadow removal for video change detection with uavs,” in _Defense + Commercial Sensing_ , 2019.
  * [14] N. Su, Y. Zhang, S. Tian, Y. Yan, and X. Miao, “Shadow detection and removal for occluded object information recovery in urban high-resolution panchromatic satellite images,” _IEEE J. of S. T. in Applied Earth Observations and Remote Sensing_ , vol. 9, pp. 2568–2582, 2016.
  * [15] W. Zhang, X. Zhao, J.-M. Morvan, and L. Chen, “Improving shadow suppression for illumination robust face recognition,” _IEEE PAMI_ , vol. 41, pp. 611–624, 2019.
  * [16] S. Surkutlawar and R. Kulkarni, “Shadow suppression using rgb and hsv color space in moving object detection,” _International Journal of Advanced Computer Science and Applications_ , vol. 4, pp. 164–169, 2013.
  * [17] S. Das, H. A. Sial, K. Ma, R. Baldrich, M. Vanrell, and D. Samaras, “Intrinsic decomposition of document images in-the-wild,” in _Proc. BMVC._ BMVA Press, 2020.
  * [18] S. Das*, K. Ma*, Z. Shu, D. Samaras, and R. Shilkrot, “Dewarpnet: Single-image document unwarping with stacked 3d and 2d regression networks,” in _Proc. ICCV_ , 2019.
  * [19] H. Le, B. Goncalves, D. Samaras, and H. Lynch, “Weakly labeling the antarctic: The penguin colony case,” in _CVPR Workshops_ , June 2019.
  * [20] H. Le, C.-P. Yu, G. Zelinsky, and D. Samaras, “Co-localization with category-consistent features and geodesic distance propagation,” in _ICCV Workshop_ , 2017.
  * [21] H. Le, V. Nguyen, C.-P. Yu, and D. Samaras, “Geodesic distance histogram feature for video segmentation,” _Proc. ACCV_ , 2016.
  * [22] J. Xu, H. Le, M. Huang, S. Athar, and D. Samaras, “Variational feature disentangling for fine-grained few-shot classification,” in _Proc. ICCV_ , 2021.
  * [23] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proc. ICCV_ , 2017.
  * [24] D. Liu, C. Long, H. Zhang, H. Yu, X. Dong, and C. Xiao, “Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes,” _Proc. CVPR_ , pp. 8136–8145, 2020.
  * [25] H. G. Barrow and J. M. Tenenbaum, “Recovering intrinsic scene characteristics from images,” _Computer Vision Systems_ , pp. 3–26, 1978.
  * [26] G. Finlayson, S. Hordley, C. Lu, and M. Drew, “On the removal of shadows from images,” _IEEE PAMI_ , 2006.
  * [27] X. Huang, G. Hua, J. Tumblin, and L. Williams, “What characterizes a shadow boundary under the sun and sky?” in _Proc. ICCV_ , 2011.
  * [28] R. Guo, Q. Dai, and D. Hoiem, “Paired regions for shadow detection and removal,” _IEEE PAMI_ , 2012.
  * [29] Y. Shor and D. Lischinski, “The shadow meets the mask: Pyramid-based shadow removal,” _Computer Graphics Forum_ , vol. 27, no. 2, pp. 577–586, Apr 2008\.
  * [30] L. Zhang, Q. Zhang, and C. Xiao, “Shadow remover: Image shadow removal based on illumination recovering optimization,” _IEEE Transactions on Image Processing_ , vol. 24, no. 11, Nov 2015.
  * [31] L. Qu, J. Tian, S. He, Y. Tang, and R. W. H. Lau, “Deshadownet: A multi-context embedding deep network for shadow removal,” in _Proc. CVPR_ , 2017.
  * [32] J. Wang, X. Li, and J. Yang, “Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal,” in _Proc. CVPR_ , 2018.
  * [33] T. F. Y. Vicente, L. Hou, C.-P. Yu, M. Hoai, and D. Samaras, “Large-scale training of shadow detectors with noisily-annotated shadow examples,” in _Proc. ECCV_ , 2016.
  * [34] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proc. CVPR_ , 2017.
  * [35] R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in _Proc. ECCV_ , 2016.
  * [36] A. Odena, V. Dumoulin, and C. Olah, “Deconvolution and checkerboard artifacts,” _Distill_ , 2016. [Online]. Available: http://distill.pub/2016/deconv-checkerboard
  * [37] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” in _Proc. CVPR_ , 2018.
  * [38] T. Porter and T. Duff, “Compositing digital images,” _Proc. ACM SIGGRAPH_ , vol. 18, no. 3, Jan 1984.
  * [39] H. Gong and D. Cosker, “Interactive removal and ground truth for difficult shadow scenes,” _J. Opt. Soc. Am. A_ , vol. 33, no. 9, pp. 1798–1811, 2016\.
  * [40] A. Levin, D. Lischinski, and Y. Weiss, “A closed-form solution to natural image matting,” _IEEE PAMI_ , vol. 30, no. 2, pp. 228–242, 2008\.
  * [41] X. Hu, C.-W. Fu, L. Zhu, J. Qin, and P.-A. Heng, “Direction-aware spatial context features for shadow detection and removal,” _IEEE PAMI_ , 2019.
  * [42] X. Hu, Y. Jiang, C.-W. Fu, and P.-A. Heng, “Mask-ShadowGAN: Learning to remove shadows from unpaired data,” in _Proc. ICCV_ , 2019.
  * [43] G. Finlayson and M. S. Drew, “4-sensor camera calibration for image representation invariant to shading, shadows, lighting, and specularities,” in _Proc. ICCV_ , vol. 2, July 2001, pp. 473–480 vol.2.
  * [44] G. Finlayson, S. D. Hordley, and M. S. Drew, “Removing shadows from images,” in _Proc. ECCV_ , ser. ECCV ’02. London, UK, UK: Springer-Verlag, 2002, pp. 823–836.
  * [45] M. S. Drew, “Recovery of chromaticity image free from shadows via illumination invariance,” in _ICCV Workshop_ , 2003, pp. 32–39.
  * [46] G. Finlayson, M. Drew, and C. Lu, “Entropy minimization for shadow removal,” _IJCV_ , 2009.
  * [47] T. F. Y. Vicente, M. Hoai, and D. Samaras, “Leave-one-out kernel optimization for shadow detection and removal,” _IEEE PAMI_ , vol. 40, no. 3, pp. 682–695, 2018.
  * [48] T. F. Y. Vicente and D. Samaras, “Single image shadow removal via neighbor-based region relighting,” in _ECCV Workshops_ , 2014.
  * [49] X. Hu, L. Zhu, C.-W. Fu, J. Qin, and P.-A. Heng, “Direction-aware spatial context features for shadow detection,” in _Proc. CVPR_ , 2018.
  * [50] G. Finlayson, M. M. Darrodi, and M. Mackiewicz, “Rank-based camera spectral sensitivity estimation,” _J. Opt. Soc. Am. A_ , vol. 33, no. 4, pp. 589–599, Apr 2016.
  * [51] A. R. Smith and J. F. Blinn, “Blue screen matting,” in _Proc. ACM SIGGRAPH_ , 1996.
  * [52] S. Wright, “Digital compositing for film and video,” in _Focal Press_ , 2001\.
  * [53] R. D. Cook, “Influential observations, high leverage points, and outliers in linear regression,” _Statistical Science_ , pp. 393–397, 1986.
  * [54] W. Sun and Y. Yuan, _Optimization Theory and Methods: Nonlinear Programming_. Springer Optimization and Its Applications, 2010.
  * [55] L. Zhu, Z. Deng, X. Hu, C.-W. Fu, X. Xu, J. Qin, and P.-A. Heng, “Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection,” in _Proc. ECCV_ , 2018.
  * [56] S. Xie, R. Girshick, P. Dollar, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _Proc. CVPR_ , July 2017.
  * [57] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. ICLR_ , 2015.
  * [58] X. Cun, C.-M. Pun, and C. Shi, “Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting gan,” in _Proc. AAAI_ , 2020.
  * [59] Q. Yang, K. Tan, and N. Ahuja, “Shadow removal using bilateral filtering,” _IEEE Transactions on Image Processing_ , vol. 21, pp. 4361–4368, 2012.
  * [60] H. Le and D. Samaras, “Shadow removal via shadow image decomposition,” in _Proc. ICCV_ , 2019.
  * [61] ——, “From shadow segmentation to shadow removal,” in _Proc. ECCV_ , 2020\.

|  Hieu Le received his B.S. degree in computer science from the Ho Chi Minh
City University of Science in 2012, and his Ph.D. degree in computer science
from the Stony Brook University in 2020. His research interest includes
illumination modelling for image analysis and generation.
---|---
|  Dimitris Samaras received the diploma degree in computer science and
engineering from the University of Patras, in 1992, the MSc degree from
Northeastern University, in 1994, and the PhD degree from the University of
Pennsylvania, in 2001. He is an associate professor of computer science with
Stony Brook University. He specializes in illumination modeling and estimation
for recognition and graphics, and biomedical image analysis. He is Program
Chair of CVPR 2022.
---|---