# Adaptive Domain Interest Network for Multi-domain Recommendation

Yuchen Jiang, Qi Li, Han Zhu, Jinbei Yu, Jin Li, Ziru Xu, Huihui Dong, Bo
Zheng  jiangyuchen.jyc, luyuan.lq, zhuhan.zh, jinbei.yjb, echo.lj, ziru.xzr,
dhh267344<EMAIL_ADDRESS>Alibaba GroupChina

(2018)

###### Abstract.

Industrial recommender systems usually hold data from multiple business
scenarios and are expected to provide recommendation services for these
scenarios simultaneously. In the retrieval step, the topK high-quality items
selected from a large number of corpus usually need to be various for multiple
scenarios. Take Alibaba display advertising system for example, not only
because the behavior patterns of Taobao users are diverse, but also
differentiated scenarios’ bid prices assigned by advertisers vary
significantly. Traditional methods either train models for each scenario
separately, ignoring the cross-domain overlapping of user groups and items, or
simply mix all samples and maintain a shared model which makes it difficult to
capture significant diversities between scenarios. In this paper, we present
Adaptive Domain Interest network that adaptively handles the commonalities and
diversities across scenarios, making full use of multi-scenarios data during
training. Then the proposed method is able to improve the performance of each
business domain by giving various topK candidates for different scenarios
during online inference. Specifically, our proposed ADI models the
commonalities and diversities for different domains by shared networks and
domain-specific networks, respectively. In addition, we apply the domain-
specific batch normalization and design the domain interest adaptation layer
for feature-level domain adaptation. A self training strategy is also
incorporated to capture label-level connections across domains. ADI has been
deployed in the display advertising system of Alibaba, and obtains 1.8%
improvement on advertising revenue.

Recommender Systems, Information Retrieval, Neural Networks, Multi-domain
Learning

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information systems Information
retrieval Retrieval models and ranking

##  1\. Introduction

Modern recommendation systems and advertising systems are usually built as a
pipeline, including retrieving, ranking, reranking and other parts. For
retrieval step (also known as matching step), the most important objective is
to retrieve topK high-quality items from a very large corpus (millions) for
downstream ranking task in limited time. Item-CF (sarwar2001item, 1) and user-
CF (han2011data, 2) are the most lightweight and common methods used in
retrieval, which leverage user/item collaborative signals. However, with the
proliferation of deep learning methods, deep models for recommendation perform
better than algorithms based solely on collaborative signals. YouTube product-
DNN (covington2016deep, 3, 4) proposes to generate user/item vector
representations, calculating their inner product and then retrieving items
using efficient approximate k-nearest neighbor searches. Facebook EBR
(huang2020embedding, 5) integrate embedding-based retrieval with boolean
matching retrieval in their search engines to address the semantic matching
issues. Further works focus on either the capability of vector representations
or searching strategies for retrieval. MIND (li2019multi, 6) proposes a multi-
interest retrieve model using a dynamic routing mechanism. In the meantime,
TDMs (zhu2018learning, 7, 8, 9) and Deep-Retrieval (gao2020deep, 10) are
proposed to increase model complexity by building the searching index for the
large corpus. Although the deep models are thriving in recommendation system,
traditional recommenders and online advertising system mainly focus on how to
model single scenario well. In this paper, we are devoted to get benefit from
multiple scenarios’ data in the retrieval stage.

Data collected from multiple business scenarios own commonalities and
diversities. For the former, there is an overlap between both users and items
for different domains, in other words, the domain-invariant user interest and
item information can be transferred from one domain to another. Taking Alibaba
display advertising production data for example, there are 49% of users and
79% of items appear at least two scenarios based on our data analysis on
traffic logs in a day. For the latter, data distributions from multiple
domains are different, since users’ preference, items’ displaying permission
and advertiser’s bidding price are quite various across domains. To address
those issues, three solutions are generally adopted in real world systems. The
first solution is to train one model per domain using training data collected
from this domain, but the drawbacks (sheng2021one, 11) are obvious: 1).
Maintaining multiple models for multiple domains need much human operation
cost and calculating resources. 2). Separately training models per domain
makes it impossible to exploit domain-invariant knowledge, especially for
those minor business scenarios where training data is limited. A step further
solution is to mix data and train a shared model. By doing this, human
operation cost and calculating resources can be saved. But if without
particular design, model performance may decrease when different domains
conflict. The last generally adopted solution is to train a unified model in a
multi-task manner. Although the multi-domain recommendation task usually
shares the similar model structure with multi-task recommendation task, we
argue that those two tasks are fundamentally different. For the former, data
distributions of inputs from multiple domains are quite different but the the
task goals are the same (such as multi-domain semantic segmentation in
(zou2019confidence, 12)). For the latter, labels from different task vary
significantly (such as CTR and CVR prediction in (ma2018entire, 13)) while the
input is the same. Therefore, a particular architecture designed for multi-
domain recommendation is needed in real world applications.

Existing efforts for multi-domain recommendation (sheng2021one, 11, 14) focus
on ranking step, while retrieval step is rarely studied (hao2021adversarial,
15). To solve the multi-domain recommendation in retrieval step, we come up
with the Adaptive Domain Interst network, which learns users’ preferences for
multiple scenarios simultaneously. Firstly, our proposed ADI models the
commonalities and diversities for different domains by common networks and
domain-specific networks, respectively. To tackle the feature-level domain
adaptation, we present two domain adaptation methods, which are domain-
specific batch normalization and domain interest adaptation layer. In
addition, to capture label-level connection across domains, a self training
method is also incorporated.

To summarize, our proposed method achieves following contributions:

  * •

We proposed a novel model architecture named Adaptive Domain Interest network
to tackle multi-domain recommendation in retrieval step. The ADI network
efficiently learn the commonalities and diversities for multiple domains,
leading to an overall performance lift for all domains.

  * •

We provide domain interest adaptation layers to dynamically transfer raw input
features to domain-related features. Extensive experiments and visualization
prove the effectiveness of the proposed domain interest adaptation layer.

  * •

We get a first attempt to apply self training method on our multi-domain
recommendation problem, capturing the potential label-level cross-domain
connection.

  * •

We conduct solid experiments on real-world industry production dataset and
deploy our proposed method in the online advertising system.

##  2\. Related Works

Domain Adaptation: Domain Adaptation $\left(DA\right)$ problem is a branch of
transfer learning, aiming to learn from source domain then get better
performance on target domain. The key to solving the DA problem lies in
transferring useful knowledge from source domain to target domain. A lot of
works (ganin2015unsupervised, 16, 17) extract domain-invariant features by
minimizing the cross-domain difference of feature distributions through
adversarial learning. DSN (bousmalis2016domain, 18) designs domain separation
network to transfer knowledge through the shared-network. Some works
(inoue2018cross, 19, 20) handle DA problem through a self-training/pseudo-
labeling strategy. Previously DA methods mostly focus on solving problems in
computer vision and neural language process, while recently there are more and
more researches on recommendation systems especially on CTR prediction.

Multi-Domain Recommendation:Multi-domain recommendation (man2017cross, 21, 22)
task aims to improve model performance on each domain using knowledge
transferred from the other domains. The difference between multi-domain
recommendation$\left(MDR\right)$ (sheng2021one, 11, 14, 15) and cross-domain
recommendation$\left(CDR\right)$ (li2020ddtcdr, 22, 23) lies in transfer
directions. CDR aims to transfer knowledge in a specific direction (for
example, using data from main scenario to improve performance in cold-start
scenario), while MDR is aiming to gain an overall performance in all domains.
SAR-Net (shen2021sar, 14) accommodates the transfer of users’ interest across
scenarios through two specific attention modules, and uses mixture of experts
to extract the required information. STAR (sheng2021one, 11) proposes star
topology consisting of shared centered parameters and domain-specific
parameters, to keep one model get refined CTR prediction for different domain.
AFT (hao2021adversarial, 15) proposes a novel adversarial learning method to
solve the MDR problem.

Multi-Task Learning: Multi-task learning (ruder2017overview, 24) is a machine
learning paradigm to learn several related tasks at the same time, leading to
a better performance on each task. There are many general multi-task model
structures, which gain significant progress in computer vision
(heuer2021multitask, 25), neural language processing (liu2019multi, 26),
information retrieval (jain2021mural, 27) and recommendation system
(ma2018entire, 13). Share-Bottom (caruana1997multitask, 28) network designs a
shared network at the bottom to learn the similarity and multiple task-
specific network at the top to learn the differences. Multi-gate Mixture-of-
Experts (MMoE) (ma2018modeling, 29) use multiple expert networks at the bottom
to capture different patterns in the data and learn a per-task and per-sample
weighting of each expert networks allowing different tasks to utilize experts
differently. However, the MMoE architecture meets worse performance when tasks
correlation is complex, which is called seesaw phenomenon. PLE
(tang2020progressive, 30) eases above issue by separating experts network into
the task-related private network and task-independent shared network. SNR-Net
(ma2019snr, 31) learns a optimized combination of several shared sub-networks.

Self Training: Self training (or self-supervised learning) is a learning
strategy, in which labeled data are limited and much more unlabeled data are
available. Exploiting unlabeled data is the key to gaining a performance boost
in self training paradigm. (zou2018unsupervised, 20, 12) apply self training
method on DA problems by producing pseudo-labels for the target domain. For
multi-domain recommendation task, most work focus on sample level
(wang2019minimax, 32), feature level (shen2021sar, 14), and parameter level
(sheng2021one, 11) transferring to gain a better performance. However, we
argue potential connection in the labels from different domains can be
employed by our proposed efficient self training method, which has been
neglected by existing works (sheng2021one, 11, 14).

Figure 1. An illustration of the overall architecture of ADI. Following the
gray arrow, a sample will be firstly embeded, then fed into the Domain
Interest Adaptation Layer, Shared Domain-Specific Network, Fusion Layer and
Domain-Specific Forward Network. After getting user/item representations
through user/item tower, the inner product will be produced and the sampled
softmax loss will be calculated in the end. The domain indicator is ultilized
for choosing which domain-related network to use.

##  3\. Preliminaries

###  3.1. Problem Formulation

In this section, we formalize the definition of multi-domain retrieval task.
Multi-domain retrieval task aims to retrieve high-quality items for multiple
domains from a very large corpus. More specifically, let $\mathcal{U}$ and
$\mathcal{V}$ denote the user set and the item set, respectively. The online
multi-domain retrieval task can be formulated as follows:

(1)
$\mathcal{S}_{u,d}=\operatornamewithlimits{argTopk}_{v\in\mathcal{V}}f_{\theta}(v|u,d),$

where $d$ denotes the domain indicator, and $f_{\theta}(v|u,d)$ is the
estimated matching function with trainable parameters $\theta$ for measuring
the quality of $u$ to $\mathcal{V}$ given the user $u$ and domain indicator
$d$. $\mathcal{S}_{u,d}$ is a set containing topK items with respect to
$f_{\theta}(v|{u,d})$.

In neural based retrieval models, learning such a model $f_{\theta}(v|u,d)$
can be regarded as an instance-level classification problem. Distribution of
positive item $v$ win from $\mathcal{V}$ is based on the softmax function,
i.e.,

(2)
$s_{\theta}(v|u,d)=\frac{\exp(f_{\theta}(v|u,d))}{\sum_{v^{\prime}\in\mathcal{V}}\exp(f_{\theta}(v^{\prime}|u,d))}.$

Then $\theta$ is trained to minimize the negative log likelihood $-\log
s_{\theta}(v|u,d)$ over the training data

(3)
$\theta^{*}=\operatornamewithlimits{argmin}_{\theta}\sum_{d}\sum_{u}\sum_{v\in\mathcal{B}_{u,d}}-\log
s_{\theta}(v|u,d),$

where $\mathcal{B}_{u,d}$ is the set of interacted items by $u$ given the user
$u$ and domain indicator $d$.

In practice, since $\mathcal{V}$ is usually extremely large, sub-sampling is
widely adopted to reduce the computational complexity of computing the
denominator of Eq. (2) . Following (covington2016deep, 3, 6), we use the
sampled softmax loss (jean2015using, 33) and replace $f_{\theta}(v|u,d)$ in
Eq. (2) with

(4) $\tilde{f}_{\theta}(v|u,d)=f_{\theta}(v|u,d)-\log Q(v).$

With the sub-sampling, we have Eq. (5). $\mathcal{N}_{u,d}$ is the set of
irrelevant items, which are sampled from $\mathcal{V}$ according to the
proposal distribution $Q:\mathcal{V}\to\mathbb{R}$ such that its size
satisfies $|\mathcal{N}_{u,d}|\ll|\mathcal{V}|$.

(5)
$\theta^{*}=\operatornamewithlimits{argmin}_{\theta}\sum_{d,u,v\in\mathcal{B}_{u,d}}-\tilde{f}_{\theta}(v|u,d)+\log\left(\exp(\tilde{f}_{\theta}(v|u,d))+\sum_{v^{\prime}\in\mathcal{N}_{u,t}}\exp(\tilde{f}_{\theta}(v^{\prime}|u,t))\right).$

##  4\. Methodology

In this section, we introduce our proposed method to tackle multi-domain
retrieval problem. The overall model architecture are shown in Figure 1. The
total model architecture is designed to commonalities and diversities for
different domains from three angles. Firstly, the backbone network extracts
parameter-level commonalities and diversities from data collected from
different domains. Secondly, the domain adaptation methods learn feature-level
diversities. Lastly, the self-training strategy captures label-level
commonalities.

Table 1. A brief feature description for Alibaba display advertising,
consisting of user features, item features, and domain features.

Feature Type | Feature Description
---|---
User Features | User profiles, User behaviors (click, add_to_cart, pay..), etc.
Item Features | Item attributes, Creative attributes, Advertiser attributes, etc.
User Domain Features | User behaviors by domain, etc.
Item Domain Features | Bidding Price by domain, Item statistics by domain (ctr, click..), etc.
Domain Indicator Feature | Domain indicator

###  4.1. Backbone Network

To efficiently learn the commonalities and diversities between data
distributions from different domains, we design the shared networks and the
domain-specific networks at the bottom with domain-specific forward networks
at the top. Such architecture is able to perform better when dealing with
multi-domain retrieval problems compared to vanilla DNN (covington2016deep,
3), share-bottom network (caruana1997multitask, 28) and MMoE (ma2018modeling,
29), which is proved in the following experiments part.

####  4.1.1. Shared Embedding Layer

As shown in Table 1, the training/testing samples contain rich feature
information. Therefore, the first step is to transfer those high dimensional
sparse one-hot vectors into low-dimensional embedding vectors, and all domains
share the same embedding layers.

(6) $F_{i}=EMBED(f_{i}),$ (7)
${\bf{F}}=concat(F_{1}~{}|\cdot\cdot\cdot|~{}F_{n}),$

where $F_{i}$ denotes $i_{th}$ embeded feature. $\bf{F}$ denotes user/item
inputs.

####  4.1.2. Shared Network & Domain-Specific Network

After obtaining encoded user representations and item representations, we
introduce the shared network and domain-specific network as shown in Figue 1.
Inspired by (bousmalis2016domain, 18), we design the shared network to learn
representations shared by all domains and the domain-specific network to learn
domain-specific representations in each domain:

(8)
$\alpha_{k}=\frac{W_{shared}^{k}(f_{domain})+b_{shared}^{k}}{\sum_{n=1}^{K}(W_{shared}^{n}(f_{domain})+b_{shared}^{n})},$
(9) ${E_{shared}}=\sum_{k=1}^{K}\alpha_{k}MLP_{shared}^{k}({\bf{F}}),$ (10)
${E_{spec}^{(d)}}=MLP_{spec}^{(d)}({\bf{F}}^{(d)}),$

where $MLP$ denotes the multilayer perceptron, $f_{domain}$,
${\textbf{F}}^{(d)}$ denote domain-related features and data collected from
domain $d$, respectively. In our practice, we use domain indicator embedding
as $f_{domain}$. $W_{shared}^{n},b_{shared}$ are weights and bias of a one-
layer shallow neural network. Data from all domains will feed into shared
networks, while data from domain $d$ will be feed into $d_{th}$ domain-
specific network. More specifically, suppose there are training data from $D$
domains, we will build $K$ shared network and $D$ specific network. The total
number of FCs is $D+K$

####  4.1.3. Fusion Layer

The fusion layer aims to learn an optimized combination of outputs from
Domain-Specific Network and Shared Network, which can be described as follows:

(11) $\beta_{1}^{(d)}=\sigma(W_{fusion\\_spec}^{(d)}(f_{domain})),$ (12)
$\beta_{2}^{(d)}=\sigma(W_{fusion\\_shared}^{(d)}(f_{domain})),$ (13)
$\small{E_{fusion}^{(d)}}=concat(\beta^{(d)}_{1}E_{spec}^{(d)}~{}|~{}\beta^{(d)}_{1}E_{spec}^{(d)}\odot{\beta^{(d)}_{2}E_{shared}}~{}|~{}\beta^{(d)}_{2}E_{shared});$

where $\sigma$ denotes sigmoid function, $\odot$ denotes hadamard product, and
$\beta_{1}^{(d)},\beta_{2}^{(d)}$ denote feature weights assigned for
$E_{spec}^{(d)},E_{shared}$, respectively. We name the proposed fusion layer
the CONCAT version. Therefore, shared and specific network will produce
domain-related $E_{fusion}^{(d)}$ for each domain. Besides, we implement two
variants, which are SUM version used by MMoE (ma2018modeling, 29), SAR-Net
(shen2021sar, 14) and Network-Mul version proposed by STAR (sheng2021one, 11).
For the SUM version, we use the gating network of MMoE as the fusion layer.
$W_{gate},b_{gate}$ denote weights and bias of the gating network:

(14) $\alpha^{(d)}=\sigma(W_{gate}^{(d)}(f_{domain})+b_{gate}),$ (15)
${E_{fusion}^{(d)}}={\alpha^{(d)}}E_{spec}^{(d)}+{(1-\alpha^{(d)})}E_{shared},$

For the Network-Mul version, we use the STAR-Topology FCN of STAR
(sheng2021one, 11) as the fusion layer (Note that STAR only has one shared
network), $W_{shared}$, $b_{shared}$ and $W_{spec}^{(d)}$, $b_{spec}^{(d)}$
denote parameters in $FC_{shared}$ and $FC_{spec}$, respectively:

(16) $FC_{Net-
Mul}(X)=(W_{shared}\odot{W_{spec}^{(d)}})\cdot{\textbf{X}}+b_{shared}+b_{spec}^{(d)},$
(17) ${E_{fusion}^{(d)}}=FC_{Net-Mul}({{\bf{F}}^{(d)}}),$

Experiments in Section 5.3.1 prove our proposed CONCAT version achieves the
best performance, which is adopted as the fusion layer.

####  4.1.4. Domain-Specific Forward Network

After obtaining domain-related $E_{fusion}^{(d)}$, finally, the outputs will
feed into domain-related forward network, which describes as follows:

(18) ${E}=FC_{forward}^{(d)}(E_{fusion}^{(d)}).$

the output $E$ produced by user tower and item tower will be used for
following inner product and sampled softmax calculating.

###  4.2. Domain Adaptation

We provide two approaches to solving domain adaptation problems in the multi-
domain recommendation task: domain-specific batch normalization and domain
interest adaptation layer.

Figure 2. An illustration of DSBN. Samples collected from different domains
will choose different branches of DSBN.

####  4.2.1. Domain-Specific Batch Normalization

Batch normalization technology (BN) (ioffe2015batch, 34) has been widely used
to train very deep neural network. Let $\mu$ denotes the mean value of input
${\bf{X}}$, while $\sigma^{2}$ denotes the variance. The batch normalization
method can be described as follows:

(19)
$\hat{{\bf{X}}}=\alpha\frac{{\bf{X}}-\mu}{\sqrt{\sigma^{2}+\epsilon}}+\beta,$

where $\alpha$ and $\beta$ are learnable parameters, $\epsilon$ is a very
small quantity to avoid the denominator being 0. BN assumes that the input
${\bf{X}}$ satisfies the assumption of independent and identical
distribution(i.i.d), which works well in single scenario. However, multi-
domain retrieval problem is facing with a mixed data distributions.
Calculating the global BN parameters and ignoring statistic discrepancies
between different domains may hurt the final performance. Inspired by
(chang2019domain, 35), We apply the domain-specific batch normalization (DSBN)
to solve the mentioned problem:

(20)
$\hat{{\bf{X}}}^{(d)}=\alpha^{(d)}\frac{{\bf{X}}^{(d)}-\mu^{(d)}}{\sqrt{(\sigma^{(d)})^{2}+\epsilon}}+\beta^{(d)},$

where ${\bf{X}}^{(d)}\in\bf{X}$ denotes collected samples from domain $d$. By
estimating domain-specific batch statistics
$\mu^{(d)},(\sigma^{(d)})^{2},\alpha^{(d)},\beta^{(d)}$ in batch
normalization, we trust that the model is able to to capture the domain-
specific information.

####  4.2.2. Domain Interest Adaptation Layer

The domain interest adaptation layer comes from the intuition that different
domains are supposed to focus on different parts of the raw features. We
implement three types of domain interest adaptation layer: linear domain
transformation, vanilla domain attention, and SE-Block based domain attention:
Linear domain transformation: Linear domain transformation used by
(shen2021sar, 14) maps original features into domain-related features. Let
$F_{i}^{(d)}$ denotes $i_{th}$ feature of embedded input collected from domain
$d$, and $N$ is total feature number. $W^{(d)},b^{(d)}$ share the same
dimension with input $F^{(d)}$. Linear domain transformation method describes
as follows:

(21) $F^{(d)}=concat(F_{1}^{(d)}~{}|~{}\cdot\cdot\cdot~{}|~{}F_{N}^{(d)}),$
(22) $\hat{F}^{(d)}=W^{(d)}\odot F^{(d)}+b^{(d)},$

Vanilla domain attention: Let $Q^{(d)}_{i}$ denotes $i_{th}$ domain-specific
query vector for attention weights calculating and $\alpha_{i}^{(d)}$ denotes
the attention weight of $i_{th}$ feature, the vanilla domain attention
mechanism describes as follows:

(23) $\alpha^{(d)}_{i}=\sigma(Q^{(d)}_{i}{F^{(d)}_{i}}),$ (24)
$\hat{F}^{(d)}=concat(\alpha_{1}^{(d)}F_{1}^{(d)}~{}|~{}\cdot\cdot\cdot~{}|~{}\alpha_{n}^{(d)}F_{N}^{(d)}),$

SE-Block based domain attention: Squeeze-and-Excitation Network (SE-Net)
(hu2018squeeze, 36) has achieved SOTA results in many computer vision tasks
(wang2019towards, 37). We argue that SE-Block is another form of attention
mechanism, capturing the difference in the importance of features in different
domains. $F_{se}$ denotes a $(FC,Relu,FC)$ block and $F_{avg}$ denotes average
pooling operator. ${\bm{\alpha}}^{(d)}$ denotes $N-dimensional$ SE attention
scores vector for domain $d$.

(25)
${\bm{\alpha}}^{(d)}=F_{se}(concat(F_{avg}({F_{1}^{(d)}})~{}|~{}\cdot\cdot\cdot~{}|~{}F_{avg}(F_{N}^{(d)}))),$
(26)
$\hat{F}^{(d)}={\bm{\alpha}}^{(d)}\odot{concat(F_{1}^{(d)}~{}|~{}\cdot\cdot\cdot~{}|~{}F_{N}^{(d)})}.$

SE-Block based domain adaptation layer learns different domain attention
weights for different domains, transferring cross-domain knowledge in a
lightweight and efficient way.

By adding domain interest adaptation layer into the backbone network, the raw
features are transferred into domain-related features. Experiments and
visualization in Section 5.3 prove the effectiveness of the proposed domain
interest adaptation layer.

###  4.3. Self Training

Input: Matching function $f_{\theta}(X,d)$, training data $(X^{(d)},d)$
sampled from domain $d$ with label $y^{(d)}$, hyper-parameter $p$ to determine
the portion of pseudo-labels.

Output: The trained parameter $\theta^{*}$.

1 Initializing $\theta$ randomly.

2 for  _$epoch\leftarrow 1$ to $EPOCHS$_ do

3 for  _$iter\leftarrow 1$ to $ITERATION\\_NUM$_ do

4 for  _$i\leftarrow 1$ to $D$_ do

5 Calculate loss for all training data $(X^{(i)},y^{(i)})$

6 for  _$j\leftarrow 1$ to $D$_ do

7 if  _$i\neq j$_ then

8 Embed $(X^{(i)},i)$ sampled from domain $i$

9 Compute prediction scores for $(X^{(i)},j)$

10 Sort by $Scores^{(j)}_{(i)}=f_{\theta}(X^{(i)},j)$

11 Select top $p$ percent samples as pseudo data $(X_{p}^{(i)},y_{p}^{(j)})$

12 Calculate loss for pseudo data $(X_{p}^{(i)},y_{p}^{(j)})$

13 end if

14

15 end for

16

17 end for

18 $p=p+\Delta{p}$

19

20 end for

21

22 end for

Algorithm 1 Self Training For Multi-Domain Retrieval Table 2. Overall
performance comparisons on Alibaba production dataset, we use Recall@N as the
metric .

Method | Domain #1 | Domain #2 | Domain #3
---|---|---|---
R@100 | R@500 | R@1000 | R@100 | R@500 | R@1000 | R@100 | R@500 | R@1000
DNN-Single | 0.1862 | 0.3934 | 0.5109 | 0.1170 | 0.2653 | 0.3647 | 0.1029 | 0.2401 | 0.3189
DNN | 0.1845 | 0.3877 | 0.5077 | 0.1548 | 0.3354 | 0.4429 | 0.1168 | 0.2606 | 0.3597
Shared-Bottom | 0.1418 | 0.3330 | 0.4323 | 0.1247 | 0.2980 | 0.3890 | 0.0796 | 0.2196 | 0.2860
Cross-Stitch | 0.1476 | 0.3245 | 0.4416 | 0.1306 | 0.2901 | 0.3978 | 0.0879 | 0.2054 | 0.3028
MMoE | 0.1860 | 0.3988 | 0.5142 | 0.1633 | 0.3549 | 0.4619 | 0.1097 | 0.2609 | 0.3530
PLE | 0.1908 | 0.4076 | 0.5248 | 0.1777 | 0.3799 | 0.4901 | 0.1102 | 0.2693 | 0.3662
ADI-LT | 0.1935 | 0.4086 | 0.5244 | 0.1691 | 0.3627 | 0.4694 | 0.1018 | 0.2483 | 0.3387
ADI-VA | 0.2062 | 0.4292 | 0.5465 | 0.1865 | 0.3905 | 0.5005 | 0.1123 | 0.2712 | 0.3672
ADI-SE | 0.2325 | 0.4683 | 0.5874 | 0.2064 | 0.4222 | 0.5337 | 0.1259 | 0.2982 | 0.3999
ADI-SE (MAX) | 0.2438 | 0.4849 | 0.6039 | 0.2219 | 0.4429 | 0.5547 | 0.1468 | 0.3292 | 0.4342

Self training methods (zou2018unsupervised, 20, 12) have been proved as an
efficient learning strategy for exploiting unlabeled data during model
training. We apply this technology on the multi-domain recommendation in
retrieval step for two reasons: 1). There is a potential label-level
connection in training data when there are data overlaps between domains. To
be more specific, a interacted item by a user in one domain may still be
interacted by the same user in another domain. This assumption works
especially when larger domain helps minor domains or even new domains where
labeled data are limited. 2). Adding pseudo-labeled data into training
inevitable change the original data distribution, however, we argue that our
proposed self training method is more suitable for retrieval models rather
than ranking models. Ranking models in advertising systems need to predict
precise CTR scores (zhou2018deep, 38), adding extra pseudo-labeled data may
lead to unknowable performance since the data distribution has been changed
and CTR models are sensitive to the data distribution. However, retrieval
models for advertising systems aim to provide candidates set for downstream
tasks. In other words, precise CTR score are not necessary for retrieval
models since multiple candidates will be generated equally. Therefore
additional potential interest signals can be added into the model even if the
data distribution is slightly changed for generating high-quality topK
candidates. Existing methods mostly focus on sample-level (wang2019minimax,
32), feature level (shen2021sar, 14), and parameter level (sheng2021one, 11)
transferring, while neglecting label-level transferring. Therefore, we
proposed this efficient self training method to mine the potential label-level
transferring knowledge though domains, which has been proved effective in our
experiments.

Given an item $v$ interacted by user $u$ in domain $d$, the self training
method follows two steps. a). freeze the model to generate pseudo-labels for
$v$ in other domains except domain $d$. b). freeze the pseudo-labels then
fine-tune the model. Following Algorithm 1, for each step, we select pseudo-
labels with highest confidence scores, and the selection portion gradually
increases during training. After obtaining pseudo-labels for $v$ in other
domains, $\theta$ in Eq.3 is trained to minimize the negative log likelihood
$-\log s_{\theta}(v|u,d)$ over the training data and pseudo-labeled data:

(27)
$\theta^{*}=\operatornamewithlimits{argmin}_{\theta}\sum_{d}\sum_{u}\sum_{v\in\mathcal{B}_{u,d}}-(\log
s_{\theta}(v|u,d)+\log s_{\theta}(\tilde{v}|u,d)).$

where $\tilde{v}$ is the selected potential positive pseudo-items given user
$u$ and domain $d$.

Table 3. Overall performance comparisons on WSDM Cross-Market dataset. We use
Recall@10 as metric and our ADI achieves average SOTA performance.

Method | Domain #1 | Domain #2 | Domain #3 | Avg
---|---|---|---|---
DNN-Single | 0.250 | 0.535 | 0.572 | 0.453
DNN | 0.193 | 0.380 | 0.460 | 0.344
Shared-Bottom | 0.261 | 0.507 | 0.537 | 0.435
Cross-Stitch | 0.252 | 0.404 | 0.510 | 0.388
MMoE | 0.273 | 0.611 | 0.368 | 0.417
PLE | 0.201 | 0.548 | 0.525 | 0.424
ADI | 0.266 | 0.614 | 0.519 | 0.466

##  5\. Experiments

###  5.1. Experimental Settings

####  5.1.1. Dataset Description

Table 4. Statistics of datasets used in experiments.

Dataset | Users | Items | Record in D#1 | Record in D#2 | Record in D#3
---|---|---|---|---|---
Alibaba Production | 92,664,693 | 2,240,723 | 1,364,983,432 | 167,494,696 | 609,611,432
WSDM Cross-Market | 16,903 | 10,994 | 77,173 | 48,302 | 23,367

Table 5. User/Item Overlap between different domains.

Dataset | User/Item | D#1 | D#2 | D#3
---|---|---|---|---
Alibaba Prodcution | D#1 | 78,336,157/1,735,326 | 6,840,732/1,295,028 | 7,043,405/1,525,081
D#2 | - | 11,147,803/1,383,050 | 5,551,052/1,279,533
D#3 | - | - | 13,099,654/1,998,209
WSDM Cross-Market | D#1 | 6466/9762 | 0/1190 | 0/752
D#2 | - | 7109/2198 | 0/816
D#3 | - | - | 3328/1245

Two datasets are used to validate our proposed ADI. One is real industrial
data named Alibaba Display Advertising Data, and the other is publicly
accessible data called WSDM Cross-Market Recommendation Data. Table 4 and
Table 5 show the statistic information of this two datasets. The full
description are described as follows.

Alibaba Display Advertising Data: The Alibaba production data regarding
advertising exposure to the consumer on 3 business domains as positive sample,
which is collected from traffic logs of the Alibaba online display advertising
system and divided by scenarios’ bid price assigned rules. The dataset
consists of billions of samples with user behavior, ad attribute and ad
domain-specific statistic features.

WSDM Cross-Market Recommendation Data: The user purchase and rating data on
various markets with a considerable number of shared item subsets, provided in
”WSDM 2022 CUP - Cross-Market Recommendation”. We regard 5 ratings samples as
positive. The training dataset consists of millions of examples with no other
features but userId and itemId. So the feature-level domain attention method
cannot be evaluate on this dataset. The original dataset contains three source
domains and two target domains. To keep a consistent model structure with the
former, without loss of generality, we use only three source domains to
evaluate our proposed method.

####  5.1.2. Comparing methods

  * •

DNN-Single: This method is a implement of YouTube DNN (yi2019sampling, 4), one
of the most well-known recommendation method in industry. In this version, we
train $D$ models for $D$ domains separately. All comparing methods train one
model for all domains except this one.

  * •

DNN: This version is YouTube DNN trained with mixed data from different
domains.

  * •

Shared-Bottum (caruana1997multitask, 28): This method is a classical
architecture for multi-task/domain adaptation, in which all tasks share one
common network at the bottom and each task utilize separated network at the
top.

  * •

Cross-Stitch (misra2016cross, 39): Cross-Stitch designs $D$ networks at the
bottom and learns a optimized linear combination of bottom-network outputs as
its task-specific outputs.

  * •

MMoE (ma2018modeling, 29): MMoE designs $N$ expert networks at the bottom to
capture task-related signals and learns a combination of $N$ expert networks
trough its gating mechanism.

  * •

PLE (tang2020progressive, 30): PLE is optimized version of MMoE. By designing
task-specific expert and common expert to alleviate the seesaw phenomenon.

  * •

ADI-LT: Our first variant version of ADI, equipped with DSBN, linear
transformation and self-training module.

  * •

ADI-VA: Our second variant version of ADI, equipped with DSBN, vanilla domain
attention and self-training module.

  * •

ADI-SE/ADI-SE (MAX): Our third variant version of ADI, equipped with DSBN, SE-
Block based domain adaptation layer and self-training module. The difference
between ADI-SE and ADI-SE (MAX) is that ADI-SE (MAX) contains more than one
shared network in the backbone network.

####  5.1.3. Evaluation and Metrics

For Alibaba Display Advertising dataset, data of one day from 3 business
domains are used for training and the data of the following day is used for
testing. For WSDM Cross-Market Recommendation dataset, testing data is given
along with training data, in which each user has 100 candidate items in
contrast to retrieval from whole corpus and only one of them is the positive
sample. Following existing works (zhu2018learning, 7), we use Recall@N as our
performance metrics.

####  5.1.4. implementation details

All comparing methods share same input and equipped with BN (ADI is equipped
with DSBN). Besides, for a fair comparison, the sizes of the model parameters
are kept the same for all methods. For example, suppose there are 3 domains,
we set 3 domain-specific networks and 1 shared network for PLE and ADI. At the
meantime we set 4 expert networks for MMoE. Each network share exactly the
same model size.

Figure 3. The impact on numbers of shared networks. (3+K shared) in x-axis
denotes 3 domain-specific and K shared networks

###  5.2. Overall Performance

Alibaba Production: As shown in table. 2, our method outperforms among
existing works significantly. It is worth noting that the performance of DNN
on domain #2, domain #3 is better than DNN-single, while performance on domain
#1 is worse. Such phenomenon shows that simply mix training data without
designing particular model architecture may hurt the model performance.

Different variants shows the impact of different domain interest adaptation
layers. ADI-LT meets the worst performance, while ADI-SE achieves the best
performance. To sum up, ADI-VA and ADI-SE both achieve overall performance
uplift for all domains.The recall of ADI-SE (MAX) is better than ADI-SE proves
that using more shared networks uplifts the model performance, but the model
complexity also increases. It is a trade-off of choosing a proper model with
good performance and affordable parameter complexity.

WSDM Cross-Market: Results are shown in table. 3. This dataset only contains
user/item ID features and ratings, making it impossible to equip ADI with
domain interest attention layer module (Because only user/item ID feature is
incorporated in the user/item tower). In addition, there are no overlap users
between domains in this dataset shown in table 5. In our final implementation
version, we use the backbone network and DSBN module. Therefore our ADI
doesn’t get an overall best performance on all domains but only achieves the
best average performance.

###  5.3. Discussions

To better understand the effectiveness of our proposed ADI, We conduct several
interesting discussions on the Alibaba Production dataset.

Table 6. Variant versions’ performances of the fusion layer.

Variants | R@500
---|---
Domain #1 | Domain #2 | Domain #3
SUM | 0.4567 | 0.4105 | 0.2686
Network-Mul | 0.4619 | 0.4023 | 0.2706
CONCAT | 0.4683 | 0.4222 | 0.2982

####  5.3.1. Variants of the fusion layer

As mentioned in Section 4.1.3, We implement different versions of the fusion
layer. It is worth noting that our proposed method equiped with Network-Mul is
exactly the same to the model architecture of STAR (sheng2021one, 11) for the
MDR problem. The results are showed in Table 6. Our proposed CONCAT achieves
the best, while the original SUM version of MMoE/PLE (ma2018modeling, 29, 30)
meets the worst performance. The performance of Network-Mul version used by
STAR is between two variants.

####  5.3.2. Ablation Study

Table 7. Ablation study.

Method | R@500
---|---
Domain #1 | Domain #2 | Domain #3
ADI-SE | 0.4683 | 0.4222 | 0.2982
w/o ST | 0.4462 | 0.4022 | 0.2744
w/o ST&DIAL | 0.4164 | 0.3784 | 0.2576
w/o ST&DIAL&DSBN | 0.3964 | 0.3555 | 0.2399

To study the effectiveness of each component of ADI, we conduct several
ablation studies. We list those models without part of components as follows.

  * •

ADI-SE: Full model version of ADI.

  * •

w/o ST: ADI trained without self training method.

  * •

w/o ST&DIAL: ADI without domain interest adaptation layer and self training
method.

  * •

w/o ST&DIAL&DSBN: ADI without DSBN, domain interest adaptation layer and self
training method.

DSBN to capture the diversity of data distributions from different domains,
domain interest adaptation layer to assign different feature weights for
different domains, and self-training method to exploit potential label-level
connections between domains, all improve the performance of the model as can
be seen in Table 7

####  5.3.3. Impact on numbers of shared experts

The numbers of shared expert networks effect the final model performance.
Therefore, we change the shared expert networks of several methods from 1 to
5. As can be seen from Figure 3, we compare MMoE, PLE and our proposed ADI
with different numbers of shared expert networks in the exactly same model
size setting (For MMoE, all experts are shared experts). The results show that
with the shared expert networks increasing, the performances of PLE and ADI
increase, while MMoE’s increases at the beginning then decreases.

Figure 4. Different variants of SE-Block usages. 1-SE version denotes applying
a global SE-Block to reweight raw features. 3-SE version denotes using the SE-
Block selected by domain indicator to process feature-level domain adaptation.
4-SE version denotes equipping each domain-specific network and shared network
with a SE-Block, in order to increasing the model capability.

####  5.3.4. Usage of SE-Block

Table 8. Variant versions’ performances of SE-Blocks.

Variants | R@500
---|---
Domain #1 | Domain #2 | Domain #3
1-SE | 0.4415 | 0.3922 | 0.2804
3-SE | 0.4683 | 0.4222 | 0.2982
4-SE | 0.4516 | 0.4041 | 0.2917

Different variants of adding SE-Block into the backbone model gain
differential improvements of the model performance. As shown in Figure 4, we
compare three usages of SE-Block, which are described as follows. Here we use
the simplest version of ADI which consist 3 domain-specific networks and 1
shared network.

  * •

1-SE: We use SE-Block as an universal domain interest adaptation layer for
dynamically feature-level domain adapting.

  * •

3-SE: We keep $D$ domain adaptation layers with $D$ SE-Blocks. Which domain
adaptation layer to use is depended on the domain indicator.

  * •

4-SE: SE-Block is added into each expert network. In this variant SE-Block is
treated as a method of increasing model complexity rather than domain
adaptation.

Based on results shown in Table 8, we argue that proper usage of SE-Block
matters for the final model performance. The version of 1-SE gets the worst
performance and 3-SE gets the best. An interesting observation is that 4-SE
version is no better than 3-SE, even though the model parameters increase.
Therefore, we suggest using 3-SE version as the domain adaptation layer, which
re-weight feature embeds at the sample domain-level rather than the network-
level.

####  5.3.5. Visualization

Figure 5. The attention weight visualization of each domain adaptation layer
on item side.

To give an intuitive evidence of effectiveness of the domain adaptation layer,
we visualize the attention weights for different domains. As shown in Figure.
5, the different darkness distributions suggest that each domain focuses on
different features. It is worth note that we draw domain-related features in
colors. To be more specific, we draw statistic features ( CTR, CLICK, COST and
PAYNUM) from domain #1 in red, domain #2 in green and domain #3 in yellow. As
we can seen, the adaptation layer automatically assign domain-related features
higher attention scores, which proved the effectiveness of our proposed domain
adaptation layer.

Table 9. Online A/B Test

Online Metric | Domain #1 | Domain #2 | Domain #3
---|---|---|---
RPM | +1.9% | +0.7% | +2.6%
PPC | +2.0% | -0.2% | +1.6%
CTR | +0.0% | +0.9% | +1.0%

##  6\. Online A/B Test

We have deployed our model on the display advertising system of Alibaba. To
get a stable conclusion, we observe the online experiment for two weeks. Four
common metrics in advertising system are used to measure the online
performance: RPM(Revenue Per Mille), PPC(Pay Per Click), CTR(Click Through
Rate) . As the result shown in Table 9, the present method ADI gets overall
improvements on three domains in our online A/B test experiment.

##  7\. Conclusion

In this paper, we have investigated the problem of multi-domain
recommendation. Compared with exsiting works, our proposed ADI attempts to
apply domain adaptation on retrieval stage of a recommendation system. The
backbone network effectively learns commonalities and diversities for multiple
domains. The DSBN component and the domain interest adaptation layer are
applied for feature-level domain adaptation. And the self training method
captures potential label-level connections across domains. Experiments on
public and industrial datasets validate the superiority of our proposed
method. Extensive discussions verify the effectiveness of the proposed method
and conduct suggestions of proper usage of each component. The proposed ADI
has been deployed on Alibaba display advertising system and gain significant
profits, validating the commercial values of the proposed method.

## References

  * (1) Badrul Sarwar, George Karypis, Joseph Konstan and John Riedl  “Item-based collaborative filtering recommendation algorithms”  In _Proceedings of the 10th international conference on World Wide Web_ , 2001, pp. 285–295
  * (2) Jiawei Han, Jian Pei and Micheline Kamber  “Data mining: concepts and techniques”  Elsevier, 2011
  * (3) Paul Covington, Jay Adams and Emre Sargin  “Deep neural networks for youtube recommendations”  In _Proceedings of the 10th ACM conference on recommender systems_ , 2016, pp. 191–198
  * (4) Xinyang Yi et al.  “Sampling-bias-corrected neural modeling for large corpus item recommendations”  In _Proceedings of the 13th ACM Conference on Recommender Systems_ , 2019
  * (5) Jui-Ting Huang et al.  “Embedding-based retrieval in facebook search”  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 2553–2561
  * (6) Chao Li et al.  “Multi-interest network with dynamic routing for recommendation at Tmall”  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , 2019, pp. 2615–2623
  * (7) Han Zhu et al.  “Learning tree-based deep model for recommender systems”  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 1079–1088
  * (8) Han Zhu et al.  “Joint optimization of tree-based index and deep model for recommender systems”  In _arXiv preprint arXiv:1902.07565_ , 2019
  * (9) Jingwei Zhuo et al.  “Learning optimal tree models under beam search”  In _International Conference on Machine Learning_ , 2020, pp. 11650–11659  PMLR
  * (10) Weihao Gao et al.  “Deep Retrieval: Learning A Retrievable Structure for Large-Scale Recommendations”  In _arXiv preprint arXiv:2007.07203_ , 2020
  * (11) Xiang-Rong Sheng et al.  “One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction”  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 4104–4113
  * (12) Yang Zou et al.  “Confidence regularized self-training”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5982–5991
  * (13) Xiao Ma et al.  “Entire space multi-task model: An effective approach for estimating post-click conversion rate”  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, 2018, pp. 1137–1140
  * (14) Qijie Shen et al.  “SAR-Net: A Scenario-Aware Ranking Network for Personalized Fair Recommendation in Hundreds of Travel Scenarios”  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 4094–4103
  * (15) Xiaobo Hao et al.  “Adversarial Feature Translation for Multi-domain Recommendation”  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 2021, pp. 2964–2973
  * (16) Yaroslav Ganin and Victor Lempitsky  “Unsupervised domain adaptation by backpropagation”  In _International conference on machine learning_ , 2015, pp. 1180–1189  PMLR
  * (17) Eric Tzeng, Judy Hoffman, Kate Saenko and Trevor Darrell  “Adversarial discriminative domain adaptation”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7167–7176
  * (18) Konstantinos Bousmalis et al.  “Domain separation networks”  In _Advances in neural information processing systems_ 29, 2016, pp. 343–351
  * (19) Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki and Kiyoharu Aizawa  “Cross-domain weakly-supervised object detection through progressive domain adaptation”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5001–5009
  * (20) Yang Zou, Zhiding Yu, BVK Kumar and Jinsong Wang  “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training”  In _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 289–305
  * (21) Tong Man, Huawei Shen, Xiaolong Jin and Xueqi Cheng  “Cross-Domain Recommendation: An Embedding and Mapping Approach.”  In _IJCAI_ 17, 2017, pp. 2464–2470
  * (22) Pan Li and Alexander Tuzhilin  “Ddtcdr: Deep dual transfer cross domain recommendation”  In _Proceedings of the 13th International Conference on Web Search and Data Mining_ , 2020, pp. 331–339
  * (23) Guangneng Hu, Yu Zhang and Qiang Yang  “Conet: Collaborative cross networks for cross-domain recommendation”  In _Proceedings of the 27th ACM international conference on information and knowledge management_ , 2018, pp. 667–676
  * (24) Sebastian Ruder  “An overview of multi-task learning in deep neural networks”  In _arXiv preprint arXiv:1706.05098_ , 2017
  * (25) Falk Heuer, Sven Mantowsky, Saqib Bukhari and Georg Schneider  “Multitask-centernet (mcn): Efficient and diverse multitask learning using an anchor free approach”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 997–1005
  * (26) Xiaodong Liu, Pengcheng He, Weizhu Chen and Jianfeng Gao  “Multi-task deep neural networks for natural language understanding”  In _arXiv preprint arXiv:1901.11504_ , 2019
  * (27) Aashi Jain et al.  “MURAL: multimodal, multitask retrieval across languages”  In _arXiv preprint arXiv:2109.05125_ , 2021
  * (28) Rich Caruana  “Multitask learning”  In _Machine learning_ 28.1 Springer, 1997, pp. 41–75
  * (29) Jiaqi Ma et al.  “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts”  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 1930–1939
  * (30) Hongyan Tang, Junning Liu, Ming Zhao and Xudong Gong  “Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations”  In _Fourteenth ACM Conference on Recommender Systems_ , 2020, pp. 269–278
  * (31) Jiaqi Ma et al.  “SNR: sub-network routing for flexible parameter sharing in multi-task learning”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 33.01, 2019, pp. 216–223
  * (32) Bo Wang et al.  “A minimax game for instance based selective transfer learning”  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2019, pp. 34–43
  * (33) Sébastien Jean, Kyunghyun Cho, Roland Memisevic and Yoshua Bengio  “On Using Very Large Target Vocabulary for Neural Machine Translation”  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ Beijing, China: Association for Computational Linguistics, 2015, pp. 1–10  DOI: 10.3115/v1/P15-1001
  * (34) Sergey Ioffe and Christian Szegedy  “Batch normalization: Accelerating deep network training by reducing internal covariate shift”  In _International conference on machine learning_ , 2015, pp. 448–456  PMLR
  * (35) Woong-Gi Chang et al.  “Domain-specific batch normalization for unsupervised domain adaptation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7354–7362
  * (36) Jie Hu, Li Shen and Gang Sun  “Squeeze-and-excitation networks”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141
  * (37) Xudong Wang, Zhaowei Cai, Dashan Gao and Nuno Vasconcelos  “Towards universal object detection by domain attention”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7289–7298
  * (38) Guorui Zhou et al.  “Deep interest network for click-through rate prediction”  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 1059–1068
  * (39) Ishan Misra, Abhinav Shrivastava, Abhinav Gupta and Martial Hebert  “Cross-stitch networks for multi-task learning”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3994–4003


Shilin<EMAIL_ADDRESS>Hieu<EMAIL_ADDRESS>Dimitris
<EMAIL_ADDRESS>Computer Science Department
Stony Brook University
New York, USA Temporal Feature warping for VSD

# Temporal Feature Warping for Video Shadow Detection

###### Abstract

While single image shadow detection has been improving rapidly in recent
years, video shadow detection remains a challenging task due to data scarcity
and the difficulty in modelling temporal consistency. The current video shadow
detection method achieves this goal via co-attention, which mostly exploits
information that is temporally coherent but is not robust in detecting moving
shadows and small shadow regions. In this paper, we propose a simple but
powerful method to better aggregate information temporally. We use an optical
flow based warping module to align and then combine features between frames.
We apply this warping module across multiple deep-network layers to retrieve
information from neighboring frames including both local details and high-
level semantic information. We train and test our framework on the ViSha
dataset. Experimental results show that our model outperforms the state-of-
the-art video shadow detection method by 28%, reducing BER from 16.7 to 12.0.

##  1 Introduction

Shadows appear in most natural images. Simply detecting shadows benefits many
computer vision tasks such as image classification [Filippi and İnci
Güneralp(2013)], image segmentation [Xu et al.(2019)Xu, Chen, Su, Ji, Xu,
Memon, and Zhou] and object tracking [Saravanakumar et al.(2010)Saravanakumar,
Vadivel, and Saneem Ahmed, Mohanapriya and Mahesh(2017)]. Therefore, shadow
detection has drawn a lot of interest in recent years especially with the
rapid development of deep-learning-based methods [Nguyen et al.(2017)Nguyen,
Yago Vicente, Zhao, Hoai, and Samaras, Zheng et al.(2019)Zheng, Qiao, Cao, and
Lau, Wang et al.(2018)Wang, Li, and Yang, Ding et al.(2019)Ding, Long, Zhang,
and Xiao, Le et al.(2018)Le, Vicente, Nguyen, Hoai, and Samaras]. However,
recent shadow detection works mostly deal with shadows in single images while
video shadow detection remains an open question despite many potential
applications [Wang et al.(2020)Wang, Curless, and Seitz, Le and
Samaras(2020a), Le and Samaras(2020b), Le et al.(2016)Le, Nguyen, Yu, and
Samaras].

A shadow video typically consists of hundreds of frames that contain shadows
varying in shapes and intensities. The detection problem is compounded by
video-specific issues such as motion blur. Thus, simply applying image shadow
detection methods frame-by-frame on video often yields inconsistent
predictions (see Fig 1.d). Instead, a common strategy to deal with video data
is to leverage temporal information across video frames [Zhu et al.(2017)Zhu,
Xiong, Dai, Yuan, and Wei, Yan et al.(2019)Yan, Li, Xie, Li, Wang, Chen, and
Lin]. Here the difficulty lies in how to incorporate information across video
where there is spatial misalignment between frames due to the movements of
objects and cameras.

In this paper, we propose a straightforward but powerful deep-learning based
method to obtain a rich feature representation for video shadow detection. We
focus on dealing with the temporal misalignment between frame representations.
Our strategy is simple: we align the features across frames by optical flow
and then linearly combine them to obtain the per-frame final feature
representation. Optical flow is easy to obtain [Liu et al.(2020)Liu, Zhang,
He, Liu, Wang, Tai, Luo, Wang, Li, and Huang] and can effectively align
spatial content, including small details. As this warping is computationally
efficient, we can apply it on multiple layers of our network.

We report that this simple optical-flow based feature aggregation scheme works
surprisingly well for shadow detection in videos. We train our model in an
end-to-end fashion on the Visha dataset [Chen et al.(2021)Chen, Wan, Zhu,
Shen, Fu, Liu, and Qin]. Our method achieves state-of-the-art video shadow
detection performance, outperforming the previous method [Chen et
al.(2021)Chen, Wan, Zhu, Shen, Fu, Liu, and Qin] by a 28% BER reduction. Fig 1
illustrates the effect of our proposed temporal feature warping method. As can
be seen, the features of two visually similar frames could be wildly different
(column c), which results in inconsistent outputs (column d). Our method warps
and then combines the features, making them consistent across frames (column
e), and finally outputs stable and temporally consistent results (last
column).

Figure 1: Motivation of our work. (a) shows a pair of consecutive video
frames, (b) shows the ground truth shadow masks, (c) shows the original
extracted high level feature maps. Even though the difference between images
is small, the difference between features is high, (d) shows shadow mask
prediction using original features, (e) is combined features obtained by our
method, (f) shows shadow prediction using combined features. Here we
demonstrate the effectiveness of our method; the combined feature maps
preserve correspondence between frames and produce more stable predictions.

##  2 Related Work

Single image shadow detection is a well studied topic. On one hand, earlier
research on image shadow detection mostly focuses on spectral or spatial
features of images such as chromaticity, physical properties, geometry, and
texture [Sanin et al.(2012)Sanin, Sanderson, and Lovell]. On the other hand,
recent shadow detection methods show tremendous success with the rapid
development of deep learning. Le _et al_[Le et al.(2018)Le, Vicente, Nguyen,
Hoai, and Samaras] propose to train a shadow detection network together with a
shadow attenuation network that generates adversarial training examples. Hu
_et al_[Hu et al.(2018)Hu, Zhu, Fu, Qin, and Heng] propose a directional-aware
feature extractor for aggregating spatial information. Zhu _et al_[Zhu et
al.(2018)Zhu, Deng, Hu, Fu, Xu, Qin, and Heng] utilize recurrent attention
residual modules to fully aggregate the global and local contexts in different
layers of the CNN to detect shadows. Chen _et al_[Chen et al.(2020)Chen, Zhu,
Wan, Wang, Feng, and Heng] further improve detection performance by
introducing a multi-task mean teacher architecture which leverages unlabeled
data. However, image methods trained on image datasets such as SBU [Vicente et
al.(2016b)Vicente, Hou, Yu, Hoai, and Samaras] and CUHK-Shadow [Hu et
al.(2021)Hu, Wang, Fu, Jiang, Wang, and Heng] do not generalize well to videos
due to the lack of temporal consistency.

Video shadow detection is a classic problem on its own. Earlier work [Jacques
et al.(2005)Jacques, Jung, and Musse, Shi and Liu(2019)] focuses on the
spectral and spatial features, which depend heavily on the quality of data.
Without temporal constraints, these methods often output inconsistent
predictions across frames. The first large-scale video shadow detection
dataset was proposed by Chen _et al_[Chen et al.(2021)Chen, Wan, Zhu, Shen,
Fu, Liu, and Qin]. The dataset contains 120 fully-annotated videos with a
total of 11,685 frames. They also proposed the first deep-learning-based
method for video shadow detection in which a dual gated co-attention module is
used to focus on common high-level features between frames. This co-attention
module allows their network to filter out temporally inconsistent information
from each frame representation to obtain more stable and consistent results.
However, this mechanism causes the method to be less sensitive to shadow areas
that substantially change across frames due to temporal misalignment. By
contrast, our method performs temporal alignment before combining features.
Besides, our temporal alignment module can be used for all layers of the
network, allowing us to pick up even small shadow areas. Co-attention can only
be applied on high-level feature maps due to its computational cost.

Optical flow is used in various high-level video tasks [Shin et al.(2005)Shin,
Kim, Kang, Lee, Paik, Abidi, and Abidi, Zhong et al.(2013)Zhong, Liu, Ren,
Zhang, and Ren, Buades et al.(2016)Buades, Lisani, and Miladinović]. Recent
deep learning based methods [Ilg et al.(2017)Ilg, Mayer, Saikia, Keuper,
Dosovitskiy, and Brox, Sun et al.(2018)Sun, Yang, Liu, and Kautz] for optical
flow estimation are fairly accurate and efficient in inference. However, most
popular datasets used in training optical flow estimation models, _e.g_ , MPI
Sintel [Butler et al.(2012)Butler, Wulff, Stanley, and Black] and KITTI 2015
[Menze and Geiger(2015)], are sufficiently different from shadow detection
datasets [Wang et al.(2018)Wang, Li, and Yang, Vicente et al.(2016a)Vicente,
Hoai, and Samaras, Hu et al.(2021)Hu, Wang, Fu, Jiang, Wang, and Heng]. To
compensate for this domain shift, we use a simple module to refine optical
flow before using it to warp our features. Liu _et al_[Liu et al.(2020)Liu,
Zhang, He, Liu, Wang, Tai, Luo, Wang, Li, and Huang] propose an unsupervised
dense optical flow estimation network with better cross dataset generalization
capability by learning from abundant augmentations of training data.

##  3 Method

###  3.1 Overview

The overall structure of our framework is illustrated in Fig 2. Our model
consists of two branches with identical architectures. The input of our model
is a RGB video frame pair. The two images are input to the two branches of the
model to extract two sets of feature maps across the three different layers of
the network. Throughout the network, these features are progressively enriched
by the information from the features of the other image via temporal warping
and linear combination.

In particular, we first obtain two dense optical flow fields from the two
images using ARFlow [Liu et al.(2020)Liu, Zhang, He, Liu, Wang, Tai, Luo,
Wang, Li, and Huang]. Following [Gadde et al.(2017)Gadde, Jampani, and Gehler,
Li et al.(2021)Li, Zhao, He, Zhu, and Liu], we train a small module to refine
these optical flow fields to better suit the feature warping task, depicted as
FlowCNN in Fig 2. At multiple layers of the network, we combine the features
of each branch with the aligned features from the other. A flow-guided warp
(FGwarp) module is used to first warp the frame feature representation to
spatially align it with the content of the other frame and then linearly
combine the original features with the warped features from the other frame.
This simple feature aggregation scheme ensures consistency between the two
frame representations. The combined features of different levels are input
through a shadow refinement module to generate the final shadow mask
prediction.

Figure 2: Simplified network architecture. Our network consists of two
branches with identical architecture. Each branch predicts the shadow mask for
the corresponding input frame. Here we only show the data flow from the “Frame
t” branch to the “Frame t+k” branch for simplicity. In practice, feature
warping and combination goes both ways. At multiple layers of the network, we
feed the outputs of both branches into a flow-guided warp (FGwarp) module
(details in Fig.3) to obtain a temporally enriched feature (colored as
orange). We use a shadow feature refinement module to predict the shadow mask
from all combined features.

###  3.2 Shadow detector network

Our network consists of two branches with identical architecture. For each
branch, we use the MobileNet V2 [Sandler et al.(2018)Sandler, Howard, Zhu,
Zhmoginov, and Chen] as the backbone feature extractor. Each branch consists
of a series of inverted residual bottlenecks (IRBs) [Sandler et
al.(2018)Sandler, Howard, Zhu, Zhmoginov, and Chen]. We input the feature maps
of the $4^{th}$, the $7^{th}$ and the last block of each branch, which encode
the low, mid, and high-level features respectively, to the FGwarp module to
obtain the corresponding combined features. Finally, these three combined
feature maps are input to a detail enhancement module [Hu et al.(2021)Hu,
Wang, Fu, Jiang, Wang, and Heng] to refine the features and predict the final
shadow mask.

###  3.3 Optical flow estimation

We use a pre-trained ARFlow [Liu et al.(2020)Liu, Zhang, He, Liu, Wang, Tai,
Luo, Wang, Li, and Huang] model to produce optical flow for our two input
frames. This network is trained on MPI Sintel [Butler et al.(2012)Butler,
Wulff, Stanley, and Black] which differs in object types and occlusions from
our shadow video data. Thus, we train a flow refinement module, FlowCNN, to
refine (by adapting the domain) the output of ARFlow to better fit the feature
warping task. The input for the FlowCNN consists of the optical flow from
ARFlow, the two input frames, and the pixel-wise difference of the two frames.
The network consists of 4 convolutional layers, in which the first two layers
are followed by a BatchNorm and a ReLU layer. The output of the third layer is
then concatenated with the original flow and passed to the last convolution
layer to obtain the refined optical flow. We train our proposed FlowCNN
together with the shadow detector network in an end-to-end manner.

###  3.4 Flow warping and combination

Figure 3: FGwarp module. The feature map from frame t is warped using the flow
field computed from our FlowCNN. The warped feature is then linearly combined
with the original feature of frame t+k to form the aggregated features. Note
that we resize the flow field to the size of the input feature maps.

We enforce consistency between frame feature representations by mutually
exchanging their intermediate features. Since motion in video causes the
spatial misalignment between the content of consecutive frames, we first need
to apply an optical flow based feature warping to align the features. A flow-
guided warp (FGwarp) module is defined for the task. Fig 3 illustrates this
scheme for transferring features from frame $t$ to frame $t+k$.

Given a pair of feature maps, $f_{1}$ and $f_{2}$, and the refined optical
flow $U_{1\rightarrow{2}}$. The feature $f_{2}$ can be warped to spatially
align with the contents of $f_{1}$. We perform this feature warping for each
channel of the feature map separately. The value at a pixel $p$, channel $c$
of the warped feature $\hat{f}_{1}^{c}$ can be computed as follows:

$\hat{f}_{1}^{c}(p)=\sum_{q}{B(q,p+U_{1\rightarrow{2}}(p))f_{2}^{c}(q)}$ (1)

where $q$ enumerates all spatial locations in the feature map and
$B(\cdot,\cdot)$ denotes the bi-linear interpolation kernel.

Given the refined optical flow ${U}_{{t}\rightarrow{t+k}}$ and the feature map
$f_{t}$, we can propagate the features from the $t^{th}$ frame to get the
aligned $\hat{f}_{t+k}$:

$\hat{f}_{t+k}=warp(f_{t},U_{{t}\rightarrow{t+k}}),\\\ $ (2)

The $warp(\cdot,\cdot)$ function is implemented by applying Eq.1 on different
feature map levels. Then, the combined feature map $\tilde{f}_{t+k}$ can be
computed as follows:

$\tilde{f}_{t+k}=w_{1}\odot f_{t+k}+w_{2}\odot\hat{f}_{t+k},\\\ $ (3)

where $w_{1}$ and $w_{2}$ represent the per-channel coefficients of the linear
combination that combines the two features, $\odot$ represents channel-wise
scalar multiplication. The resulting $\tilde{f}_{t+k}$ is then passed to the
following layers in the feature extractor backbone.

###  3.5 Training and Inference

We implement our framework using PyTorch. The feature extractor backbone is
initialized via pre-trained MobileNet V2 on ImageNet [Deng et al.(2009)Deng,
Dong, Socher, Li, Li, and Fei-Fei] while other components are trained from
scratch. We use stochastic gradient descent (SGD) with momentum of $0.9$ and
weight decay of $0.0005$ to optimize the whole network. The training is in an
end-to-end fashion where the objective function is to minimize the mean
squared error (MSE) between the ground-truth and predicted shadow masks. The
initial learning rate is set to $0.005$, updated by poly strategy [Liu et
al.(2015)Liu, Rabinovich, and Berg] with the power of $0.9$. We train the
model for $20$k iterations. All input frames are resized to $512\times 512$.
The coefficient vectors $w_{1}$ and $w_{2}$ are initialized as
$w_{1}=[1,1..,1]$ and $w_{2}=[0,0..,0]$, i.e., the temporal warping is not
enforced at start. $k$ is set to 1 in training, i.e., adjacent frames are used
as input pairs to train our model.

For inference, we resize the inputs to $512\times 512$. We predict shadow
masks for each pair of adjacent frames. Thus, each frame, except the first and
last one, will be used in two different inference passes. The final shadow
mask of each frame is the average of the frame’s two output shadow masks.

##  4 Experiments and Results

###  4.1 Evaluation datasets and metrics

Benchmark dataset We use the ViSha dataset [Chen et al.(2021)Chen, Wan, Zhu,
Shen, Fu, Liu, and Qin] to evaluate our proposed method. The ViSha dataset has
4788 frames from 50 videos for training and 6897 frames from 70 videos for
testing. All methods are trained on the training set and evaluated on the
testing set for a fair comparison.

Evaluation metric We employ the commonly-used balanced error rate (BER) to
evaluate shadow detection performance, which is defined as: BER
$=100\times(1-\frac{1}{2}(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}))$, where
$TP,TN,FP,FN$ are the total numbers of true positive, true negative, false
positive, and false negative pixels respectively. Since shadow pixels are
usually minority in natural images, the BER is less biased than mean pixel
accuracy. In general, lower BER indicates better shadow detection performance.
We also provide separate mean pixel error rates for the shadow and non-shadow
classes.

###  4.2 Comparison with the state-of-the-art

There is only one CNN-based video shadow detection method,TVSD [Chen et
al.(2021)Chen, Wan, Zhu, Shen, Fu, Liu, and Qin]. Besides, we compare our
method with state-of-the-art image shadow detection methods including BDRAR
[Zhu et al.(2018)Zhu, Deng, Hu, Fu, Xu, Qin, and Heng], MTMT [Chen et
al.(2020)Chen, Zhu, Wan, Wang, Feng, and Heng] and FSDNet [Hu et al.(2021)Hu,
Wang, Fu, Jiang, Wang, and Heng]. We obtain the results of TVSD from their
provided pre-trained model. We train BDRAR, MTMT, and FSDNet on the Visha
dataset using their official source-codes with their default settings.

Table 1: Comparison with the state-of-the-art shadow detection methods on the ViSha [Chen et al.(2021)Chen, Wan, Zhu, Shen, Fu, Liu, and Qin] dataset. Both Balanced Error Rate (BER) and per class error rates are shown. Best performances are printed in bold. Method | BER | Shadow | Non Shad.
---|---|---|---
BDRAR[Zhu et al.(2018)Zhu, Deng, Hu, Fu, Xu, Qin, and Heng] | 13.34 | 20.80 | 5.89
MTMT[Chen et al.(2020)Chen, Zhu, Wan, Wang, Feng, and Heng] | 14.55 | 21.15 | 7.94
FSDNet[Hu et al.(2021)Hu, Wang, Fu, Jiang, Wang, and Heng] | 14.59 | 18.88 | 10.30
TVSD[Chen et al.(2021)Chen, Wan, Zhu, Shen, Fu, Liu, and Qin] | 16.76 | 31.78 | 1.75
Ours | 12.02 | 11.88 | 12.17

####  4.2.1 Quantitative results

Table 1 reports the performance of all methods on the ViSha dataset. Our
method performs best on overall BER, we obtain a 10% error reduction and a 18%
error reduction compared with BDRAR and FSDNet, respectively. TVSD has the
best performance on non-shadow error. However, this is mainly because the
method is insensitive to shadow areas, especially the fast changing shadows
and the shadows with small areas. For the shadow areas, our method achieves
the lowest error rate. The results show that leveraging temporal information
on intermediate representations can boost the shadow detection performance for
videos significantly.

Figure 4: Comparison of shadow predictions by selected methods. (a) shows the
input images, (b) is the ground truth shadow masks and (c)-(g) are shadow
masks predicted by our method and BDRAR [Zhu et al.(2018)Zhu, Deng, Hu, Fu,
Xu, Qin, and Heng], MTMT [Chen et al.(2020)Chen, Zhu, Wan, Wang, Feng, and
Heng], FSDNet [Hu et al.(2021)Hu, Wang, Fu, Jiang, Wang, and Heng] and TVSD
[Chen et al.(2021)Chen, Wan, Zhu, Shen, Fu, Liu, and Qin]. Figure 5: Video
shadow detection results. Comparison between our proposed method, FSDNet and
TVSD. First row shows the input frames, second row shows the ground truth
shadow masks and the bottom two are the results of comparison methods.

####  4.2.2 Qualitative results

In Fig 4, we show some shadow detection results of our method in comparison
with other methods. In the first row, we can see that all methods produce
fairly accurate shadow masks with clear boundaries between shadow and non-
shadow areas. Comparing with FSDNet, our method generates shadow masks with
smoother boundaries without disjoint areas. Comparing with TVSD, our method is
able to capture more shadow details, as shown in the last two rows. Fig 5
shows the performance of our method on three consecutive video frames of two
example videos. As can be seen, TVSD is not robust in detecting moving shadows
and not sensitive to small shadow areas. By contrast, our method predicts more
accurate shadow masks in all cases. As we can see in the right three columns,
only our method is able to detect the fast moving shadow regions.

####  4.2.3 Failure cases

Figure 6: Failure cases. Failure cases caused by non-shadow dark objects and
limited training data diversity. (a) shows the input image, (b) shows the
ground truth and (c) shows the shadow mask predicted by our method.

Some failure cases of our method are shown in Fig 6. Many of them are caused
by dark objects being incorrectly classified as shadows. Dark objects are
naturally difficult for shadow detection, especially when the dark object is
the majority of scene. Correctly classifying these cases requires contextual
understanding of the scenes as well as sufficient training data. Note that the
training set of the Visha dataset has only 50 videos. Improving the
generalization of models trained on such small-sized dataset is challenging.

###  4.3 Application on Video Shadow Removal

Shadow masks can be further used to remove shadows to create shadow-free
images. We test our predicted shadow masks on the model proposed in [Le and
Samaras(2019), Le and Samaras(2020b)] which takes the original image and the
shadow mask as input and produces the shadow-free version of the image. Fig 7
shows the performance of using our predicted shadow mask and using the ground
truth shadow mask, results show that our video shadow detection method gets
reasonable results and can be used downstream to create shadow-free videos.

Figure 7: Application on shadow removal model. Shadow-free image generation
using shadow masks predicted by our method versus the ground truth. (a) shows
the input image, (b) shows the ground truth shadow mask, (c) shows the shadow
removal results using ground truth, (d) shows the shadow mask predicted by our
method and (e) shows the results using our shadow mask.

##  5 Summary

Large-scale video shadow detection is still in an early stage. In this paper,
we show that a basic and simple idea can be extremely effective for the task.
We deployed an optical-flow based feature warping and combination scheme to
enforce correspondence between representations. Highly correlated intermediate
representations led to an improvement of shadow prediction accuracy and
consistency in the videos. In experimental results our model was able to
handle large shadow appearance changes and capture small shadow regions, and
outperformed the state-of-the-art methods on the existing video dataset. We
believe our method will serve as an important stepping stone for future work.

## References

  * [Buades et al.(2016)Buades, Lisani, and Miladinović] Antoni Buades, Jose-Luis Lisani, and Marko Miladinović.  Patch-based video denoising with optical flow estimation.  _IEEE Transactions on Image Processing_ , 25(6):2573–2586, 2016.
  * [Butler et al.(2012)Butler, Wulff, Stanley, and Black] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black.  A naturalistic open source movie for optical flow evaluation.  In A. Fitzgibbon et al. (Eds.), editor, _European Conf. on Computer Vision (ECCV)_ , Part IV, LNCS 7577, pages 611–625. Springer-Verlag, October 2012.
  * [Chen et al.(2020)Chen, Zhu, Wan, Wang, Feng, and Heng] Zhihao Chen, Lei Zhu, Liang Wan, Song Wang, Wei Feng, and Pheng-Ann Heng.  A multi-task mean teacher for semi-supervised shadow detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [Chen et al.(2021)Chen, Wan, Zhu, Shen, Fu, Liu, and Qin] Zhihao Chen, Liang Wan, Lei Zhu, Jia Shen, Huazhu Fu, Wennan Liu, and Jing Qin.  Triple-cooperative video shadow detection.  In _CVPR_ , 2021.
  * [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255, 2009.
  * [Ding et al.(2019)Ding, Long, Zhang, and Xiao] Bin Ding, Chengjiang Long, Ling Zhang, and Chunxia Xiao.  Argan: Attentive recurrent generative adversarial network for shadow detection and removal.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [Filippi and İnci Güneralp(2013)] Anthony M. Filippi and İnci Güneralp.  Influence of shadow removal on image classification in riverine environments.  _Opt. Lett._ , 38(10):1676–1678, May 2013.
  * [Gadde et al.(2017)Gadde, Jampani, and Gehler] Raghudeep Gadde, Varun Jampani, and Peter V. Gehler.  Semantic video cnns through representation warping.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * [Hu et al.(2018)Hu, Zhu, Fu, Qin, and Heng] Xiaowei Hu, Lei Zhu, Chi-Wing Fu, Jing Qin, and Pheng-Ann Heng.  Direction-aware spatial context features for shadow detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [Hu et al.(2021)Hu, Wang, Fu, Jiang, Wang, and Heng] Xiaowei Hu, Tianyu Wang, Chi-Wing Fu, Yitong Jiang, Qiong Wang, and Pheng-Ann Heng.  Revisiting shadow detection: A new benchmark dataset for complex world.  _IEEE Transactions on Image Processing_ , 30:1925–1934, 2021.
  * [Ilg et al.(2017)Ilg, Mayer, Saikia, Keuper, Dosovitskiy, and Brox] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * [Jacques et al.(2005)Jacques, Jung, and Musse] J.C.S. Jacques, C.R. Jung, and S.R. Musse.  Background subtraction and shadow detection in grayscale video sequences.  In _XVIII Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI’05)_ , pages 189–196, 2005.
  * [Le and Samaras(2019)] Hieu Le and Dimitris Samaras.  Shadow removal via shadow image decomposition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [Le and Samaras(2020a)] Hieu Le and Dimitris Samaras.  From shadow segmentation to shadow removal.  In _ECCV_ , 2020a.
  * [Le and Samaras(2020b)] Hieu Le and Dimitris Samaras.  Physics-based shadow image decomposition for shadow removal, 2020b.
  * [Le et al.(2016)Le, Nguyen, Yu, and Samaras] Hieu Le, Vu Nguyen, Chen-Ping Yu, and D. Samaras.  Geodesic distance histogram feature for video segmentation.  _ACCV_ , 2016.
  * [Le et al.(2018)Le, Vicente, Nguyen, Hoai, and Samaras] Hieu Le, Tomas F. Yago Vicente, Vu Nguyen, Minh Hoai, and Dimitris Samaras.  A+d net: Training a shadow detector with adversarial shadow attenuation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [Li et al.(2021)Li, Zhao, He, Zhu, and Liu] Jiangyun Li, Yikai Zhao, Xingjian He, Xinxin Zhu, and Jing Liu.  Dynamic Warping Network for Semantic Video Segmentation.  _Complexity_ , 2021:1–10, February 2021.
  * [Liu et al.(2020)Liu, Zhang, He, Liu, Wang, Tai, Luo, Wang, Li, and Huang] Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang.  Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [Liu et al.(2015)Liu, Rabinovich, and Berg] Wei Liu, Andrew Rabinovich, and Alexander C. Berg.  Parsenet: Looking wider to see better, 2015.
  * [Menze and Geiger(2015)] Moritz Menze and Andreas Geiger.  Object scene flow for autonomous vehicles.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * [Mohanapriya and Mahesh(2017)] D. Mohanapriya and K. Mahesh.  A video target tracking using shadow suppression and feature extraction.  In _2017 International Conference on Information Communication and Embedded Systems (ICICES)_ , pages 1–6, 2017.
  * [Nguyen et al.(2017)Nguyen, Yago Vicente, Zhao, Hoai, and Samaras] Vu Nguyen, Tomas F. Yago Vicente, Maozheng Zhao, Minh Hoai, and Dimitris Samaras.  Shadow detection with conditional generative adversarial networks.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * [Sandler et al.(2018)Sandler, Howard, Zhu, Zhmoginov, and Chen] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [Sanin et al.(2012)Sanin, Sanderson, and Lovell] Andres Sanin, Conrad Sanderson, and Brian C. Lovell.  Shadow detection: A survey and comparative evaluation of recent methods.  _Pattern Recognition_ , 45(4):1684–1695, 2012\.
  * [Saravanakumar et al.(2010)Saravanakumar, Vadivel, and Saneem Ahmed] S. Saravanakumar, A. Vadivel, and C.G Saneem Ahmed.  Multiple human object tracking using background subtraction and shadow removal techniques.  In _2010 International Conference on Signal and Image Processing_ , pages 79–84, 2010.
  * [Shi and Liu(2019)] Hang Shi and Chengjun Liu.  Moving cast shadow detection in video based on new chromatic criteria and statistical modeling.  In _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_ , pages 196–201, 2019.
  * [Shin et al.(2005)Shin, Kim, Kang, Lee, Paik, Abidi, and Abidi] Jeongho Shin, Sangjin Kim, Sangkyu Kang, Seong-Won Lee, Joonki Paik, Besma Abidi, and Mongi Abidi.  Optical flow-based real-time object tracking using non-prior training active feature model.  _Real-Time Imaging_ , 11(3):204–218, 2005.  ISSN 1077-2014.  Special Issue on Video Object Processing.
  * [Sun et al.(2018)Sun, Yang, Liu, and Kautz] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [Vicente et al.(2016a)Vicente, Hoai, and Samaras] Tomas F. Yago Vicente, Minh Hoai, and Dimitris Samaras.  Noisy label recovery for shadow detection in unfamiliar domains.  In _CVPR_ , 2016a.
  * [Vicente et al.(2016b)Vicente, Hou, Yu, Hoai, and Samaras] Tomás F Yago Vicente, Le Hou, Chen-Ping Yu, Minh Hoai, and Dimitris Samaras.  Large-scale training of shadow detectors with noisily-annotated shadow examples.  In _European Conference on Computer Vision_ , pages 816–832. Springer, 2016b.
  * [Wang et al.(2018)Wang, Li, and Yang] Jifeng Wang, Xiang Li, and Jian Yang.  Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [Wang et al.(2020)Wang, Curless, and Seitz] Yifan Wang, Brian Curless, and S. Seitz.  People as scene probes.  In _ECCV_ , 2020.
  * [Xu et al.(2019)Xu, Chen, Su, Ji, Xu, Memon, and Zhou] Weiyue Xu, Huan Chen, Qiong Su, Changying Ji, Weidi Xu, Muhammad-Sohail Memon, and Jun Zhou.  Shadow detection and removal in apple image segmentation under natural light conditions using an ultrametric contour map.  _Biosystems Engineering_ , 184:142–154, 2019.  ISSN 1537-5110.
  * [Yan et al.(2019)Yan, Li, Xie, Li, Wang, Chen, and Lin] Pengxiang Yan, Guanbin Li, Yuan Xie, Zhen Li, Chuan Wang, Tianshui Chen, and Liang Lin.  Semi-supervised video salient object detection using pseudo-labels.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [Zheng et al.(2019)Zheng, Qiao, Cao, and Lau] Quanlong Zheng, Xiaotian Qiao, Ying Cao, and Rynson W.H. Lau.  Distraction-aware shadow detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [Zhong et al.(2013)Zhong, Liu, Ren, Zhang, and Ren] Sheng-hua Zhong, Yan Liu, Feifei Ren, Jinghuan Zhang, and Tongwei Ren.  Video saliency detection via dynamic consistent spatio-temporal attention modelling.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 27(1), Jun. 2013.
  * [Zhu et al.(2018)Zhu, Deng, Hu, Fu, Xu, Qin, and Heng] Lei Zhu, Zijun Deng, Xiaowei Hu, Chi-Wing Fu, Xuemiao Xu, Jing Qin, and Pheng-Ann Heng.  Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [Zhu et al.(2017)Zhu, Xiong, Dai, Yuan, and Wei] Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, and Yichen Wei.  Deep feature flow for video recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.