# EENMF: An End-to-End Neural Matching Framework for E-Commerce Sponsored
Search

Wenjin Wu, Guojun Liu, Hui Ye, Chenshuang Zhang, Tianshu Wu, Daorui Xiao, Wei
Lin, Xiaoyu Zhu  1234-5678-9012 Alibaba Group
kevin.wwj,guojun.liugj,yehui.yh,chenshuang.zcs,shuke.wts,daorui.xdr,yangkun.lw,benjamin.zxy@alibaba-
inc.com

(2019)

###### Abstract.

E-commerce sponsored search contributes an important part of revenue for the
e-commerce company. In consideration of effectiveness and efficiency, a large-
scale sponsored search system commonly adopts a multi-stage architecture. We
name these stages as ad retrieval, ad pre-ranking and ad ranking. Ad retrieval
and ad pre-ranking are collectively referred to as ad matching in this paper.
In the ad matching stage, there are two important problems that need to be
addressed. First, in the keyword-based mechanism of traditional sponsored
search, it is a great challenge for advertisers to identify and collect lots
of relevant bid keywords for their ads. Due to the improper keyword bidding,
advertisers cannot get their desired ad impressions; meanwhile, sometimes
there are no ads displayed to user for long-tail queries. These issues lead to
inefficiency. Second, deep models with personalized features have been
successfully employed for click prediction in the ranking stage. However, for
the reason of computing complexity, deep models with personalized features are
not effectively and efficiently applied in the ad matching stages. To address
these two problems, we propose an end-to-end neural matching framework (EENMF)
to model two tasks—vector-based ad retrieval and neural networks based ad pre-
ranking. Under the deep matching framework, vector-based ad retrieval
harnesses user recent behavior sequence to retrieve relevant ad candidates
without the constraint of keyword bidding. Simultaneously, the deep model is
employed to perform the global pre-ranking of ad candidates from multiple
retrieval paths effectively and efficiently. Besides, the proposed model tries
to optimize the pointwise cross-entropy loss which is consistent with the
objective of predict models in the ranking stage. We conduct extensive
evaluation to validate the performance of the proposed framework. In the real
traffic of a large-scale e-commerce sponsored search, the proposed approach
significantly outperforms the baseline.

Sponsored search, Ad retrieval, Ad matching, Ad pre-ranking, Deep learning

††copyright: acmcopyright††copyright: acmlicensed††conference: DAPA 2019 WSDM
Workshop on Deep matching in Practical Applications; February 15th, 2019;
Melbourne, Australia††journalyear: 2019

##  1\. Introduction

When users search in the search engine, sponsored search platform enables
advertisers to target advertisements (ads111In the remainder, ad(s) is used to
refer to advertisement(s)) to users’ search requests. Along with organic
results, search engine presents the sponsored results to users in response to
their search requests. Precisely, in e-commerce sponsored search, organic
results are the products named ”item” on Taobao platform, while the ads are
also a special kind of item.

In consideration of effectiveness and efficiency, large-scale search systems
or recommendation systems often adopt a multi-stage search architecture (Liu
et al., 2017). In our e-commerce sponsored search system, a three-stage
architecture has been adopted over the past few years. Sequentially we name
these three stages as ad retrieval, ad pre-ranking and ad ranking. In this
paper, we refer the ad  retrieval and pre-ranking stage both as matching. We
mainly focus on proposing an efficient and effective neural matching model for
the ad matching stage in sponsored search.

In our e-commerce sponsored search, there exist two types of problems in the
ad matching stage. Firstly, in the traditional sponsored search system,
keyword-based mechanism provides a simple ad retrieval method where the whole
burden is on advertisers, making it a big challenge for advertisers to
optimize bids. It is quite impossible for advertisers to identify and collect
lots of relevant bid keywords to target their ads. Due to the improper keyword
bidding, advertisers possibly can not get the desired ad impressions;
meanwhile, there are no ads displayed in the search result pages for long-tail
queries. To alleviate this problem, the search engines often provide an
advanced matching service to advertisers, which rewrites user’s original query
to many different related bid keywords, enriching the connections between
user’s query and bid keywords of ads. The query rewriting approach is limited
to matching only against predefined bid keywords of ads. Thus, the keyword-
based mechanism is still unable to achieve a good match between user query
requests and advertisements. Secondly, various types of personalized
information such as user profile, user long-time click behaviors and real-time
click behaviors, have been proved to be effective for click prediction models
in the ranking stage (Cheng and Cantú-Paz, 2010; Zhang et al., 2014; Shan et
al., 2016). However, for the sake of computing complexity, deep models
exploiting personalized information such as user recent click behaviors, is
not utilized in the matching stage.

To address the inefficiency of keyword-based mechanism and lack of uniformly
user personalized information modeling in the matching stage, inspired by
recent work on multi-task learning (Liu et al., 2015; Bai et al., 2009), we
propose a practical neural matching model to fulfill these two tasks: vector-
based ad retrieval and neural networks based ads pre-ranking respectively.
Vector-based ad retrieval exploits user recent behavior sequence to select
relevant ad candidates without the constraint of keyword bidding. For the
vector retrieved ads without bidding information, a bid optimizing strategy
called Optimized Cost Per Click (OCPC)(Su et al., 2018) is applied to
determine how much the advertisers will be charged if their ads are clicked.
Simultaneously, there often exist various ads retrieval paths in sponsored
search system such as keyword-based retrieval and vector-based retrieval. We
adapt the model to perform the global pre-ranking of ad candidates. The model
is trained to optimize the cross-entropy loss under the guide of search
impression logs, which makes the optimized objects of matching and ranking
stages consistent. Finally, we conduct online evaluation in the real-world
operational environment of our large-scale e-commerce sponsored search.

The main contributions of this paper are summarized as follows:

  * •

We propose a novel multi-task neural matching framework for a large-scale
e-commerce sponsored search, which is trained on user search sessions. The
proposed matching model tries to optimize the cross-entropy loss function
which is consistent with the objective of predicting models in the ranking
stage.

  * •

Under the framework, we implement the vector-based ad retrieval to overcome
the shortness of the keyword-based ad retrieval and provide advertisers a
keyword-free way to advertise their products.

  * •

The proposed approach is deployed in a large-scale e-commerce sponsored search
platform. The online evaluation reveals that the proposed method retrievals
and selects more relevant ads than the baseline methods.

Figure 1. An E-commerce Sponsored Search System Overview. This paper focuses
on two parts: vector-based ad retrieval and ad pre-ranking, which are
highlighted in red. Ad retrieval and ad pre-ranking are collectively referred
to as  ad matching in this paper.

##  2\. Methodology

Figure 2. Neural Matching Model. Attentive GRU-RNN is adapted to model user
behavior sequence, and this model fulfills two tasks: vector-based retrieval
and deep ad pre-ranking.

In order to better understand the proposed neural matching framework, Figure 1
the overall architecture and data flow of the sponsored search system.

###  2.1. Model Architecture

The proposed model architecture is shown in Figure 2. Horizontally, the
architecture consists of two parallel sub neural networks, one network for
user requests (which we term Qu-Net) in the left side and the other for
advertisements (which we term Ad-Net) in the right side. User request features
and advertisement features are fed to Qu-Net and Ad-Net as inputs
respectively, and ad clicked labeled 1 or not clicked labeled 0 is produced as
output. Vertically, the underlying model architecture can be divided into four
parts from bottom to top including input and embedding layers, encoding
layers, shared layers and task-specified layers. We detail these layers in the
following.

###  2.2. Input and embedding layers

The input instance of the proposed models is consisting of four types of
features: query feature, user profile feature, user previous behaviors feature
and the target ad item feature. User behaviors is a behavior sequence
$X=\left\\{q_{0},i_{1},i_{2},...,i_{m}\right\\}$ where $q_{0}$ is user’s
current query, and $i_{k}$ indicates the $k$th behavior item that the user
acted before this search request. $X$ is ordered by the time of user behavior.
Each behavior item is represented by ID features including item ID, shop ID,
brand ID, term IDs of the item’s title and the corresponding search query
feature. In Ad-Net, ad item is also represented by ID features like the
behavior item, except the search query feature. Each unique ID space has a
separately learned embedding matrix. Very large cardinality ID space (e.g.
product item Ids and search query terms) are truncated by keeping top ones
after sorting based on their frequency in search logs. Out-of-vocabulary
values are set to zero and mapped to the embedding of zero. Multivalued ID
feature embeddings, such as word IDs of item title, are summed before fed to
the next layer.

Remarkably, sparse features in the same ID space share the same underlying
embedding matrix. Since an ad item is also a product, ad item features in Ad-
Net share all embedding matrices with behavior item features in Qu-Net.
Sharing embedding is important for improving generalization, speeding up model
training and reducing model parameters.

###  2.3. Encoding layers

When a user search in e-commerce mobile app, she browses and clicks product
items in the form of streaming. For instance, when a user want to buy a shoes
and search ”shoes” , she usually browses the result list, clicks shoes which
she likes and compares them before adding to cart. Intuitively, items in the
same behavior sequence are correlated. In other words, user previous behavior
items are predictable for the next behavior item. This type of relation has
been proved to be effective in recommendation system (Covington et al., 2016;
Zhou et al., 2017)

In the encoding layer, we use recurrent neural network (RNN) to encode user
behavior sequence. We consider the latest previous $m$ behaviors, padding the
default symbol to the fixed size m if length of previous behaviors is less
than $m$. We adopt GRUs, since GRUs have fewer parameters and competitive
performance to LSTMs(Chung et al., 2014).

In the e-commerce sponsored search, there may exist items in previous behavior
sequence unrelated to current search query. For example, user current search
query is ”red dress”, while in her previous behavior sequence there are dress
product items searched by ”dress” and shoes product items searched by ”shoes”.
Obviously, these two category of product items are of different relevance to
current search query ”red dress”. Thus, we adopt query based attention nets to
address this problem. Vector $h_{j}$ is the GRU hidden output at the step $j$.
We take $h_{j}$ as the representation of $j$-th behavior item, and represent
the behavior sequence as a weighted sum of the vector representation of all
behavior items. The attention weight makes it possible to assign proper credit
to items according to their importance to current query request.
Mathematically, it takes the formulates as follows:

(1) $h=\sum_{t=1}^{m}w_{t}h_{t}$

(2)
$w_{t}=\dfrac{\exp(net(h_{t},Q;\theta))}{\sum_{i=1}^{m}\exp(net(h_{i},Q;\theta))}$

where $w_{t}$ is the weight for $h_{t}$, $net(h_{t},Q;\theta)$ is a two-layer
attention network parameterized by $\theta$ with hidden state $h_{t}$ and
query feature embeddings $Q$ as inputs. $h$ is the vector representation of
user previous behavior items. As for Ad-Net, we directly use one fully
connected layer to map ad item embeddings to a vector with the same dimension
as the encoding vector of user previous behavior items $h$.

###  2.4. Shared layers

After the encoding layers, we get a user query request vector output and an ad
vector output with the same dimension, which may not well fit in the same
vector space. Inspired by DSSM models (Huang et al., 2013), we stack two
shared nonlinear fully connected layers over the encoding layer of Qu-Net and
Ad-Net to bind these two types of vectors. Besides, hard parameter sharing
greatly reduces the risk of overfitting (Ruder, 2017). Furthermore, let
$h_{l}$ denotes the corresponding output of the $l$-th hidden layers.

(3) $h_{l}=f(h_{l-1})$

where $f(\cdot)$ is an non-linear activation function. The output of the
shared layers are query request’s representation and ad’s with dimension $d$
(e.g. 128), which are fed to the next multi-task specific layers.

###  2.5. Multi-task specific layers

Through the previous layers, we obtain representations of both query request
and ad. Our model has two tasks to fulfill: vector-based ads retrieval and ads
pre-ranking. For these two different tasks, we apply specific network layers
and optimization objectives.

#### Vector-based ad retrieval

For the vector-based ad retrieval task, with $V_{qu}$ and $V_{a}$ as inputs
the relevance score is computed by cosine similarity as:

(4) $cosine(V_{qu},V_{a})=\dfrac{V_{qu}\cdot V_{a}}{||V_{qu}||\cdot||V_{a}||}$

The larger the cosine value, the more similar is between $V_{qu}$ and $V_{a}$.
We use the cross-entropy loss as the objective to train model:

(5)
$C_{v}(\theta)=-\frac{1}{N}\sum_{i=1}^{N}(y_{i}\log(P(V_{qu,i},V_{a,i})))+(1-y_{i})\log(1-P(V_{qu,i},V_{a,i}))$

(6) $P(V_{qu,i},V_{a,i})=\dfrac{\exp(\gamma
cosine(V_{qu,i},V_{a,i}))}{1+\exp(\gamma cosine(V_{qu,i},V_{a,i})}$

where $\gamma$ is a tuning factor determined by validation set.
$y_{i}\in\\{0,1\\}$ is the target. If user clicked the current ad, the
instance is positive, otherwise negative. The loss is summed over all samples
in a mini-batch (128 samples in our experiments). At serving time, ad
retrieval is reduced to a nearest neighbor search problem. Product
quantization is used to implement K nearest neighbor search, which is
efficiently supported in Faiss library (Johnson et al., 2017). The details is
that we apply forward inference of the current query request through Qu-Net
obtaining the $V_{qu}$ vector which is normalized, and then we use $V_{qu}$ to
search ads’ Faiss index to obtain relevant ads.

#### Ad pre-ranking

In our scenario, thousands of ads are recalled through bidword-based ad
retrieval and vector-based retrieval. The ads pre-ranking stage needs to score
and select top N (e.g. 200) ad candidates for the ranking stage. Different
from the baseline approach which uses static score built in ads inverted index
to select top N ads, we employ a deep model with personalized features to
score and select top N ads. Here we still model it as a click-through rate
prediction (CTR) problem, and use the cross-entropy loss as the objective
which is consistent with the objective of CTR predicting model in the ranking
stage. In order to model the interaction between query request features and ad
features, we add a nonlinear fully connected layer for them. The loss function
is descirbed in Equation 7.

(7)
$C_{r}(\theta)=-\frac{1}{N}\sum_{i=1}^{N}(y_{i}\log(P^{\prime}(V_{qu,i},V_{a,i})))+(1-y_{i})\log(1-P^{\prime}(V_{qu,i},V_{a,i}))$

(8) $P^{\prime}(V_{qu},V_{a})=FC(V_{qu},V_{a};\theta)$

where the lightweight networks $FC(V_{qu},V_{a};\theta)$ is consisting of one
fully connected layer and a logits regression layer, with the concatenation of
$V_{qu}$ and $V_{a}$ as inputs. The reason for choosing
$FC(V_{qu},V_{a};\theta)$ is concluded as follows. First, most of the matrix
computation $(V_{qu},V_{a})W$ in the fully connected layer can be calculated
offline, since as described in Equation 9, $W$ is the parameters matrix of
this layer, $(V_{qu},\vec{\textbf{0}})W$ is computed only one time per query
request and $(\vec{\textbf{0}},V_{a})W$ can be computed offline for ads in
advance. Second, the lightweight networks $FC(V_{qu},V_{a};\theta)$ are
flexible to incorporate other effective features such as id features or
statistic features, which is similar to the wide part of Heng-Tze Cheng et
al.’s deep & wide model (Cheng et al., 2016).

(9) $(V_{qu},V_{a})W=(V_{qu},\vec{\textbf{0}})W+(\vec{\textbf{0}},V_{a})W$

Finally, our model is trained jointly with the objective:

(10) $C_{joint}(\theta)=\alpha C_{v}(\theta)+(1-\alpha)C_{r}(\theta)$

where $\alpha$ is a hyperparameter that balances the effects of two tasks.

##  3\. Experiments

###  3.1. Dataset description

We use the search logs from both sponsored search and organic search to
reorder each user’s historical behaviors according to the timeline, and then
construct the train dataset and test dataset. An instance records the complete
information about an ad impression including user profile, query, user recent
behaviors and corresponding behavior of the current ad (click or non-click).
And the window size of user recent behaviors is 6. In our experiment, if user
clicked the current ad then the instance is positive, otherwise negative.
About $3\times 10^{9}$ instances from search logs per day are sampled as data
set. We use samples of every three consecutive days for training and test on
the samples of the next day. We divide the search logs of 12 consecutive days
to three groups of training and testing data sets for training and evaluation.
We employ distributed Tensorflow 222https://www.tensorflow.org/machine
learning platform deployed on a large-scale computing cluster to train our
neural networks

###  3.2. Offline Evaluation

####  3.2.1. Comparison of different network architectures

In order to investigate the effectiveness of the proposed model, we compare
five network architectures and the baseline model DSSM (Huang et al., 2013).
These models are described as follows:

  * •

DNN: it employs the mean pooling to represent user previous behaviors,
ignoring the order information and taking all behavior items equally.

  * •

GRU-RNN: it applies GRU cell based RNN to model user previous behavior
sequence.

  * •

Attention-DNN: it adds a query based attention network over DNN, and
distinguishes the different role that each behavior item plays when predicting
the current interest.

  * •

Attention-GRU-RNN: Similarily, it adds the query based attentive network GRU-
RNN, considering both order and importance information.

  * •

Concatenate-DNN: it directly concatenates embeddings of user previous
behaviors, letting the raw information feed into later layers.

  * •

DSSM (Huang et al., 2013): In DSSM (Huang et al., 2013), a query is parallel
to the titles of the ad documents clicked on for that query. We extracted the
query-title pairs as positive samples for model training from ads click logs
using a procedure similar to (Huang et al., 2013), and randomly sampled four
negative documents per positive sample. Since user’s search query is often
short, we enrich the query with the title of recent behavior items. We also
use terms in the query and ad’s title as input features.

  * •

Search2Vec (Grbovic et al., 2016), which is the state-of-art approach for
board match in sponsored search. Since our offline evaluation is based on the
search session logs and user request is sparse, we do not conduct offline
evaluation for Search2Vec (Grbovic et al., 2016). However, we trained the
Search2Vec model and conducted the online evaluation with the real search
traffic, which is described in Subsection 3.3

To measure the overall performance of each model, we employ Area Under ROC
Curve (AUC) as the evaluation metric, which is widely used in industry. AUC
measures whether the clicked instances are ranked higher than the non-clicked
ones. For the fair model comparison, we tune model parameters using validate
dataset (5% samples from training dataset not used for training models) to
ensure these models to achieve its best performance respectively.

Table 1. Comparison of Different Models for User Behavior Sequence on Task1 vector-based ads retrieval and Task2 ads pre-ranking model type | Task1 AUC | Task2 AUC
---|---|---
DNN | 0.6657 | 0.6655
GRU-RNN | 0.6760 | 0.6758
Attention-DNN | 0.6762 | 0.6760
Attention-GRU-RNN | 0.6885 | 0.6886
Concatenate-DNN | 0.6795 | 0.6796
DSSM (Huang et al., 2013) | 0.6200 | -

Table 1 reports the overall AUC of all models on the test dataset. These
results demonstrate that RNN based models outperform DNN based models, and
models with attention mechanism outperform the ones without respectively for
both tasks. Specifically, RNN brings about 0.01 AUC improvement comparing with
DNN. Attention mechanism brings about 0.01 AUC improvement for DNN and 0.012
for RNN. Concatenate-DNN outperforms DNN with about 0.01 AUC improvement.
These results conform the hypothesis that user previous behavior items
sequence are predictable for the next behavior item, but previous behavior
items are not equally important. The overall evaluation results show the
effectiveness of the GRU-RNN with attention model. Besides, the result of the
original DSSM (Huang et al., 2013) method indicates that it is not fit for ad
retrieval very well. There are two reasons for that. First, DSSM (Huang et
al., 2013) solely employing the term features can not distinguish between
positive and negative samples very well, and we find that in the training
dataset the positive and negative samples seem to be relevant to their
corresponding query in the textual content. Second, the DSSM (Huang et al.,
2013) method employs a negative sampling based pairwise loss which does not
directly aim at optimizing CTR prediction.

In the following, we conduct more detailed analysis of our model in order to
analyze the individual effect of different components or parameters on the
performance. In each experiment, we only check one component or parameter,
while the rest will be fixed.

####  3.2.2. Influence of jointly training

We compare our proposed jointly training model with the single training model.
As described in Section 2.5, our model is trained to fulfill two tasks:
vector-based ads retrieval and ads pre-ranking. We conduct this comparison
based on the GRU-RNN with attention model. As shown in Table 2, the jointly
training model achieves better performance than the single trained ones. For
the vector-based ads retrieval task, the joint model leads to about 0.0120 AUC
improvement. As for the ads pre-ranking task, the joint model leads to about
0.0088 AUC improvement. Besides, when these two tasks are trained
individually, Task2 ’s AUC is larger than Task1’s, which is consistent with
the empirical idea that a lightweight networks $FC(V_{qu},V_{a};\theta)$ is
more powerful than the cosine interaction between $V_{qu}$ and $V_{a}$. One
possible reason is that the joint training tends to learn more expressive
representations of user request and ads. However, we find that two tasks
almost have no significant difference in AUC metric when they are jointly
trained.

Table 2. Comparison of jointly traning and single training on two tasks: Task1 vector-based ads retrieval and Task2 ads pre-ranking | Task1 AUC | Task2 AUC
---|---|---
single training task1 | 0.6765 | -
single training task2 | - | 0.6798
jointly training | 0.6885 | 0.6886

Importantly, when the sponsored search system serves online, these two tasks
(vector-based ads retrieval and ads pre-ranking) are needed to serve together.
Consideration of online serving’s efficiency and convenience, one joint model
is a better choose than two single models.

####  3.2.3. Influence of shared layers

In order to analysis the effect of shared layers described in Section 2.4, we
carry out the comparison experiment and the result is shown in Table 3. It can
be observed that share layers are consistently better than the non-share ones
in both tasks. In one sense, sharing layers is a type of interaction between
query request and ads. Besides, hard parameter sharing greatly reduces the
risk of overfitting (Ruder, 2017).

Table 3. Comparison of share vs. non-share layers for Task1 vector-based ads retrieval and Task2 ads pre-ranking | Task1 AUC | Task2 AUC
---|---|---
share | 0.6820 | 0.6821
non-share | 0.6761 | 0.6794

####  3.2.4. Influence of hyperparameter $\gamma$

For the vector-based ad retrieval task, the relevance score between $V_{qu}$
and $V_{a}$ is computed by cosine similarity. As shown in Equation 6 of
Section 4.5, $P(V_{qu,i},V_{a,i})$ is the predict value. We train the models
using different $\gamma$ values and evaluation results are shown in 4. The
performance of model is very sensitive to parameter $\gamma$. When parameter
$\gamma$ is 6, the model achieves the best performance. The mean of the final
predict value is 0.0570 and minimum and maximum values are 0.0060 and 0.3000
respectively. And we find that the mean predict value 0.0570 is nearest to the
ratio of positive instances in the training dataset.

Table 4. Comparison of different parameter $\gamma$ for vector-based ad retrieval task $\gamma$ | (mean,var,[min,max]) of predict value | AUC
---|---|---
1 | (0.2690, 0.0000, [0.2690, 0.2710]) | 0.4999
3 | (0.0680, 0.0260, [0.0475, 0.2300]) | 0.6642
6 | (0.0570, 0.0443, [0.0060, 0.3000]) | 0.6866
9 | (0.0580, 0.0300, [0.0120, 0.2400]) | 0.6533

###  3.3. Online Evaluation

In this subsection we conduct online A/B testing on the e-commerce sponsored
search platform with 1% of overall search traffic lasting three days. Four
metrics are used to evaluate the performance of the proposed approach as
following:

  * •

$CTR=AdClickCount/AdPresentCount$

  * •

$PR=AdPresentCount/RequestCount$

  * •

$CPC=AdCostAmount/AdClickCount$

  * •

$RPM=CTR*PPC$

We deploy the proposed model EENMF for these two tasks: vector-based ads
retrieval and ads pre-ranking in the system.

####  3.3.1. Evaluation of vector-based ads retrieval

In our system, there exists two retrieval methods: the first one is a graph
covering based query rewriting method similar to (Malekian et al., 2008) used
to retrieval the keyword-based ads; and the second one is BKR which is a
variant of the method proposed in (Su et al., 2018) for broad match.
Search2Vec (Grbovic et al., 2016), which is the state-of-art approach for
board match in sponsored search. Therefore, we conduct four groups of
experiments: (1) Search2Vec; (2) BKR; (3) EENMF; and (4) EENMF combines with
BKR. To compare fairly, all these methods employ the same user behavior
information including queries and recent clicked items to recall ads. In the
online A/B test, users, together with their queries, are randomly and evenly
distributed to four buckets. Each experimental group is deployed in one
bucket.

As shown in Table 5, The improvement of metrics PR and CTR demonstrates that
all these keyword-free and vector-based ad retrieval methods can recall more
and better ad candidates in the ad matching stage. The lifts also illustrates
that advertisers are able to receive more valuable traffic even if they choose
the keyword-free bidding advertising. Meanwhile, the lift of RPM metric
indicates the improvement of the sponsored search platform’s monetization
ability. Specifically, EENMF outperforms Search2Vec significantly. ENMF’s
performance is a little better than BKR. The reason may be that EENMF model is
deeper and more expressive than both Search2Vec and BKR, and the optimized
objective of EENMF is more consistent with the object of ranking stage.
Besides, the combination of EENMF and BKR achieves higher PR and RPM metrics,
which is valuable to the ad platform.

####  3.3.2. Evaluation of ads pre-ranking

As for the ads pre-ranking task, the baseline is a heuristics method based on
the static ad-level quality scores from indexes and Jaccard similarity between
query and ad in categories and properties.

Table 5. Comparison of Average Online Metric Lift Rates for Task1 vector-based ads retrieval Methods | PR | CTR | CPC | RPM
---|---|---|---|---
Search2Vec (Grbovic et al., 2016) | 2.2% | 0.5% | 1.9% | 2.4%
BKR (Su et al., 2018) | 2.1% | 2.0% | 2.1% | 4.1%
EENMF | 2.5% | 1.9% | 3.2% | 5.1%
EENMF + BKR (Su et al., 2018) | 3.5% | 2.7% | 4.0% | 6.7%

As shown in Table 6, the CTR increases 3.1% and CPC decrease 3.3%, which
proves the effectiveness of the proposed pre-ranking deep model. Overall,
these online evaluation results demonstrate the significant effectiveness of
the proposed approach.

Table 6. Average Online Metric Lift Rates for Task2 ads pre-ranking Metric | Lift Rate
---|---
PR | 0.0%
CTR | 3.1%
CPC | -3.3%
RPM | -0.2%

##  4\. Discussion

In this section, we discuss the efficiency of online inference. In a large-
scale e-commerce sponsored search system, it is critical to response to user
query request timely. Usually, given a query request, it is very time-
consuming to perform ad ranking in a deep learning based sponsored search
system. Since there exist quite a lot of ad candidates in the matching stage,
the proposed deep models need to be served online efficiently. Reviewing the
model architecture described in Section 2, we find that it is divided to two
sub networks: Qu-Net and Ad-Net. When the model serves online request, for the
user query request side, we just need one forward inference per query request
through Qu-Net and obtain the $V_{qu}$ vector. For the ad side, we conduct
forward inference for all ads in the advertising repository using the trained
network model offline and these inferenced ad vectors $V_{a}$ are built into
index. When a new ad arrives, its vector $V_{a}$ is infereced and written into
index in the updating system.

##  5\. Related work

Query rewriting has been well studied in the literature. These works on query
rewriting fall into two categories: one is based on the relevance matching
among queries and ads (Broder et al., 2008, 2009), and the other is based on
mining the co-occurrence among queries and ads from the historical ad click
logs (Sussillo, 2014; Boldi et al., 2008). However, these methods can not
overcome the limitation of keyword-based ad retrieval. To address these
problems, the most related works were conducted by Mihajlo Grbovic et al.
(Grbovic et al., 2016) . Mihajlo Grbovic et al. (Grbovic et al., 2016)
designed a query-ad semantic matching approach based on embeddings of queries
and ads, namely Search2Vec. The embeddings were learned by the skip-gram model
(Mikolov et al., 2013) on user search session data in an unsupervised manner.
Different from their works, we propose a deep leaning based matching framework
to realize the vector-based ad retrieval and the global ad pre-ranking in an
end-to-end manner, which is more compatible with new and long-tail ads and
queries.

Another related work is leveraging deep learning techniques for the semantic
matching problem in the information retrieval and recommendation systems. In
the context of information retrieval, many representation focused approaches
based on the Siamese architecture have been explored especially for short text
matching, such as DSSM (Huang et al., 2013). DeepCrossing model (Shan et al.,
2016) are some of the methods that learn query and document text embedding to
predict click-through rate. Zhang et al. (Zhang et al., 2014) proposed a
framework directly modeling the dependency on user’s sequential behaviors into
the click prediction process through the recurrent network. The interaction
focused neural models (Guo et al., 2016) learn query-document matching
patterns from word-level interactions. Wasi Ahmad et al. (Ahmad et al., 2018)
proposed a joint framework trained on search sessions to predict next query
and rank corresponding documents. Our work falls into the representation
focused approach to learn user request and document representations and
jointly models the two tasks: retrieval ads and pre-ranking ads in sponsored
search. In the context of recommendation, deep learning based models learn a
better representation of user’s demands, item’s characteristics and historical
interactions between them. Cheng et al. (Cheng et al., 2016) proposed an App
recommender system for Google Play with a wide & deep model. Covington et al.
(Covington et al., 2016) posed recommendation as extreme multiclass
classification and presented a deep neural network based recommendation
algorithm for video recommendation on YouTube. DeepIntent model proposed in
Zhai et al. (Zhai et al., 2016) comprises a Bidirectional Recurrent Neural
Network (BRNN) combined with an attention module. Among these previous works,
DeepIntent model (Zhai et al., 2016) and DSSM (Huang et al., 2013) are mostly
similar to our work. However, There are two important different aspects.
First, our model tries to minimize the pointwise cross-entropy loss which is
consistent with the objective of CTR predicting model in the ranking stage.
Second, we focus on employing an attention based GRU-RNN model to learn user
search request which consists of user query and recent behavior sequence.

##  6\. Conclusions and Future Work

The paper contributes an efficient and effective ad matching framework based
on neural networks for the ad matching phrase in large-scale e-commerce
sponsored search. The optimized objective of the proposed matching model is
consistent with the predict model in the ranking phrase, which makes the
performance of the multi-stage architecture better. The neural network model
introduces personalized and real-time features to the ad matching stage. And
we jointly fulfill the vector-based ad retrieval task and the global ad pre-
ranking task in e-commerce sponsored search. Comparing with baseline methods,
experiment results show that our ad matching framework achieves better
performance. In the near future, we will introduce more features to the model
such as image features and explore external memory networks (Sukhbaatar et
al., 2015) to model user behaviors.

## Acknowledgment

The authors would like to thank the colleagues for their valuable supports,
such as Yan Zhang, Genbao Chen, Yuping Jiang, Hao Wan, Sheng Xu, Zhenkui
Huang, Qing Ye, Tao Ma, Hang Xiang, Di Zhang, Hongbin Zhao, Jinhui Li, Bo Wu.

## References

  * (1)
  * Ahmad et al. (2018) Wasi Ahmad, Kai-Wei Chang, and Hongning Wang. 2018\.  Multi-Task Learning for Document Ranking and Query Suggestion. In ICLR.
  * Antonellis et al. (2008) Ioannis Antonellis, Hector Garcia Molina, and Chi Chao Chang. 2008.  Simrank++: Query Rewriting Through Link Analysis of the Click Graph. In VLDB, Vol. 1. VLDB Endowment, 408–421.
  * Bai et al. (2009) Jing Bai, Ke Zhou, Guirong Xue, Hongyuan Zha, Gordon Sun, Belle Tseng, Zhaohui Zheng, and Yi Chang. 2009\.  Multi-task Learning for Learning to Rank in Web Search. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM ’09). ACM, New York, NY, USA, 1549–1552.
  * Boldi et al. (2008) Paolo Boldi, Francesco Bonchi, Carlos Castillo, Debora Donato, Aristides Gionis, and Sebastiano Vigna. 2008.  The query-flow graph: model and applications. In CIKM ’08: Proceeding of the 17th ACM conference on Information and knowledge management. ACM, New York, NY, USA, 609–618.
  * Broder et al. (2009) Andrei Broder, Peter Ciccolo, Evgeniy Gabrilovich, Vanja Josifovski, Donald Metzler, Lance Riedel, and Jeffrey Yuan. 2009\.  Online expansion of rare queries for sponsored search. In WWW ’09: Proceedings of the 18th international conference on World wide web. ACM, New York, NY, USA, 511–520.
  * Broder et al. (2008) Andrei Z. Broder, Peter Ciccolo, Marcus Fontoura, Evgeniy Gabrilovich, Vanja Josifovski, and Lance Riedel. 2008.  Search advertising using web relevance feedback. In CIKM (2008-11-10), James G. Shanahan, Sihem Amer-Yahia, Ioana Manolescu, Yi Zhang, David A. Evans, Aleksander Kolcz, Key-Sun Choi, and Abdur Chowdhury (Eds.). ACM, 1013–1022.
  * Cheng and Cantú-Paz (2010) Haibin Cheng and Erick Cantú-Paz. 2010.  Personalized click prediction in sponsored search. In WSDM, Brian D. Davison, Torsten Suel, Nick Craswell, and Bing Liu (Eds.). ACM, 351–360.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016\.  Wide & Deep Learning for Recommender Systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems (DLRS 2016). ACM, New York, NY, USA, 7–10.
  * Choi et al. (2010) Yejin Choi, Marcus Fontoura, Evgeniy Gabrilovich, Vanja Josifovski, Maurício R. Mediano, and Bo Pang. 2010.  Using landing pages for sponsored search ad selection.. In WWW (2010-04-28), Michael Rappa, Paul Jones, Juliana Freire, and Soumen Chakrabarti (Eds.). 251–260.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.
  * Clevert et al. (2015) Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs).  CoRR abs/1511.07289 (2015).  arXiv:1511.07289
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep Neural Networks for YouTube Recommendations. In RecSys, Shilad Sen, Werner Geyer, Jill Freyne, and Pablo Castells (Eds.). ACM, 191–198.
  * Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.  J. Mach. Learn. Res. 12 (July 2011), 2121–2159.
  * Grbovic et al. (2016) Mihajlo Grbovic, Nemanja Djuric, Vladan Radosavljevic, Fabrizio Silvestri, Ricardo Baeza-Yates, Andrew Feng, Erik Ordentlich, Lee Yang, and Gavin Owens. 2016.  Scalable Semantic Matching of Queries to Ads in Sponsored Search Advertising. In SIGIR. ACM, New York, NY, USA, 375–384.
  * Grbovic et al. (2015) Mihajlo Grbovic, Nemanja Djuric, Vladan Radosavljevic, Fabrizio Silvestri, and Narayan Bhamidipati. 2015\.  Context- and Content-aware Embeddings for Query Rewriting in Sponsored Search. In SIGIR. ACM, 383–392.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016\.  Semantic Matching by Non-Linear Word Transportation for Information Retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (CIKM ’16). ACM, New York, NY, USA, 701–710.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015\.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.  CoRR abs/1502.01852 (2015).  arXiv:1502.01852
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data. In CIKM. ACM, 2333–2338.
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017\.  Billion-scale similarity search with GPUs.  arXiv preprint arXiv:1702.08734 (2017).
  * Klambauer et al. (2017) Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017.  Self-Normalizing Neural Networks.  CoRR abs/1706.02515 (2017).
  * Liu et al. (2017) Shichen Liu, Fei Xiao, Wenwu Ou, and Luo Si. 2017\.  Cascade Ranking for Operational E-commerce Search. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’17). ACM, New York, NY, USA, 1557–1565.
  * Liu et al. (2015) Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. 2015\.  Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval. In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 - June 5, 2015. 912–921.
  * Malekian et al. (2008) Azarakhsh Malekian, Chi-Chao Chang, Ravi Kumar, and Grant Wang. 2008.  Optimizing Query Rewrites for Keyword-based Advertising. In Proceedings of the 9th ACM Conference on Electronic Commerce (EC ’08). ACM, New York, NY, USA, 10–19.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed Representations of Words and Phrases and their Compositionality. In NIPS. Curran Associates, Inc., 3111–3119.
  * Mitra et al. (2017) Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017\.  Learning to Match Using Local and Distributed Representations of Text for Web Search. In Proceedings of the 26th International Conference on World Wide Web (WWW ’17). 1291–1299.
  * Nair and Hinton (2010) Vinod Nair and Geoffrey E. Hinton. 2010.  Rectified Linear Units Improve Restricted Boltzmann Machines. In ICML, Johannes Fürnkranz and Thorsten Joachims (Eds.). Omnipress, 807–814.
  * Okura et al. (2017) Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017\.  Embedding-based News Recommendation for Millions of Users. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’17). ACM, New York, NY, USA, 1933–1942.
  * Palangi et al. (2015) Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab K. Ward. 2015\.  Deep Sentence Embedding Using the Long Short Term Memory Network: Analysis and Application to Information Retrieval.  CoRR abs/1502.06922 (2015).
  * Pang et al. (2016) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016\.  Text Matching As Image Recognition. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI’16). AAAI Press, 2793–2799.
  * Ramachandran et al. (2017) Prajit Ramachandran, Barret Zoph, and Quoc V. Le. 2017\.  Searching for Activation Functions.  CoRR abs/1710.05941 (2017).  arXiv:1710.05941
  * Ruder (2017) Sebastian Ruder. 2017\.  An Overview of Multi-Task Learning in Deep Neural Networks.  CoRR abs/1706.05098 (2017).
  * Shan et al. (2016) Ying Shan, T. Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. 2016\.  Deep Crossing: Web-Scale Modeling Without Manually Crafted Combinatorial Features. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16). ACM, New York, NY, USA, 255–262.
  * Shen et al. (2014) Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014.  Learning semantic representations using convolutional neural networks for web search. In WWW (Companion Volume), Chin-Wan Chung, Andrei Z. Broder, Kyuseok Shim, and Torsten Suel (Eds.). ACM, 373–374.
  * Su et al. (2018) Yan Su, Lin Wei, Wu Tianshu, Xiao Daorui, Wu Bo, and Liu Kaipeng. 2018\.  Beyond Keywords and Relevance: A Personalized Ad Retrieval Framework in E-Commerce Sponsored Search.  (2018).  https://arxiv.org/abs/1712.10110
  * Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015.  End-To-End Memory Networks. In NIPS. 2440–2448.
  * Sussillo (2014) David Sussillo. 2014\.  Random Walks: Training Very Deep Nonlinear Feed-Forward Networks with Smart Initialization.  CoRR abs/1412.6558 (2014).
  * Zhai et al. (2016) Shuangfei Zhai, Keng-hao Chang, Ruofei Zhang, and Zhongfei Mark Zhang. 2016.  DeepIntent: Learning Attentions for Online Advertising with Recurrent Neural Networks. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16). ACM, New York, NY, USA, 1295–1304.
  * Zhang et al. (2014) Yuyu Zhang, Hanjun Dai, Chang Xu, Jun Feng, Taifeng Wang, Jiang Bian, Bin Wang, and Tie-Yan Liu. 2014\.  Sequential Click Prediction for Sponsored Search with Recurrent Neural Networks. In AAAI. AAAI Press, 1369–1375.
  * Zhou et al. (2017) Guorui Zhou, Chengru Song, Xiaoqiang Zhu, Xiao Ma, Yanghui Yan, Xingya Dai, Han Zhu, Junqi Jin, Han Li, and Kun Gai. 2017.  Deep Interest Network for Click-Through Rate Prediction.  (2017).  https://arxiv.org/abs/1706.06978


# DRSM: EFFICIENT NEURAL 4D DECOMPOSITION FOR DYNAMIC RECONSTRUCTION IN
STATIONARY MONOCULAR CAMERAS

###### Abstract

With the popularity of monocular videos generated by video sharing and live
broadcasting applications, reconstructing and editing dynamic scenes in
stationary monocular cameras has become a special but anticipated technology.
In contrast to scene reconstructions that exploit multi-view observations, the
problem of modeling a dynamic scene from a single view is significantly more
under-constrained and ill-posed. Inspired by recent progress in neural
rendering, we present a novel framework to tackle 4D decomposition problem for
dynamic scenes in monocular cameras. Our framework utilizes decomposed static
and dynamic feature planes to represent 4D scenes and emphasizes the learning
of dynamic regions through dense ray casting. Inadequate 3D clues from a
single-view and occlusion are also particular challenges in scene
reconstruction. To overcome these difficulties, we propose deep supervised
optimization and ray casting strategies. With experiments on various videos,
our method generates higher-fidelity results than existing methods for single-
view dynamic scene representation.

Index Terms—  Single-view Reconstruction, Dynamic Scene Reconstruction, Neural
Radiance Field

##  1 introduction

In recent years, the popularity of short videos and live broadcasts has led to
the generation of a lot of video data, most of which are dynamic content from
a single perspective of a fixed camera. We try to efficiently reconstruct and
realistically render dynamic scenes in single view videos. Dynamic scenes in
the video may be disturbed or obscured by other objects, such as hands (Fig.
2) and wires (Fig. 3). Our goal is to accurately recover the entire static and
dynamic scene of interest to the viewer while removing occluding objects.

Fig. 1: Framework of the proposed DRSM.

Neural Radiation Fields (NeRF) [1] tackles novel view synthesis of static
scene by learning implicit representations of objects from multiple captured
views. To model dynamic scenes, many works [2] propose the ray deformation
paradigms that parameterizes a deformed scene as a NeRF in canonical space
with a time-dependent deformation for dynamic reconstruction [3, 4, 5, 6, 7].
Other works learn the 4D scene representation by decoupling static and dynamic
scenes with different NeRFs [8, 9]. For example, D2NeRF [8] achieves dynamic
and static decoupling, which can remove all dynamic objects in the scene. But
this does not solve the problem where we want to reconstruct dynamic and
static scenes simultaneously.

Typically, dynamic NeRFs rely on video flow captured by multi-view cameras
[10, 11] or one free-viewpoint camera [3, 4, 5, 6, 7] to get full view
perception of dynamic scenes. Different with them, we aim to solve the problem
of modeling dynamic scene in single view, which is ill-posed and challenging
due to limited geometric perception. Many works exploit auxiliary information
to help understand the structure of the scene, such as SMPL [12] prior to help
constrain human motion space [13, 14] or depth prior to help recover geometry
of objects [15, 16]. Among them, NDR [15] solves the geometric reconstruction
of moving objects and can be modified as a background reconstruction technique
to solve our problem.

Moreover, the optimization for dynamic NeRFs is computationally intensive
since it requires multiple MLP evaluations. To avoid huge memory footprint of
previous methods [17], we decouple spatial and temporal features via planar
factorization [18] to model 4D field for single-view videos.

Overall, our technical contributions are as follows: 1) We propose an
efficient 4D decomposition framework (DRSM) with planar factorization for fast
Dynamic Reconstruction in Stationary Monocular Cameras; 2) we address the
inherent motion-appearance ambiguity for single-view using depth prior; 3) we
propose an efficient importance sampling strategy (ISDM) based on dynamic and
mask regions to improve the reconstruction quality for time-variant and
occluded regions; 4) we demonstrate a convincing rendering quality and smooth
point clouds on multiple short-form videos.

##  2 Method

The architecture of our network DRSM is shown in Fig. 1. We take a video
$V=\\{\mathbf{I}_{i},\mathbf{D}_{i},\mathbf{M}_{i}:i\in[1,T]\\}$ from a single
viewpoint as input, where $\mathbf{I}_{i}$ is the $i$-th frame image,
$\mathbf{D}_{i}$ is the corresponding depth image and $\mathbf{M}_{i}$ is the
mask of occluded objects to be removed. The object mask can be obtained by
combining the Segment Anything Method (SAM) [19] with the OSTrack tracking
model [20]. The video duration is normalized to [0, 1]. Thus, time of the
$i$-th frame is $i/T$.

Our network starts by randomly picking a frame for training. We employ the
ISDM sampling strategy to identify high-priority region and build casting
rays. For sampling points along casting ray, we use bilinear interpolation to
query their features on spatial and temporal tri-planes and construct the
fused features, which are then passed to MLP decoders to predict color and
density. We apply volume rendering to generate color and depth for each
casting ray, and design rendering losses for supervision. After training, the
network learns 4D representation and can reconstruct video, point cloud and
synthesize novel views.

###  2.1 Preliminaries

NeRF [1] learns a regression function F that takes the encoded coordinates of
a 3D point $\textbf{x}=(\textit{x, y, z})$ observed from a view direction
$\textbf{d}=(\theta,\phi)$ as input, and outputs the corresponding radiance c
and volume density $\sigma$:
$\textit{F}_{\text{NeRF}}:(\mathbf{x},\mathbf{d})\rightarrow(\mathbf{c},\sigma)$.
The estimated color $\mathit{\hat{C}}(\mathbf{r})$ and depth
$\mathit{\hat{D}}(\mathbf{r})$ of a pixel can be rendered by integrating the
radiance by tracking a ray $\mathbf{r}(s)=\mathbf{o}+s\mathbf{d}$, cast from
the camera toward the center of the pixel:

$\begin{split}\hat{{C}}(\mathbf{r})=\int_{s_{n}}^{s_{f}}T(s)\sigma(\mathbf{r}(s))\mathbf{c}(\mathbf{r}(s),\mathbf{d}),\end{split}$
(1)
$\begin{split}\hat{D}(\mathbf{r})=\int_{s_{n}}^{s_{f}}T(s)\sigma(\mathbf{r}(s))s\,ds,\end{split}$
(2) $\begin{split}T(s)=\exp\left(-\int_{s_{n}}^{s}\sigma(\mathbf{r}(p))\
dp\right).\end{split}$ (3)

$T(s)$ is the accumulated transmittance along the ray r up to $s$.

###  2.2 4D decomposition for dynamic scenes

A dynamic scene could be naively represented as a 4D volume $\mathbf{V}$.
Inspired by [21], we decompose V into a static volume and a dynamic volume by
planar factorization:

$\begin{split}\mathbf{V}=\\{\mathbf{V}_{s}\\{P_{XY},P_{XZ},P_{YZ}\\},\mathbf{V}_{d}\\{P_{XT},P_{YT},P_{ZT}\\}\\}.\end{split}$
(4)

Here static volume $\mathbf{V}_{s}$ is projected to a tri-plane representing
only spaces of $xy$, $xz$, and $yz$. The dynamic volume $\mathbf{V}_{d}$ is
projected to a tri-plane representing spaces and time, $xt$, $yt$, and $zt$.
Each plane has dimension $N\times N\times W$, where $N$ is the resolution and
$W$ is the number of feature channels. This approach allows us to represent a
4D volume efficiently using six planes (Fig. 1b). For a 4D point
$v=(x,y,z,t)$, we can query its features $\mathbf{f}(v)$ by projecting it onto
these planes and use bilinear interpolation $\psi$ to obtain the corresponding
values:

$\begin{split}\mathbf{f}(v)=\psi(P_{XY},x,y)\odot\psi(P_{XZ},x,z)\odot\psi(P_{YZ},y,z)\\\
\odot\psi(P_{XT},x,t)\odot\psi(P_{YT},y,t)\odot\psi(P_{ZT},z,t),\end{split}$
(5)

where $\psi(P_{XY},x,y)$ means given regularly spaced feature plane $P_{XY}$
and the $x,y$ coordinates, using bilinear interpolation to calculate the plane
feature of $v$. The $\odot$ represents Hadamard product to get fused features.

We use two small MLPs (Fig. 1c) to decode the fused features $\mathbf{f}(v)$
like Instant-NGP [22]. The features and positional encoding are concatenated
and fed into the geometry MLP $E_{g}$ to obtain density $\sigma$ and high
dimensional features $\mathbf{f}^{\prime}(v)$:

$\begin{split}\sigma(v),\mathbf{f}^{\prime}(v)=E_{g}(\mathbf{f}(v),\gamma(v)).\end{split}$
(6)

Here, $\gamma(\cdot)$ is an encoding function [1]. Then, we concatenate the
feature $\mathbf{f}^{\prime}(v)$ with the positional encoding of view
direction $(\theta,\phi)$ and feed it into the color MLP $E_{c}$ to obtain the
radiance:

$\begin{split}\mathbf{c}(r,g,b)=E_{c}(\mathbf{f}^{\prime}(v),\gamma(\theta,\phi)).\end{split}$
(7)

Fig. 2: Comparison of DRSM and other methods on dynamic reconstruction
results. We remove the hand in video and show PSNR metric of each method.

###  2.3 ISDM sampling strategy

Previous scene representation methods [2] usually randomly sample a batch of
pixels/rays on the whole input image for training. In our work, we focus on
learning the representation for dynamic scene of interest while removing
occluding objects. Uniform sampling is no longer suitable for our method
because dynamic areas and occluded areas require higher sampling weights.

We propose the importance sampling strategy based on dynamic and mask regions.
For the occlusion mask $\mathbf{M}_{i}$ of frame $i$ (0 for occluded pixels),
we ignore those pixels in occluded region in the ray selection. We create an
importance map $\tilde{\mathbf{P}}_{i}$ to guide the pixel sampling, assigning
higher probability for those regions with higher occlusion frequencies across
all frames. The sampling importance map is calculated according to element-
wise division:

$\begin{split}\tilde{\mathbf{P}}_{i}={\mathbf{M}_{i}T}/({\sum_{k=1}^{T}\mathbf{M}_{k}+\boldsymbol{\varepsilon}}).\end{split}$
(8)

In addition to occlusion areas, we should also prioritize sampling dynamic
areas. In uniform sampling, a large proportion of selected pixels may fall
into the static background, which contributes less to the dynamic
reconstruction. To identify the dynamic region, we calculate temporal
difference of pixels on frames $i$ and $j$ [11]:

$\begin{split}\mathbf{P}_{i}=\tilde{\mathbf{P}}_{i}\odot\min(\frac{1}{3}\left\|\mathbf{I}_{i}-\mathbf{I}_{j}\right\|_{1},\alpha),j\in(i-\tau,i+\tau),\end{split}$
(9)

where $\alpha$ is a lower-bound parameter controlling the sampling weights of
the dynamic region and $\tau$ is set to 25 in the experiment. ISDM sampling
adjusts the sampling probability of time-varying and occlusion areas, which
helps improve reconstruction quality and speed up training.

###  2.4 Optimization

We supervise scene reconstruction in terms of reconstructed image $\hat{C}$,
depth $\hat{D}$, and regularization loss to optimize the parameters of feature
planes and MLPs. For each batch of training data, there are $R$ rays sampled
by ISDM strategy on one frame. We first minimize the difference between the
ground truth color and the predicted color, as shown in Eq. (1). To assist in
scene representation for single-view input, we further optimize the geometry
using depth supervision. The color loss and depth loss are shown in the
following equations:

$\begin{split}\mathcal{L}_{\text{color}}=\frac{1}{|\mathcal{R}|}\sum_{\mathbf{r}\in\mathcal{R}}\left\|C(\mathbf{r})-\hat{C}(\mathbf{r})\right\|_{2}^{2},\end{split}$
(10)
$\begin{split}\mathcal{L}_{\text{depth}}=\frac{1}{|\mathcal{R}|}\sum_{\mathbf{r}\in\mathcal{R}}\left\|D(\mathbf{r})-\hat{D}(\mathbf{r})\right\|_{2}^{2}.\end{split}$
(11)

Dynamic scene reconstruction in stationary monocular camera is a severely ill-
posed problem. To achieve robust reconstruction, we apply strong regularizers.
We adopt 2D total variation (TV) loss $\mathcal{L}_{\text{TV-2D}}$ for space
planes in [22] and 1D TV loss $\mathcal{L}_{\text{TV-1D}}$ on the space axis
for space-time planes and a similar smooth loss $\mathcal{L}_{\text{smooth}}$
on the time axis. The total optimization objective is:

$\begin{split}\mathcal{L}=\mathcal{L}_{\text{color}}+\lambda_{\text{1}}\mathcal{L}_{\text{depth}}+\lambda_{\text{2}}\mathcal{L}_{\text{2D}}+\lambda_{\text{3}}\mathcal{L}_{\text{1D}}+\lambda_{\text{4}}\mathcal{L}_{\text{smooth}}.\end{split}$
(12)

Fig. 3: Ablation study on a marionette dancing video. We remove manipulating wires and show the reconstructed point clouds. Table 1: Quantitative comparisons on our collected dataset. We report PSNR$\uparrow$, SSIM$\uparrow$, LPIPS$\downarrow$ and training time (minutes). Model | “Box” | “Marionette” | “Web page” | “Xiangqi” | “Calligraphy” | “Toy” | Time
---|---|---|---|---|---|---|---
PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS
D-NeRF* | 37.02 | 0.945 | 0.084 | 33.62 | 0.885 | 0.058 | 35.80 | 0.947 | 0.068 | 35.21 | 0.959 | 0.049 | 36.27 | 0.911 | 0.095 | 37.14 | 0.944 | 0.088 | 477min
NDR | 34.95 | 0.942 | 0.094 | 33.74 | 0.909 | 0.053 | 35.37 | 0.944 | 0.086 | 35.24 | 0.955 | 0.062 | 35.02 | 0.890 | 0.139 | 35.03 | 0.940 | 0.097 | 666min
Ours-5K | 38.07 | 0.945 | 0.071 | 34.12 | 0.918 | 0.044 | 37.79 | 0.954 | 0.047 | 35.20 | 0.930 | 0.048 | 37.20 | 0.928 | 0.073 | 37.05 | 0.942 | 0.089 | 15min
Ours-10K | 39.36 | 0.953 | 0.070 | 36.42 | 0.945 | 0.023 | 39.38 | 0.964 | 0.043 | 37.27 | 0.964 | 0.037 | 38.51 | 0.943 | 0.058 | 38.42 | 0.950 | 0.079 | 35min

##  3 experiments

Experimental settings. We normalize the scene into device coordinates (NDC) to
handle monocular videos and then sample casting rays within the NDC space. We
use a model with four symmetric spatial resolutions 64, 128, 256 and 512. The
feature length $W$ at each scale is set to 32. We set the frequencies of
positional encoding $\gamma(\cdot)$ for sampling points and view direction to
4. In each training iteration, a batch contains $\mathcal{R}=2048$ sampling
rays. The loss weights in Eq.(12) are empirically set as $\lambda_{1}=1.0$,
$\lambda_{2}=0.0002$, $\lambda_{3}=0.0001$, $\lambda_{4}=0.001$. Adam [23]
optimizer is adopted for training, and the initial learning rate is set to
$0.01$. We train all scenes with $5k$ and $10k$ iterations on a single RTX
3090 GPU, which take around $15$ and $35$ minutes, respectively. We build a
video dataset, including life videos related to box, marionette, web page,
xiangqi, calligraphy and toy. We use the “Record3D” app on iPhone and RGBD
camera to record videos. Each video lasts for $5\sim 7$ seconds and we sample
$10$ frames per second for training.

Comparison experiments. We compare our method with other dynamic scene
reconstruction methods for monocular videos, such as D-NeRF [3] and NDR [16].
D-NeRF builds a deformable neural radiance field based on a canonical 3D
representation and time-guided motion fields. However, the model performance
of D-NeRF depending on a canonical frame suffers when objects exhibit long-
distance translations [24]. NDR focuses on modeling dynamic foreground objects
based on bijective motion map and implicit representations of MLPs. Using an
MLP with a specific bandwidth to learn both spatial and temporal variations
simultaneously results in suboptimal reconstruction of complex scenes.

As shown in Fig. 2, D-NeRF* and NDR are modified version of original models
with depth supervision for fair comparison with our method. D-NeRF* failed to
capture the deformation of long-distance moving objects, i.e., the characters
on the box. The predicted color of the doll predicted by NDR is affected by
the movement of the box. This is because NDR’s bijective map focuses on
learning the geometric changes of moving objects not the high-frequency
details of static part. Our network is specifically designed for the
reconstruction of combined static and dynamic scenes, resulting in better
video appearance reconstructions.

In Table 1, we show the indicators such as PSNR, SSIM and LPIPS of 6 videos to
quantitatively evaluation the reconstruction. Our model outperforms the
existing methods on multiple aspects and requires shorter training time.

Ablation study. We present ablation experiments on network modules and the
reconstructed point clouds in Fig. 3. The marionette dancing video contains
some manipulating wires to be removed. Without planar factorization, our
network failed to reconstruct high quality texture details in static region
(color prediction error in yellow ellipse) as well as dynamic region (sleeves
and decorations in yellow box). Furthermore, we observe severe distortions in
the reconstructed point cloud when depth supervision is disabled, indicating
that the network is unable to learn the correct geometry from single-view
input without prior. Without ISDM sampling, predicted high-frequency textures
also become blurry. We provide the corresponding PSNR indicator to further
demonstrate the effectiveness of proposed modules. Our complete model produces
high-fidelity reconstructions.

##  4 CONCLUSION

This paper presents a novel neural 4D decomposition for dynamic reconstruction
from single-view videos. Without observation from multi-viewpoints, the
problem of modeling dynamic scenes is typically quite challenging. We apply
planar decomposition to static and dynamic scenes respectively to improve the
model’s modeling ability of 4D scenes. To address the ambiguous geometry, we
utilize depth prior to constrain the motion space. The adaptive sampling
strategies aid the reconstruction on moving objects and occluding regions. The
ablation study demonstrates the effectiveness of proposed network. We conduct
rich experiments to show the superiority of our network than existing methods
on the special task of single-view dynamic scene construction.

## References

  * [1] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng,  “Nerf: Representing scenes as neural radiance fields for view synthesis,”  in ECCV, 2020.
  * [2] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa,  “Monocular dynamic view synthesis: A reality check,”  NeurIPS, 2022.
  * [3] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer,  “D-nerf: Neural radiance fields for dynamic scenes,”  in CVPR, 2021.
  * [4] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla,  “Nerfies: Deformable neural radiance fields,”  ICCV, 2021.
  * [5] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang,  “Dynamic view synthesis from dynamic monocular video,”  in ICCV, 2021.
  * [6] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang,  “Neural scene flow fields for space-time view synthesis of dynamic scenes,”  in CVPR, 2021.
  * [7] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt,  “Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video,”  in ICCV, 2021.
  * [8] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli,  “D2NeRF: Self-supervised decoupling of dynamic and static objects from a monocular video,”  NeurIPS, 2022.
  * [9] Boyu Zhang, Wenbo Xu, Zheng Zhu, and Guan Huang,  “Detachable novel views synthesis of dynamic scenes using distribution-driven neural radiance fields,”  arXiv preprint arXiv:2301.00411, 2023.
  * [10] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou,  “Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans,”  in CVPR, 2021.
  * [11] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al.,  “Neural 3d video synthesis from multi-view video,”  in CVPR, 2022.
  * [12] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black,  “SMPL: A skinned multi-person linear model,”  ACM Trans. Graphics (Proc. SIGGRAPH Asia), vol. 34, no. 6, pp. 248:1–248:16, Oct. 2015.
  * [13] Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher-Shlizerman,  “Humannerf: Free-viewpoint rendering of moving people from monocular video,”  in CVPR, 2022.
  * [14] Hao Zhao, Jinsong Zhang, Yu-Kun Lai, Zerong Zheng, Yingdi Xie, Yebin Liu, and Kun Li,  “High-fidelity human avatars from a single rgb camera,”  in CVPR, 2022.
  * [15] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim,  “Space-time neural irradiance fields for free-viewpoint video,”  in CVPR, 2021.
  * [16] Hongrui Cai, Wanquan Feng, Xuetao Feng, Yan Wang, and Juyong Zhang,  “Neural surface reconstruction of dynamic scenes with monocular rgb-d camera,”  NeurIPS, 2022.
  * [17] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa,  “Plenoctrees for real-time rendering of neural radiance fields,”  in ICCV, 2021, pp. 5752–5761.
  * [18] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.,  “Efficient geometry-aware 3d generative adversarial networks,”  in CVPR, 2022.
  * [19] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick,  “Segment anything,”  arXiv:2304.02643, 2023.
  * [20] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen,  “Joint feature learning and relation modeling for tracking: A one-stream framework,”  in ECCV, 2022.
  * [21] Ang Cao and Justin Johnson,  “Hexplane: A fast representation for dynamic scenes,”  in CVPR, 2023.
  * [22] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller,  “Instant neural graphics primitives with a multiresolution hash encoding,”  ACM Transactions on Graphics (ToG), vol. 41, no. 4, pp. 1–15, 2022.
  * [23] D Kinga, Jimmy Ba Adam, et al.,  “A method for stochastic optimization,”  in ICLR. San Diego, California;, 2015.
  * [24] Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, and Anton Van Den Hengel,  “Bali-rf: Bandlimited radiance fields for dynamic scene modeling,”  arXiv preprint arXiv:2302.13543, 2023.