studied to exclude obsolete tweets from being considered. Therefore, authors
propose to analyze hashtags333Hashtags are prefixed expressions using the
symbol of $\\#$ to be used for marking a specific topic in Twitter in social
media as the indicators of user interests to provide a personalized ranking of
topics. They present an online collaborative filtering method following
pairwise ranking approach for matrix factorization (Stream Ranking Matrix
Factorization), and propose a pairwise learning to optimize an ordinal loss
and a selective negative sampling based on a selective active learning, using
three objective losses, including hinge loss, SVM, and RankSVM for training.

Recently, authors in (Ktena et al., 2019) have centered their work on delayed
positive feedback at stream media to study the effect of two factors, such as
the trend and seasonality, in online advertising. In live streams, the
predicting models are dealt with the cold start issue. This is because in
online real-time scenarios, fresh data lack enough label information and the
few appearance of the positive response of users, which leads to the
underestimation of CTR values. They conduct experiments to estimate CTR values
for video ads in Twitter platform, and examine predicting models with logistic
regression and Wide& Deep (Cheng et al., 2016) models using five loss function
designed for delayed positive samples to identify the combination of learners
and loss functions for continuous stream data.

###  4.6. Summary

To summarize different framework covered in above sections, Table 5 outlines
main learning strategies used by different methods. In Table 6, we also
outline studied methods from a different aspects, including feature
engineering, downstream tasks used for the evaluation of models, and domain
applications. Recent years have witnessed a significant growth in networking
technologies and a larger number of online users across the world. As a
result, scalability is a major challenge for recommender and online
advertising. In Table 7 we overview different efforts made to provide
technical solutions for user response prediction in real world applications.
Comparing with academic scale solutions, models deployed for production system
need massive resources to store and execute internal processes. To address
these requirements, industry attempts to devise paralleled model and data
architectures that data can be processed with high throughput and remarkably
low latency. Recently some work (Mattson et al., 2019; Naumov et al., 2019)
focus on developing benchmark framework suites to provide adequate flexibility
along with good test results to make fair comparisons between academic and
industrial models. In Appendix D.4, we also outline some potential directions
for future studies.

Table 5. Overview of main ideas of user response prediction methods along with pros and cons Learning Strategy |  Algorithm |  Advantages |  Disadvantages
---|---|---|---
Data Hierarchy Analysis |  Clustering based |  \+ Using clusters as an auxiliary information for samples with insufficient observations |  \- May have a variation in user response rates
Matrix Factorization |  Collaborative Filtering |  \+ Good scalability along with simplicity
\+ It can provide robust performance against sparse data |  -Explore all historical data -Weak on anonymous behaviour sequences \- No promised performance in case of lack of user info due to privacy issues
Training a classifier |  LR |  \+ Scalability |  \- Needs feature engineering
|  FM based |  \+ Have a closed form equation that can be calculated in a linear time |  \- Limited to model 2nd order feature interactions
Feature Learning + Training a classifier |  DNN based |  \+ end-to-end interface with representation learning and non-linear transformation
\+ High flexibility using a modular implementation via open-source frameworks |  \- Interpretability \- Prone to over-fitting due to requirement of large amount of input data
\- Hyper-parameter tuning issue
|  RNN based |  \+ Can learn from sequential data with variable lengths
\+ Robust performance with regard to data sparsity |  \- Rely on linear sequential structure; \- Hard to take full advantages of GPU/TPU computing architectures; \- Long training time
|  GNN based |  \+ Addressing the network structure in the input data
to aggregate feature information of neighboring nodes
\+ Joining with attention mechanism to provide good interpretability |  \- A model trained cannot be directly applied to an input graph with different structure \- Computational cost
Stream based framework |  \+ Adjust prediction in user preferences over time
\+ Joining with external memory network for increment updates
\+ Reservoir technique to use more samples to update the model |  \- Unpractical to stack up the training data for modeling
Table 6. Comparing user response prediction methods in terms of feature characteristics, application domains, and downstream tasks. Feature Types /Organization | Application Domain |  Prediction Task (Publications)
---|---|---
E-commerce |  Display Advertising |  Recomm. Systems
Feature Engineering |  ✓ |  |  (1) CVR ((Lee et al., 2012; Vasile et al., 2017; Liu et al., 2020)); (2) CTR ((Chapelle et al., 2014; Dalessandro et al., 2015; He et al., 2014; Kumar and Salo, 2016; Ren et al., 2016; Zhang et al., 2016c; Zhu et al., 2017b)); (3) Ranking((Covington et al., 2016)
Feature Learning |  Collaborative Filtering Based |  ✓ |  ✓ |  ✓ |  (1) CTR ((Liu et al., 2015a; Menon et al., 2011)); (2) Ranking((Liu et al., 2015a));
(3) Product Rating((Wu et al., 2019))
|  Multi-field |  |  |  |
(categorical) |  |  |  ✓ |  (1) CTR((Cheng et al., 2016; Huang et al., 2019)),
(2) Ranking((He et al., 2017)) |
|  Textual |  |  ✓ |  ✓ |  (1) CTR ((Baqapuri and Trofimov, 2014; Effendi and Ali, 2017), Deep(Char) WordMatch (Edizel et al., 2017), DSM (Gligorijevic et al., 2018)); (2) Ranking ((Gligorijevic et al., 2018))
|  Visual |  ✓ |  |  |  (1) CTR(DICM(Ge et al., 2018),(Chen et al., 2016)), (2) Ranking(ACF(Chen et al., 2017), PinSage(Ying et al., 2018))
|  Sequential |  ✓ |  ✓ |  ✓ |  (1) CTR(DSIN(Feng et al., 2019), DIEN(Zhou et al., 2019a), DIN(Zhou et al., 2018b), RNN(Zhang et al., 2014), MIMN(Pi et al., 2019)),
(2) CVR(GMP(Chao Huang and Chawla, 2019), DTAIN(Gligorijevic et al., 2019)),
(3) Ranking(FGNN(Qiu et al., 2020), GAG(Ruihong et al., 2020), LGSR(Yong et
al., 2020))
|  Network based |  ✓ |  ✓ |  |  (1) CTR(FiGNN(Li et al., 2019), GIN(Li et al., 2019),(Wang et al., 2018), KNI(Qu et al., 2019)); (2) Ranking(KGNN-LS(Wang et al., 2019c))
|  Hybrid |  ✓ |  |  ✓ |  (1) CTR((Aryafar et al., 2017), RippleNet(Wang et al., 2018a), MKR(Wang et al., 2019d), DKN(Wang et al., 2018b)),
(2) Ranking(RippleNet(Wang et al., 2018a), MKR(Wang et al., 2019d))
Table 7. Summary of selected practical solution applied in industrial environments Algorithm |  Challenge |  Introduced Strategy |  Application Domain |  Provider
---|---|---|---|---
AdPredictor(Graepel et al., 2010) |  Scalability |  Bayesian probit regression model, Weight pruning, |  |
Parallel training |  Sponsored search ads |  Microsoft Bing |  |
EtsyCTR(Aryafar et al., 2017) |  Dealing with image data |  Transfer learning, Feature Hashing, Ensemble model |  E-commerce |  Etsy
FBCTR(He et al., 2014) |  Massive data |  Uniform sub-sampling, Cascade of classifiers, Ensemble model |  Display ads |  Facebook
DLRM(Naumov et al., 2019) |  Memory constraints in embeddings and computational costs of DL components |  Using PyTorch and Caffe2 for model and data parallelism |  Recomm. Sys |  Facebook
DeepFM(Guo et al., 2018) |  |  |  |
PIN(Qu et al., 2018) |  Insensitive gradient issue in DNN based models and space complexity of FFM-based models |  Shared embedding vectors, an end-to-end prediction model(Guo et al., 2018), Net-in-Net architecture to combine FM and DNN units(Qu et al., 2018) |  Recomm. Sys |  Huawei
DIN(Zhou et al., 2018b) |  |  |  |
DIEN(Zhou et al., 2019a) |  Large number of DNN parameters, Addressing temporal drifts in user interests representation |  Mini-batch aware regularization and local adaptive activation function(Zhou et al., 2018b), Attention based user interest extractor layer(Zhou et al., 2019a) |  Display ads |  Alibaba
MIMN(Pi et al., 2019) |  Handling long user behavior sequences |  Multi-channel memory network |  Display ads |  Alibaba
HPMN(Ren et al., 2019) |  |  |  |
UBR4CTR(Qin et al., 2020) |  |  |  |
SIM(Qi et al., 2020) |  Tackling long sequential user behaviours |  Memory network model along with a GRU network(Ren et al., 2019). Self-attentive retrieval module to select relevant user behaviors(Qin et al., 2020). Cascaded two-stage search model(Qi et al., 2020). |  E-commerce |  Alibaba
EGES(Wang et al., 2018) |  Scalability |  Graph Embedding Using XTensorflow |  Recomm. Sys |  Taobao
DICM(Ge et al., 2018) |  Dealing with images to represent user behaviours |  A distributed model server to handle image data embedding and reduce the communication latency |  Display ads |  Taobao
RAM(Zhao et al., 2020b) |  Balance immediate advertising revenue and long-run user experience |  Joint optimization using two-level reinforcement learning |  Recomm. Sys |  ByteDance
HPS-4(Zhao et al., 2020a) |  Massive model with large number of parameters |  A distributed hierarchical GPU parameter server |  Display ads |  Baidu
PinSage(Ying et al., 2018) |  Massive input graph with billions of links |  Highly scalable graph convolutional network |  Recomm. Sys |  Pinterest
DCN_V2(Wang et al., 2020) |  Controlling the number of model parameters to learn feature interactions for real-time data |  Mixture of low-rank approximation of DCN method(Wang et al., 2017a) organized in stacked and parallel structures |  Recomm. Sys |  Google

##  5\. Conclusion

This survey provides a comprehensive overview of computational methods for
user response prediction in online advertising. Our goal is to provide a
detailed review and categorization of the online advertising ecosystem,
stakeholders, data sources, and technical solutions. To achieve the goal, we
review and categorize online advertising platforms, type of user responses,
data sources and features, and propose a taxonomy to characterize main stream
approaches for user response prediction. For each type of user response
prediction methods, we also briefly study technical details of representative
methods, with a focus on machine learning, especially deep learning, based
approaches. In addition to the algorithms, we also review user response
prediction applications, benchmark data, and open source codes. The survey
delivers a first-hand guideline for industry and academia to comprehend the
state-of-the-art. It also serves as a technical reference for practitioners
and developers to design their own computational approaches for user response
prediction.

##  6\. Acknowledgments

This work is partially sponsored by the U.S. National Science Foundation
through Grant Nos. IIS-1763452 and CNS-1828181 and by the Bidtellect Inc.
through a sponsorship agreement.

## References

  * (1)
  * Agarwal et al. (2010) Deepak Agarwal, Rahul Agrawal, Rajiv Khanna, and Nagaraj Kota. 2010.  Estimating Rates of Rare Events with Multiple Hierarchies through Scalable Log-Linear Models. In _KDD_. 213–222.
  * Agarwal et al. (2007) Deepak Agarwal, Andrei Zary Broder, Deepayan Chakrabarti, Dejan Diklic, Vanja Josifovski, and Mayssam Sayyadian. 2007.  Estimating rates of rare events at multiple resolutions. In _KDD_. 16–25.
  * Aliannejadi et al. (2018) Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2018.  Target Apps Selection: Towards a Unified Search Framework for Mobile Devices.  _SIGIR_ , 215–224.
  * Alpaydin (2010) Ethem Alpaydin. 2010\.  _Introduction to Machine Learning_ (2nd ed.).  The MIT Press.
  * Aryafar et al. (2017) Kamelia Aryafar, Devin Guillory, and Liangjie Hong. 2017\.  An Ensemble-based Approach to Click-Through Rate Prediction for Promoted Listings at Etsy.  _CoRR_ abs/1711.01377 (2017).
  * Baqapuri and Trofimov (2014) Afroze Ibrahim Baqapuri and Ilya Trofimov. 2014.  Using Neural Networks for Click Prediction of Sponsored Search.  _CoRR_ abs/1412.6601 (2014).
  * Barbará et al. (2002) D. Barbará, Y. Li, and J. Couto. 2002.  COOLCAT: an entropy-based algorithm for categorical clustering. In _CIKM_. 582–589.
  * Barbaro et al. (2019) Eduardo Barbaro, Eoin Martino Grua, Ivano Malavolta, Mirjana Stercevic, Esther Weusthof, and Jeroen van den Hoven. 2019.  Modelling and predicting User Engagement in mobile applications.  _J. of Data Science_ (2019), 1–17.
  * Barbieri et al. (2016) Nicola Barbieri, Fabrizio Silvestri, and Mounia Lalmas. 2016\.  Improving Post-Click User Engagement on Native Ads via Survival Analysis. In _WWW_. 761–770.
  * Baron et al. (2014) Shawn D Baron, Caryn Brouwer, and Amaya Garbayo. 2014\.  A model for delivering branding value through high-impact digital advertising.  _J. of Advertising Research_ , 286–291.
  * Bidmon and Röttl (2018) Sonja Bidmon and Johanna Röttl. 2018.  _Advertising Effects of In-Game-Advertising vs. In-App-Advertising_.  Springer, 73–86.
  * Bigon et al. (2019) L. Bigon, G. Cassani, C. Greco, L. Lacasa, M. Pavoni, A. Polonioli, and J. Tagliabue. 2019.  Prediction is very hard, especially about conversion. Predicting user purchases from clickstream data in fashion e-commerce.  _CoRR_ abs/1907.00400 (2019).
  * Blondel et al. (2016) M. Blondel, A. Fujino, N. Ueda, and M. Ishihata. 2016\.  Higher-Order Factorization Machines. In _NIPS_. 3359–3367.
  * Chan et al. (2018) Patrick P. K. Chan, Xian Hu, Lili Zhao, Daniel S. Yeung, Dapeng Liu, and Lei Xiao. 2018\.  Convolutional Neural Networks based Click-Through Rate Prediction with Multiple Feature Sequences. In _IJCAI_. 2007–2013.
  * Chao Huang and Chawla (2019) Xuchao Zhang Chuxu Zhang Jiashu Zhao Dawei Yin Chao Huang, Xian Wu and Nitesh Chawla. 2019\.  Online Purchase Prediction via Multi-Scale Modeling of Behavior Dynamics.  _KDD_ (2019), 2613–2622.
  * Chapelle (2014) Olivier Chapelle. 2014\.  Modeling delayed feedback in display advertising. In _KDD_. 1097–1105.
  * Chapelle et al. (2014) Olivier Chapelle, Eren Manavoglu, and Rómer Rosales. 2014\.  Simple and Scalable Response Prediction for Display Advertising.  _J. of TIST_ 5 (2014), 61:1–61:34.
  * Chen et al. (2016) Gong Chen, Jacob H. Cox, A. Selcuk Uluagac, and John A. Copeland. 2016.  In-Depth Survey of Digital Advertising Technologies.  _J. of IEEE COMST_ 18 (2016), 2124–2148.
  * Chen et al. (2016) Junxuan Chen, Baigui Sun, Hao Li, Hongtao Lu, and Xian-Sheng Hua. 2016.  Deep CTR Prediction in Display Advertising.  _CoRR_ abs/1609.06018 (2016).
  * Chen et al. (2017) Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat-Seng Chua. 2017\.  Attentive Collaborative Filtering: Multimedia Recommendation with Item- and Component-Level Attention. In _SIGIR_. 335–344.
  * Chen et al. (2019) Wenqiang Chen, Lizhang Zhan, Yuanlong Ci, Minghua Yang, Chen Lin, and Dugang Liu. 2019\.  FLEN: Leveraging Field for Scalable CTR Prediction.  _CoRR_ abs/1911.04690 (2019).
  * Cheng et al. (2016) H. Cheng, L. Koc, J. Harmsen, and et al. 2016\.  Wide & Deep Learning for Recommender Systems. In _DLRS_. 7–10.
  * Choi et al. (2019) Hana Choi, Carl F. Mela, Santiago R. Balseiro, and Adam Leary. 2019\.  Online Display Advertising Markets: A Literature Review and Future Directions.  _J. of Information Systems Research_ 31, 556–575.
  * Chu (2011) Shu-Chuan Chu. 2011\.  Viral advertising in social media: Participation in Facebook groups and responses among college-aged users.  _J. of Interactive Advertising_ 12 (2011), 30–43.
  * Covington et al. (2016) P. Covington, J. Adams, and E. Sargin. 2016.  Deep Neural Networks for YouTube Recommendations. In _RecSys_. 191–198.
  * Dalessandro et al. (2014) Brian Dalessandro, Daizhuo Chen, Troy Raeder, Claudia Perlich, Melinda Han Williams, and Foster Provost. 2014.  Scalable hands-free transfer learning for online advertising. In _KDD_. 1573–1582.
  * Dalessandro et al. (2015) Brian Dalessandro, Rod Hook, Claudia Perlich, and Foster Provost. 2015.  Evaluating and Optimizing Online Advertising: Forget the Click, but There Are Good Proxies.  _J. of Big data_ 3 (2015), 90–102.
  * Deng et al. (2017) Y. Deng, Y. Shen, and H. Jin. 2017.  Disguise Adversarial Networks for Click-through Rate Prediction. In _IJCAI_. 1589–1595.
  * Diaz-Aviles et al. (2012) Ernesto Diaz-Aviles, Lucas Drumond, Lars Schmidt-Thieme, and Wolfgang Nejdl. 2012.  Real-time top-n recommendation in social streams. In _RecSys_. 59–66.
  * Dunlavy et al. (2011) Daniel M. Dunlavy, Tamara G. Kolda, and Evrim Acar. 2011\.  Temporal Link Prediction Using Matrix and Tensor Factorizations.  _J. of ACM TKDD_ 5 (2011), 10:1–10:27.
  * Ebadi et al. (2019) N. Ebadi, B. Lwowski, M. Jaloli, and P. Rad. 2019\.  Implicit Life Event Discovery From Call Transcripts Using Temporal Input Transformation Network.  _IEEE Access_ 7 (2019), 172178–172189.
  * Edizel et al. (2017) Bora Edizel, Amin Mantrach, and Xiao Bai. 2017\.  Deep Character-Level Click-Through Rate Prediction for Sponsored Search.  _CoRR_ abs/1707.02158 (2017).
  * Effendi and Ali (2017) Muhammad Junaid Effendi and Syed Abbas Ali. 2017.  Click Through Rate Prediction for Contextual Advertisment Using Linear Regression.  _CoRR_ abs/1701.08744 (2017).
  * F. et al. (2020) Maurizio F., Paolo C., and Dietmar J. 2020.  Methodological Issues in Recommender Systems Research. In _IJCAI_. 4706–4710.
  * Fan et al. (2019) Shaohua Fan, Junxiong Zhu, Xiaotian Han, Chuan Shi, Linmei Hu, Biyu Ma, and Yongliang Li. 2019.  Metapath-guided Heterogeneous Graph Neural Network for Intent Recommendation.  _KDD_ (2019), 2478–2486.
  * Feng et al. (2019) Yufei Feng, Fuyu Lv, Weichen Shen, Menghan Wang, Fei Sun, Yu Zhu, and Keping Yang. 2019.  Deep Session Interest Network for Click-Through Rate Prediction. In _IJCAI_. 2301–2307.
  * Gao et al. (2018) Hongchang Gao, Deguang Kong, Miao Lu, Xiao Bai, and Jian Yang. 2018.  Attention Convolutional Neural Network for Advertiser-level Click-through Rate Forecasting. In _WWW_. 1855–1864.
  * Ge et al. (2018) T. Ge, L. Zhao, G. Zhou, and et al. 2018\.  Image Matters: Visually Modeling User Behaviors Using Advanced Model Server.  _CIKM_ (2018), 2087–2095.
  * Gharibshah et al. (2020) Zhabiz Gharibshah, Xingquan Zhu, Arthur Hainline, and M. Conway. 2020.  Deep Learning for User Interest and Response Prediction in Online Display Advertising.  _J. of Springer DSE_ 5 (2020), 12–26.
  * Gligorijevic et al. (2019) Djordje Gligorijevic, Jelena Gligorijevic, and A. Flores. 2019\.  Time-Aware Prospective Modeling of Users for Online Display Advertising.  _CoRR_ abs/1911.05100 (2019).
  * Gligorijevic et al. (2018) Jelena Gligorijevic, Djordje Gligorijevic, Ivan Stojkovic, Xiao Bai, Amit Goyal, and Zoran Obradovic. 2018.  Deeply Supervised Semantic Model for Click-Through Rate Prediction in Sponsored Search.  _CoRR_ abs/1803.10739 (2018).
  * Graepel et al. (2010) Thore Graepel, Joaquin Quiñonero Candela, Thomas Borchert, and Ralf Herbrich. 2010.  Web-Scale Bayesian Click-through Rate Prediction for Sponsored Search Advertising in Microsoft’s Bing Search Engine. In _ICML_. 13–20.
  * Guo and Berkhahn (2016) Cheng Guo and Felix Berkhahn. 2016.  Entity Embeddings of Categorical Variables.  _CoRR_ abs/1604.06737 (2016).
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In _IJCAI_. 1725–1731.
  * Guo et al. (2018) H. Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, X. He, and Zhenhua Dong. 2018\.  DeepFM: An End-to-End Wide & Deep Learning Framework for CTR Prediction.  _CoRR_ abs/1804.04950 (2018).
  * Gupta and Pal (2019) Rajan T. Gupta and Saibal K. Pal. 2019.  Click-Through Rate Estimation Using CHAID Classification Tree Model. In _Advances in Analytics and Applications_. 45–58.
  * H. et al. (2010) Dustin H., Stefan S., Eren M., Hema R., and Chirs L. 2010.  Improving ad relevance in sponsored search. In _WSDM_. 361–370.
  * He et al. (2014) Xinran He, Stuart Bowers, Joaquin Quiñonero Candela, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, and Ralf Herbrich. 2014.  Practical Lessons from Predicting Clicks on Ads at Facebook. In _ADKDD_. 1–9.
  * He and Chua (2017) Xiangnan He and Tat-Seng Chua. 2017.  Neural Factorization Machines for Sparse Predictive Analytics. In _SIGIR_. 355–364.
  * He et al. (2020) X. He, K. Deng, X. Wang, Y. Li, Yongdong Zhang, and Meng Wang. 2020.  LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation.  _SIGIR_ (2020), 639–648.
  * He et al. (2017) X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T. Chua. 2017.  Neural Collaborative Filtering. In _WWW_. 173–182.
  * Hu et al. (2008) Y. Hu, Y. Koren, and C. Volinsky. 2008.  Collaborative Filtering for Implicit Feedback Datasets.  _ICDM_ (2008), 263–272.
  * Huang et al. (2019) Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. 2019\.  FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction. In _RecSys_. 169–177.
  * Jannach et al. (2020) Dietmar Jannach, Gabriel de Souza P. Moreira, and Even Oldridge. 2020\.  Why Are Deep Learning Models Not Consistently Winning Recommender Systems Competitions Yet? A Position Paper. In _RecSys_. 44–49.
  * Jiang et al. (2016) Zilong Jiang, Shu Gao, and Wei Dai. 2016.  Research on CTR Prediction for Contextual Advertising Based on Deep Architecture Model.  _Control Engineering and Applied Informatics_ 18 (03 2016), 11–19.
  * Jiang et al. (2018) Zilong Jiang, S. X. Gao, and Mingjiang Li. 2018.  An improved advertising CTR prediction approach based on the fuzzy deep neural network. In _PloS one_.
  * Juan et al. (2017) Yuchin Juan, Damien Lefortier, and Olivier Chapelle. 2017\.  Field-aware Factorization Machines in a Real-world Online Advertising System.  _CoRR_ abs/1701.04099 (2017).
  * Karmaker et al. (2017) Shubhra Karmaker, Parikshit Sondhi, and ChengXiang Zhai. 2017\.  On Application of Learning to Rank for E-Commerce Search.  _Proc. of the 40th ACM SIGIR Conference_ (2017).
  * Kim et al. (2019) K-M Kim, D. Kwak, H. Kwak, Y-J Park, S. Sim, J-H Cho, M. Kim, J. Kwon, Nako Sung, and J-W Ha. 2019.  Tripartite Heterogeneous Graph Propagation for Large-scale Social Recommendation. In _RecSys_. 56–60.
  * King et al. (2015) Michael A. King, Alan S. Abrahams, and Cliff T. Ragsdale. 2015\.  Ensemble learning methods for pay-per-click campaign management.  _Expert Syst. Appl._ 42 (2015), 4818–4829.
  * Koren (2008) Y. Koren. 2008.  Factorization Meets the Neighborhood: A Multifaceted Collaborative Filtering Model. In _KDD_. 426–434.
  * Kota and Agarwal (2011) N. Kota and D. Agarwal. 2011.  Temporal multi-hierarchy smoothing for estimating rates of rare events. In _KDD_. 1361–1369.
  * Krishnan and Sitaraman (2013) S. Krishnan and R. Sitaraman. 2013.  Understanding the effectiveness of video ads: a measurement study. In _IMC_. 149–162.
  * Ktena et al. (2019) S. Ktena, A. Tejani, L. Theis, P. Myana, D. Dilipkumar, F. Huszar, S. Yoo, and W. Shi. 2019\.  Addressing delayed feedback for continuous training with neural networks in CTR prediction.  _CoRR_ abs/1907.06558 (2019).
  * Kumar and Salo (2016) Ashish Kumar and Jari Salo. 2016.  Effects of link placements in email newsletters on their click-through rate.  _Journal of Marketing Communications_ 24, 5 (Mar 2016), 535–548.
  * Kumar et al. (2018) Rohan Kumar, Mohit Kumar, Neil Shah, and Christos Faloutsos. 2018\.  Did We Get It Right? Predicting Query Performance in e-Commerce Search.  _CoRR_ abs/1808.00239 (2018).
  * Lalmas et al. (2015) Mounia Lalmas, Janette Lehmann, Guy Shaked, Fabrizio Silvestri, and Gabriele Tolomei. 2015.  Promoting Positive Post-Click Experience for In-Stream Yahoo Gemini Users. In _KDD_. 1929–1938.
  * Lee et al. (2012) Kuang-chih Lee, Burkay Orten, Ali Dasdan, and Wentong Li. 2012\.  Estimating conversion rate in display advertising from past erformance data. In _KDD_. 768–776.
  * Leong et al. (2014) S. Leong, M. Mahdian, and S. Vassilvitskii. 2014.  Advertising in a Stream. In _Proc. of WWW Conf._
  * Li et al. (2015) Cheng Li, Yue Lu, Qiaozhu Mei, Dong Wang, and Sandeep Pandey. 2015.  Click-through Prediction for Advertising in Twitter Timeline. In _KDD_. 1959–1968.
  * Li et al. (2019) Feng Li, Zhenrui Chen, Pengjie Wang, Yi Ren, Di Zhang, and Xiaoyu Zhu. 2019.  Graph Intention Network for Click-through Rate Prediction in Sponsored Search.  _ACM SIGIR_ (2019), 961–964.
  * Li et al. (2020) Xiang Li, Chao Wang, Jiwei Tan, Xiaoyi Zeng, Dan Ou, and Bo Zheng. 2020\.  Adversarial Multimodal Representation Learning for Click-Through Rate Prediction.  _CoRR_ abs/2003.07162 (2020).
  * Li et al. (2020) Zeyu Li, Wei Cheng, Yang Chen, Haifeng Chen, and Wei Wang. 2020.  Interpretable Click-Through Rate Prediction through Hierarchical Attention.  _WSDM_ (2020), 313–321.
  * Li et al. (2019) Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, and Liang Wang. 2019.  Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction. In _CIKM_. 539–548.
  * Lian et al. (2018) J. Lian, X. Zhou, F. Zhang, Z. Chen, X. Xie, and G. Sun. 2018\.  xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems.  _KDD_ (2018).
  * Lin et al. (2019) X. Lin, H. Chen, C. Pei, F. Sun, X. Xiao, H. Sun, Y. Zhang, W. Ou, and P. Jiang. 2019\.  A Pareto-Efficient Algorithm for Multiple Objective Optimization in e-Commerce Recommendation. In _RecSys_. 20–28.
  * Ling et al. (2017) X. Ling, W. Deng, C. Gu, and et al. 2017\.  Model Ensemble for Click Prediction in Bing Search Ads. In _WWW_. 689–698.
  * Liu et al. (2019) Bin Liu, Ruiming Tang, Yingzhi Chen, Jinkai Yu, Huifeng Guo, and Yuzhou Zhang. 2019\.  Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction. In _WWW_. 1119–1129.
  * Liu et al. (2020a) Bin Liu, Niannan Xue, Huifeng Guo, Ruiming Tang, Stefanos Zafeiriou, Xiuqiang He, and Zhenguo Li. 2020a.  _AutoGroup: Automatic Feature Grouping for Modelling Explicit High-Order Feature Interactions in CTR Prediction_.
  * Liu et al. (2020b) Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, X. He, Z. Li, and Y. Yu. 2020b.  AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction. In _KDD_. 2636–2645.
  * Liu et al. (2017) Hui Liu, Xingquan Zhu, Kristopher Kalish, and Jeremy Kayne. 2017\.  ULTR-CTR: Fast Page Grouping Using URL Truncation for Real-Time Click Through Rate Estimation. In _IEEE International Conference on Information Reuse and Integration_.
  * Liu et al. (2015a) Qiang Liu, Shu Wu, and Liang Wang. 2015a.  Collaborative Prediction for Multi-entity Interaction With Hierarchical Representation. In _CIKM_. 613–622.
  * Liu et al. (2015b) Qiang Liu, Feng Yu, Shu Wu, and Liang Wang. 2015b.  A Convolutional Click Prediction Model. In _CIKM_. 1743–1746.
  * Liu et al. (2017) Xun Liu, Wei Xue, Lei Xiao, and Bo Zhang. 2017\.  PBODL : Parallel Bayesian Online Deep Learning for Click-Through Rate Prediction in Tencent Advertising System.  _CoRR_ abs/1707.00802 (2017).
  * Liu et al. (2019) Yozen Liu, Xiaolin Shi, Lucas Pierce, and Xiang Ren. 2019\.  Characterizing and Forecasting User Engagement with In-App Action Graph: A Case Study of Snapchat.  _KDD_ (2019), 2023–2031.
  * Liu et al. (2020) Zhe Liu, Xianzhi Wang, Lina Yao, Jake An, Lei Bai, and Ee-Peng Lim. 2020\.  Face to Purchase: Predicting Consumer Choices with Structured Facial and Behavioral Traits Embedding.  _CoRR_ abs/2007.06842 (2020).
  * Livne et al. (2020) Amit Livne, Roy Dor, Eyal Mazuz, Tamar Didi, Bracha Shapira, and Lior Rokach. 2020\.  Iterative Boosting Deep Neural Networks for Predicting Click-Through Rate.  _CoRR_ abs/2007.13087 (2020).
  * Ma et al. (2018) Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018.  Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. In _SIGIR_. 1137–1140.
  * Marciel et al. (2016) Miriam Marciel, Rubén Cuevas, Albert Banchs, Roberto González, Stefano Traverso, Mohamed Ahmed, and Arturo Azcorra. 2016\.  Understanding the Detection of View Fraud in Video Content Portals. In _WWW_. 357–368.
  * Mattson et al. (2019) P. Mattson, C. Cheng, C. Coleman, G. Diamos, and et al. 2019.  MLPerf Training Benchmark.  _CoRR_ abs/1910.01500.
  * McCreery and Krugman (2017) Stephen McCreery and Dean M. Krugman. 2017.  Tablets and TV Advertising: Understanding the Viewing Experience.  _Journal of Current Issues & Research in Advertising_ 38, 2 (Mar 2017), 197–211.
  * McMahan et al. (2013) B. McMahan, G. Holt, D. Sculley, and et al. 2013\.  Ad Click Prediction: A View from the Trenches. In _KDD_. 1222–1230.
  * Mei et al. (2007) Tao Mei, Xian-Sheng Hua, Linjun Yang, and Shipeng Li. 2007\.  VideoSense: towards effective online video advertising. In _the 15th Intl Conference on Multimedia_. 1075–1084.
  * Meng et al. (2014) Wei Meng, Xinyu Xing, Anmol Sheth, Udi Weinsberg, and Wenke Lee. 2014.  Your Online Interests: Pwned! A Pollution Attack Against Targeted Advertising. In _CCS_. 129–140.
  * Menon et al. (2011) Aditya Krishna Menon, Krishna-Prasad Chitrapura, Sachin Garg, Deepak Agarwal, and Nagaraj Kota. 2011\.  Response prediction using collaborative filtering with hierarchies and side-information. In _KDD_. 141–149.
  * Naumov et al. (2019) M. Naumov, D. Mudigere, H. Michael Shi, and et. al. 2019.  Deep Learning Recommendation Model for Personalization and Recommendation Systems.  _CoRR_ abs/1906.00091 (2019).
  * Niu et al. (2018) Chenglei Niu, Guojing Zhong, Y. Liu, Yandong Zhang, Y. Sun, Ailong He, and Zhaoji Chen. 2018.  Unstructured Semantic Model supported Deep Neural Network for Click-Through Rate Prediction.  _CoRR_ abs/1812.01353 (2018).
  * Oentaryo et al. (2014a) R. Oentaryo, E. Lim, M. Finegold, and et al. 2014a.  Detecting click fraud in online advertising: a data mining approach.  _J. Mach. Learn. Res._ 15 (2014), 99–140.
  * Oentaryo et al. (2014b) R. Oentaryo, E. Lim, J. Low, D. Lo, and M. Finegold. 2014b.  Predicting response in mobile advertising with hierarchical importance-aware factorization machine. In _WSDM_. 123–132.
  * Ouyang et al. (2019a) Wentao Ouyang, Xiuwu Zhang, Li Li, Heng Zou, Xin Xing, Zhaojie Liu, and Yanlong Du. 2019a.  Deep Spatio-Temporal Neural Networks for Click-Through Rate Prediction.  _KDD_ (2019), 2078–2086.
  * Ouyang et al. (2019b) Wentao Ouyang, Xiuwu Zhang, Shukui Ren, Li Li, Zhaojie Liu, and Yanlong Du. 2019b.  Click-through rate prediction with the user memory network.  _CoRR_ abs/1907.04667 (2019).
  * Ouyang et al. (2019c) Wentao Ouyang, Xiuwu Zhang, Shukui Ren, Chao Qi, Zhaojie Liu, and Yanlong Du. 2019c.  Representation Learning-Assisted Click-Through Rate Prediction. In _IJCAI_.
  * Pan et al. (2019a) Feiyang Pan, Shuokai Li, Xiang Ao, Pingzhong Tang, and Qing He. 2019a.  Warm Up Cold-start Advertisements: Improving CTR Predictions via Learning to Learn ID Embeddings.  _SIGIR_ (2019), 695–704.
  * Pan et al. (2019b) Junwei Pan, Yizhi Mao, Alfonso Lobos Ruiz, Yu Sun, and Aaron Flores. 2019b.  Predicting Different Types of Conversions with Multi-Task Learning in Online Advertising.  _KDD ’19_ (2019), 2689–2697.
  * Pan et al. (2019) Jing Pan, Weian Sheng, and Santanu Dey. 2019.  Order Matters at Fanatics Recommending Sequentially Ordered Products by LSTM Embedded with Word2Vec.  _CoRR_ abs/1911.09818 (2019).
  * Pan et al. (2018) Junwei Pan, Jian Xu, Alfonso Lobos Ruiz, Wenliang Zhao, Shengjun Pan, Yu Sun, and Quan Lu. 2018.  Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising. In _WWW_. 1349–1357.
  * Pan et al. (2016) Zhen Pan, Enhong Chen, Qi Liu, Tong Xu, Haiping Ma, and Hongjie Lin. 2016.  Sparse Factorization Machines for Click-through Rate Prediction. 400–409.  https://doi.org/10.1109/ICDM.2016.0051
  * Pei et al. (2019) Changhua Pei, Xinru Yang, Qing Cui, Xiao Lin, Fei Sun, Peng Jiang, Wenwu Ou, and Yongfeng Zhang. 2019.  Value-Aware Recommendation Based on Reinforcement Profit Maximization. In _WWW_. 3123–3129.
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014\.  DeepWalk: online learning of social representations.  _CoRR_ abs/1403.6652 (2014).
  * Pi et al. (2019) Qi Pi, Weijie Bian, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. 2019.  Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction.  _Proc. of the 25th ACM SIGKDD Conf._ (2019).
  * Pour et al. (2019) M. Safaei Pour, A. Mangino, K. Friday, Matthias Rathbun, E. Bou-Harb, F. Iqbal, Kh. Shaban, and A. Erradi. 2019\.  Data-Driven Curation, Learning and Analysis for Inferring Evolving IoT Botnets in the Wild. In _ARES_. Article 6.
  * Punjabi and Bhatt (2018) S. Punjabi and P. Bhatt. 2018.  Robust Factorization Machines for User Response Prediction. In _WWW_. 669–678.
  * Qi et al. (2020) P. Qi, X. Zhu, G. Zhou, Y. Zhang, Z. Wang, L. Ren, Y. Fan, and K. Gai. 2020.  Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction.  _CoRR_ arXiv:2006.05639 (June 2020).
  * Qin et al. (2020) Jiarui Qin, Weinan Zhang, Xin Wu, Jiarui Jin, Yuchen Fang, and Yong Yu. 2020\.  User Behavior Retrieval for Click-Through Rate Prediction.  _CoRR_ arXiv:2005.14171 (May 2020).
  * Qiu et al. (2020) Ruihong Qiu, Zi Huang, Jingjing Li, and Hongzhi Yin. 2020\.  Exploiting Cross-Session Information for Session-Based Recommendation with Graph Neural Networks.  _J. of ACM TIOS_ 38 (2020).
  * Qu et al. (2019) Yanru Qu, Ting Bai, Weinan Zhang, Jianyun Nie, and Jian Tang. 2019.  An end-to-end neighborhood-based interaction model for knowledge-enhanced recommendation.  _CoRR_ abs/1908.04032 (2019).
  * Qu et al. (2016) Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. 2016.  Product-Based Neural Networks for User Response Prediction.  _2016 IEEE 16th International Conference on Data Mining (ICDM)_ (2016), 1149–1154.
  * Qu et al. (2018) Yanru Qu, Bohui Fang, Wei-Nan Zhang, Ruiming Tang, Minzhe Niu, Huifeng Guo, yong Yu, and Xiuqiang He. 2018\.  Product-Based Neural Networks for User Response Prediction over Multi-Field Categorical Data.  _J. of ACM TIOS_ (2018).
  * Regelson et al. (2006) Regelson, Moira, Fain, and Daniel C. 2006\.  Predicting click-through rate using keyword clusters.
  * Ren et al. (2019) K. Ren, J. Qin, Y. Fang, W. Zhang, L. Zheng, W. Bian, G. Zhou, J. Xu, Y. Yu, X. Zhu, and K. Gai. 2019\.  Lifelong Sequential Modeling with Personalized Memorization for User Response Prediction. In _SIGIR’19_.
  * Ren et al. (2016) Kan Ren, Weinan Zhang, Yifei Rong, Haifeng Zhang, Yong Yu, and Jun Wang. 2016\.  User Response Learning for Directly Optimizing Campaign Performance in Display Advertising. In _CIKM_. 679–688.
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization Machines.  _2010 IEEE International Conference on Data Mining_ (2010), 995–1000.
  * Rendle et al. (2019) S. Rendle, L. Zhang, and Y. Koren. 2019.  On the Difficulty of Evaluating Baselines: A Study on Recommender Systems.  _CoRR_ abs/1905.01395 (2019).
  * Reps et al. (2014) Jenna Reps, Uwe Aickelin, Jonathan Garibaldi, and Chris Damski. 2014\.  Personalising Mobile Advertising Based on Users’ Installed Apps.  _ICDM Workshop_ , 338–345.
  * Richardson et al. (2007) Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007\.  Predicting clicks: estimating the click-through rate for new ads. In _WWW_. 521–530.
  * Ruihong et al. (2020) Qiu Ruihong, Yin Hongzhi, Huang Zi, and Tong Chen. 2020.  GAG: Global Attributed Graph Neural Network for Streaming Session-based Recommendation.  _CoRR_ abs/2007.02747 (2020).
  * Rutz et al. (2019) Oliver Rutz, Ashwin Aravindakshan, and Olivier Rubel. 2019\.  Measuring and forecasting mobile game app engagement.  _International Journal of Research in Marketing_ 36, 2 (Jun 2019), 185–199.
  * Rutz and Bucklin (2011) Oliver J. Rutz and Randolph E. Bucklin. 2011.  From Generic to Branded: A Model of Spillover in Paid Search Advertising.  _J. of Marketing Research_ (2011), 87–102.
  * Saborido et al. (2017) Rubén Saborido, Foutse Khomh, Giuliano Antoniol, and Yann-Gaël Guéhéneuc. 2017\.  Comprehension of Ads-Supported and Paid Android Applications: Are They Different?  _2017 IEEE/ACM (ICPC)_ (2017), 143–153.
  * Shan et al. (2016) Ying Shan, T. Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. 2016\.  Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features. In _KDD_. 255–262.
  * Sharma and Gaud (2015) Neha Sharma and Nirmal Gaud. 2015.  K-modes Clustering Algorithm for Categorical Data.  _International Journal of Computer Applications_ 127 (2015), 1–6.
  * Shen (2018) Weichen Shen. 2018\.  Easy-to-use,Modular and Extendible package of deep-learning based CTR models.  https://github.com/shenweichen/DeepCTR.
  * Shen (2019) Weichen Shen. 2019\.  (PyTorch) Easy-to-use,Modular and Extendible package of deep-learning based CTR models.  https://github.com/shenweichen/DeepCTR-Torch.
  * Shi et al. (2020) Shu-Ting Shi, Wenhao Zheng, Jun Tang, Qing-Guo Chen, Yao Hu, Jianke Zhu, and Ming Li. 2020.  Deep Time-Stream Framework for Click-Through Rate Prediction by Tracking Interest Evolution.  _CoRR_ abs/2001.03025 (2020).
  * Shioji and Arai (2017) Enno Shioji and Masayuki Arai. 2017.  Neural Feature Embedding for User Response Prediction in Real-Time Bidding (RTB).  _CoRR_ abs/1702.00855 (2017).
  * Song et al. (2020) Qingquan Song, Dehua Cheng, Hanning Zhou, Jiyan Yang, Yuandong Tian, and Xia Hu. 2020\.  Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction.  _CoRR_ abs/2007.06434 (2020).
  * Song et al. (2019) Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019.  AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks.  _CoRR_ abs/1810.11921 (2019).
  * Su et al. (2017) Yuhan Su, Zhongming Jin, Ying Chen, Xinghai Sun, Yaming Yang, Fangzheng Qiao, Fen Xia, and Wei Xu. 2017\.  Improving click-through rate prediction accuracy in online advertising by transfer learning. In _Intl. conference on WI_. 1018–1025.
  * Ta (2015) Anh-Phuong Ta. 2015\.  Factorization machines with follow-the-regularized-leader for CTR prediction in display advertising.  _IEEE Big Data_ (2015), 2889–2891.
  * Thejas et al. (2019) G. S. Thejas, Kianoosh G. Boroojeni, Kshitij Chandna, Isha Bhatia, S. S. Iyengar, and N. R. Sunitha. 2019\.  Deep Learning-based Model to Fight Against Ad Click Fraud. In _ACM SE_. 176–181.
  * Tian et al. (2015) T. Tian, J. Zhu, F. Xia, X. Zhuang, and T. Zhang. 2015.  Crowd Fraud Detection in Internet Advertising. In _WWW_. 1100–1110.
  * Tolomei et al. (2018) Gabriele Tolomei, Mounia Lalmas, Ayman Farahat, and Andrew Haines. 2018.  You must have clicked on this ad by mistake! Data-driven identification of accidental clicks on mobile ads with applications to advertiser cost discounting and click-through rate prediction.  _Springer - J. of Data Science and Analytics_ 7 (2018), 53–66.
  * Torres et al. (2014) Sergio Duarte Torres, Ingmar Weber, and Djoerd Hiemstra. 2014\.  Analysis of Search and Browsing Behavior of Young Users on the Web.  _J. of ACM TWEB_ 8 (2014), 7:1–7:54.
  * Truong et al. (2019) Vinh Truong, Mathews Nkhoma, and Wanni Pansuwong. 2019\.  An Integrated Effectiveness Framework of Mobile In-App Advertising.  _Australasian J. of Inf. Systems_ 23 (2019).
  * Vasile et al. (2017) Flavian Vasile, Damien Lefortier, and Olivier Chapelle. 2017\.  Cost-sensitive Learning for Utility Optimization in Online Advertising Auctions. 1–6.  https://doi.org/10.1145/3124749.3124751
  * Wang et al. (2018a) Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. 2018a.  RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems.  _CIKM ’18_ (2018), 417–426.
  * Wang et al. (2018b) Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018b.  DKN: Deep Knowledge-Aware Network for News Recommendation.  _CoRR_ abs/1801.08284 (2018).
  * Wang et al. (2019c) H. Wang, F. Zhang, M. Zhang, J. Leskovec, M. Zhao, W. Li, and Z. Wang. 2019c.  Knowledge-aware Graph Neural Networks with Label Smoothness Regularization for Recommender Systems.  _KDD_ (2019), 968–977.
  * Wang et al. (2019d) Hongwei Wang, Fuzheng Zhang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. 2019d.  Multi-Task Feature Learning for Knowledge Graph Enhanced Recommendation. In _WWW_. 2000–2010.
  * Wang et al. (2018) Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun Lee. 2018\.  Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba.  _KDD Intl. Conf._ (2018), 839–848.
  * Wang et al. (2016) Jun Wang, Weinan Zhang, and Shuai Yuan. 2016.  Display Advertising with Real-Time Bidding (RTB) and Behavioural Targeting.  _Found. Trends Inf. Retr._ 11 (2016), 297–435.
  * Wang et al. (2018) Qianqian Wang, Fang’ai Liu, Shuning Xing, and Xiaohui Zhao. 2018\.  A New Approach for Advertising CTR Prediction Based on Deep Neural Network via Attention Mechanism. In _Comp. Math. Methods in Medicine_.
  * Wang et al. (2017b) Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017b.  Knowledge Graph Embedding: A Survey of Approaches and Applications.  _IEEE Transactions on Knowledge and Data Engineering_ 29 (2017), 2724–2743.
  * Wang et al. (2017a) Ruoxi Wang, Bin Fu, Gang Fu, et al. 2017a.  Deep & Cross Network for Ad Click Predictions.  _CoRR_ abs/1708.05123 (2017).
  * Wang et al. (2020) Ruoxi Wang, Rakesh Shivanna, D. Cheng, S. Jain, D. Lin, L. Hong, and Ed Huai hsin Chi. 2020.  DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems.  _CoRR_ (2020).
  * Wang et al. (2019a) X. Wang, X. He, M. Wang, F. Feng, and T. Chua. 2019a.  Neural Graph Collaborative Filtering.  _CoRR_ abs/1905.08108 (2019).
  * Wang et al. (2019b) Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Peng Cui, P. Yu, and Yanfang Ye. 2019b.  Heterogeneous Graph Attention Network.  _CoRR_ abs/1903.07293 (2019).
  * Wang et al. (2011) X. Wang, W. Li, Y. Cui, R. Zhang, and J. Mao. 2011.  Click-Through Rate Estimation for Rare Events in Online Advertising.
  * Wen et al. (2018) Hong Wen, Jing Zhang, Quan Lin, Keping Yang, and Pipei Huang. 2018.  Multi-Level Deep Cascade Trees for Conversion Rate Prediction.  _CoRR_ abs/1805.09484 (2018).
  * Wen et al. (2019) Hong Wen, Jing Zhang, Yuan Wang, Wentian Bao, Quan Lin, and Keping Yang. 2019\.  Entire Space Multi-Task Modeling via Post-Click Behavior Decomposition for Conversion Rate Prediction.  _CoRR_ abs/1910.07099 (2019).
  * Wu et al. (2019) Xian Wu, Baoxu Shi, Yuxiao Dong, Chao Huang, and Nitesh V. Chawla. 2019.  Neural Tensor Factorization for Temporal Interaction Learning. In _WSDM_. 537–545.
  * Xiao et al. (2017) Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. 2017\.  Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks.  _CoRR_ abs/1708.04617 (2017).
  * Xu et al. (2020) Chen Xu, Quan Li, Junfeng Ge, Jinyang Gao, Xiaoyong Yang, Changhua Pei, Fei Sun, Jian Wu, Hanxiao Sun, and Wenwu Ou. 2020\.  Privileged Features Distillation at Taobao Recommendations.  _CoRR: Information Retrieval_ (2020).
  * Xu and Wu (2020) C. Xu and M. Wu. 2020\.  Learning Feature Interactions with Lorentzian Factorization Machine. In _AAAI_. 6470–6477.
  * Yang et al. (2016) Hongxia Yang, Quan Lu, Angus Xianen Qiu, and Chun Han. 2016\.  Large Scale CVR Prediction through Dynamic Transfer Learning of Global and Local Features, Vol. 53. PMLR, 103–119.
  * Yang et al. (2019) Xiao Yang, Tao Deng, Weihan Tan, Xutian Tao, Junwei Zhang, Shouke Qin, and Zongyao Ding. 2019.  Learning Compositional, Visual and Relational Representations for CTR Prediction in Sponsored Search. In _CIKM_. 2851–2859.
  * Yang et al. (2002) Y. Yang, X. Guan, and J. You. 2002.  CLOPE: a fast and effective clustering algorithm for transactional data. In _KDD_.
  * Yang et al. (2019) Yi Yang, Baile Xu, Furao Shen, and Jian Zhao. 2019\.  Operation-aware Neural Networks for User Response Prediction.  _J. of Elsevier Neural networks_ 121 (2019), 161–168.
  * Ying et al. (2018) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems.  _CoRR_ abs/1806.01973 (2018).
  * Yong et al. (2020) Xu Yong, Chen Jiahui, Huang Chao, Zhang Bo, Xing Hao, Dai Peng, and Bo Liefeng. 2020.  Joint Modeling of Local and Global Behavior Dynamics for Session-Based Recommendation*.  _J. of FAIA_ 325, ECAI 2020 (2020), 545–552.
  * Yuan et al. (2020) Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. 2020.  Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation. In _SIGIR_. 1469–1478.
  * Yuan et al. (2018) Yuan Yuan, Xiaojing Dong, Chen Dong, Yiwen Sun, Zhenyu Yan, and Abhishek Pani. 2018\.  Dynamic Hierarchical Empirical Bayes: A Predictive Model Applied to Online Advertising.  _CoRR_ abs/1809.02213 (2018).
  * Yuan et al. (2014) Y. Yuan, F. Wang, J. Li, and R. Qin. 2014\.  A survey on real time bidding advertising. In _IEEE SOLI_. 418–423.
  * Zhang et al. (2019a) C. Zhang, D. Song, C. Huang, A. Swami, and N. Chawla. 2019a.  Heterogeneous Graph Neural Network. In _KDD_. 793–803.
  * Zhang et al. (2019b) Chuxu Zhang, A. Swami, and Nitesh V. Chawla. 2019b.  SHNE: Representation Learning for Semantic-Associated Heterogeneous Networks.  _WSDM_ (2019).
  * Zhang et al. (2019) J. Zhang, T. Huang, and Z. Zhang. 2019.  FAT-DeepFFM: Field Attentive Deep Field-aware Factorization Machine. In _ICDM_.
  * Zhang et al. (2019) Li Zhang, Weichen Shen, Shijian Li, and Gang Pan. 2019\.  Field-Aware Neural Factorization Machine for Click-Through Rate Prediction.  _IEEE Access_ 7 (2019), 75032–75040.
  * Zhang et al. (2016a) Weinan Zhang, Lingxi Chen, and Jun Wang. 2016a.  Implicit Look-Alike Modelling in Display Ads - Transfer Collaborative Filtering to CTR Estimation.  _CoRR_ abs/1601.02377 (2016).
  * Zhang et al. (2016b) Weinan Zhang, Tianming Du, and Jun Wang. 2016b.  Deep Learning over Multi-field Categorical Data - - A Case Study on User Response Prediction. In _ECIR_.
  * Zhang et al. (2016c) Weinan Zhang, Tianxiong Zhou, Jun Wang, and Jian Xu. 2016c.  Bid-aware Gradient Descent for Unbiased Learning with Censored Data in Display Advertising. In _KDD_. 665–674.
  * Zhang et al. (2014) Yuyu Zhang, Hanjun Dai, Chang Xu, Jun Feng, Taifeng Wang, Jiang Bian, Bin Wang, and Tie-Yan Liu. 2014\.  Sequential Click Prediction for Sponsored Search with Recurrent Neural Networks. In _AAAI_. 1369–1375.
  * Zhang et al. (2020) Yang Zhang, Fuli Feng, Chenxu Wang, Xiangnan He, Meng Wang, Yan Li, and Yongdong Zhang. 2020.  How to Retrain Recommender System? A Sequential Meta-Learning Method. In _SIGIR_. 1479–1488.
  * Zhang et al. (2020) Y. Zhang, P. Zhao, Y. Guan, L. Chen, K. Bian, L. Song, B. Cui, and X. Li. 2020.  Preference-Aware Mask for Session-Based Recommendation with Bidirectional Transformer. In _ICASSP_. 3412–3416.
  * Zhao et al. (2020a) Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li. 2020a.  Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems.  _CoRR_ abs/2003.05622 (2020).
  * Zhao et al. (2020b) Xiangyu Zhao, Xudong Zheng, Xiwang Yang, Xiaobing Liu, and Jiliang Tang. 2020b.  Jointly Learning to Recommend and Advertise.  _CoRR_ abs/2003.00097 (2020).
  * Zhao et al. (2020) Yifei Zhao, Yu-Hang Zhou, Mingdong Ou, Huan Xu, and Nan Li. 2020.  Maximizing Cumulative User Engagement in Sequential Recommendation: An Online Optimization Perspective. In _KDD_. 2784–2792.
  * Zheng et al. (2010) Hua Zheng, Dong Wang, Qi Zhang, Hang Li, and Tinghao Yang. 2010.  Do Clicks Measure Recommendation Relevancy? An Empirical User Study. In _RecSys_. 249–252.
  * Zhou et al. (2018a) Guorui Zhou, Ying Fan, Runpeng Cui, Weijie Bian, Xiaoqiang Zhu, and Kun Gai. 2018a.  Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net.  _CoRR_ abs/1708.04106 (2018).
  * Zhou et al. (2019a) Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019a.  Deep Interest Evolution Network for Click-Through Rate Prediction.  _CoRR_ abs/1809.03672 (2019).
  * Zhou et al. (2018b) Guorui Zhou, Chengru Song, Xiaoqiang Zhu, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018b.  Deep Interest Network for Click-Through Rate Prediction.  _KDD_ (2018), 1059–1068.
  * Zhou et al. (2019b) Guorui Zhou, Kailun Wu, Weijie Bian, Zhao Yang, Xiaoqiang Zhu, and Kun Gai. 2019b.  Res-embedding for deep learning based click-through rate prediction modeling.  _CoRR_ abs/1906.10304 (2019).
  * Zhou et al. (2016) K. Zhou, M. Redi, A. Haines, and et al. 2016\.  Predicting Pre-click Quality for Native Advertisements. In _WWW_. 299–310.
  * Zhu et al. (2017b) Wen-Yuan Zhu, Chun-Hao Wang, Wen-Yueh Shih, W. Peng, and J. Huang. 2017b.  SEM: A Softmax-based Ensemble Model for CTR estimation in Real-Time Bidding advertising.  _IEEE J. of BigComp._ (2017), 5–12.
  * Zhu and Davidson (2007) Xingquan Zhu and Ian Davidson. 2007.  _Knowledge Discovery and Data Mining: Challenges and Realities_.  IGI Global.
  * Zhu et al. (2017a) X. Zhu, H. Tao, Z. Wu, J. Cao, K. Kalish, and J. Kayne. 2017a.  _Fraud Prevention in Online Digital Advertising_.  Springer.
  * Zoph and Le (2016) B. Zoph and Q. Le. 2016\.  Neural Architecture Search with Reinforcement Learning.  _CoRR_ abs/1611.01578 (2016).

##  Appendix A Appendix

###  A.1. Term Definitions

In Table A.1, we introduce some important terms used in the context of online
advertising, and also summarize common terminologies used in the paper.

Table A.1. Summary of key terms and notations in online advertising Notation |  Descriptions
---|---
Display Ads |  The form of ads containing rich media content that are displayed in reserved spaces in websites
Search Ads |  The form of ads that are displayed in the search result page triggered based on user search query. They are sorted and displayed to users to attract their attention to commercial products.
Native Ads |  The form of ads which are similar to regular posts in stream media platforms including video and images commercial contents.
Advertiser |  The stakeholders which promote products and services in online advertising, by serving ads to online users
Publisher |  The stakeholders which run websites with potential placements to show ads to online ads
Audience |  The online users who are exposed to ads in a respective online context
Ad Exchange |  The marketplace where advertisers and publishers are connected, through DSP and SSP, to negotiate selling and buying price using real-time bidding (auctions)
Auction |  A virtual real-time sale being held upon SSP ad request to gather bids of advertisers for an ad impression.
Demand Side Platform (DSP) |  A software platform in online advertising eco-system working on the behalf of advertisers to manage campaigns and submit a relevant bid price to bid requests
Supplier Side Platform (SSP) |  A software platform in online advertising working on the behalf of publishers to send a bid request to Ad exchange network. It loads the ad creative by calling ad server which has a winning ad in an auction
Ad Campaign |  A set of advertisements with a common theme (or objective) in targeting similar group of users
Ad Creative |  The message or artwork (advertising object), designed by advertisers, to be served to the audience’ devices
Ad Placement |  It refers to the location where an ad is displayed on the web-page. The common places that are available for advertiser to run an ads are the footer, the header and sidebars of website or anywhere in the beginning or middle of article and videos.
Banner Ad |  A rectangular graphic window, such as an iFrame, dedicated to show ad creative (image or text content) in the publisher’s web-page
Impression |  The rendering/presence of an ad on the user’s device
Landing page |  A web-page used to show to viewers, after they click on the ad
Click |  A user mouse click event or user tap event on the ad when they visiting an ad on desktop or mobile devices
Conversion |  The user actions, such as purchasing a product or subscribing to a service, after clicking an ad creative and being directed to advertiser’s landing web-pages
Gross Merchandise Volume |  A e-commerce metric considered by businesses to indicate the amount of sales made by users over a specific period before deducting any expenses like those associated with online marketing
Data Management Platform
(DMP) |  A software platform in online advertising designed to collect and analyze data for both advertisers and publishers. DMPs provide services to DSPs or SSPs to improve ad campaign efficiency
First party Data |  First party is referred to as the stakeholder itself, so first party data is referred to as data collected from the activities of business users of respective stakeholders. The data ranges from user profile information like demography user historical behaviors such as visited pages and purchase history, user subscription data or their activities in social media
Second party Data |  Second party is referred to as the other party of each stakeholder. The first party data of a company is referred to as the second party data of other companies
Third party Data |  Data gathered from outside sources to be packaged and sold to others. The data is organized to clusters and segments in terms of page information, user characteristics, and audience interest to be chosen by buyers. Third party data are typically gathered from DMP by analyzing user cookie information

##  Appendix B Media Types, Devices, and Platforms

Driven by the communication and networking technologies, online advertising
has been continuously evolving in the past two decades. Starting from static
banners on websites, the industry is now dynamically serving ads based on
types of media platforms, user devices, and media types (Wang et al., 2016).
In this section, we briefly summarize types of media platforms, user devices,
and ad types in the online advertising eco-system.

###  B.1. Media Platforms

There are various advertising platforms that have been proposed to serve
advertisements to users, depending on the context of users accessing to the
network.

####  B.1.1. Sponsored Search Marketing

Sponsored search is a search engine based advertising platform that uses user
query as the context and returns a list of ads related to user queries for
advertising. The result of ads are usually displayed as a sorted list of
commercial hyperlinks among the search result page on the basis of similarity
to the search result. To differentiate ads from regular list of search
results, an “Ad” tag is usually marked to promoted records. Following the pay-
per-click mechanism, the advertisers pay the search engine platform when user
click on ads. Then the potential revenue for publishers is calculated using
click-through rate values.

####  B.1.2. Display Advertising

Display advertising is one of latest forms of online advertising working on
the basis of real-time bidding to select and show personalized ads in the
dedicated areas of web pages. Different from search advertising where user
queries provide clear context of user preference, the context of display
advertising is often limited to the pages visited/requested by users. Given a
user requesting to visit a URL in the web browser, ads are delivered from
real-time auctions set up by ad exchange network. The main approach in this
advertising is targeting individuals using information available from them in
different internet sources (like cookies for browsing history). The main goal
for advertisers is to find strategies to reach to right customers to engage
them to take desirable action. The performance of marketing is evaluated based
on user responses like click or conversion which can determine display-related
advertising revenues and cost for publishers and advertisers.

####  B.1.3. Social Media Advertising

Social media networks are becoming common elements of daily life, and
providing an opportunity for people to interact and share information. The
popularity of these platforms motivate businesses and companies to target
their potential customers among online users. To this end, advertisers aim to
provide personalized posts or tweets in social media platforms, like Twitter,
Facebook, and Instagram, to attract user responses, which are usually measured
through click-through rate or conversion rate. To address the quality of ads,
some metrics like post-click-experience are also introduced to analyze the
dwell time users may take on a landing page following the click event. Some
studies (Lalmas et al., 2015; Barbieri et al., 2016) show that post-click-
experience can be used to gain additional knowledge about user preferences and
modeling user responses.

###  B.2. Ad Types

In order to promote products and services in online advertising, different
types of advertisements have been designed as the means of advertising.

####  B.2.1. Banner Ads

Banner ads, which have existed since the very beginning of online advertising
in 1990s, incorporate the main standard media type in the Internet for
advertising. Organized by compounds of text, image, or animated contents
placed in the specific area of web pages, banner ads present the advertisers’
message to users to attract their attention about the promoted content.
Clicking on the areas generally indicate some sort of matching between ad
content and user preferences. As a result, a click leads to a transition of
users from the publisher web pages to advertisers’ websites for marketing
purposes.

####  B.2.2. Textual Ads

Text ads are the most well-known type of ads in many advertising platforms
like sponsored search advertising, short message (SMS) marketing, email
advertising, or display advertising (Jiang et al., 2016). A text ad includes a
textual creative ad shown alongside search results (Edizel et al., 2017) or
part of the emails or text message sent to subscribed users to show
promotional messages. The elements of ads are organized to lead to a click
response directing users to promoted pages (Kumar and Salo, 2016).

####  B.2.3. Video Ads

With the increase of broadband Internet, the usage of video ads to deliver
promotional content has gained increasing popularity and becoming an effective
way of interact with audience. Today the striking amount of video ads are
employed to transfer commercial messages to online users. In the form of live
videos or downloadable video content, this type of advertisements can be
presented to users like banners in websites, tweets, or feeds in social media
platforms.

##### Roll-out Ads

Roll-out ad is a specific form of video ads that are joined to the other video
content in streaming websites like YouTube or Vudu. These videos automatically
appear before playing the original content can be seen in either skippable or
non-skippable formats. According to the place, the video ads being attached to
the beginning or the end of original one are called pre-roll or post-roll
advertisements. The mid-roll version of video ads is defined to augment to
some points in the middle of original videos which is contextually relevant
and less intrusive to users (Mei et al., 2007).

####  B.2.4. In-App Ads

This type of ads represents the version of advertisements which appear within
applications like video games in desktop systems or smartphones. Like digital
advertising, mobile apps are designed to reserve some space for ads. There are
generally two ways used to display and place ads in apps. The first is
considered as static brands involved in the background and integrated as the
part of app. This type follows the guaranteed contract setting and similar to
the primitive version of ads in the past which cannot directly receive user
responses. Therefore, they can be considered superficial or even not
recognizable by users (Bidmon and Röttl, 2018). The second type, which is
considered more interactive ones, places ads in transitions. They usually
appear in two forms of full screen or regular rich media ads. Following ad
real-time bidding model, relevant ads are selected from negotiation between
DSP and SSP to be displayed across the screen (Truong et al., 2019).

###  B.3. Source of Features

The source of features for user response prediction task is related to
information transferred in online advertising ecosystem. In the ad network
workflow in Figure 1, actors, such as advertiser and publisher, play a role
together with intermediary nodes to provide users with relevant commercial
content. In order to acquire positive user feedback, various data features are
used to represent users and describe advertisers and online content providers.
The data sources for feature extraction are information from demand side or
supplier side, or even external third party sources. The representative list
of features regarding supplier-side platform, demand-side platform and third-
party groups are shown in Table B.1.

####  B.3.1. Suppler Side Features

As showing in Figure 1, SSP or supplier side platforms are intermediary nodes
in the ad network which work on behalf of publisher to manage the inventory of
available ad placements in web pages. As soon as users submitting a keyword in
search engine or visiting a website in display advertising, an auction is
triggered to find ad to be served to the users. In this case, information
regarding visiting user and ad slots are transferred to relevant DSP nodes
through ad exchange network. As shown in Table B.1, this information is
characterized via features describing ad creatives and their appearance in the
publisher web-pages, such as features about the content of web-pages,
placement id, size, width and height, visibility status and format, as well as
user information such as device types, user agent, browser information etc.

Source |  Features
---|---
Supplier Side Platform |  Page URL, Device type, Devide Id, HTTP cookie, OS Version, Browser type/Version, User agent, Geo location, Ad slot (ID, width, height, visibility, format), placement ID, Publisher ID, User ID
Demand-Side Platform |  Bidding price, Paying price, Campaign Category, creative ID, Advertiser ID
Third-party |  User segmentation, User demography, Site information, Page information, Page categorization
Table B.1. A summary of major sources of categorical features in online
advertising

####  B.3.2. Demand Side Features

Another important party in the online advertising ecosystem is DSP (Demand
side platform) which intervenes in connections between advertisers and ad
exchange network. An ad exchange casts auction for bid request triggered by
SSPs to DSPs to select the display of ad on the publisher’s website.The Ad
exchange network collects bid prices offered by connected advertisers through
their DSPs, and selects ads corresponding to higher bid to present in
publisher websites. The bid price calculated by advertisers depending on three
set of main factors such as information available about users, the constraints
of publisher pages, and ad campaigns. The information is characterized using
features such as ID and the floor price of ad creatives, the URL of landing
page, the segmented user profiles, and ad campaign categories to describe the
campaign and content of ads. They are augmented with the online user browsing
history captured by DMP(Data Management Platform) nodes to facilitate decision
making to choose matching ads for users.

####  B.3.3. Third-Party Side Features

Features used to select the best matching ads to users’ interests are not only
limited to those from by DSPs and SSPs. Instead, external sources can also
provide compliment and valuable information needed to describe objects
involved in advertising scenarios. In this context, they are known as third
party data which are captured in various websites and social media platforms,
using data aggregation, cookie, or machine learning approaches. These data can
include different information exploited ranging from user web cookies, to
meta-data of devices and geographic information. For example, White Ops, an
online ad verification and fraud detection company, provides third-party side
services allowing users to query whether the traffic (i.e. the page visit) is
initialized from a genuine human user or a bot in real-time, using data
collected from “trillions of transactions”. Such services allow an advertiser
to determine whether an auction is potentially fraudulent and stop bidding on
fraud auctions (Zhu et al., 2017a). Online advertising systems can leverage
third party data to organize new features used to target users for their
campaigns.

###  B.4. Device Platforms

####  B.4.1. Desktop Advertising

Since the very beginning of banner ads on the web, desktop was and still is
the dominate device platform for advertising. Available through desktop
systems, desktop advertising entails expanded version of ads including text-
based advertisement, roll-out video ads, and in-app advertisements appearing
in search engine results, streaming web services or software. The capabilities
of smart phones make these devices as the predominant opponent of desktop
systems since it can be used for same purposes. However, it does not discount
the value of desktop branding as long as the desktop and laptop systems stays
on.

####  B.4.2. Mobile Advertising

As smartphones and mobile devices are becoming essential tools for
communication, online advertising also quickly adapts to mobile devices for
marketing. In early days of mobile phones, the common advertising form was SMS
advertising in which advertisers send the textual ads to customers. The rise
in popularity of multi-purpose mobile phones and wearable devices made an
opportunity for online companies to use a new way for advertising and
targeting audience. Nowadays, mobile advertising makes up a significant
portion of online advertising (Oentaryo et al., 2014b; Tolomei et al., 2018)
which can be roughly classified into the following two types:

##### Mobile Web Advertising

Like desktop workstations, one of the major advantages of smartphones is web
browsing, which relies on search engines to get relevant information for user
needs. There is no wonder that users take advantage of their phones to look
for services and products located near them. In era of virtual assistants,
voice search has seen significant growth using smartphones. Now, mobile web
advertising is being developed to leverage new ways, like natural language
based optimization, to provide sponsored search for verbal personalized ads
for users. Using cross-platform compatibility followed in mobile devices make
it possible for advertisers to focus on the content of their ads for potential
customers with low cost about the ways that are published in different devices
(Punjabi and Bhatt, 2018). But advertising is not limited to voice data. The
transcript of verbal or written conversations between customer-service agents
and people consist of valuable information about users implicit and explicit
behavior which can be analyzed to predict user life events and campaign
relevant products (Ebadi et al., 2019).

##### Mobile App Advertising

With the development of smartphone devices, there is an continually increase
in the production of mobile applications. The bulk of research confirmed the
fact that users today spend more time in online applications than web browsing
on cellphones (Aliannejadi et al., 2018). Mobile app advertising, known as in-
app advertising, is the specific type of advertising where ads are served
within smartphone app eco-system. In this way, there is a trend that
smartphone app developers prefer more to produce free applications by
supporting an ad model to gain revenue from presenting ads on their apps
(Saborido et al., 2017). This type of advertising deals with different
advertisement units like banners ads which are shown at top or bottom of apps
and transitional apps that are places for transitions inside applications.

####  B.4.3. Tablet Advertising

Tablet advertising is designed to present relevant ads for tablet users.
Although both tablet and smartphones are essentially mobile devices, they are
quite different in terms of quality of display and goals of usages. Some
studies highlighted the difference in user behavior regarding advertisement
when using tablet devices (McCreery and Krugman, 2017). The recent years have
seen a decline in using tablets since brands continue to create bigger screen
smartphones. But there are still many people use this device for entertainment
activities444http://googlemobileads.blogspot.com/2011/11/consumers-on-tablet-
devices-having-fun.html like playing online video games and second screen
viewing555https://www.statista.com/topics/2531/second-screen-usage/ to watch
the main media stream or live event and use tablets at the same time. Business
users also use it to complement office activities. It can lead to generate
plenty of opportunities to design distinctive tablet-based business-themed
approaches for ad campaigns. In these devices, advertisements take advantage
of high resolutions banners and videos to provide a better user experience and
grab more user engagement.

##  Appendix C Evaluation procedure of user response prediction models

The evaluation of user response prediction can be categorized into offline
(simulation based) evaluation and online test, which are from academia and
industry perspectives, respectively.

###  C.1. Offline Evaluation

Majority papers published in this domain are developed by researchers from
academia. Common approaches used by them to assess model performance are to
use a simulation of the real-world environment. In this condition, sample data
from different stakeholders in recommender and online advertising systems are
gathered to follow offline trials. As shown in Figure 2, the studies conducted
about recommendation systems and online advertising typically aim to provide
two types of outputs corresponding to different stages in the system. These
studies are categorized into two type of point-wise and list-wise methods. In
the former, the models predict the single numeric estimated value of user
feedback score such as product rates, click-through rate, or conversion rate
values. The latter prepares the ranked list of products ordered by predicted
user interaction scores. In point-wise models, models are evaluated using
different metrics such as root mean square error for regression problems
(product rating), Area over ROC curve (AUC) and Accuracy metrics for binary
classification methods i.e. CTR prediction, CVR (purchase) prediction. The
output in list-wise scenarios are optimized via ranking metrics like Precision
(Precision@K) and Recall (Recall@K) over top $k$ recommended list. The
performance of these systems are also evaluated from other aspects such as
Normalized Discounted Cumulative Gain (NDCG) and Mean Reciprocal Rank (MRR). A
common approach to evaluate the quality of a recommended item list with $k$
elements is evaluating the predicted score of user interaction for test
samples located among random selected set of unvisited items(He et al., 2017).
Table D.2 compares the performance of different methods for a binary
classification task to predict click events. Although the success of model
designed based on deep learning make it cornerstone to develop models in
academia and industry for user response prediction, it cannot be neglected
that in some scenarios like recommendation system contests (RecSys conference)
(Jannach et al., 2020) (F. et al., 2020) which evaluate the methodologies for
offline experiments and indicate that winning solutions could be selected from
the old techniques like SVMs, KNN, Logisitic Regression and Decision using
ensemble based models. Preparing proper environments to assess baselines is
also important (Rendle et al., 2019), and the research has shown that early
methods could outperform recent proposed algorithms as long as they are well
set up for experiments. These results provide some indications about the
evaluation procedures. It first can show the significance of feature
engineering and knowledge about of the application domain to use appropriate
features in modeling. It also raises the reliability issue in experiments.
Although many reported results are based on cross-validation, statistical
significance tests and availability of their code for reproducibility (Some of
which are gathered in Table D.3), several arbitrariness in experiment designs
should be addressed by research community. In the following sub-section, we
discuss evaluation methodologies followed in the reviewed papers covering
common approaches to set up experiments.

####  C.1.1. Experimental Setup

The evaluation step in prediction and classification tasks aim to assess how
well model can cope with new unseen samples. Typical data splitting is
performed by randomly selecting samples without replacement from datasets to
create three partitions: training, validation, and test. However, in
recommendation and online advertising systems, the dataset include data logs
of user interactions with online systems in which each sample come with a
timestamp. So considering the sequence of samples in term of time and applying
a chronological order constraint in the data splitting in datasets is a
rational expectation. The data-splitting in this case is followed by choosing
a arbitrary cut-off point in the dataset to prepare training and test subsets.
A typical approach is leave-one-out method which assigns the latest data to
the test-set while the reminder data are dedicated for training and validation
sets (He et al., 2017). To address short-term and long-term sequential data,
there are two form of event-based and session based datasets. In the event-
based dataset, the common idea is selecting randomly a couple of time
intervals where samples before the split point are assigned for training set
whereas the one after splitting time point are for test set (ex. in a dataset
including 7 subsequent days, the first subsequent days are assigned to
training while the last day data is for test set (Guo et al., 2017; Qu et al.,
2016)). In the session-based datasets, the same procedure is applied with a
difference that the samples before or after the split point include the
session of events. There are different approaches to define the session notion
to represent short-term user interaction logs. In (Feng et al., 2019), the
cut-off to split data logs into session is determined as the gap of 30 minutes
between subsequent interactions. Section D.1 summarizes a number of benchmark
datasets presented in different existing academic and industrial experiments.

###  C.2. Online A/B Test

A/B test is an evaluation mechanism which provides controlled environment to
compare models by splitting user traffic into two different portions, A vs. B,
to compare model performance. So models are assessed using real-world system
users to drive desired user feedback signals. To avoid misleading evaluation
results using the knowledge about the field, the statistical significance
tools are applied to create variation of data, and make statistical test to
evaluate confidence intervals. The rate of improvement compared to the
baselines and rate of error are central in results based on A/B test. A common
metric used to present online comparison is to calculate the relative
improvement like Normalized Log Loss (NLL) metric as
$\frac{{LL(\overline{p})-LL(p)}}{LL(\overline{p})}$ where $LL(\overline{p})$
is the log loss value of the best predictor on the test dataset while $LL(p)$
is for any baseline predictor $p$. Typically baseline models are built based
on logistic Regression (LR) models that are highly engineered with rich
features in production simulations (Juan et al., 2017; Cheng et al., 2016; Guo
et al., 2018; Qu et al., 2018). In recent developed models, the baseline
models are chosen from successful models like DeepFM (Guo et al., 2017) or
DIN(Zhou et al., 2018b) for online experiments (Liu et al., 2020b, a; Zhou et
al., 2019a). However, previously, authors in (Juan et al., 2017) compare the
performance of the proposed model, i.e. FFM with the baseline LR models via
calculating Return on Investment (ROI) relative improvement in display
advertising. Some other works like (Guo et al., 2018) calculate the relative
improvement in recommendation metric values like Coverage, Popularity and
Personalization to compare DeepFM model with a baseline LR model in Huawei app
market environment. Because of complexities in online businesses, there is no
standardized plan to follow online experiments. So the number of research
papers conducting this evaluation procedure are relatively limited in the
literature.

##  Appendix D Applications, Resources & Future Directions

For the online advertising, the click-through rate and conversion rate are
common user response metrics to evaluate the marketing performance and
determine the revenue of advertisers and publishers in online advertising and
recommender systems. In this section, we review some applications and tasks
related to user interactions in online content providers. We will introduce
the benchmark user-log datasets and the publicly available source-codes
provided by researchers. Such information is useful in a case we need to deal
with baseline methods in the domain of user response prediction.

###  D.1. Benchmark datasets for the user response prediction

Benchmark datasets are helpful type of data sources to carry out a fair
comparison between different methods. A list of publicly available datasets
that were examined in previous studies are summarized in Table D.1. Several
benchmarking datasets have been prepared and made publicly available to
conduct studies for the user response prediction. Table D.2 demonstrates the
performance of different user response prediction methods on selected
datasets. We describe two representative datasets in the following.

##### Criteo Dataset

This is one of the important benchmark datasets gathered from the seven days
real data logs by Criteo company. It was initially prepared for a competition
held by Kaggle in 2014 to encourage the development of approaches for click-
through rate prediction task. Criteo dataset includes 39 high cardinal
features which are consisting of 26 categorical features and 13 continuous
features for each pair of ad and user describing the event that ad visited by
the user. Each row corresponds to one impression. Each instance of user and ad
has a label to indicate whether the impressed ad receives ad click response or
not. It have a full and condensed version containing around 500 and 45 million
samples respectively.

##### Avazu Dataset

The second dataset prepared by Kaggle for a competition in 2014 including user
click behavior gathered from Avazu mobile advertising platform. Each row in
this dataset describes an impression (an ad displayed to users). Each
impression event is attributed by a set of features such as categorical
features for user, device and advertisement like hour of day, banner position,
site id and device model. Avazu dataset contains around ten days click-through
data of mobile ads following the chronological order. For experiments, two
subsets of dataset including 24 fields from the first 9 days of logs for
training and remaining for test and evaluations are also available.

Table D.1. Summary of benchmark datasets for online advertising user response
prediction.

Category | Dataset | |Feature| | |Impression| | |User| | |Item| | |Click| | |Conversion| |  Citation
---|---|---|---|---|---|---|---|---
Display Advertising |  Criteo${}^{\text{a}}$ | 39 | 45,840,617 | - | - | - | - |  (Chapelle et al., 2014; Juan et al., 2017; Ktena et al., 2019; Li et al., 2019; Lian et al., 2018; Liu et al., 2019; Pan et al., 2018; Punjabi and Bhatt, 2018; Qu et al., 2016, 2018; Song et al., 2020, 2019; Chapelle, 2014; Vasile et al., 2017; Wang et al., 2017a; Wang et al., 2020; Yang et al., 2019; Zhang et al., 2019; Guo et al., 2017, 2018; Huang et al., 2019; Li et al., 2020; Liu et al., 2020a)
Avazu(In-app)${}^{\text{b}}$ | 33 | 40,428,967 | - | - | - | - |  (Gupta and Pal, 2019; Huang et al., 2019; Chan et al., 2018; Li et al., 2019; Liu et al., 2019; Liu et al., 2015b; Punjabi and Bhatt, 2018; Qu et al., 2018; Song et al., 2020, 2019; Xu and Wu, 2020; Zhang et al., 2019, 2019; Chen et al., 2019; Deng et al., 2017; Li et al., 2020; Liu et al., 2020a, b, 2015a; Liu et al., 2017; Livne et al., 2020; Niu et al., 2018)
iPinYou${}^{\text{c}}$ | 16 | 19.5M | - | - | 14.79K | - |  (Pan et al., 2016; Qu et al., 2016, 2018; Ren et al., 2016; Zhang et al., 2016b, c; Zhu et al., 2017b; Liu et al., 2020a, b)
Taobao${}^{\text{d}}$ | - | - | 987K | 4.1M | - | - |  (Pi et al., 2019; Ren et al., 2019; Qin et al., 2020; Wang et al., 2018)
Conversion post-click logs |  Yoochoose${}^{\text{e}}$ | 9 | - | - | - | - | - |  (Liu et al., 2015b; Gligorijevic et al., 2019; Qiu et al., 2020; Yong et al., 2020)
Taobao${}^{\text{f}}$ | - | 84M | 0.4M | 4.3M | 3.4M | 18K |  (Ma et al., 2018)
Tencent${}^{\text{g}}$ | 12 | - | - | - | 50M | - |  (Yang et al., 2019; Pan et al., 2019a)
E-commerce Platform |  Amazon (Review logs)${}^{\text{h}}$ | - | - | - | - | - | - |  (Ren et al., 2019; Qi et al., 2020; Pi et al., 2019; Qu et al., 2019; Shi et al., 2020; Wang et al., 2018, 2019a; Zhou et al., 2019a, 2018b, b; He et al., 2020; Li et al., 2020)
Avito${}^{\text{i}}$ (Classified Ad portal) | 27 | 170,588,667 | - | - | - | - |  (Ouyang et al., 2019b, a, c)
Frappe${}^{\text{j}}$ (mobile app) | 10 | 288609 | 957 | 4082 | - | - |  (Wang et al., 2018; Xiao et al., 2017; Li et al., 2020)

${}^{\text{a}}$ http://labs.criteo.com/2014/02/kaggle-display-advertising-
challenge-dataset/
---
${}^{\text{b}}$ https://www.kaggle.com/c/avazu-ctr-prediction
${}^{\text{c}}$ https://contest.ipinyou.com/
${}^{\text{d}}$
https://tianchi.aliyun.com/dataset/dataDetail?dataId=649&userId=1
${}^{\text{e}}$ https://recsys.yoochoose.net/challenge.html
${}^{\text{f}}$
https://tianchi.aliyun.com/dataset/dataDetail?dataId=408&userId=1
${}^{\text{g}}$ https://algo.qq.com/
${}^{\text{h}}$ https://nijianmo.github.io/amazon/index.html
${}^{\text{i}}$ https://www.kaggle.com/c/avito-context-ad-clicks/data
${}^{\text{j}}$
https://github.com/hexiangnan/neural_factorization_machine/tree/master/data/frappe

Table D.2. Reported user response prediction (click-through rate prediction)
results in different experimets for eight commonly used datasets. The
performances were evaluated using AUC-ROC score metric.

Algorithm | Dataset(AUC-score)
---|---
Criteo | Avazu | Avito | Amazon |  Taobao  Alibaba | MovieLens | iPinYou |  Bing News
MA-W&D(Ouyang et al., 2019b) |  |  |  0.7976${}^{\text{a}}$ |  |  |  |  |
W&D-SSM(Niu et al., 2018) |  |  |  |  |  |  0.7754${}^{\text{d}}$ |  |
DeepMCP(Ouyang et al., 2019c) |  |  | 0.7927 |  |  |  |  |
DTSN-I(Ouyang et al., 2019a) |  |  | 0.8395 |  |  |  |  |
DIEN(Zhou et al., 2019a) |  |  |  |  0.7792${}^{\text{b}}$, 0.8453${}^{\text{c}}$ |  |  |  |
DIN(Zhou et al., 2018b) |  |  |  |  0.8818${}^{\text{b}}$ | 0.6031 |  0.7337${}^{\text{d}}$ |  |
DIN-ATT(Zhou et al., 2019b) |  |  |  |  0.9106${}^{\text{b}}$, 0.9404${}^{\text{c}}$ |  |  0.7429${}^{\text{d}}$ |  |
DSIN(Feng et al., 2019) |  |  |  |  | 0.6375 |  |  |
DeepFM(Guo et al., 2017) | 0.8016 |  |  |  |  |  |  |
xDeepFM(Lian et al., 2018) | 0.8052 |  |  |  |  |  |  | 0.8400
FNFM(Zhang et al., 2019) | 0.7470 |  |  |  |  |  |  |
FiBiNET(Huang et al., 2019) | 0.8021 | 0.7803 |  |  |  |  |  |
PNN(Qu et al., 2016) | 0.7700 |  |  |  |  |  | 0.7661 |
FNN(Zhang et al., 2016b) |  |  |  |  |  |  | 0.7071 |
AutoInt(Song et al., 2019) | 0.8061 | 0.7752 |  |  |  |  0.8456${}^{\text{e}}$ |  |
FAT-DeepFFM(Zhang et al., 2019) | 0.8104 | 0.7863 |  |  |  |  |  |
FGCNN(Liu et al., 2019) | 0.8022 | 0.7883 |  |  |  |  |  |
RippleNet(Wang et al., 2018a) |  |  |  |  |  |  0.9210${}^{\text{e}}$ |  | 0.6780
DKN(Wang et al., 2018b) |  |  |  |  |  |  |  | 0.6570
Fi-GNN(Li et al., 2019) | 0.8082 | 0.8120 |  |  |  |  |  |
KNI(Qu et al., 2019) |  |  |  |  0.9238${}^{\text{c}}$ |  |  0.9704${}^{\text{d}}$, 0.9449${}^{\text{e}}$ |  |
AutoGroup(Liu et al., 2020a) | 0.8028 | 0.7915 |  |  |  |  | 0.7859 |
AutoFIS(Liu et al., 2020b) | 0.8009 | 0.7852 |  |  |  |  |  |

${}^{\text{a}}$ The experiment was performed with no sampling approach on the
dataset
---
${}^{\text{b}}$ Electronics section ${}^{\text{c}}$ Book section
${}^{\text{d}}$ Section includes 20M rating instances ${}^{\text{e}}$ Section
includes 1M rating instances

###  D.2. Open-source Implementations

In this subsection, we collect a set of methods introduced in different
published papers in one place. Table D.3 denotes a list of presented methods
covered in this paper. It includes the presented method along with an employed
methodology and a link to the corresponding GitHub page of their
implementation.

Table D.3. Summary of classification based methods with open-source implementations Category | Model |  Framework |  Link
---|---|---|---
Factorization machines |  PNN (Qu et al., 2016), KFM (Qu et al., 2018) |  Tensorflow,libFFM |  https://github.com/Atomu2014/product-nets-distributed
NFM (He and Chua, 2017) |  libFM,Tensorflow |  https://github.com/hexiangnan/neural_factorization_machine
FiBiNET (Huang et al., 2019) |  DeepCTR, DeepCTR-Torch |
AFM (Xiao et al., 2017) |  Tensorflow |  https://github.com/hexiangnan/attentional_factorization_machine
Robust-FM (Punjabi and Bhatt, 2018) |  Spark |  https://www.dropbox.com/sh/ny6puvtopl98339/AACExLZ0waDL_ibWhfNItJfGa?dl=0
FLEN (Chen et al., 2019) |  Tensorflow |  https://github.com/aimetrics/jarvis
AutoFIS (Liu et al., 2020b) |  Tensorflow |  https://github.com/zhuchenxv/AutoFIS
Others(Rendle, 2010; Juan et al., 2017; Guo et al., 2017; Zhang et al., 2019) |  PyTorchFM |  https://github.com/rixwew/pytorch-fm
Deep Learning |  AutoInt (Song et al., 2019) |  Tensorflow |  https://github.com/DeepGraphLearning/RecommenderSystems
FGCNN (Liu et al., 2019) |  DeepCTR (Shen, 2018), Tensorflow |  https://github.com/shenweichen/DeepCTR
CCPM (Liu et al., 2015b) |  DeepCTR-Torch (Shen, 2019) PyTorch |  https://github.com/shenweichen/DeepCTR-Torch
xDeepFM (Lian et al., 2018) |  Tensorflow |  https://github.com/Leavingseason/xDeepFM
DIN (Zhou et al., 2018b) |  Tensorflow-gpu |  https://github.com/zhougr1993/DeepInterestNetwork
DSIN (Feng et al., 2019) |  DeepCTR (Shen, 2018), Tensorflow-gpu |  https://github.com/shenweichen/DSIN
DIEN (Zhou et al., 2019a) |  Tensorflow |  https://github.com/mouna99/dien
Wide&Deep (Cheng et al., 2016) |  Tensorflow |  https://github.com/Tensorflow/models/tree/master/official/r1/wide_deep
FNN (Zhang et al., 2016b) |  Theano |  https://github.com/wnzhang/deep-ctr
DSTN (Ouyang et al., 2019a) |  Tensorflow |  https://github.com/oywtece/dstn
W&D SSM (Niu et al., 2018) |  Tensorflow |  https://github.com/niuchenglei/ssm-dnn
Meta-embedding (Pan et al., 2019a) |  Tensorflow |  https://github.com/Feiyang/MetaEmbedding
DeepMCP (Ouyang et al., 2019c) |  Tensorflow |  https://github.com/oywtece/deepmcp
FAT-DeepFFM (Zhang et al., 2019) |  PyTorch |  (unofficial) https://github.com/p768lwy3/torecsys
MA-DNN (Ouyang et al., 2019b) |  Tensorflow |  https://github.com/rener1199/deep_memory
RocketLaunching (Zhou et al., 2018a) |  PyTorch |  https://github.com/zhougr1993/Rocket-Launching
GMP (Chao Huang and Chawla, 2019) |  Tensorflow |  https://github.com/graphmp/graphmp
MIMN (Pi et al., 2019) |  Tensorflow |  https://github.com/UIC-Paper/MIMN
UBR4CTR (Qin et al., 2020) |  Tensorflow |  https://github.com/qinjr/UBR4CTR
Network based |  KNI (Qu et al., 2019) |  Tensorflow |  https://github.com/Atomu2014/KNI
Fi-GNN (Li et al., 2019) |  Tensorflow |  https://github.com/CRIPAC-DIG/Fi_GNNs
RippleNet (Wang et al., 2018a) |  Tensorflow |  https://github.com/hwwang55/RippleNet
DKN (Wang et al., 2018b) |  Tensorflow |  https://github.com/hwwang55/DKN
KGNN-LS(Wang et al., 2019c) |  Tensorflow |  https://github.com/hwwang55/KGNN-LS

We intend to facilitate it for research communities who need to follow the
same program setting to compare and evaluate different methods. The majority
of the implementation are developed by using Python programming language. In
addition to official implementations, there are some toolkits like DeepCTR
(Shen, 2018) and DeepCTR-Torch (Shen, 2019) implemented based on Tensorflow
and PyTorch platforms. They are not only provided users with third party
implementations of contemporary methods in literature, but also they prepared
a platform including several software components to build customized models.
In this case researchers can examine different methods under the same input
and output interface and apply the similar setting to all approaches.

###  D.3. Applications

In this subsection, we briefly review applications typically developed in
relation to the user click-through rate prediction. The evaluation of the
probability that users make interactions more than a click on promoted item or
an ad have been studied from different aspect in many research.

##### Revenue per click prediction

This is an application to show earning of advertisers made from ad campaigns.
In this application a metric called RPC(revenue per click) is used to compute
the advertising interest based on user feedback in the form of clicks or
conversions. Like the click-through rate prediction, the data sparsity is the
common issue from advertiser perspectives to estimate the revenue based on
user click responses. It means that not only the proportion of click events
over impression are so small in user behavior histories, but also the number
of bid units which receive the click response and lead to revenue is also very
few. Although this metric is essentially important to analyze the performance
of the online advertising, there are a few studies published in literature
regarding this metric because of revenue confidentiality policy adopted by
many companies. In the following part, we go over an important work in this
domain:

Authors in (Yuan et al., 2018) proposed a model to dynamically determine the
data-driven hierarchy defined for the ad group and campaign and advertisers
account. Meanwhile, they presented an empirical bayes method to get inferences
through the hierarchical structure. In the context of the sponsored search
advertising, bid values typically assigned for keywords to calculate the
potential advertiser revenue. However, bid units are formed as the atomic
units for the combination of a keyword in addition to match type and an ad
group. The performance data were collected on the advertisers’ side for
experiments contain daily impressions, clicks, conversions and attributed
revenue at the bid-unit level. Therefore, the prediction problem was defined
as the prediction of the next day’s RPC for given bid unit, given the
historical clicks and revenue data. The features are the hierarchical
structure information of the bid units. It encompasses corresponding
campaigns, ad groups, and keywords, as well as geo-targeting information at
the campaign level, which are shared by the bid units under each campaign.
Following an empirical approach to get inferences through the hierarchical
structure, they proposed an extended empirical bayes method that was capable
of dynamically constructing the hierarchy and used the loss concept in
decision tree models for optimization.

##### Mistaken click prediction

In performance-based business models like the pay-per-click, the quality of
click event plays an important role for the revenue made for advertisers.
Relying on click events, there is a probability that advertisers are charged
by Ad Exchange Network using valueless clicks. These type of clicks is
generally considered as accidental clicks quite often happen in mobile
devices, when users are confronting interstitial ads. They are interrupted by
the ads covering the whole screen. They may click on ads and be directed to a
website and bounce back without spending a considerable time. Ignoring these
type of click may lead to overestimation of click-through rate values for the
mobile advertising.

To this end, authors in (Tolomei et al., 2018) proposed a data-driven method
to detect mistaken clicks on ads. Although from advertisers perspective,
valuable clicks are those followed by conversions, but it is not always true
for all click events. In the context of Yahoo mobile apps, they categorized
the clicks using extra information about the time users spending on the
advertiser landing page into three sections accidental, short and long.
Decomposing a dwell time distribution into three above classes, they proposed
a technique to apply a smooth discounting factor to charge less advertisers
with regard to accidental clicks.

##### Fraud in Online Advertising

Today, the main principle to target users for online advertising is tailoring
ads to user interest profiles. It consequently leads to a billing model to
charge advertisers and pay publishers based on how many times targeted users
interact with ads. In this business model, publishers register themselves in
the ad network to host ad placements and advertisers organize ad campaigns for
target users. From publisher’s perspective, the revenue from advertising is
directly dependent to number of users interacting with ads in web-pages and
the cost paid by advertisers for displaying targeted ads. The rate of
investment in online advertising is ascending annually. It tempts some people
to commit fraudulent activities. According to common performance based
business models like the cost-per-click and cost-per-impression for the
sponsored search and display advertising, fraudulent form of clicks and views
for textual rich media and video ads has gained a lot of attentions (Marciel
et al., 2016).

In study conducted in (Meng et al., 2014), it is demonstrated that organizing
user interest profile relying on user visits have a vulnerability can be used
for web-based fraud activities like the cross-site request forgery scripting
and click-jacking to embed hidden requests not initiated by users. The
increase in adopting IoT based solutions let users connected to Internet from
different devices. Compromised devices which do not follow basic security
measures can be easy target for such exploitations (Pour et al., 2019). The
fraud activities could orchestrate an attack against DoubleClick Ad Exchange
Network to a manipulate user interest profile which can modify the publisher
revenue. The main attribute of this attack is preparing a mechanism to modify
user interest profiles without explicit interactions with ad exchange network
and further knowledge about external involving factors. It was planned to
generate polluted user profiles worked based on the behavioral targeting and
re-targeting which led to present biased targeted ads that in turn need the
higher bid price made by advertisers and revenue for the publisher.

To deal with these smart threatening attacks, different studies have been
conducted to take the advantage of machine learning based methods to address
challenges from different aspects (Tian et al., 2015; Thejas et al., 2019;
Oentaryo et al., 2014a). In (Thejas et al., 2019) authors investigated to use
an auto-encoder neural network and GAN to regenerate click events. They
designed a neural network model to predict a fake click events by adding some
extent of noise to input data. In the context of sponsored search advertising,
the threat of fraud crowdsourcing activities was discussed in (Tian et al.,
2015). In this case, the fraudulent behaviors including a series of search and
click ads is distributed among vast number of web-publishers where fraudulent
traffic is buried in the majority of normal traffic. They constructed a graph
to represent a click history of users. They then applied a clustering
algorithm using a dispersity filter to find the coalitions that attacks are
concerted against a common set of advertisers.

##### Others

The application of predicting user response rates is not limited to above
applications. They are some studies consider the user response from different
aspects to predict short-term final desirable purchase activities
(Gligorijevic et al., 2019; Chao Huang and Chawla, 2019) or long-term
consistent influence like a branding (Baron et al., 2014) in common mobile and
display advertising.

###  D.4. Future Directions

User response prediction is considered as an important task to evaluate the
effectiveness of online advertising and recommender systems. It has led to
various research studies to address this problem. In this section, we present
a number of possible future directions identified in the recent studies. They
can be found useful by researchers in the community to develop the next
solutions.

##### Joint optimization models based on business value metrics

Implicit feedback like CTR value prediction is commonly used as an objective
to select candidate ads for end-users (Wang et al., 2018b; He et al., 2014).
An accurate prediction of CTR values can make a direct impact in the biding
value of ad in the display advertising (Chapelle et al., 2014). Although CTR
values can potentially show the user tendency with regard to items and
commercials in online provider systems, it is not a definite indicator to show
the success of business to gain new customers. Accordingly, business oriented
conversion rate values, which convey additional information about user intent
with regard to products and services in online systems, are proposed. For
example, by comparing different user conversion feedback values initiated by
click incident on items such as add-to-cards, add-to-wishlist and order
productsa, order rate is found to be the most robust objective rather compared
to add-to-cart to provide relevant personalized recommended list made by
e-commerce search (Karmaker et al., 2017). Pareto efficient learning-to-rank
algorithm (Lin et al., 2019) is also proposed to calculate solution by
aggregating two loss functions for CTR and Gross Merchandise Value (GMV)
prediction using proper constraints. Studies such as (Pei et al., 2019; Zhao
et al., 2020b) encompass a couple of user interactions as the conversion user
feedback. These actions are mapped to profit value which are considered as the
reward for the system. In the model training following reinforcement learning,
a policy to find better user actions is learnt to maximize accumulated profits
and accommodate user preferences. Learning multiple factors with regard to
optimize user preferences and commercial profits have a potential to shape the
new trend in developing recommendation system algorithms.

##### Neural Architecture Search for User Response Prediction

Deep neural networks are becoming increasingly popular in recommendation and
online advertising, but training process for them are potentially expensive.
For many methods, the structure of neural network are manually set via
extensive empirical studies. In our review, we have seen few attempts like
(Zhang et al., 2016b; Qu et al., 2016) to present fully connected neural
network for user response prediction. To model user interactions with other
components in the context of display advertising, diamond shape of multi-layer
perception network to have larger hidden layer is suggested by comparing
different structures. But broadly speaking, this process is not well-studied
in the literature and heavily relies on intuitive understanding of developers
and knowledge from the application domain like (Liu et al., 2020b, a). Neural
architecture search is to automate the architecture selection process to find
the best neural network architecture via an optimization procedure (Zoph and
Le, 2016). Multi-objective evolutionary optimization (Song et al., 2020) has
recently been proposed to select network architecture, by organizing the
search space as a direct acyclic graph, and utilizing learning-to-rank to
search and filter out a selection of architectures in each iteration. Future
research may consider domain knowledge and business metrics to design
efficient and effective neural architecture search strategies for user
response prediction.

##### Online Learning User Response Prediction and Recommender Systems

Online environment is inherently dynamic that the stream of data are changing
over time. It may lead a gradual shift in user preferences which can affect
the performance of predictive models. So recommendation systems generally need
to apply online learning (retraining mechanisms) to update and tackle new user
interactions. This is partially related to cross-domain recommendation
approaches that a pre-trained recommendation model is applied to different
downstream tasks (Yuan et al., 2020). The full training of model is the
straight-forward solution which is only helpful when we have a limited amount
of data (Livne et al., 2020). For online scenarios, selection based retraining
and fine-tuning are other types of retraining solutions. The former applies a
selection method for sampling of older user interaction and new data to create
an updated training data while the latter proposes a transferring strategy to
train a model using the new user interaction information. Very recently
authors in (Zhang et al., 2020) propose to learn transfer component in a
cyclic fashion using meta-learning approach for sequential input data.
Nonetheless, this topic is one of the important subjects for practical
recommendation systems that needs to be well-addressed in future.