* Ruprechter et al. (2019) Thorsten Ruprechter, Tiago Santos, and Denis Helic. 2019.  On the Relation of Edit Behavior, Link Structure, and Article Quality on Wikipedia. In _COMPLEX NETWORKS ’19: International Workshop on Complex Networks & Their Applications_. Springer, Cham, Switzerland, 242–254.  https://link.springer.com/chapter/10.1007/978-3-030-36683-4_20
  * Ruprechter et al. (2020) Thorsten Ruprechter, Tiago Santos, and Denis Helic. 2020.  Relating Wikipedia article quality to edit behavior and link structure.  _Applied Network Science_ 5 (2020), 1–20.  Issue 61. https://appliednetsci.springeropen.com/articles/10.1007/s41109-020-00305-y
  * Ruvo and Santone (2015) Giuseppe De Ruvo and Antonella Santone. 2015.  Analysing wiki quality using probabilistic model checking. In _WET ICE ’15: IEEE International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises_. Institute of Electrical and Electronic Engineers, New York City, United States, 224–229.  https://ieeexplore.ieee.org/document/71943655
  * Saengthongpattana and Soonthornphisaj (2014) Kanchana Saengthongpattana and Nuanwan Soonthornphisaj. 2014.  Assessing the Quality of Thai Wikipedia Articles Using Concept and Statistical Features. In _WorldCIST ’14: World Conference on Information Systems and Technologies_. Springer, Cham, Switzerland, 513–523.  https://link.springer.com/chapter/10.1007/978-3-319-05951-8_49
  * Saengthongpattana et al. (2017) Kanchana Saengthongpattana, Thepchai Supnithi, and Nuanwan Soonthornphisaj. 2017.  Ontology-Based Classifiers for Wikipedia Article Quality Classification. In _iSAI-NLP ’17: International Joint Symposium on Artificial Intelligence and Natural Language Processing_. Springer, Cham, Switzerland, 68–81.  https://link.springer.com/chapter/10.1007/978-3-319-94703-7_7
  * Saengthongpattana et al. (2018) Kanchana Saengthongpattana, Thepchai Supnithi, and Nuanwan Soonthornphisaj. 2018.  Quality Classification of ASEAN Wikipedia Articles using Statistical Features. In _iSAI-NLP ’18: International Joint Symposium on Artificial Intelligence and Natural Language Processing_. Institute of Electrical and Electronic Engineers, New York City, United States, 1–6.  https://ieeexplore.ieee.org/document/8692954/
  * Salutari et al. (2019) Flavia Salutari, Diego Da Hora Gilles, Dubuc, and Dario Rossi. 2019.  A Large-Scale Study of Wikipedia Users’ Quality of Experience. In _The World Wide Web Conference_ (San Francisco, CA, USA) _(WWW ’19)_. Association for Computing Machinery, New York, NY, USA, 3194–3200.  https://doi.org/10.1145/3308558.3313467
  * Sarkar et al. (2019) Soumya Sarkar, Bhanu Prakash Reddy Guda, Sandipan Sikdar, and Animesh Mukherjee. 2019.  StRE: Self Attentive Edit Quality Prediction in Wikipedia. In _ACL ’19: Annual Meeting of the Association for Computational Linguistics_. ACL Anthology, Online, 3962–3972.  https://aclanthology.org/P19-1387/
  * Schapire (2003) Robert E. Schapire. 2003.  _The Boosting Approach to Machine Learning: An Overview_.  Springer New York, New York, NY, 149–171.  https://doi.org/10.1007/978-0-387-21579-2_9
  * Schmidt and Zangerle (2019) Manuel Schmidt and Eva Zangerle. 2019.  Article quality classification on Wikipedia: introducing document embeddings and content features. In _OpenSym ’19: International Symposium on Open Collaboration_. Association for Computing Machinery, New York City, United States, 1–8.  https://dl.acm.org/doi/10.1145/3306446.3340831
  * Seyedsadr et al. (2016) Seyedtaha Seyedsadr, Mohammadali Afsharkazemi, and Hashem Nikoomaram. 2016.  Qualifying Articles of Persian Wikipedia Encyclopedia Through J48 Algorithm, ANFIS and Subtractive Clustering.  _Automation_ 3 (2016), 141–153.  Issue 6. https://www.sciencepublishinggroup.com/journal/paperinfo?journalid=134%26doi=10.11648/j.acis.20150306.18
  * Shen et al. (2017) Aili Shen, Jianzhong Qi, and Timothy Baldwin. 2017.  A Hybrid Model for Quality Assessment of Wikipedia Articles. In _ALTA ’17: Australasian Language Technology Association Workshop_. ACL Anthology, Online, 43–52.  https://aclanthology.org/U17-1005/
  * Shen et al. (2019) Aili Shen, Bahar Salehi, Timothy Baldwin, and Jianzhong Qi. 2019.  A joint model for multimodal document quality assessment. In _JCDL ’19: Joint Conference on Digital Libraries_. Association for Computing Machinery, New York City, United States, 107–110.  https://dl.acm.org/doi/10.1109/JCDL.2019.00024
  * Shen et al. (2020) Aili Shen, Bahar Salehi, Jainzhong Qi, and Timothy Baldwin. 2020.  A multimodal approach to assessing document quality.  _Journal of Artificial Intelligence Research_ 68 (2020), 607–632.  https://www.jair.org/index.php/jair/article/view/11647
  * Soonthornphisaj and Paengporn (2017) Nuanwan Soonthornphisaj and Peerapoom Paengporn. 2017.  Thai Wikipedia article quality filtering algorithm. In _IMECS ’17: International MultiConference of Engineers and Computer Scientists_. International Association of Engineers, Hong Kong, China.  https://www.iaeng.org/publication/IMECS2017/IMECS2017_pp299-305.pdf
  * Stein and Hess (2007) Klaus Stein and Claudia Hess. 2007.  Does it matter who contributes: a study on featured articles in the german wikipedia. In _HT ’07: Conference on Hypertext and Hypermedia_. Association for Computing Machinery, New York City, United States, 171–174.  https://dl.acm.org/doi/10.1145/1286240.1286290
  * Stvilia et al. (2009) Besiki Stvilia, Abdullah Al-Faraj, and Yong Jeong Yi. 2009.  Issues of cross-contextual information quality evaluation—The case of Arabic, English, and Korean Wikipedias.  _Library & Information Science Research_ 31 (2009), 232–239.  Issue 4. https://www.sciencedirect.com/science/article/pii/S0740818809000954
  * Stvilia et al. (2007) Besiki Stvilia, Les Gasser, Michael B. Twidale, and Linda C. Smith. 2007.  A framework for information quality assessment.  _Journal of the Association for Information Science and Technology_ 58 (2007), 1720–1733.  Issue 12. https://onlinelibrary.wiley.com/doi/10.1002/asi.20652
  * Stvilia et al. (2005a) Besiki Stvilia, Michael B. Twidale, Les Gasser, and Linda C. Smith. 2005a.  Information quality discussions in wikipedia. In _ICKM ’05: International Conference on Knowledge Management_. Universiti Putra Malaysia, Seri Kembangan, Malaysia.  https://www.researchgate.net/publication/200773232_Information_Quality_Discussions_in_Wikipedia
  * Stvilia et al. (2005b) Besiki Stvilia, Michael B. Twidale, Linda C. Smith, and Les Gasser. 2005b.  Assessing information quality of a community-based encyclopedia. In _ICIQ ’05: International Conference on Information Quality_. Massachusetts Institute of Technology, Cambridge, Massachusetts, 442–454.  https://www.semanticscholar.org/paper/Assessing-Information-Quality-of-a-Community-Based-Stvilia-Twidale/dd888dddccc2075a44f99ec2380fda652040afaf
  * Su and Liu (2015) Qi Su and Pengyuan Liu. 2015.  A Psycho-Lexical Approach to the Assessment of Information Quality on Wikipedia. In _WI-IAT ’15: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology_. Institute of Electrical and Electronic Engineers, New York City, United States, 184–187.  https://ieeexplore.ieee.org/document/7397452
  * Sugandhika and Ahangama (2022) Chinthani Sugandhika and Supunmali Ahangama. 2022.  Assessing Information Quality of Wikipedia Articles through Google’s E-A-T Model.  _IEEE Access_ 10 (2022), 52196–52209.  https://ieeexplore.ieee.org/document/9770051
  * Sugandhika et al. (2021) Chinthani Sugandhika, Supunmali Ahangama, and Sapumal Ahangama. 2021.  Modelling Wikipedia’s Information Quality using Informativeness, Reliability and Authority. In _ICAC ’21: International Conference on Advancements in Computing_. Institute of Electrical and Electronic Engineers, New York City, United States, 169–174.  https://ieeexplore.ieee.org/document/9671092
  * Suzuki (2012) Yu Suzuki. 2012.  Assessing Quality Values of Wikipedia Articles Using Implicit Positive and Negative Ratings. In _WAIM ’12: International Conference on Web-Age Information Management_. Springer, Berlin, Heidelberg, 127–138.  https://link.springer.com/chapter/10.1007/978-3-642-32281-5_13
  * Suzuki (2013) Yu Suzuki. 2013.  Effects of Implicit Positive Ratings for Quality Assessment of Wikipedia Articles.  _Journal of Information Processing_ 21 (2013), 342–348.  Issue 2. https://www.jstage.jst.go.jp/article/ipsjjip/21/2/21_342/_article
  * Suzuki (2015) Yu Suzuki. 2015.  Quality Assessment of Wikipedia Articles Using h-index.  _Journal of Information Processing_ 23 (2015), 22–30.  Issue 1. https://www.jstage.jst.go.jp/article/ipsjjip/23/1/23_22/_article
  * Suzuki and Yoshikawa (2012a) Yu Suzuki and Masatoshi Yoshikawa. 2012a.  Mutual evaluation of editors and texts for assessing quality of Wikipedia articles. In _WikiSym ’12: International Symposium on Wikis and Open Collaboration_. Association for Computing Machinery, New York City, United States, 1–10.  https://dl.acm.org/doi/10.1145/2462932.2462956
  * Suzuki and Yoshikawa (2012b) Yu Suzuki and Masatoshi Yoshikawa. 2012b.  QualityRank: assessing quality of wikipedia articles by mutually evaluating editors and texts. In _HT ’12: ACM Conference on Hypertext & Social Media_. Association for Computing Machinery, New York City, United States, 307–308.  https://dl.acm.org/doi/10.1145/2309996.2310047
  * Suzuki and Yoshikawa (2013) Yu Suzuki and Masatoshi Yoshikawa. 2013.  Assessing quality score of Wikipedia article using mutual evaluation of editors and texts. In _CIKM ’13: ACM International Conference on Information & Knowledge Management_. Association for Computing Machinery, New York City, United States, 1722–1732.  https://dl.acm.org/doi/10.1145/2505515.2505610
  * Sydow et al. (2017) Marcin Sydow, Katarzyna Baraniak, and Paweł Teisseyre. 2017.  Diversity of editors and teams versus quality of cooperative work: experiments on wikipedia.  _Journal of Intelligent Information Systems_ 48 (2017), 601–632.  https://link.springer.com/article/10.1007/s10844-016-0428-1
  * Sáez-Trumper (2021) Diego Sáez-Trumper. 2021.  Disinformation and AI: The Differences Between Wikipedia and Social Media.  https://diff.wikimedia.org/2021/09/15/disinformation-and-ai-the-differences-between-wikipedia-and-social-media/.  Accessed: 2023-04-04.
  * Teblunthuis (2021) Nathan Teblunthuis. 2021.  Measuring Wikipedia Article Quality in One Dimension by Extending ORES with Ordinal Regression. In _Proceedings of the 17th International Symposium on Open Collaboration_ (Online, Spain) _(OpenSym ’21)_. Association for Computing Machinery, New York, NY, USA, Article 5, 10 pages.  https://doi.org/10.1145/3479986.3479991
  * Tsikerdekis (2017) Michail Tsikerdekis. 2017.  Cumulative Experience and Recent Behavior and their Relation to Content Quality on Wikipedia.  _Interacting with Computers_ 29 (2017), 737–754.  Issue 5. https://academic.oup.com/iwc/article/29/5/737/3885842
  * Urquiza et al. (2016) Guido Urquiza, Matías Soria, Sebastián Pérez Casseignau, Edgardo Ferretti, Sergio Alejandro Gómez, and Marcelo Errecalde. 2016.  On the Assessment of Information Quality in Spanish Wikipedia. In _CACIC ’19: Argentine Congress of Computer Science_. National University of La Plata, La Plata, Argentina, 702–711.  http://sedici.unlp.edu.ar/handle/10915/56750
  * Velichety (2019) Srikar Velichety. 2019.  Quality Assessment of Peer-Produced Content in Knowledge Repositories Using Big Data and Social Networks: The Case of Implicit Collaboration in Wikipedia.  _ACM SIGMIS Database: The DATABASE for Advances in Information Systems_ 50 (2019), 28–51.  Issue 4. https://dl.acm.org/doi/10.1145/3371041.3371045
  * Velichety et al. (2019) Srikar Velichety, Sudha Ram, and Jesse Bockstedt. 2019.  Quality Assessment of Peer-Produced Content in Knowledge Repositories using Development and Coordination Activities.  _Journal of Management Information Systems_ 36 (2019), 478–512.  Issue 2. https://www.tandfonline.com/doi/full/10.1080/07421222.2019.1598692
  * Velázquez et al. (2017) Carlos G. Velázquez, Leticia C. Cagnina, and Marcelo Errecalde. 2017.  On the Feasibility of External Factual Support as Wikipedia’s Quality Metric.  _Processamiento del Lenguaje Natural_ 58 (2017), 93–100.  http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/5417
  * Vincent and Hecht (2021) Nicholas Vincent and Brent Hecht. 2021.  A Deeper Investigation of the Importance of Wikipedia Links to Search Engine Results.  _Proceedings of the ACM on Human-Computer Interaction_ 5 (2021), 18.  Issue CSCW1. https://doi.org/10.1145/3449078
  * Wang et al. (2021) Ping Wang, Muyan Li, Xiaodan Li, Heshen Zhou, and Jingrui Hou. 2021.  A hybrid approach to classifying Wikipedia article quality flaws with feature fusion framework.  _Expert Systems with Applications_ 181 (2021), 115089.  Issue 1. https://www.sciencedirect.com/science/article/pii/S0957417421005303?via%253Dihub
  * Wang and Li (2020) Ping Wang and Xiaodan Li. 2020.  Assessing the quality of information on wikipedia: A deep‐learning approach.  _Journal of the Association for Information Science and Technology_ 71 (2020), 16–28.  Issue 1. https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24210
  * Wang et al. (2019) Ping Wang, Xiaodan Li, and Renli Wu. 2019.  A deep learning-based quality assessment model of collaboratively edited documents: A case study of Wikipedia.  _Journal of Information Science_ 47 (2019), 176 – 191.  Issue 2. https://journals.sagepub.com/doi/10.1177/0165551519877646
  * Wang and Iwaihara (2010) Se Wang and Mizuho Iwaihara. 2010.  Quality Evaluation of Wikipedia Articles through Edit History and Editor Groups. In _APWeb ’11: Asia-Pacific Web Conference_. Springer, Berlin, Heidelberg, 188–199.  https://link.springer.com/chapter/10.1007/978-3-642-20291-9_20
  * Warncke-Wang et al. (2013) Morten Warncke-Wang, Dan Cosley, and John Riedl. 2013.  Tell me more: an actionable quality model for Wikipedia. In _WikiSym ’13: International Symposium on Open Collaboration_. Association for Computing Machinery, New York City, United States, 1–10.  https://dl.acm.org/doi/10.1145/2491055.2491063
  * Wikimedia (2022a) Wikimedia. 2022a.  Wikipedia Statistics - Edit and Revert Trends.  https://stats.wikimedia.org/EN/EditsRevertsEN.htm.  Accessed: 2023-04-03.
  * Wikimedia (2022b) Wikimedia. 2022b.  Wikistats - Statistics For Wikimedia Projects.  https://stats.wikimedia.org.  Accessed: 2023-04-03.
  * Wikipedia (2022a) Wikipedia. 2022a.  List of Wikipedias.  https://meta.wikimedia.org/wiki/List_of_Wikipedias.  Accessed: 2023-04-03.
  * Wikipedia (2022b) Wikipedia. 2022b.  Wikipedia.  https://en.wikipedia.org/wiki/Wikipedia.  Accessed: 2023-04-03.
  * Wikipedia (2022c) Wikipedia. 2022c.  Wikipedia: Content assessment.  https://en.wikipedia.org/wiki/Wikipedia:Content_assessment.  Accessed: 2023-04-03.
  * Wikipedia (2022d) Wikipedia. 2022d.  Wikipedia: Size of Wikipedia.  https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia.  Accessed: 2023-04-03.
  * Wikipedia (2023) Wikipedia. 2023.  Wikipedia: Manual of Style.  https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/.  Accessed: 2023-04-04.
  * Wilkinson and Huberman (2007) Dennis Wilkinson and Bernardo Huberman. 2007.  Cooperation and quality in wikipedia. In _WikiSym ’07: International Symposium on Wikis_. Association for Computing Machinery, New York City, United States, 157–164.  https://dl.acm.org/doi/10.1145/1296951.1296968
  * Wolpert and Macready (1997) David H Wolpert and William G Macready. 1997.  No Free Lunch Theorems for Optimization.  _IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION_ 1 (1997), 67.  Issue 1.
  * Wu et al. (2011) Guangyu Wu, Martin Harrigan, and Pádraig Cunningham. 2011.  Characterizing Wikipedia pages using edit network motif profiles. In _SMUC ’11: International Workshop on Search and Mining User-generated Contents_. Association for Computing Machinery, New York City, United States, 45–52.  https://dl.acm.org/doi/10.1145/2065023.2065036
  * Wu et al. (2012) Guangyu Wu, Martin Harrigan, and Pádraig Cunningham. 2012.  Classifying Wikipedia articles using network motif counts and ratios. In _WikiSym ’12: International Symposium on Wikis and Open Collaboration_. Association for Computing Machinery, New York City, United States, 1–12.  https://dl.acm.org/doi/10.1145/2462932.2462948
  * Wu et al. (2010) Kewen Wu, Qinghua Zhu, Yuxiang Zhao, and Hua Zheng. 2010.  Mining the Factors Affecting the Quality of Wikipedia Articles. In _ISME ’10: International Conference of Information Science and Management Engineering_. Institute of Electrical and Electronic Engineers, New York City, United States, 343–346.  https://ieeexplore.ieee.org/document/5572324
  * Wöhner et al. (2015) Thomas Wöhner, Sebastian Köhler, and Ralf Peters. 2015.  Good Authors = Good Articles? - How Wikis Work. In _WI ’15: International Conference on Wirtschaftsinformatik_. Association for Information Systems, Atlanta, Georgia.  https://aisel.aisnet.org/wi2015/59/?utm_source=aisel.aisnet.org%252Fwi2015%252F59%26utm_medium=PDF%26utm_campaign=PDFCoverPages
  * Wöhner and Peters (2009) Thomas Wöhner and Ralf Peters. 2009.  Assessing the quality of Wikipedia articles with lifecycle based metrics. In _WikiSym ’09: International Symposium on Wikis and Open Collaboration_. Association for Computing Machinery, New York City, United States, 1–10.  https://dl.acm.org/doi/10.1145/1641309.1641333
  * Węcel and Lewoniewski (2015) Krzysztof Węcel and Włodzimierz Lewoniewski. 2015.  Modelling the Quality of Attributes in Wikipedia Infoboxes. In _BIS ’15: International Conference on Business Information Systems_. Springer, Cham, Switzerland, 308–320.  https://link.springer.com/chapter/10.1007/978-3-319-26762-3_27
  * Xiao et al. (2013) Kui Xiao, Bing Li, Peng He, and Xi hui Yang. 2013.  Detection of Article Qualities in the Chinese Wikipedia Based on C4.5 Decision Tree. In _KSEM ’13: International Conference on Knowledge Science_. Springer, Berlin, Heidelberg, 444–452.  https://link.springer.com/chapter/10.1007/978-3-642-39787-5_36
  * Xu et al. (2019) Feiyu Xu, Hans Uszkoreit, Yangzhou Du, Wei Fan, Dongyan Zhao, and Jun Zhu. 2019.  Explainable AI: A brief survey on history, research areas, approaches and challenges. In _Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part II 8_. Springer International Publishing, Cham, 563–574.
  * Xu and Luo (2011) Yanxiang Xu and Tiejian Luo. 2011.  Measuring article quality in Wikipedia: Lexical clue model. In _SWS ’11: Symposium on Web Society_. Institute of Electrical and Electronic Engineers, New York City, United States, 141–146.  https://ieeexplore.ieee.org/document/6101286
  * Yahya et al. (2020) Adnan Yahya, Afnan Ahmad, Alaa Assaf, Rawan Khater, and Ali Salhi. 2020.  Models for Arabic Document Quality Assessment. In _BIS ’20: International Conference on Business Information Systems_. Springer, Cham, Switzerland, 297–310.  https://link.springer.com/chapter/10.1007/978-3-030-61146-0_24
  * Yahya and Salhi (2014) Adnan Yahya and Ali Salhi. 2014.  Quality assessment of Arabic web content: The case of the Arabic Wikipedia. In _IIT ’14: International Conference on Innovations in Information Technology_. Institute of Electrical and Electronic Engineers, New York City, United States, 36–41.  https://ieeexplore.ieee.org/document/6987558
  * Yang et al. (2016) Diyi Yang, Aaron L. Halfaker, Robert Kraut, and Eduard Hovy. 2016.  Who Did What: Editor Role Identification in Wikipedia. In _ICWSM ’16: International AAAI Conference on Web and Social Media_. Association for the Advancement of Artificial Intelligence, Palo Alto, California, 446–455.  https://ojs.aaai.org/index.php/ICWSM/article/view/14732
  * Yapıcı et al. (2019) M. Mutlu Yapıcı, Adem Tekerek, and Nurettin Topaloğlu. 2019.  Literature Review of Deep Learning Research Areas.  _Gazi Mühendislik Bilimleri Dergisi_ 5 (2019), 188 – 215.  Issue 3. https://doi.org/10.30855/gmbd.2019.03.01
  * Yu and Iwaihara (2018) Linfeng Yu and Mizuho Iwaihara. 2018.  Finding high quality documents through link and click graphs. In _IIAI-AAI ’18: International Congress on Advanced Applied Informatics_. Institute of Electrical and Electronic Engineers, New York City, United States, 49–54.  https://ieeexplore.ieee.org/abstract/document/8693372
  * Zeng et al. (2006) Honglei Zeng, Maher A Alhossaini, Li Ding, Richard Fikes, and Deborah L. McGuinness. 2006.  _Computing trust from revision history_.  Technical Report. Stanford Univ Ca Knowledge Systems LAB.  https://apps.dtic.mil/sti/citations/ADA454704
  * Zhang et al. (2015) Ning Zhang, Lingyun Ruan, and Luo Si. 2015.  _Predicting Low-Quality Wikipedia Articles Using User’s Judgements_.  Springer, Cham, Switzerland.  https://link.springer.com/chapter/10.1007/978-3-319-05467-4_6
  * Zhang et al. (2018) Shiyue Zhang, Zheng Hu, Chunhong Zhang, and Ke Yu. 2018.  History-Based Article Quality Assessment on Wikipedia. In _BIGCOMP ’18: International Conference on Big Data and Smart Computing_. Institute of Electrical and Electronic Engineers, New York City, United States, 1–8.  https://ieeexplore.ieee.org/document/8367090
  * Zhu et al. (2020) Rui Zhu, Yiwen Guo, and Jing-Hao Xue. 2020.  Adjusting the imbalance ratio by the dimensionality of imbalanced data.  _Pattern Recognition Letters_ 133 (2020), 217–223.  https://doi.org/10.1016/j.patrec.2020.03.004
  * Ölçer and Temizel (2022) Didem Ölçer and Tuğba Taşkaya Temizel. 2022.  Quality assessment of web-based information on type 2 diabetes.  _Online Information Review_ 46 (2022), 715–732.  Issue 4. https://www.emerald.com/insight/content/doi/10.1108/OIR-02-2021-0089/full/html


11institutetext: Math and CS, Rutgers University Newark, Newark, New Jersey,
USA 11email<EMAIL_ADDRESS>
22institutetext: School of Mathematics, Institute for Advanced Study,
Princeton, New Jersey, USA

# On Connecting Deep Trigonometric Networks with Deep Gaussian Processes:
Covariance, Expressivity, and Neural Tangent Kernel

Chi-Ken Lu 11 Patrick Shafto 1122

###### Abstract

Deep Gaussian Process as a Bayesian learning model is promising because it is
expressive and capable of uncertainty estimation. With Bochner’s theorem, we
can view the deep Gaussian process with squared exponential kernels as a deep
trigonometric network consisting of the random feature layers, sine and cosine
activation units, and random weight layers. Focusing on this particular class
of models allows us to obtain analytical results. We shall show that the
weight space view yields the same effective covariance functions which were
obtained previously in function space. The heavy statistical tails can be
studied with multivariate characteristic function. In addition, the trig
networks are flexible and expressive as one can freely adopt different prior
distributions over the parameters in weight and feature layers. Lastly, the
deep trigonometric network representation of deep Gaussian process allows the
derivation of its neural tangent kernel, which can reveal the mean of
predictive distribution from the intractable inference.

##  1 Introduction

Deep learning is powerful but opaque. Recent advances have shown that such
models are powerful enough to achieve or exceed human level performance in
classification of images [60] and games such as Go [40] and DOTA [55]. As a
parametric representation for function, deep neural networks also see broad
applications in solving differential equation [33, 41]. Recent advances have
also that these models are opaque in ways that prevent application in high-
stakes domains [34], and leave them vulnerable to adversaries. Moreover,
applying models to problem domains remains a challenge because we lack
theoretical motivation or understanding of the consequences of different
choices. An important direction for the field is to develop tools by which we
may obtain a clearer understanding of the power and limits of deep learning.

Two main routes to clarifying our understanding of deep learning are analytic
mathematical formulations of specific models and theoretical connections
between models of deep learning. Gaussian processes (GPs) [35] hold promise
both due to their compact analytic representation and their connection to
shallow neural networks [26] and kernel method [51, 6, 10]. However, these
strengths do not translate directly to deep Gaussian Processes (DGPs) [9] due
to the non-Gaussian nature of composition of GPs [12, 23, 30] and the
intractable marginalization in Bayesian learning. A second compelling
candidate is the Neural Tangent Kernel (NTK) [16], which like GPs relies on
large width limits to develop analytic representations for deep learning
models and learning dynamics. In this work, we build on these foundations to
further develop tools for analytic characterization of, and connections
between, deep learning models.

In this paper, we analytically investigate the deep trigonometriconometric
network, an approximate representation of the infinite dimensional DGPs [32,
7], and make three main contributions. (i) Covariance: we show that the deep
trigonometric networks are equivalent to the DGPs with squared exponential
kernels as the former in the wide limit yield the same covariance as the
latter [23]. In weight space view, the derivation of covariance relies on the
spectrum of a unique class of random matrices. We also study the multivariate
characteristic function for understanding its heavy statistical tails. (ii)
Expressivity: we show the trig networks can approximate an expressive GP with
spectra mixture kernel [52] if the features are samples from mixture of
Gaussians. A deep kernel learning model [53] is connected with a deep trig
network containing a correlated weight layer. In addition, marginal prior
distribution [59] of a shallow trig net can be non-Gaussian if an embedding
phase shift network is incorporated. (iii) NTK: translating DGPs to the deep
trigonometric network representation we derive a closed form NTK for the
corresponding DGPs. Exact DGP inference is impossible due to intractable
marginalization in function space, and the NTK can shed light on the
predictive mean from weight space view. By connecting DGP with the deep
trigonometric networks from covariance perspective, the latter representation
can be more flexible, expressive, and transparent than the former through our
analytical results including the NTK.

The paper has the following organizations. A background for the trigonometric
networks, deep Gaussian processes, and the random feature expansion of kernels
is introduced in Sec. 2. In Sec. 3, covariance of shallow trig networks with
different parameter distributions and its non-Gaussian function distribution
are discussed. The derivation of effective kernels for deep trigonometric
networks with various parameter distributions is given in Sec. 4. The heavy-
tailed property is discussed in Sec. 5. As the connection between deep
trigonometric network and DGP is built, Sec. 6 devotes to the derivation of
neural tangent kernel for the trig network. The context of literature in which
the present work should be placed can be found in Sec. 7, and a conclusion in
Sec. 8 is provided.

##  2 Background

As the basis for theoretical findings in this paper, we outline three prior
theoretical results: marginal prior distribution for deep linear neural
network [59], exact covariance of two-layer DGP with squared exponential
kernel [23], and the random feature expansion of squared exponential kernel
[32].

###  2.1 Random neural networks

Neural networks are a class of parametric models in which one can regard the
function output as the outcome of propagating the input through a
computational graph consisting of multiple layers of linear and nonlinear
mappings. For example, a shallow network can have the following form,

$f({\bf x})={\bf w}\Phi(\Omega{\bf x})\>,$ (1)

where the input ${\bf x}\in\mathbb{R}^{D}$ is sequentially propagated through
the feature layer (producing preactivation from multiplying
$\Omega\in\mathbb{R}^{n\times d}$ with input), activation units (element-wise
nonlinear mapping $\Phi(\cdot)$), and weight layer (multiplying ${\bf
w}\in\mathbb{R}^{1\times n}$). A deep neural network has similar structure.
For instance,

$f({\bf x})={\bf w}_{2}\Phi\big{(}\Omega_{2}{\bf W}_{1}\Phi(\Omega_{1}{\bf
x})\big{)}\>,$ (2)

where the matrices in feature layers have
$\Omega_{1}\in\mathbb{R}^{n_{1}\times D}$ and
$\Omega_{2}\in\mathbb{R}^{n_{2}\times H}$, and in weight layers ${\bf
W}_{1}\in\mathbb{R}^{H\times n_{1}}$ and ${\bf w}_{2}\in\mathbb{R}^{1\times
n_{2}}$. The integer $H$ represents the width of latent layer output in deep
networks.

The inductive bias associated with neural networks is connected to the prior
distributions from which the random parameters in the computational graph are
sampled. How well a model can generalize in Bayesian learning is critically
related to its inductive bias [54]. While it is usually difficult to describe
the inductive bias of neural networks quantitatively, some special cases do
permit analytic investigation. [59] analytically investigated the marginal
distribution over the output of deep linear and ReLu networks. The following
remark is about a particular shallow linear network.

###### Remark 1

Consider the linear network $f({\bf x})={\bf W}{\Omega}{\bf x}$, a special
case of Eq. (1) with $\Phi$ being identity mapping, and the entries in the
random matrices ${\Omega}\in\mathbb{R}^{2\times D}$ and ${\bf
W}\in\mathbb{R}^{1\times 2}$ are independent normal, i.e.
$\Omega_{ij}\sim\mathcal{N}(0,\sigma_{1}^{2})$ and ${\bf
W}_{ij}\sim\mathcal{N}(0,\sigma_{2}^{2})$. Then, the marginal distribution
over the output is a Laplace distribution $p(f({\bf x}))=\exp(-|f({\bf
x})|/\kappa)/2\kappa$ with $\kappa:=\sigma_{1}\sigma_{2}|{\bf x}|$. The heavy-
tailed character is consistent with the findings in [49].

###### Proof

It is easy to observe that the latent output ${\bf h}=\Omega{\bf x}$ has
independent components $h_{i}\sim\mathcal{N}(0,\sigma_{1}^{2}|{\bf x}|^{2})$.
Similarly, conditional on ${\bf h}$, the output has $f|{\bf
h}\sim\mathcal{N}(0,\sigma_{2}^{2}|{\bf h}|^{2})$. To obtain the marginal
distribution $p(f):=\operatorname{\mathbb{E}}_{{\bf W},\Omega}[p(f|{\bf
W},\Omega)]=\operatorname{\mathbb{E}}_{\bf h}[p(f|{\bf h})]$, one can
integrate out ${\bf h}$ during the Fourier transformation and then apply the
inverse transform [59]. Namely, in this particular case with hidden dimension
$n={\rm dim}({\bf h})=2$, we can get,

$p(f)=\int\frac{dq}{2\pi}d{\bf h}\ e^{iqf}\tilde{p}(q|{\bf h})p({\bf
h})=\int\frac{dq}{2\pi}\frac{e^{iqf}}{1+\sigma_{1}^{2}\sigma_{2}^{2}|{\bf
x}|^{2}q^{2}}=\frac{e^{-|f|/\kappa}}{2\kappa}\>.$ (3)

In deriving above, we have used the fact that the Fourier transformation of
$p(f|{\bf h})$ is $\tilde{p}(q|{\bf
h})=\exp(-\frac{1}{2}q^{2}\sigma_{2}^{2}|{\bf h}|^{2})$ and the residue
theorem is applied to complete the last equality.

As the outputs of neural network are not independent given the shared
parameters, another perspective of studying the inductive bias is to
investigate the distribution over the function values, i.e. $p(f({\bf
x}_{1}),f({\bf x}_{2}),\cdots,f({\bf x}_{N}))$, indexed by the set of inputs.
This is a more challenging task than the above marginal distribution over the
function at single input. Fortunately, the central limit theorem applies when
the number of activation units becomes infinity, the multivariate distribution
converges to Gaussian, and the limiting statistics only depends on the mean
$\operatorname{\mathbb{E}}[f({\bf x})]$ and covariance
$\operatorname{\mathbb{E}}[f({\bf x})f({\bf y})]$. Closed form covariance
functions can be derived for shallow networks with sigmoidal and ReLu
activations [51, 6], but the same techniques do not seem to carry to the
deeper networks. As for the deep networks of finite width, various techniques
from statistical physics [13, 56, 36] have been employed to compute the
corrections.

###  2.2 Gaussian process and deep Gaussian process

In parallel, Gaussian Processes [35] (GPs) directly model the set of function
values with a Gaussian, $p(f({\bf x}_{1:N})|\theta)=\mathcal{N}(\mu({\bf
x}_{1:N}),\Sigma({\bf x}_{1:N},{\bf x}_{1:N}))$, with $\theta$ being the
hyper-parameters in the mean function $\mu$ and covariance matrix $\Sigma$,
which fully specify the model. Being Gaussian allows analytic marginalization,
which leads to the defining property of the mean function
$\operatorname{\mathbb{E}}[f({\bf x}_{i})]=\mu({\bf x}_{i})$ and the
covariance function,

$\Sigma_{ij}=\operatorname{\mathbb{E}}\\{[f({\bf x}_{i})-\mu({\bf
x}_{i})][f({\bf x}_{j})-\mu({\bf x}_{j})]\\}=k({\bf x}_{i},{\bf x}_{j})\>,$

where $k$ is a predetermined kernel function, e.g. squared exponential
function. In addition, a closed form for the marginal likelihood $p({\bf
y|X},\theta)=\operatorname{\mathbb{E}}_{f\sim\mathcal{N}(\mu,\Sigma)}[p({\bf
y}|f({\bf X}))]$ can be obtained if a Gaussian likelihood is adopted, with
which the optimal hyper-parameters is determined. Conditional on the prior
observations, the responses ${\bf y}_{*}$ at a set of inputs ${\bf X}_{*}$
then follows another normal distribution $\mathcal{N}({\bf
y}_{*}|\mu_{*},\Sigma_{*})$ with conditional mean,

$\mu_{*}=\Sigma({\bf X}_{*},{\bf X})[\Sigma({\bf X})+\sigma_{s}^{2}I]^{-1}{\bf
y}\>,$ (4)

and conditional covariance,

$\Sigma_{*}=\Sigma({\bf X}_{*})-\Sigma({\bf X}_{*},{\bf X})[\Sigma({\bf
X})+\sigma_{s}^{2}I]^{-1}\Sigma({\bf X},{\bf X}_{*})\>,$ (5)

where we take the prior mean to be zero, $\mu=0$, for easing the notation, and
hyper-parameter $\sigma_{s}^{2}$ denoting the noise variance connecting $f$ to
the observations.

Among many extensions of GPs for enhancing expressivity, e.g. warped GP in
[42], Deep Gaussian Processes (DGPs) [9] are a general hierarchical
composition of GPs. The compositional structure enhances its expressive power,
e.g. a GP with SE kernel can not fit a step function well but a DGP can.
Consider for simplicity a two-layer function $f({\bf x})=f_{2}({\bf
f}_{1}({\bf x}))$ where the input ${\bf x}\in\mathbb{R}^{D}$ is mapped to the
hidden output ${\bf h=f}_{1}({\bf x})\in\mathbb{R}^{H}$ and then to a real
output $f_{2}({\bf h})$. The hidden layer with finite $H$ is referred to as
the bottleneck in [1, 2]. DGP is defined by the joint density $p(f_{2}({\bf
f}_{1}({\bf x}_{1:N}))$,

$\mathcal{N}(f_{2}({\bf h}_{1:N})|0,\Sigma_{2}({\bf
H}))\prod_{i=1}^{H}\mathcal{N}(h_{i}({\bf x}_{1:N})|0,\Sigma_{1}({\bf X}))\>,$

where subscripts in covariance matrices remind us that the covariance
functions in different layers can be different. The hidden output ${\bf H}$ is
a data matrix consisting of vector-valued hidden functions ${\bf h}({\bf
x}_{1:N})$, entering as input to second GP. In Bayesian inference, the
marginalization of the hidden random variables ${\bf h}$ is not tractable,
which leads to various approximation schemes including variational inference
[37, 38, 58, 47, 27] and expectation propagation [5].

One advantage of modeling with the function space view, such as GP, is that we
can augment the model by imposing constraint on the function through inducing
points [45, 44], i.e. the random function has to pass through a set of points,
$f({\bf z}_{1:M})=u_{1:M}$, in the absence of noise. Those points can be
treated as additional hyper-parameters to be optimized (empirical Bayes), or
can be treated as random variable so that one has to infer their distribution
in a full Bayes setting. In the context of DGP, these inducing points can
serve as hidden function’s support in variational inference [37], or they can
be interpreted as the low fidelity observations in multi-fidelity regression
problems [17, 8]. However, it becomes less straightforward to incorporate
these inducing points into the deep neural networks from a random weight space
view [27].

An alternative scheme for inference with DGP models is to view DGP as a GP at
the level of the marginal prior, i.e. the hidden function ${\bf f}_{1}$ being
marginalized out from the joint, which is similar to the partially collapsed
inference in Gibbs sampling [48] and deep kernel learning [50]. The idea was
motivated by the observation that the covariance of the marginal prior
distribution over the array of function values taken at inputs ${\bf X}$,

$p({\bf f}|{\bf X})=\int d{\bf F}_{1}\ p({\bf f}_{2}|{\bf F}_{1})p({\bf
F}_{1}|{\bf X})\>,$ (6)

can be computed analytically [23]. As such, an approximating distribution
$q({\bf f|X})=\mathcal{N}(0,\Sigma_{\rm eff})$ with the matched covariance
$[\Sigma_{\rm eff}]_{ij}=\operatorname{\mathbb{E}}[f_{2}({\bf f}_{1}({\bf
x}_{i}))f_{2}({\bf f}_{1}({\bf x}_{j}))]$ can be plugged into the standard GP
inference pipeline. The compositional hierarchy incorporates all scales from
layers into the effective kernels, e.g. $k_{\rm
eff}=\sigma_{2}^{2}\big{\\{}1+2\frac{\sigma_{1}^{2}}{\ell_{2}^{2}}[1-\exp(-\frac{d^{2}({\bf
x}_{i},{\bf x}_{j})}{2\ell_{1}^{2}})]\big{\\}}^{-\frac{1}{2}}$ for 2-layer DGP
with SE kernels in both zero-mean GPs, and the multi-scale character enables
capturing complex patterns in some time series data [21]. Moreover, the model
augmentation incorporating latent function supports as additional hyper-
parameters was shown to have better generalization [21]. The closed form
kernel for the 2-layer DGP with learnable latent function support is in the
following lemma. The proof can be found in [22].

###### Lemma 1

Consider the two-layer DGP, $f({\bf x})=f_{2}({\bf f}_{1}({\bf x}))$, where
the latent functions, ${\bf f}_{1}:\mathbb{R}^{D}\mapsto\mathbb{R}^{H}$ being
a vector-valued GP and $f_{2}:\mathbb{R}^{H}\mapsto\mathbb{R}$ being a GP with
SE kernel. The latent function is conditioned on the support, ${\bf
f}_{1}({\bf z}_{1:M})={\bf u}_{1:M}$. The covariance has the following closed
form [22],

$\operatorname{\mathbb{E}}_{{\bf f}_{1}}\operatorname{\mathbb{E}}_{f_{2}|{\bf
f}_{1}}[f({\bf x})f({\bf y})]=\prod_{i=1}^{H}\frac{e^{-\frac{[\mu_{*,i}({\bf
x})-\mu_{*,i}({\bf
y})]^{2}}{2(1+\delta_{i}^{2})}}}{\sqrt{1+\delta^{2}_{i}}}\>,$ (7)

where the conditional means $\mu_{*,i}({\bf x})$ and $\mu_{*,i}({\bf y})$ are
associated with the conditional Gaussian density $p(f_{1,i}({\bf
x}),f_{1,i}({\bf y})|{\bf z}_{1:p},u_{i,1:p})$, and the positive value
$\delta^{2}_{i}=\Sigma_{*}({\bf x,x})+\Sigma_{*}({\bf y,y})-2\Sigma_{*}({\bf
x,y})$.

###  2.3 Random feature expansion

To connect neural networks and above GPs with SE kernel, the following theorem
based on the Bochner’s theorem is needed. Its proof was provided in [32].

###### Theorem 2.1

The shallow cosine network [43, 14],

$f({\bf x})=\sqrt{\frac{2}{n}}\sum_{i=1}^{n}w_{i}\cos[\omega_{i}\cdot({\bf
x}-{\bf z}_{i})+b_{i}]\>,$ (8)

is a random parametric function mapping an input ${\bf x}\in\mathbb{R}^{D}$ to
$\mathbb{R}$. The collection of independent and normal weight variables,
$w_{1:n}\sim\mathcal{N}(0,\sigma^{2})$, and bias $b_{1:n}\sim{\rm
Unif}[0,\pi]$. In above expression, ${\bf z}_{1:n}\in\mathbb{R}^{D}$ are a set
of shift vectors, and are referred to as inducing points in GP literature
[14]. The random network has zero mean, and the covariance converges to,

$\operatorname{\mathbb{E}}[f({\bf x})f({\bf
y})]\rightarrow\sigma^{2}\exp[-\frac{1}{2}({\bf x-y})^{T}\Lambda({\bf
x-y})]\>,$ (9)

in the limit $n\rightarrow\infty$ if the random vectors
$\\{\omega_{1:n}\in\mathbb{R}^{D}\\}$ are samples from a multivariate normal
distribution $\mathcal{N}(0,\Lambda)$.

##  3 Shallow trigonometric network

An alternative for the shallow networks in Eq. (8) which yields the same SE
covariance was proposed in [7]. With the feature vectors
$\omega_{1:n}\in\mathbb{R}^{D}$, and the random variables $w^{c}_{1:M}$ and
$w^{s}_{1:M}$ associated with the cosine and sine postactivation,
respectively, we can write the random function as,

$\displaystyle f({\bf x})$
$\displaystyle=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}w^{c}_{i}\cos(\omega_{i}\cdot{\bf
x})+w^{s}_{i}\sin(\omega_{i}\cdot{\bf x})\>,$ (10) $\displaystyle={\bf
w}\Phi(\Omega{\bf x})\>,$ (11)

in which the compact notation in the second line has ${\bf
w}=[w^{c}_{1},w^{c}_{2},\cdots,w^{c}_{n},w^{s}_{1},\cdots,w^{s}_{n}]\in\mathbb{R}^{1\times
2n}$ and $\Omega=[\omega_{1},\cdots,\omega_{n}]^{T}\in\mathbb{R}^{n\times D}$.
Activation here is a doublet which reads
$\Phi(~{})=\left(\begin{smallmatrix}\cos(~{})\\\
\sin(~{})\end{smallmatrix}\right)$.

Based on the same argument in [32], Eq. (10) represents a finite-basis model
for random smooth functions whose covariance converges to some fixed form in
the limit of large $n$. If the features in $\Omega$ are sampled from a
distribution and remain fixed, then one can infer the weight parameters ${\bf
w}$ given the data (or hyperdata in [21]) ${\bf Z,u}$, the prior distribution
$p({\bf w})=\mathcal{N}(0,\sigma^{2}I_{2n})$, and observation noise variance
$\sigma_{s}^{2}$. The notation means ${\bf Z}=\left(\begin{smallmatrix}{\bf
z}_{1},&\cdots,&{\bf z}_{M}\end{smallmatrix}\right)\in\mathbb{R}^{D\times M}$
and ${\bf
u}=\left(\begin{smallmatrix}u_{1},&\cdots,&u_{M}\end{smallmatrix}\right)^{T}\in\mathbb{R}^{M\times
1}$. Following the linear Bayesian learning [35], the posterior reads

$p({\bf w}|{\bf Z,u})=\mathcal{N}({\bf w}|{\bar{\bf w}},A^{-1})\>,$ (12)

with the conditional mean and precision matrix,

$\displaystyle{\bar{\bf w}}^{T}$
$\displaystyle=\sigma_{s}^{-2}A^{-1}\Phi(\Omega{\bf Z}){\bf u}\>,$ (13)
$\displaystyle A$ $\displaystyle=\sigma_{s}^{-2}\Phi(\Omega{\bf
Z})\Phi^{T}(\Omega{\bf Z})+\sigma^{-2}I_{2n}\>,$ (14)

where the postactivation matrix $\Phi(\Omega{\bf Z})$ has shape $(2n,M)$.
Furthermore, the distribution over the predicted value at a new input,
$y_{*}={\bf w}\Phi(\Omega{\bf x}_{*})$, is still a Gaussian with mean,

$\bar{f}_{*}=K_{*}\big{(}\sigma_{s}^{2}I_{2n}+K\big{)}^{-1}{\bf u}\>,$

and variance

$\sigma^{2}_{*}=\sigma_{s}^{2}+K_{**}-K_{*}\big{(}\sigma_{s}^{2}I_{2n}+K\big{)}^{-1}K_{*}^{T}\>,$

where we have used the kernel expression $K_{*}=\sigma^{2}\Phi_{*}^{T}\Phi$,
$K_{**}=\sigma^{2}\Phi_{*}^{T}\Phi_{*}$ and $K=\sigma^{2}\Phi^{T}\Phi$ [35].
The shorthand notation has $\Phi_{*}=\Phi(\Omega{\bf x}_{*})$ and
$\Phi=\Phi(\Omega{\bf Z})$. The above result is thus an approximation for GP
regression.

In the framework of GP regression, one way to enhance the expressive power of
the nonparametric model is, ironically, to form a linear combination of
different kernels and treat the coefficients as hyper-parameters optimizing
the evidence. The classic regression on Mauna Loa dataset in [35] adopts the
SE kernel along with rational quadratic and periodic kernels. One may also
view the spectra mixture kernel [52] as a special kernel composition. For
Bayesian neural network, on the other hand, the prior function distribution
induced by prior parameter distribution [54, 59] encodes the expressive power.
In practice, design of activation in a network was shown to yield good
extrapolation [29]. In the following two subsections, we shall introduce two
ideas improving the expressivity associated with the trig network
representation of GP.

###  3.1 Features from mixture of Gaussians and spectra mixture kernel

Following the work of [32], one can obtain a shallow trig network
representation of GP regression model with SE kernel if the features $\Omega$
are sampled from a normal distribution. Similarly, the GP regression models
with Laplacian and Cauchy kernels can have their network representation if the
features are sampled from different single-mode distributions. The following
lemma show that the model with spectra mixture kernel is equivalent to the
shallow trig network if the features are sampled from a mixture of Gaussians.

###### Lemma 2

If the features are sampled from a mixture of multivariate Gaussians,
$\omega_{1:n}\sim\sum_{a}\pi_{a}\mathcal{N}(\mu_{a},\Lambda_{a})$ with
positive $\pi$’s, and the weight ${\bf w}\sim\mathcal{N}(0,\sigma^{2}I_{2n})$,
then the covariance of outputs in Eq. (10) converges to the spectrum mixture
kernel,

$k({\bf x},{\bf y})=\sigma^{2}\sum_{a}\pi_{a}\cos[\mu_{a}^{T}({\bf
x-y})]e^{-\frac{({\bf x-y})^{T}\Lambda_{a}({\bf x-y})}{2}}\>,$ (15)

in the wide network limit $n\rightarrow\infty$.

###### Proof

As the weight parameters are independent, one can easily see that the
covariance in the large $n$ limit converges to

$\displaystyle\operatorname{\mathbb{E}}[f({\bf x})f({\bf y})]$
$\displaystyle\rightarrow\sigma^{2}\int d\omega\
p(\omega)\cos[\omega\cdot({\bf x-y})]$ $\displaystyle=\sigma^{2}{\rm
Re}\sum_{a}\pi_{a}\int d\omega\
\mathcal{N}(\omega|\mu_{a},\Lambda_{a})e^{i\omega\cdot({\bf x-y})}$
$\displaystyle=\sigma^{2}\sum_{a}\pi_{a}\cos[\mu_{a}\cdot({\bf
x-y})]\exp[-\frac{1}{2}({\bf x-y})^{T}\Lambda_{a}({\bf x-y})]\>.$

In the first equality, ${\rm Re}$ refers to as the operation of taking real
part.

###  3.2 Prior distribution over the network output

Here, we investigate the marginal prior function distribution $p(f)=\int d{\bf
w}\ p(f|{\bf w})p({\bf w})$ induced by the prior weight distribution $p({\bf
w})$. Following the technique in Remark 1, we can conclude that the prior
function distribution associated with the shallow trig network in Eq. (10) is
Gaussian, independent of the feature number $n$.

###### Remark 2

The probability density over the function Eq. (10) for a single input is
always a Gaussian with zero mean and variance $\sigma^{2}$, independent of the
width $n$ and of the sampling distribution $p(\Omega)$.

###### Proof

Given ${\bf w}$ is independent normal with variance $\sigma^{2}$, the
conditional distribution $p(f|\Phi)$ is also a normal with variance
$\frac{\sigma^{2}}{n}\sum_{i=1}^{n}\cos^{2}\omega_{1}\cdot{\bf
x}+\cdots\cos^{2}\omega_{n}\cdot{\bf x}+\sin^{2}\omega_{1}\cdot{\bf
x}+\cdots+\sin^{2}\omega_{n}\cdot{\bf x}=\sigma^{2}$. Thus, $p(f({\bf
x}))=\mathcal{N}(0,\sigma^{2})$.

It was suggested that the superior expressive power of deep linear network and
ReLu network is related to their non-Gaussian prior function distribution [49,
36, 59]. Besides the network with finite width which lifts the Gaussianity
[56], the following shallow network $f_{\psi}:\mathbb{R}^{D}\mapsto\mathbb{R}$
with modified preactivation,

$f_{\psi}({\bf
x})=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}w^{c}_{i}\cos[\omega_{i}\cdot{\bf
x}+\psi({\bf x})]+w^{s}_{i}\sin[\omega_{i}\cdot{\bf x}-\psi({\bf x})]\>,$ (16)

incorporating a phase shift network $\psi({\bf x})$ is proposed to lift the
Gaussianity.

###### Lemma 3

The Fourier transformed $\tilde{p}(q)$ associated with the prior distribution
over the output in Eq. (16) is,

$\tilde{p}(q)=e^{-\frac{1}{2}q^{2}\sigma^{2}}\int d\omega
p(\omega)e^{\frac{1}{2}q^{2}\sigma^{2}\sin\psi({\bf x})\sin 2\omega\cdot{\bf
x}}\>,$ (17)

where the feature $\omega\in\mathbb{R}^{D}$ are sampled from the high
dimensional normal distribution
$p(\omega)=\prod_{d=1}^{D}\mathcal{N}(\omega_{d}|0,\sigma_{d}^{2})$.

It can be seen that the phase shift network $\psi({\bf x})$ lifts the Gaussian
character of the prior distribution, but the intractable high-dimensional
integral in Eq. (17) stands in the way of obtaining a closed form for its
characteristic function. Nevertheless, one can proceed with the approximation
of Gauss-Hermite quardature [15]. Consider the case where the variances
$\sigma_{1:D}^{2}=\sigma_{F}^{2}$ associated with the features in all
dimensions are the same, and after including the most relevant terms,

$\tilde{p}(q)\approx
e^{-\frac{1}{2}q^{2}\sigma^{2}}(\frac{\lambda_{0}}{\sqrt{\pi}})^{D}\big{\\{}1+2\frac{\lambda_{1}}{\lambda_{0}}\sum_{d=1}^{D}\cosh[{\frac{1}{2}q^{2}\sigma^{2}\sin\psi({\bf
x})\sin(2\sqrt{2}\sigma_{F}z_{1}x_{d})}]\big{\\}}\>,$ (18)

where the coefficients $\lambda_{0}\approx 1.18$ and $\lambda_{1}\approx 0.30$
are given in [15] and $z_{1}\approx 1.22$ is the nonzero root of the third
order Hermite polynomial. Consequently, the characteristic $\tilde{p}$ obtains
a non-Gaussian correction $\propto q^{4}e^{-q^{2}\sigma^{2}}$ for small
Fourier component $q$.

##  4 Deep trigonometric network

Now we proceed to consider the deep trigonometric network proposed in [7].
With the same notation as the shallow network, the deep trigonometric network
of interest has the following matrix representation,

$f({\bf x})={\bf w}_{2}\Phi(\Omega_{2}{\bf W}_{1}\Phi(\Omega_{1}{\bf x}))\>,$
(19)

in which the random weight matrices ${\bf w}_{2}\in\mathbb{R}^{1\times
2n_{2}}$, ${\bf W}_{1}\in\mathbb{R}^{H\times 2n_{1}}$ and the feature matrices
$\Omega_{2}\in\mathbb{R}^{n_{2}\times H}$,
$\Omega_{1}\in\mathbb{R}^{n_{1}\times D}$. Here, the hidden output ${\bf
h}={\bf W}_{1}\Phi(\Omega_{1}{\bf x})$ has bottleneck [1] dimension $H$
collecting the $n_{1}$ postactivations. Besides the compositional hierarchy
which makes the function more expressive than its shallow counterpart, one can
also adopt different prior distribution over the weight and feature matrices.
In the following three subsections, we shall discuss the cases of (i) the
entries in ${\bf W}_{1}$, ${\bf w}_{2}$, $\Omega_{1}$, and $\Omega_{2}$ are
all independent normal, which corresponds to the zero-mean two-layer DGP with
SE kernels [23], (ii) same as in (i) but in the first layer the weight entries
in ${\bf W}_{1}\sim p({\bf w}_{1}|{\bf Z,U})$ are not independent, which
corresponds to the two-layer DGP for multi-fidelity regression [22] and hyper-
data learning [21] with ${\bf Z,U}$ acting as the support in the latent
function, and (iii) same as in (ii) but the feature matrix $\Omega_{2}$
consists of samples from the mixture of Gaussians, which corresponds to the
two-layer DGP with outer GP using the SM kernel.

###  4.1 Deep trig net covariance and random matrix spectrum

To show that the deep trigonometric network yields the same covariance as the
two-layer DGP when the entries in weight matrices have independent normal
distribution, the spectrum of the following square random matrix with
dimension $2n_{1}$,

$G=[\Phi(\Omega_{1}{\bf x})-\Phi(\Omega_{1}{\bf y})][\Phi(\Omega_{1}{\bf
x})-\Phi(\Omega_{1}{\bf y})]^{T}\>,$ (20)

is critical in determining the statistics of network outputs.

###### Remark 3

The square matrix $G$ has $(2n_{1}-1)$ zero eigenvalues and one nonzero
eigenvalue. If the set of feature vectors $\\{\omega_{1:n_{1}}\\}$ are sampled
from $\mathcal{N}(\omega|0,I_{D})$, then the nonzero eigenvalue shall converge
to the following,

$\displaystyle\big{|}\Phi({\bf x})-\Phi({\bf y})\big{|}^{2}$
$\displaystyle=\frac{1}{n_{1}}\sum_{i=1}^{n_{1}}(\cos\omega_{i}{\bf
x}-\cos\omega_{i}{\bf y})^{2}+(\sin\omega_{i}{\bf x}-\sin\omega_{i}{\bf
y})^{2}$ $\displaystyle\rightarrow 2-2k_{\rm SE}({\bf x,y})\>,$

in the limit $n_{1}\rightarrow\infty$. $k_{\rm SE}({\bf
x,y})=\exp[-\frac{1}{2}|{\bf x-y}|^{2}]$ stands for the squared exponential
covariance function with all hyper-parameters set to unity.

###### Proof

First, one can view ${\bf v}=\Phi(\Omega_{1}{\bf x})-\Phi(\Omega_{1}{\bf y})$
as a fixed vector in the $2n_{1}$ dimensional space. The entries read
$\frac{1}{\sqrt{n_{1}}}[\cos\omega_{1}\cdot{\bf x}-\cos\omega_{1}\cdot{\bf
y},\cdots,\cos\omega_{n_{1}}\cdot{\bf x}-\cos\omega_{n_{1}}\cdot{\bf
y},\sin\omega_{1}\cdot{\bf x}-\sin\omega_{1}\cdot{\bf y},\cdots]$. Then, one
can in principle find out the orthogonal subspace, spaned by the set of
vectors $\\{{\bf v}_{\perp,1:(2n_{1}-1)}\\}$, to ${\bf v}$ in the space. Thus,
we have ${\bf v}^{T}{\bf v}_{\perp}=0$, one can write the zero eigenvalue
equations,

$G{\bf v}_{\perp}={\bf v}{\bf v}^{T}{\bf v}_{\perp}=0\>,$

and the only nonzero eigenvalue one,

$G{\bf v}=|{\bf v}|^{2}{\bf v}\>.$

With the knowledge of the spectrum of random matrix $G$, now we can go on to
derive the desired covariance of deep trigonometric network.

###### Lemma 4

The covariance of the deep trigonometric network in Eq. (19),

$\operatorname{\mathbb{E}}_{{\bf
W}_{1}}\bigg{\\{}\operatorname{\mathbb{E}}_{{\bf w}_{2}|{\bf
W}_{1}}\big{[}f({\bf x})f({\bf y})\big{]}\bigg{\\}}\rightarrow k_{\rm
DGP}({\bf x,y})=\big{\\{}1+2[1-k_{\rm SE}({\bf
x,y})]\big{\\}}^{-\frac{H}{2}}\>,$ (21)

as the numbers of features $n_{1}$ and $n_{2}$ both approach infinity.

###### Proof

The independence among the zero-mean random weights ${\bf w}_{2}$ and uniform
variance leads to $\operatorname{\mathbb{E}}_{{\bf w}_{2}}[({\bf
w}_{2}\Phi(\Omega_{2}{\bf h}_{x}))({\bf w}_{2}\Phi(\Omega_{2}{\bf
h}_{y}))]=\Phi(\Omega_{2}{\bf h}_{x})^{T}\Phi(\Omega_{2}{\bf h}_{y})$, which
at the limit $n_{2}\rightarrow\infty$ results in,

$\operatorname{\mathbb{E}}[f({\bf x})f({\bf
y})]\rightarrow\operatorname{\mathbb{E}}_{{\bf
W}_{1}}\big{[}e^{-\frac{d^{2}({\bf x,y})}{2}}\big{]}\>,$

where the squared distance between the latent outputs ${\bf h}({\bf x})$ and
${\bf h}({\bf y})$ in the exponent can be rewritten as,

$\displaystyle d^{2}({\bf x,y})$ $\displaystyle=[{\bf h(x)-h(y)}]^{T}[{\bf
h(x)-h(y)}]$ $\displaystyle={\rm Tr}\big{\\{}{\bf W}_{1}[\Phi(\Omega_{1}{\bf
x})-\Phi(\Omega_{1}{\bf y})][\Phi(\Omega_{1}{\bf x})-\Phi(\Omega_{1}{\bf
y})]^{T}{\bf W}_{1}^{T}\big{\\}}$ $\displaystyle=\sum_{i=1}^{H}{\bf
w}_{1,i}G{\bf w}_{1,i}^{t}\>,$

where the rows of ${\bf W}_{1}$ are written as $\\{{\bf w}_{1,1:H}\\}$ in the
last line. Lastly, the determinant of $(I_{2n_{1}}+G)$ enters as a result of

$\operatorname{\mathbb{E}}_{{\bf
w}_{1,1:H}\sim\mathcal{N}(0,I_{2n_{1}})}[e^{-d^{2}({\bf
x,y})}]=\Pi_{i=1}^{H}\frac{1}{\sqrt{{\rm det}[I_{2n_{1}}+G]}}\>.$ (22)

###  4.2 Deep trig net with weights representing latent function support

In above subsection, the deep trigonometric net with centered and independent
Gaussian weights ${\bf W}_{1}$ and ${\bf w}_{2}$ is equivalent to composition
of two zero-mean GPs. In [21], it was shown that treating the support in
latent function, i.e. $M$ hyper-data points with ${\bf h}({\bf z}_{1:M})={\bf
u}_{1:M}$, as additional hyper-parameters can enhance generalization of DGPs.
${\bf z}\in\mathbb{R}^{D}$ and ${\bf u}\in\mathbb{R}^{H}$. Here, the function
space view translates to the weight parameters, ${\bf W}_{1}|{\bf
Z,U}\sim\prod_{i=1}^{H}\mathcal{N}({\bf w}_{1,i}|{\bar{\bf w}}_{i},A^{-1})$,
conditional on the hyper-input and output matrices, ${\bf
Z}:=\left(\begin{smallmatrix}{\bf z}_{1},&\cdots,&{\bf
z}_{M}\end{smallmatrix}\right)\in\mathbb{R}^{D\times M}$ and ${\bf
U}=\left(\begin{smallmatrix}{\bf u}_{1},&\cdots,&{\bf
u}_{M}\end{smallmatrix}\right)\in\mathbb{R}^{H\times M}$, respectively. The
conditional precision matrix,

$A=\big{[}I_{2n_{1}}+\Phi(\Omega_{1}{\bf Z})\Phi^{T}(\Omega_{1}{\bf
Z})\big{]}$ (23)

and the conditional mean for each output dimension,

${\bar{\bf w}}_{1,i}=A^{-1}\Phi(\Omega_{1}{\bf Z}){\bf U}_{i,:}^{T}\>,$ (24)

which can be found in Ch.2.1.2 in [35] [also in [27]].

###### Lemma 5

If the latent layer weights ${\bf W}_{1}$ in Eq. (19) have the correlated
prior distribution ${\bf W}_{1}\sim\prod_{i=1}^{H}p({\bf w}_{1,i}|{\bf
Z,U}_{i,:})$, then the covariance converges to the DGP covariance in Eq. (7).

###### Proof

The proof follows the previous one except that we are evaluating the following
expectation,

$\displaystyle\operatorname{\mathbb{E}}[f({\bf x})f({\bf y})]$
$\displaystyle=\operatorname{\mathbb{E}}_{{\bf
w}_{1,1:H}\sim\mathcal{N}({\bar{\bf w}}_{1:H},A^{-1})}\bigg{[}e^{-\frac{{\bf
w}_{1,1}G{\bf w}_{1,1}^{T}}{2}}e^{-\frac{{\bf w}_{1,2}G{\bf
w}_{1,2}^{T}}{2}}\cdots e^{-\frac{{\bf w}_{1,H}G{\bf
w}_{1,H}^{T}}{2}}\bigg{]}$
$\displaystyle=\prod_{i=1}^{H}\frac{e^{-\frac{1}{2}{\bf\bar{w}}_{i}(I+GA^{-1})^{-1}G{\bf\bar{w}}_{i}^{T}}}{\sqrt{|I+A^{-1}G|}}\>,$

in which we just focus on one term in the product. By writing the matrix
$G={\bf vv}^{T}$ related to the inputs ${\bf x,y}$ (see Remark 7) and using
the matrix inversion lemma, the exponent in above expression becomes
$-\frac{1}{2}{\bf\bar{w}}_{i}{\bf v}(1+{\bf v}^{T}A^{-1}{\bf v})^{-1}{\bf
v}^{T}{\bf\bar{w}}_{i}^{T}$. As for the determinant in denominator, the matrix
$A^{-1}G$ does not couple the vector ${\bf v}$ with its orthogonal subspace
${\bf v}_{\perp}$, leading to $|I+A^{-1}G|=1+({\bf v}^{T}A^{-1}G{\bf v})/({\bf
v}^{T}{\bf v})$. With some manipulation and lengthy calculation,

$\displaystyle\big{|}I+A^{-1}G\big{|}$
$\displaystyle=1+[\Phi_{x}-\Phi_{y}]^{T}[\Phi_{x}-\Phi_{y}]-[\Phi_{x}-\Phi_{y}]^{T}\Phi_{Z}[I+\Phi^{T}_{Z}\Phi_{Z}]^{-1}\Phi^{T}_{Z}[\Phi_{x}-\Phi_{y}]$
$\displaystyle\rightarrow
1+k_{xx}+k_{yy}-2k_{xy}-k_{xZ}K_{ZZ}^{-1}k_{Zx}-k_{yZ}K_{ZZ}^{-1}k_{Zy}+2k_{xZ}K_{ZZ}^{-1}k_{Zy}\>.$

It can also be seen that the above result is identical to $(1+{\bf
v}^{T}A^{-1}{\bf v})$. Similarly, one can show the scalar ${\bar{\bf
w}}_{i}{\bf vv}^{T}{\bar{\bf w}}_{i}^{T}=(m_{x}-m_{y})^{2}$ with the limiting
form $m_{x}\rightarrow k_{xZ}K^{-1}_{ZZ}{\bf U}_{i,:}$.

###  4.3 Deep trig net with mixed spectrum features

The deep trigonometric networks are expressive as the choices over the
weights’ prior distribution are flexible. One may also consider employing
different distributions over the features as we do in the shallow nets. Here,
we are interested in the resultant covariance when the outer features
$\Omega_{2}$ consist of samples from mixture of Gaussians at different centers
and the inner weights ${\bf W}_{1}$ representing the latent function support.

###### Lemma 6

When the deep trigonometric network in Eq. (19) has fixed features
$\omega_{2}\in\mathbb{R}$ from samples of a mixed distribution
$\sum_{a}\pi_{a}\mathcal{N}(\mu_{a},\sigma_{a}^{2})$ and the random variables
${\bf w}_{1}$ represent the weight space view of latent function support ${\bf
w}_{1}\Phi(\Omega_{1}{\bf z}_{1:M})=u_{1:M}$, it is equivalent to the DGP
$f({\bf x})=f_{2}(f_{1}({\bf x}))$ with $f_{1}|{\bf
Z,u}\sim\mathcal{GP}(m,\Sigma)$ and $f_{2}|f_{1}\sim\mathcal{GP}(0,k_{\rm
SM})$. $m$ and $\Sigma$ are the conditional mean and covariance matrix given
the hyper-data ${\bf Z,u}$. The covariance is,

$\operatorname{\mathbb{E}}[f({\bf x})f({\bf
y})]=\sum_{a}\frac{\pi_{a}}{(1+\sigma_{a}^{2}\delta^{2})^{1/2}}\exp\big{[}-\frac{\sigma_{a}^{2}(m_{x}-m_{y})^{2}+\delta^{2}\mu_{a}^{2}}{2(1+\sigma_{a}^{2}\delta^{2})}\big{]}\cos\big{[}\frac{\mu_{a}(m_{x}-m_{y})}{1+\sigma_{a}^{2}\delta^{2}}\big{]}\>.$
(25)

###### Proof

It is easier to work out the covariance in the function space. Observing that

$\operatorname{\mathbb{E}}_{f_{1}|{\bf
Z,u}}\big{\\{}\operatorname{\mathbb{E}}_{f_{2}|f_{1}}[f_{2}(f_{1}({\bf
x}))f_{2}(f_{1}({\bf y}))]\big{\\}}={\rm
Re}\operatorname{\mathbb{E}}_{f_{1}|{\bf
Z,u}}[\operatorname{\mathbb{E}}_{\omega_{2}}e^{i\omega_{2}[f_{1}({\bf
x})-f_{1}({\bf y})]}]\>,$

one can compute the expectation with respect to the latent function $f_{1}$
first, followed by that of feature $\omega_{2}$. Thus, we get the covariance,

$\operatorname{\mathbb{E}}_{\omega_{2}\sim\sum_{a}\pi_{a}\mathcal{N}(\mu_{a},\sigma^{2}_{a})}\big{\\{}\operatorname{\mathbb{E}}_{(f_{1}({\bf
x}),f_{1}({\bf y}))^{T}\sim\mathcal{N}(m,\Sigma)}[\cos\omega_{2}(f_{1}({\bf
x})-f_{1}({\bf y}))]\big{\\}}\>,$

which can be computed analytically.

Such deep trigonometric net is closely related to the deep kernel learning
with the SM kernel [53]. Now it becomes clear that the outer network
represents the random function $f_{2}\sim\mathcal{GP}(0,k_{\rm SM})$. The
hyper-data ${\bf Z,u}$ constrain the inner function $f_{1}$, and in the limit
when the hyper-data are dense the function ${\bf f}_{1}$ becomes deterministic
[21]. Such situation is equivalent to passing the inputs to a parametric
function and then to a GP. However, the probabilistic nature in $f_{1}$ in the
sparse hyper-data limit is helpful for preventing overfitting in deep kernel
learning with over-parameterized $f_{1}$ [28].

##  5 Characteristic function of deep trigonometric network

[30] demonstrated that zero-mean DGPs, $f_{L}\circ{\bf
f}_{L-1}\circ\cdots\circ{\bf f}_{1}$, with any depth $L$ have heavy-tailed
statistics. Their argument is based on the fact that the characteristic
function for DGP has a lower bound which corresponds to the characteristic
function of a GP with the limiting kernel when the width in each layer
approaches infinity. It is not clear whether we can directly apply their
argument to the deep trigonometric network from the weight space view.
Focusing on the network defined in Eq. (19) with iid ${\bf w}_{2}$ and ${\bf
W}_{1}$, the characteristic function for a set of $N$ inputs can be computed
as the followings,

$\displaystyle\operatorname{\mathbb{E}}[\exp(i{\bf t}\cdot{\bf f})]$
$\displaystyle=\operatorname{\mathbb{E}}\\{\exp[it_{1}f({\bf
x}_{1})+\cdots+it_{N}f({\bf x}_{N})]\\}$
$\displaystyle=\operatorname{\mathbb{E}}\bigg{\\{}i{\bf
w}_{2}\big{[}t_{1}\Phi(\Omega_{2}{\bf h}({\bf x}_{1}))+\cdots
t_{N}\Phi(\Omega_{2}{\bf h}({\bf x}_{N}))\big{]}\bigg{\\}}$
$\displaystyle=\operatorname{\mathbb{E}}[\exp(-\frac{1}{2}{\bf t}K{\bf t})]$
$\displaystyle\geq\exp\bigg{[}-\frac{1}{2}{\bf
t}^{T}\operatorname{\mathbb{E}}_{{\bf W}_{1}}(K){\bf
t}\bigg{]}=\exp\big{(}-\frac{1}{2}{\bf t}^{T}K_{\rm DGP}{\bf t}\big{)}$

where the $N\times N$ matrix $K$ has scalar entries
$K_{ij}=\Phi^{T}(\Omega_{2}{\bf h}({\bf x}_{i}))\Phi(\Omega_{2}{\bf h}({\bf
x}_{j}))$. The inequality following the convexity of exponential function is
important to the heavy-tailed statistics as, by using Theorem 2 in [30], we
can conclude that

$\lim_{n_{1},n_{2}\rightarrow\infty}p(f({\bf x}_{1})=f({\bf
x}_{2})=\cdots=f({\bf x}_{N})=0)>\mathcal{N}(0_{N}|0_{N},K_{\rm DGP})\>.$ (26)

As for the shallow trig net in Eq. (10), given $\Omega$ fixed, the
characteristic function is exact Gaussian.

##  6 Neural tangent kernel for trigonometric networks

For probabilistic regression problems with data matrix ${\bf X}$ and
observations ${\bf y}$, one has two choices over the models for prediction.
The first choice is function-based models, such as GPs and DGPs. The exact GP
inference produces a predictive distribution $p(y_{*}|{\bf X,y,x}_{*})$ with
closed form mean and variance that only depends on the covariance function and
hyper-parameters. However, such luxury is not carried over to DGP as there is
no corresponding exact inference. The second choice is weight-based models:
the shallow Bayesian neural network, Eq. (10), and its deep version, Eq. (19).
For shallow trig network with fixed feature matrix $\Omega$, then it becomes a
Bayesian linear regression problem (see Sec. 3), and the predictive mean and
variance converge to the GP’s result as the number of features
$n\rightarrow\infty$.

It is not clear whether the appealing correspondence between shallow trig
network and GP with SE kernel can carry to the deep trigonometric network and
2-layer DGP discussed in this paper. Nevertheless, the perspective from neural
tangent kernel [16, 3] may shed some light on this issue. For gradient based
learning of infinite and deep neural network $f({\bf x}|\theta)$, the network
function shall eventually converge to the predictive mean of GP with the
following kernel,

$k({\bf x,y})=\operatorname{\mathbb{E}}_{\theta}\big{[}\frac{\partial f({\bf
x}|\theta)}{\partial\theta}\cdot\frac{\partial f({\bf
y}|\theta)}{\partial\theta}\big{]}\>,$ (27)

where the derivative operation $\partial/\partial\theta$ with respect to all
weight parameters in $\theta$ generates a vector. Moreover, the neural tangent
kernel remains a constant during the gradient descent, so its value is
determined by the initial distribution over $\theta$ (a recent study [39]
suggested that the neural tangent kernels for deeper model may still evolve
during training).

Now, given the fact that the deep trigonometric network yields the same
covariance as the two-layer DGP with SE kernels, it is interesting to derive
the neural tangent kernel associated with Eq. (19), which may reveal some
insights into the correspondence between deep trigonometric network and DGP.

###### Lemma 7

Assume that the features $\Omega_{1,2}$ in the deep trigonometric network in
Eq. (19) are fixed and the weights ${\bf w}_{1,2}$ are learned through
gradient descent with squared loss function. Then the associated neural
tangent kernel reads,

$k_{\rm NTK}({\bf x,y})=k_{\rm DGP}+k_{\rm SE}k_{\rm DGP}^{3}\>,$ (28)

where $k_{\rm SE}$ is the SE covariance function and $k_{\rm DGP}$ is the
exact covariance of the two-layer DGP. Note that we have set all hyper-
parameters to unit for ease of notation.

###### Proof

As only the weight parameters are learned, the neural tangent kernel has the
following expression,

$\displaystyle k_{NTK}({\bf x,y})$
$\displaystyle=\operatorname{\mathbb{E}}_{{\bf
W}_{1}}\bigg{\\{}\operatorname{\mathbb{E}}_{{\bf W}_{2}|{\bf
W}_{1}}\big{[}\frac{\partial f({\bf x})}{\partial{\bf W}_{2}}\frac{\partial
f({\bf y})}{\partial{\bf W}_{2}}+\frac{\partial f({\bf x})}{\partial{\bf
W}_{1}}\frac{\partial f({\bf y})}{\partial{\bf W}_{1}}\big{]}\bigg{\\}}$
$\displaystyle=K_{\rm DGP}+K_{e}\>,$

where we observe that the first term (derivative wrt second weight ${\bf
w}_{2}$) is the same as the covariance of DGP (see Sec. 4.1). Next, we shall
focus on the second term, $K_{e}$. Notice that the order of differentiation
$\partial f/{\bf w}_{1}$ and the expectation $\operatorname{\mathbb{E}}_{{\bf
w}_{2}|{\bf w}_{1}}$ can be switched. To facilitate the computation, we can
temporarily write $f({\bf x})={\bf w}_{2}\Phi(\Omega_{2}{\bf
w}_{a}\Phi(\Omega_{1}{\bf x}))$ and $f({\bf y})={\bf w}_{2}\Phi(\Omega_{2}{\bf
w}_{b}\Phi(\Omega_{1}{\bf y}))$ so that we can first compute the expectation
and then take the derivatives. The rest of derivations just rest on some
simple tricks,

$\displaystyle K_{e}$ $\displaystyle=\operatorname{\mathbb{E}}_{{\bf
w}_{1}}\bigg{\\{}\sum_{i=1}^{n_{1}}\frac{\partial^{2}}{\partial
w_{a,i}\partial w_{b,i}}\operatorname{\mathbb{E}}_{{\bf w}_{2}|{\bf
w}_{a,b}}[f_{a}({\bf x})f_{b}({\bf y})]\big{|}_{{\bf w}_{a}={\bf w}_{b}={\bf
w}_{1}}\bigg{\\}}$ $\displaystyle=\operatorname{\mathbb{E}}_{{\bf
w}_{1}}\bigg{[}e^{-\frac{{\bf w}_{1}G{\bf
w}_{1}^{T}}{2}}\Phi^{T}(\Omega_{1}{\bf x})\Phi(\Omega_{1}{\bf y})(1-{\bf
w}G{\bf w}_{1}^{T})\bigg{]}$ $\displaystyle=e^{-\frac{|{\bf
x-y}|^{2}}{2}}(1+2\frac{\partial}{\partial\lambda})\operatorname{\mathbb{E}}_{{\bf
w}_{1}}\big{[}e^{-\lambda\frac{{\bf w}_{1}G{\bf
w}_{1}^{T}}{2}}\big{]}\big{|}_{\lambda=1}$ $\displaystyle=k_{{\rm SE}}({\bf
x,y})\big{[}1+2(1-k_{\rm SE}({\bf x,y}))\big{]}^{-\frac{3}{2}}\>.$

To arrive at the second equality, we have used

$\sum_{i}\partial^{2}_{w_{a,i}w_{b,i}}e^{-\frac{1}{2}({\bf
w}_{a}\cdot\Phi_{x}-{\bf w}_{b}\cdot\Phi_{y})^{2}}=e^{-\frac{1}{2}({\bf
w}_{a}\cdot\Phi_{x}-{\bf
w}_{b}\cdot\Phi_{y})^{2}}(\Phi_{x}\cdot\Phi_{y})\big{[}1-({\bf
w}_{a}\cdot\Phi_{x}-{\bf w}_{b}\cdot\Phi_{y})^{2}\big{]}\>.$

As for the shallow trig net in Eq. (10), it is easy to show that the NTK is
the same as $k_{\rm SE}$ if $\Omega$ are independent and normal. Hence, the
predictive distribution for $y_{*}|{\bf x}_{*},\Omega,{\bf X,y}$ is the same
for the shallow Bayesian trig network in wide limit and GP with SE kernel.
Moreover, the mean of this distribution shall coincide with the prediction
obtained using gradient descent as the equivalence between NTK and $k_{\rm
SE}$ suggests. However, the correspondence is much intriguing between DGP and
deep trigonometric network as there is no exact inference for both models. If
one adopts the moment matching inference [23] which treats the marginal prior
distribution of DGP as a GP [21], then the predictive distribution is the same
as the GP with $k_{\rm DGP}$. With the equivalence between DGP and deep
trigonometric net, one can say that the single prediction made by gradient
descent algorithm shall converge to the predictive mean of a GP with $k_{\rm
NTK}$ in Eq. (28). The origin for the discrepancy between $k_{\rm DGP}$ and
$k_{\rm NTK}$ is a very interesting question as the exact DGP inference is
impossible and the optimization of deep trigonometric network is not convex.

##  7 Related work

While [26] first pointed out the general correspondence between an infinite
neural network and Gaussian process (GP), [51] demonstrated that neural
networks with iid Gaussian weights and sigmoidal activation units are a
representation of random functions drawn from a GP with arcsine covariance
function. Later, [6] obtained the arccosine kernel from computing the
covariance of outputs from the ReLu neural networks. With the closed form
kernels, one is able to transform the prediction task based on a wide Bayesian
neural network [24] to a GP regression problem. Moreover, the correspondence
holds beyond the shallow neural networks. [25] and [19] studied the deep and
wide neural networks and obtained a recursive relation for the emergent
kernels. Similar techniques appeared in earlier work [31] on derivation of
analytic kernels with which the authors identified a transition from a smooth
into a chaotic phase.

Deep Gaussian Process [9] (DGP) is a hierarchical composition of GPs.
Theoretical progresses regarding understanding DGPs have been made via several
important observations. In the deep limit, DGPs collapse to a constant
function for some subspace of hyperparameters [12, 11, 46] and carry a heavy-
tailed distribution over function derivatives [12]. [23] showed that the
covariance and kurtosis are analytical characteristics of some two-layer DGPs,
and a similar transition into chaotic phase with heavy-tailed multivariate
statistics.

The connections between deep random networks and GPs have been studied
extensively in literature [57, 18, 20], but the link between deep random
networks and DGP was lacking. In fact, deep neural networks can have some
narrow layers connecting their neighboring wide ones. [1] demonstrated that
such deep neural networks with bottlenecks are equivalent to DGPs. [30]
exploited the convexity of exponential function, showing bounded
characteristic function of DGPs, and its convergence to that of a GP with some
limiting kernel as the bottleneck width approaches infinity. Moreover, heavier
statistical tails occur in the deeper models, and their conclusion is very
general, not depending on the kernels of DGP or activation units of deep
neural network. The deep trigonometriconometric network with bottleneck latent
layer was proposed by [7] as a proxy for the DGPs with SE kernels, based on
the random Fourier expansion of SE kernels [32, 14]. Improvements in
generalization over expectation propagation [5] were demonstrated by employing
a dedicated variational inference scheme [4, 7].

##  8 Conclusion

More precise understanding of deep learning is critical for exploiting its
expressive power and potential applications in high-stakes domains. We
analytically investigated the covariance, marginal distribution, statistical
tail, and neural tangent kernel of the trigonometric networks, connecting them
with the deep Gaussian processes which can carry squared exponential kernel,
spectral mixture kernel, and a combinations thereof. We have shown that deep
Gaussian processes and deep trigonometric networks, one in function space and
the other in weight space, yield the same covariance. The derivation for the
deep models in weight space is less intuitive, because it relies on an
infinite dimensional Gaussian integral and knowledge of the spectrum of a
particular random matrix. We have open a door to analyzing the effect of the
non-Gaussianity of deep Gaussian process on its modeling power. Specifically,
the derived neural tangent kernel $k_{\rm NTK}$ with deep trigonometric net
representation allows the possibility of analyzing the implication of
differences between $k_{\rm NTK}$ and the exact kernel $k_{\rm DGP}$ of deep
Gaussian process and whether the predictive mean of intractable DGP inference
can be obtained with $k_{\rm NTK}$.

## Acknowledgement

This research was funded by the Air Force Research Laboratory and DARPA under
agreement number FA8750-17-2-0146. Insightful conversation with Daniel Roberts
was acknowledged.

## References

  * [1] Devanshu Agrawal, Theodore Papamarkou, and Jacob D Hinkle.  Wide neural networks with bottlenecks are deep Gaussian processes.  J. Mach. Learn. Res., 21:175–1, 2020.
  * [2] Laurence Aitchison.  Why bigger is not always better: on finite and infinite neural networks.  In International Conference on Machine Learning, pages 156–164. PMLR, 2020.
  * [3] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang.  On exact computation with an infinitely wide neural net.  Advances in Neural Information Processing Systems, 32:8141–8150, 2019.
  * [4] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In International Conference on Machine Learning, pages 1613–1622. PMLR, 2015.
  * [5] Thang Bui, Daniel Hernández-Lobato, Jose Hernandez-Lobato, Yingzhen Li, and Richard Turner.  Deep Gaussian processes for regression using approximate expectation propagation.  In International Conference on Machine Learning, pages 1472–1481, 2016.
  * [6] Youngmin Cho and Lawrence K Saul.  Kernel methods for deep learning.  In Advances in neural information processing systems, pages 342–350, 2009.
  * [7] Kurt Cutajar, Edwin V Bonilla, Pietro Michiardi, and Maurizio Filippone.  Random feature expansions for deep Gaussian processes.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 884–893. JMLR. org, 2017.
  * [8] Kurt Cutajar, Mark Pullin, Andreas Damianou, Neil Lawrence, and Javier González.  Deep Gaussian processes for multi-fidelity modeling.  arXiv preprint arXiv:1903.07320, 2019.
  * [9] Andreas Damianou and Neil Lawrence.  Deep Gaussian processes.  In Artificial Intelligence and Statistics, pages 207–215, 2013\.
  * [10] Amit Daniely, Roy Frostig, and Yoram Singer.  Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity.  In NIPS, 2016.
  * [11] Matthew M Dunlop, Mark A Girolami, Andrew M Stuart, and Aretha L Teckentrup.  How deep are deep Gaussian processes?  The Journal of Machine Learning Research, 19(1):2100–2145, 2018\.
  * [12] David Duvenaud, Oren Rippel, Ryan Adams, and Zoubin Ghahramani.  Avoiding pathologies in very deep networks.  In Artificial Intelligence and Statistics, pages 202–210, 2014\.
  * [13] Ethan Dyer and Guy Gur-Ari.  Asymptotics of wide networks from Feynman diagrams.  In International Conference on Learning Representations, 2019.
  * [14] Yarin Gal and Richard Turner.  Improving the Gaussian process sparse spectrum approximation by representing uncertainty in frequency inputs.  In International Conference on Machine Learning, pages 655–664. PMLR, 2015.
  * [15] Robert E Greenwood and JJ Miller.  Zeros of the Hermite polynomials and weights for Gauss’ mechanical quadrature formula.  Bulletin of the American Mathematical Society, 54(8):765–769, 1948\.
  * [16] Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  In Advances in neural information processing systems, pages 8571–8580, 2018.
  * [17] Marc C Kennedy and Anthony O’Hagan.  Predicting the output from a complex computer code when fast approximations are available.  Biometrika, 87(1):1–13, 2000.
  * [18] Mohammad Emtiyaz E Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa.  Approximate inference turns deep networks into Gaussian processes.  In Advances in neural information processing systems, pages 3094–3104, 2019.
  * [19] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein.  Deep neural networks as Gaussian processes.  In International Conference on Learning Representations, 2018.
  * [20] Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein.  Finite versus infinite neural networks: an empirical study.  Advances in Neural Information Processing Systems, 33:15156–15172, 2020.
  * [21] Chi-Ken Lu and Patrick Shafto.  Conditional deep Gaussian processes: Empirical bayes hyperdata learning.  Entropy, 23(11):1387, 2021.
  * [22] Chi-Ken Lu and Patrick Shafto.  Conditional deep Gaussian processes: Multi-fidelity kernel learning.  Entropy, 23(11):1545, 2021.
  * [23] Chi-Ken Lu, Scott Cheng-Hsin Yang, Xiaoran Hao, and Patrick Shafto.  Interpretable deep Gaussian processes with moments.  In International Conference on Artificial Intelligence and Statistics, pages 613–623, 2020.
  * [24] David JC MacKay.  Information theory, inference and learning algorithms.  Cambridge university press, 2003.
  * [25] Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Ghahramani.  Gaussian process behaviour in wide deep neural networks.  In International Conference on Learning Representations, 2018.
  * [26] Radford M Neal.  Monte carlo implementation of Gaussian process models for Bayesian regression and classification.  arXiv preprint physics/9701026, 1997.
  * [27] Sebastian W Ober and Laurence Aitchison.  Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes.  In International Conference on Machine Learning, pages 8248–8259. PMLR, 2021.
  * [28] Sebastian W Ober, Carl E Rasmussen, and Mark van der Wilk.  The promises and pitfalls of deep kernel learning.  In Uncertainty in Artificial Intelligence, pages 1206–1216. PMLR, 2021.
  * [29] Tim Pearce, Russell Tsuchida, Mohamed Zaki, Alexandra Brintrup, and Andy Neely.  Expressive priors in Bayesian neural networks: Kernel combinations and periodic functions.  In Uncertainty in Artificial Intelligence, pages 134–144. PMLR, 2020.
  * [30] Geoff Pleiss and John P Cunningham.  The limitations of large width in neural networks: A deep Gaussian process perspective.  Advances in Neural Information Processing Systems, 34, 2021.
  * [31] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli.  Exponential expressivity in deep neural networks through transient chaos.  In NIPS, 2016.
  * [32] Ali Rahimi and Benjamin Recht.  Random features for large-scale kernel machines.  In Advances in neural information processing systems, pages 1177–1184, 2008.
  * [33] Maziar Raissi, Paris Perdikaris, and George E Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational physics, 378:686–707, 2019.
  * [34] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  arXiv preprint arXiv:1711.05225, 2017.
  * [35] C. E. Rasmussen and C. K. I. Williams.  Gaussian Process for Machine Learning.  MIT press, Cambridge, MA, 2006.
  * [36] Daniel A Roberts, Sho Yaida, and Boris Hanin.  The principles of deep learning theory.  arXiv preprint arXiv:2106.10165, 2021.
  * [37] Hugh Salimbeni and Marc Deisenroth.  Doubly stochastic variational inference for deep Gaussian processes.  In Advances in Neural Information Processing Systems, 2017.
  * [38] Hugh Salimbeni, Vincent Dutordoir, James Hensman, and Marc Peter Deisenroth.  Deep Gaussian processes with importance-weighted variational inference.  arXiv preprint arXiv:1905.05435, 2019.
  * [39] Mariia Seleznova and Gitta Kutyniok.  Analyzing finite neural networks: Can we trust neural tangent kernel theory?  Proceedings of Machine Learning Research vol, 145:1–28, 2021.
  * [40] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484–489, 2016.
  * [41] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  Advances in Neural Information Processing Systems, 33:7462–7473, 2020.
  * [42] Edward Snelson, Zoubin Ghahramani, and Carl E Rasmussen.  Warped Gaussian processes.  In Advances in neural information processing systems, pages 337–344, 2004.
  * [43] Josep M Sopena, Enrique Romero, and Rene Alquezar.  Neural networks with periodic and monotonic activation functions: a comparative study in classification problems.  1999\.
  * [44] M. Titsias and N. Lawrence.  Bayesian Gaussian process latent variable model.  In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 844–851, 2010.
  * [45] Michalis Titsias.  Variational learning of inducing variables in sparse Gaussian processes.  In Artificial Intelligence and Statistics, pages 567–574, 2009\.
  * [46] Anh Tong and Jaesik Choi.  Characterizing deep Gaussian processes via nonlinear recurrence systems.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9915–9922, 2021.
  * [47] Ivan Ustyuzhaninov, Ieva Kazlauskaite, Markus Kaiser, Erik Bodin, Neill Campbell, and Carl Henrik Ek.  Compositional uncertainty in deep Gaussian processes.  In Conference on Uncertainty in Artificial Intelligence, pages 480–489. PMLR, 2020.
  * [48] David A Van Dyk and Taeyoung Park.  Partially collapsed Gibbs samplers: Theory and methods.  Journal of the American Statistical Association, 103(482):790–796, 2008.
  * [49] Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel.  Understanding priors in Bayesian neural networks at the unit level.  In International Conference on Machine Learning, pages 6458–6467. PMLR, 2019.
  * [50] Zheng Wang, Wei Xing, Robert Kirby, and Shandian Zhe.  Physics informed deep kernel learning.  arXiv preprint arXiv:2006.04976, 2020.
  * [51] Christopher KI Williams.  Computing with infinite networks.  In Advances in neural information processing systems, pages 295–301, 1997.
  * [52] Andrew Wilson and Ryan Adams.  Gaussian process kernels for pattern discovery and extrapolation.  In International Conference on Machine Learning, pages 1067–1075, 2013.
  * [53] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing.  Deep kernel learning.  In Artificial Intelligence and Statistics, pages 370–378, 2016\.
  * [54] Andrew Gordon Wilson and Pavel Izmailov.  Bayesian deep learning and a probabilistic perspective of generalization.  arXiv preprint arXiv:2002.08791, 2020.
  * [55] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang.  Dota: A large-scale dataset for object detection in aerial images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3974–3983, 2018.
  * [56] Sho Yaida.  Non-Gaussian processes and neural networks at finite widths.  In Mathematical and Scientific Machine Learning, pages 165–192. PMLR, 2020.
  * [57] Greg Yang.  Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation.  arXiv preprint arXiv:1902.04760, 2019.
  * [58] Haibin Yu, Yizhou Chen, Bryan Kian Hsiang Low, Patrick Jaillet, and Zhongxiang Dai.  Implicit posterior variational inference for deep Gaussian processes.  In Advances in Neural Information Processing Systems, pages 14502–14513, 2019.
  * [59] Jacob Zavatone-Veth and Cengiz Pehlevan.  Exact marginal prior distributions of finite Bayesian neural networks.  Advances in Neural Information Processing Systems, 34, 2021.
  * [60] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  arXiv preprint arXiv:1611.03530, 2016.