* Lowd (2005) Daniel Lowd. 2005\.  Good word attacks on statistical spam filters. In _In Proceedings of the Second Conference on Email and Anti-Spam (CEAS_. Citeseer.
  * Lum and Johndrow (2016) Kristian Lum and James Johndrow. 2016.  A statistical framework for fair predictive algorithms.  _arXiv preprint arXiv:1610.08077_ (2016).
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions. In _Advances in Neural Information Processing Systems_. 4765–4774.
  * Luong et al. (2011) Binh Thanh Luong, Salvatore Ruggieri, and Franco Turini. 2011\.  k-NN as an implementation of situation testing for discrimination discovery and prevention. In _Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 502–510.
  * Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, Nov (2008), 2579–2605.
  * Madras et al. (2019) David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. 2019.  Fairness through causal awareness: Learning causal latent-variable models for biased data. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_. 349–358.
  * Maiorca et al. (2013) Davide Maiorca, Igino Corona, and Giorgio Giacinto. 2013\.  Looking at the bag is not enough to find the bomb: an evasion of structural methods for malicious pdf files detection. In _Proceedings of the 8th ACM SIGSAC symposium on Information, computer and communications security_. ACM, 119–130.
  * Martens et al. (2007) David Martens, Bart Baesens, Tony Van Gestel, and Jan Vanthienen. 2007.  Comprehensible credit scoring models using rule extraction from support vector machines.  _European journal of operational research_ 183, 3 (2007), 1466–1476.
  * McSherry (2017) Frank McSherry. 2017\.  Statistical inference considered harmful.  https://github.com/frankmcsherry/blog/blob/master/posts/2016-06-14.md
  * Mehrabi et al. (2019) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019\.  A survey on bias and fairness in machine learning.  _arXiv preprint arXiv:1908.09635_ (2019).
  * Mei and Zhu (2015) Shike Mei and Xiaojin Zhu. 2015.  The security of latent dirichlet allocation. In _Artificial Intelligence and Statistics_. 681–689.
  * Metzen et al. (2017) Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. 2017.  On detecting adversarial perturbations.  _arXiv preprint arXiv:1702.04267_ (2017).
  * Milich and Karr ([n. d.]) Andrew Milich and Michael Karr. [n. d.].  Eluding Mass Surveillance: Adversarial Attacks on Facial Recognition Models.  ([n. d.]).
  * Miller (2018) Tim Miller. 2018\.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial Intelligence_ (2018).
  * Missier et al. (2015) P. Missier, J. Bryans, C. Gamble, V. Curcin, and R. Danger. 2015.  _Provabs: Model, policy, and tooling for abstracting PROV graphs_. Vol. 8628.  3–15 pages.  https://doi.org/10.1007/978-3-319-16462-5_1
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model cards for model reporting. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_. ACM, 220–229.
  * Mohri et al. (2018) Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. 2018\.  _Foundations of machine learning_.  MIT press.
  * Montavon et al. (2018) Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. 2018.  Methods for interpreting and understanding deep neural networks.  _Digital Signal Processing_ 73 (2018), 1–15.
  * Moosavi-Dezfooli et al. (2017) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2017.  Universal adversarial perturbations. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1765–1773.
  * Moosavi-Dezfooli et al. (2016) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016\.  Deepfool: a simple and accurate method to fool deep neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2574–2582.
  * Mopuri et al. (2017) Konda Reddy Mopuri, Utsav Garg, and R Venkatesh Babu. 2017\.  Fast Feature Fool: {A} data independent approach to universal adversarial perturbations.  _CoRR_ abs/1707.0 (2017).  http://arxiv.org/abs/1707.05572
  * Moreau et al. (2011) Luc Moreau, Ben Clifford, Juliana Freire, Joe Futrelle, Yolanda Gil, Paul Groth, Natalia Kwasnikowska, Simon Miles, Paolo Missier, Jim Myers, Beth Plale, Yogesh Simmhan, Eric Stephan, and Jan Van Den Bussche. 2011\.  The Open Provenance Model core specification (v1.1).  _Future Generation Computer Systems_ 27, 6 (2011), 743–756.  https://doi.org/10.1016/j.future.2010.07.005
  * Moreau et al. (2012) Luc Moreau, Paolo Missier, Khalid Belhajjame, Reza B’Far, James Cheney, Sam Coppens, Stephen Cresswell, Yolanda Gil, Paul Groth, Graham Klyne, Timothy Lebo, Jim McCusker, Simon Miles, James Myers, Satya Sahoo, and Curt Tilmes. 2012\.  _PROV-DM: The PROV Data Model_.  Technical Report. World Wide Web Consortium.  http://www.w3.org/TR/prov-dm/
  * Namaki et al. (2020) Mohammad Hossein Namaki, Avrilia Floratou, Fotis Psallidas, Subru Krishnan, Ashvin Agrawal, and Yinghui Wu. 2020.  Vamsa: Tracking Provenance in Data Science Scripts.  arXiv:cs.LG/2001.01861
  * Narayanan (2018) Arvind Narayanan. 2018\.  Translation tutorial: 21 fairness definitions and their politics. In _Proc. Conf. Fairness Accountability Transp., New York, USA_.
  * Narayanan and Shmatikov (2008) Arvind Narayanan and Vitaly Shmatikov. 2008.  Robust de-anonymization of large datasets (how to break anonymity of the Netflix prize dataset).  (2008).
  * Narodytska and Kasiviswanathan (2017) Nina Narodytska and Shiva Kasiviswanathan. 2017.  Simple black-box adversarial attacks on deep neural networks. In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 1310–1318.
  * Nelson et al. (2009) Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D Joseph, Benjamin I P Rubinstein, Udam Saini, Charles Sutton, J D Tygar, and Kai Xia. 2009.  Misleading learners: Co-opting your spam filter.  In _Machine learning in cyber trust_. Springer, 17–51.
  * Newsome et al. (2005) James Newsome, Brad Karp, and Dawn Song. 2005.  Polygraph: Automatically generating signatures for polymorphic worms. In _2005 IEEE Symposium on Security and Privacy (S &P’05)_. IEEE, 226–241.
  * Neyman (1992) Jerzy Neyman. 1992\.  On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection.  In _Breakthroughs in Statistics_. Springer, 123–150.
  * Nguyen et al. (2015) Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015\.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 427–436.
  * Nori et al. (2019) Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. 2019\.  InterpretML: A Unified Framework for Machine Learning Interpretability.  (sep 2019).  arXiv:1909.09223 http://arxiv.org/abs/1909.09223
  * Núñez et al. (2002) Haydemar Núñez, Cecilio Angulo, and Andreu Català. 2002.  Rule extraction from support vector machines.. In _Esann_. 107–112.
  * Ohrimenko et al. (2016) Olga Ohrimenko, Felix Schuster, Cédric Fournet, Aastha Mehta, Sebastian Nowozin, Kapil Vaswani, and Manuel Costa. 2016.  Oblivious multi-party machine learning on trusted processors. In _25th ${$USENIX$}$ Security Symposium (${$USENIX$}$ Security 16)_. 619–636.
  * Papernot et al. (2017) Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. 2017\.  Practical black-box attacks against machine learning. In _Proceedings of the 2017 ACM on Asia conference on computer and communications security_. ACM, 506–519.
  * Papernot et al. (2016a) Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. 2016a.  The limitations of deep learning in adversarial settings. In _2016 IEEE European symposium on security and privacy (EuroS &P)_. IEEE, 372–387.
  * Papernot et al. (2018) Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P Wellman. 2018.  SoK: Security and privacy in machine learning. In _2018 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 399–414.
  * Papernot et al. (2016b) Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. 2016b.  Distillation as a defense to adversarial perturbations against deep neural networks. In _2016 IEEE Symposium on Security and Privacy (SP)_. IEEE, 582–597.
  * Pedreshi et al. (2008) Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. 2008\.  Discrimination-aware data mining. In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 560–568.
  * Perdisci et al. (2006) Roberto Perdisci, David Dagon, Wenke Lee, Prahlad Fogla, and Monirul Sharif. 2006.  Misleading worm signature generators using deliberate noise injection. In _2006 IEEE Symposium on Security and Privacy (S &P’06)_. IEEE, 15–pp.
  * Perreault (2015) Lucille Perreault. 2015\.  Big Data and Privacy: Control and Awareness Aspects.. In _CONF-IRM_. 15.
  * Polyzotis et al. (2018) Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich. 2018.  Data lifecycle challenges in production machine learning: a survey.  _ACM SIGMOD Record_ 47, 2 (2018), 17–28.
  * Qian et al. (2019) Bin Qian, Jie Su, Zhenyu Wen, Devki Nandan Jha, Yinhao Li, Yu Guan, Deepak Puthal, Philip James, Renyu Yang, Albert Y Zomaya, et al. 2019\.  Orchestrating Development Lifecycle of Machine Learning Based IoT Applications: A Survey.  _arXiv preprint arXiv:1910.05433_ (2019).
  * Quadrianto and Sharmanska (2017) Novi Quadrianto and Viktoriia Sharmanska. 2017.  Recycling privileged learning and distribution matching for fairness. In _Advances in Neural Information Processing Systems_. 677–688.
  * Ribeiro et al. (2015) Mauro Ribeiro, Katarina Grolinger, and Miriam AM Capretz. 2015\.  Mlaas: Machine learning as a service. In _2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 896–902.
  * Ribeiro et al. (2016a) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016a.  Model-Agnostic Interpretability of Machine Learning. In _2016 ICML Workshop on Human Interpretability in Machine network Learning (WHI 2016)_.  https://doi.org/10.1145/2858036.2858529 arXiv:1606.05386
  * Ribeiro et al. (2016b) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016b.  Model-agnostic interpretability of machine learning.  _arXiv preprint arXiv:1606.05386_ (2016).
  * Ribeiro et al. (2016c) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016c.  Why should i trust you?: Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. ACM, 1135–1144.
  * Rivest (1987) Ronald L Rivest. 1987\.  Learning decision lists.  _Machine learning_ 2, 3 (1987), 229–246.
  * Rizzi (2009) S Rizzi. 2009.  What-if analysis. Encyclopedia of database systems.
  * Romei and Ruggieri (2014) Andrea Romei and Salvatore Ruggieri. 2014.  A multidisciplinary survey on discrimination analysis.  _The Knowledge Engineering Review_ 29, 5 (2014), 582–638.
  * Rubinstein et al. (2009a) Benjamin I P Rubinstein, Peter L Bartlett, Ling Huang, and Nina Taft. 2009a.  Learning in a large function space: Privacy-preserving mechanisms for SVM learning.  _arXiv preprint arXiv:0911.5708_ (2009).
  * Rubinstein et al. (2009b) Benjamin I P Rubinstein, Blaine Nelson, Ling Huang, Anthony D Joseph, Shing-hon Lau, Satish Rao, Nina Taft, and J Doug Tygar. 2009b.  Antidote: understanding and defending against poisoning of anomaly detectors. In _Proceedings of the 9th ACM SIGCOMM conference on Internet measurement_. ACM, 1–14.
  * Sacks (2018) Samm Sacks. 2018\.  New China Data Privacy Standard Looks More Far-Reaching than GDPR.  _Center for Strategic and International Studies_ 29 (2018).
  * Samek (2019) Wojciech Samek. 2019\.  _Explainable AI: interpreting, explaining and visualizing deep learning_. Vol. 11700.  Springer Nature.
  * Samek et al. (2016) Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. 2016\.  Evaluating the visualization of what a deep neural network has learned.  _IEEE transactions on neural networks and learning systems_ 28, 11 (2016), 2660–2673.
  * Schelter et al. (2017) Sebastian Schelter, Joos-Hendrik Boese, Johannes Kirschnick, Thoralf Klein, and Stephan Seufert. 2017\.  Automatically tracking metadata and provenance of machine learning experiments. In _Machine Learning Systems Workshop at NIPS_. 27–29.
  * Schetinin et al. (2007) Vitaly Schetinin, Jonathan E Fieldsend, Derek Partridge, Timothy J Coats, Wojtek J Krzanowski, Richard M Everson, Trevor C Bailey, and Adolfo Hernandez. 2007.  Confident interpretation of Bayesian decision tree ensembles for clinical applications.  _IEEE Transactions on Information Technology in Biomedicine_ 11, 3 (2007), 312–319.
  * Schmidt and Biessmann (2019) Philipp Schmidt and Felix Biessmann. 2019.  Quantifying Interpretability and Trust in Machine Learning Systems.  _arXiv preprint arXiv:1901.08558_ (2019).
  * Settles (2009) Burr Settles. 2009\.  _Active learning literature survey_.  Technical Report. University of Wisconsin-Madison Department of Computer Sciences.
  * Shalin et al. (1988) Valerie L Shalin, Edward J Wisniewski, Keith R Levi, and Paul D Scott. 1988.  A formal analysis of machine learning systems for knowledge acquisition.  _International Journal of Man-Machine Studies_ 29, 4 (1988), 429–446.
  * Sharif et al. (2016) Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. 2016.  Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_. ACM, 1528–1540.
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017.  Membership inference attacks against machine learning models. In _2017 IEEE Symposium on Security and Privacy (SP)_. IEEE, 3–18.
  * Shrikumar et al. (2017) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017\.  Learning important features through propagating activation differences. In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 3145–3153.
  * Singh et al. (2018) Jatinder Singh, Jennifer Cobbe, and Chris Norval. 2018\.  Decision Provenance: Harnessing data flow for accountable systems.  _IEEE Access_ 7 (2018), 6562–6574.
  * Šrndic and Laskov (2013) Nedim Šrndic and Pavel Laskov. 2013.  Detection of malicious pdf files based on hierarchical document structure. In _Proceedings of the 20th Annual Network & Distributed System Security Symposium_. Citeseer, 1–16.
  * Stamp (2011) Mark Stamp. 2011\.  _Information security: principles and practice_.  Wiley Online Library.
  * Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013\.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ (2013).
  * Tan et al. (2016) Hui Fen Tan, Giles Hooker, and Martin T Wells. 2016\.  Tree space prototypes: Another look at making tree ensembles interpretable.  _arXiv preprint arXiv:1611.07115_ (2016).
  * Tan (2018) Pang-Ning Tan. 2018\.  _Introduction to data mining_.  Pearson Education India.
  * Teo et al. (2008) Choon H Teo, Amir Globerson, Sam T Roweis, and Alex J Smola. 2008.  Convex learning with invariances. In _Advances in neural information processing systems_. 1489–1496.
  * Tolomei et al. (2017) Gabriele Tolomei, Fabrizio Silvestri, Andrew Haines, and Mounia Lalmas. 2017.  Interpretable predictions of tree-based ensembles via actionable feature tweaking. In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_. ACM, 465–474.
  * Toreini et al. (2020) Ehsan Toreini, Mhairi Aitken, Kovila Coopamootoo, Karen Elliott, Carlos Gonzalez Zelaya, and Aad van Moorsel. 2020.  The relationship between trust in AI and trustworthy machine learning technologies. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 272–283.
  * Tramèr et al. (2016) Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. 2016.  Stealing machine learning models via prediction apis. In _25th ${$USENIX$}$ Security Symposium (${$USENIX$}$ Security 16)_. 601–618.
  * Uysal and Gunal (2014) Alper Kursat Uysal and Serkan Gunal. 2014.  The impact of preprocessing on text classification.  _Information Processing and Management_ 50, 1 (2014), 104–112.  https://doi.org/10.1016/j.ipm.2013.08.006
  * Veale et al. (2018) Michael Veale, Max Van Kleek, and Reuben Binns. 2018\.  Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. In _Proceedings of the 2018 chi conference on human factors in computing systems_. ACM, 440.
  * Verma and Rubin (2018) Sahil Verma and Julia Rubin. 2018.  Fairness definitions explained. In _2018 IEEE/ACM International Workshop on Software Fairness (FairWare)_. IEEE, 1–7.
  * Wachenfeld and Winner (2016) Walther Wachenfeld and Hermann Winner. 2016.  Do autonomous vehicles learn?  In _Autonomous Driving_. Springer, 451–471.
  * Wachter et al. (2017) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017\.  Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR.  (nov 2017).  arXiv:1711.00399 http://arxiv.org/abs/1711.00399
  * Wang et al. (2014) Qia Wang, Wenjun Zeng, and Jun Tian. 2014.  Compressive sensing based secure multiparty privacy preserving framework for collaborative data-mining and signal processing. In _2014 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 1–6.
  * Whittlestone et al. (2019) Jess Whittlestone, Rune Nyrup, Anna Alexandrova, and Stephen Cave. 2019.  The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions. In _Proceedings of the AAAI/ACM Conference on AI Ethics and Society, Honolulu, HI, USA_. 27–28.
  * Wittel and Wu (2004) Gregory L Wittel and Shyhtsun Felix Wu. 2004.  On Attacking Statistical Spam Filters.. In _CEAS_.
  * Woodworth et al. (2017) Blake Woodworth, Suriya Gunasekar, Mesrob I Ohannessian, and Nathan Srebro. 2017.  Learning non-discriminatory predictors.  _arXiv preprint arXiv:1702.06081_ (2017).
  * Wu et al. (2016) Xi Wu, Matthew Fredrikson, Somesh Jha, and Jeffrey F Naughton. 2016.  A methodology for formalizing model-inversion attacks. In _2016 IEEE 29th Computer Security Foundations Symposium (CSF)_. IEEE, 355–370.
  * Xiao et al. (2015) Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. 2015\.  Is feature selection secure against training data poisoning?. In _International Conference on Machine Learning_. 1689–1698.
  * Xu et al. (2017) Weilin Xu, David Evans, and Yanjun Qi. 2017.  Feature squeezing: Detecting adversarial examples in deep neural networks.  _arXiv preprint arXiv:1704.01155_ (2017).
  * Xu et al. (2016) W Xu, Y Qi, and D Evans. 2016.  Automatically Evading Classifiers: A Case Study on PDF Malware Classifiers. NDSS.
  * Yang et al. (2018) Ke Yang, Julia Stoyanovich, Abolfazl Asudeh, Bill Howe, H V Jagadish, and Gerome Miklau. 2018\.  A nutritional label for rankings. In _Proceedings of the 2018 International Conference on Management of Data_. ACM, 1773–1776.
  * Yucheng et al. (2017) Yao Yucheng, Song Ling, and E Chi. 2017.  Investigation on distributed K-means clustering algorithm of homomorphic encryption.  _Computer Technology and Development_ 27, 2 (2017), 81–85.
  * Zafar (2019) Muhammad Bilal Zafar. 2019\.  Discrimination in Algorithmic Decision Making: From Principles to Measures and Mechanisms.  (2019).
  * Zafar et al. (2015) Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2015.  Fairness constraints: Mechanisms for fair classification.  _arXiv preprint arXiv:1507.05259_ (2015).
  * Zelaya et al. (2019) Vladimiro G Zelaya, Paolo Missier, and Dennis Prangle. 2019\.  Parametrised Data Sampling for Fairness Optimisation.  (2019).
  * Zemel et al. (2013) Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013.  Learning fair representations. In _International Conference on Machine Learning_. 325–333.
  * Zhang et al. (2018) Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018\.  Mitigating unwanted biases with adversarial learning. In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_. ACM, 335–340.
  * Zhang et al. (2015) Fei Zhang, Patrick P K Chan, Battista Biggio, Daniel S Yeung, and Fabio Roli. 2015.  Adversarial feature selection against evasion attacks.  _IEEE transactions on cybernetics_ 46, 3 (2015), 766–777.
  * Zhao et al. (2017) Mengchen Zhao, Bo An, Wei Gao, and Teng Zhang. 2017\.  Efficient Label Contamination Attacks Against Black-Box Learning Models.. In _IJCAI_. 3945–3951.
  * Zhao et al. (2012) Wentao Zhao, Jun Long, Jianping Yin, Zhiping Cai, and Geming Xia. 2012.  Sampling attack against active learning in adversarial environment. In _International Conference on Modeling Decisions for Artificial Intelligence_. Springer, 222–233.
  * Zhou and Hooker (2016) Yichen Zhou and Giles Hooker. 2016.  Interpreting models via single tree approximation.  _arXiv preprint arXiv:1610.09036_ (2016).
  * Zhou et al. (2003) Zhi-Hua Zhou, Yuan Jiang, and Shi-Fu Chen. 2003.  Extracting symbolic rules from trained neural network ensembles.  _Ai Communications_ 16, 1 (2003), 3–15.
  * Žliobaitė (2015) Indrė Žliobaitė. 2015\.  On the relation between accuracy and fairness in binary classification.  _arXiv preprint arXiv:1505.05723_ (2015).


# Sequential Spatial Network for Collision Avoidance in Autonomous Driving
††thanks: This research is supported by Sino German fond (5091331).

1st Haichuan Li  Chair of Robotics, Artificial
Intelligence and Real-Time Systems
Technical University of Munich
Garching, Germany
<EMAIL_ADDRESS>2nd Liguo Zhou  Chair of Robotics, Artificial
Intelligence and Real-Time Systems
Technical University of Munich
Garching, Germany
<EMAIL_ADDRESS>3rd Zhenshan Bing  Chair of Robotics, Artificial
Intelligence and Real-Time Systems
Technical University of Munich
Munich, Germany
<EMAIL_ADDRESS>4th Marzana Khatun  Institute for Driver Assistance and
Connected Mobility
Hochschule Kempten
Kempten, Germany
<EMAIL_ADDRESS>5th Rolf Jung  Institute for Driver Assistance
and
Connected Mobility
Hochschule Kempten
Kempten, Germany
<EMAIL_ADDRESS>6th Alois Knoll  Chair of Robotics, Artificial
Intelligence
and Real-Time Systems
Technical University of Munich
Garching, Germany
<EMAIL_ADDRESS>
###### Abstract

Several autonomous driving strategies have been applied to autonomous
vehicles, especially in the collision avoidance area. The purpose of collision
avoidance is achieved by adjusting the trajectory of autonomous vehicles (AV)
to avoid intersection or overlap with the trajectory of surrounding vehicles.
A large number of sophisticated vision algorithms have been designed for
target inspection, classification, and other tasks, such as ResNet, YOLO,
etc., which have achieved excellent performance in vision tasks because of
their ability to accurately and quickly capture regional features. However,
due to the variability of different tasks, the above models achieve good
performance in capturing small regions but are still insufficient in
correlating the regional features of the input image with each other. In this
paper, we aim to solve this problem and develop an algorithm that takes into
account the advantages of CNN in capturing regional features while
establishing feature correlation between regions using variants of attention.
Finally, our model achieves better performance in the test set of L5Kit
compared to the other vision models. The average number of collisions is 19.4
per 10000 frames of driving distance, which greatly improves the success rate
of collision avoidance.

###### Index Terms:

Collision Avoidance, Computer Vision, Autonomous Driving, Trajectory
Prediction, Sequential Spatial

##  I Introduction

In the past many years, researchers have focused on how to turn vehicles from
assisted driving to more intelligent autonomous driving. Due to the iteration
of intelligent hardware and the improvement of chip computing power, a large
amount of data collected by sensors can be quickly converted and fed into
models to make decisions. In the driving process, the safety factor is the
first consideration for users and researchers. Therefore, how AV should avoid
collisions has become a top priority. Concepts such as probabilistic methods
(eg.: Markov chains [1] and Monte

(a) Front Collision

(b) Side Collision

(c) Rear Collision

Figure 1: Three different situations of collisions. The front collision case
is shown in (a). The side collision case is shown in (b). The rear collision
case is shown in (c). Our purpose is to let autonomous driving vehicles avoid
collisions such as these.

Carlo [2]), safety distance-based control methods [3], and trajectory
prediction methods [3] have been designed in recent years to cope with complex
traffic conditions. In terms of vision, CNN [4] has made outstanding
contributions and has been applied to a large number of road condition
inspection tasks due to its excellent regional feature extraction
capabilities. The local feature information obtained by CNN will be used for
obstacle detection. Secondly, because the motion trajectory is planned for AV,
the relationship between each local feature of the image obtained by CNN needs
to be established. Some strategies are based on CNN plus RNN [5] so that they
can deal with sequential graphs as input, eg.: STDN [6].

Although the above strategies have performed well in a large number of vision
tasks, their performances are still far inferior to similar-sized
convolutional neural networks counterparts, such as EfficientNets [7] and
RepVGG [8]. We believe this is due to the following aspects. First, the huge
differences between the sequential tasks of NLP and the image tasks of CV are
ignored. For example, when the local feature information acquired in a two-
dimensional image is compressed into one-dimensional time series information,
how to achieve accurate mapping becomes a difficult problem. Second, it is
difficult to keep the original information of inputs since after RNN layers,
we need to recover the dimension from one to three. Besides, due to the
several transformations between different dimensions, that process becomes
even harder, especially since our input size is 224×224×5. Third, the
computational and memory requirement of switching between layers are extremely
heavy tasks, which also becomes a tricky point for the algorithm to run.
Higher hardware requirements as well as more running time arise when running
the attention part.

In this paper, we propose a new network structure based on CNN and attention
to vision tasks in autonomous driving. The new network structure overcomes
these problems by using Sequential Spatial Network (SSN) blocks. As shown in
Fig. 3, input images first go through the convolution stem for fine-grained
feature extraction, and are then fed into a stack of SSN blocks for further
processing. The Upsampling Convolutional Decreasing (UCD) blocks are
introduced for the purpose of local information enhancement by deep
convolution, and in SSN block of features generated in the first stage can be
less loss of image resolution, which is crucial for the subsequent trajectory
adjustment task.

In addition, we adopt a staged architecture design using five convolutional
layers with different kernel sizes and steps gradually decreasing the
resolution (sequence length) and flexibly increasing the dimensionality. Such
a design helps to extract local features of different scales and, since the
first stage retains high resolution, our design can effectively reduce the
resolution of the output information in the first layer at each convolutional
layer, thus reducing the computational effort of subsequent layers. The
Reinforcement Region Unit (RRU) and the Fast MultiHead Self-Attention (FMHSA)
in the SSN block can help obtain global and local structural information
within the intermediate features and improve the normalization capability of
the network. Finally, average pooling is used to obtain better trajectory
tuning.

Extensive experiments on the lykit dataset demonstrate the superiority of our
SSN network in terms of accuracy. In addition to image classification, SSN
block can be easily transferred to other vision tasks and serve as a versatile
backbone.

##  II Related Works

Over the past few decades, autonomous driving has flourished in the wave of
deep learning, where a large number of solution strategies are based on
computer vision, using images as the primary input. The prevailing visual
neural networks are typically built on top of a basic block in which a series
of convolutional layers are stacked sequentially to capture local information
in intermediate features. However, the limited receptive field of the small
convolution kernel makes it difficult to obtain global information, which
hinders the high performance of the network on highly feature-dependent tasks
(such as trajectory prediction and planning). In view of this dilemma, many
researchers have begun to deeply study self-attention-based [9] networks with
the ability to capture long-distance information. Here, we briefly review
traditional CNNs and recently proposed visual networks. Convolutional neural
network. The first standard CNN was proposed by LeCun [10] et al. was used for
handwritten character recognition. Based on this foundation, a large number of
visual models have achieved cross-generational success in a variety of tasks
with images as the main input. Google Inception Net [11] and DenseNet [12]
showed that deep neural networks consisting of convolutional and pooling
layers can yield adequate results in recognition. SENet [13] and MobileNetV3
[14] demonstrate the effectiveness of multiple paths within a basic block.

ResNet [15] is a classic structure that has a better generalization ability by
adding shortcut connections to the underlying network. To alleviate the
limited acceptance domain in previous studies, some studies used the attention
mechanism as an operator for adapting patterns.

(a) ResNet-50

(b) RepVGG

(c) ViT

Figure 2: Three popular network structures in vision areas. The structure of
ResNet-50 is shown in (a). The structure of RepVGG is shown in (b). The
structure of ViT is shown in (c).

##  III Approach

###  III-A Network Structure Overview

Our strategy is to take advantage of both CNN and attention by building a
hybrid network. An overview of ResNet-50 [15], RepVGG [8], ViT [16] and our
network are shown in Fig. 2 and Fig. 3.

Resnet-50 consists of five stages, stage0 consists of a convolutional layer, a
batch normalization layer and a maxpooling layer. stage1, stage2, stage3, and
stage4 consist of bottleneck blocks, and the output is fed to a fully
connected layer for classification. The advantage of this design is that
resnet50 can efficiently handle the classification problem between different
images. However, since the stage0 only uses one convolutional layer with a
large convolutional kernel, the large field of view can quickly complete the
initial processing of the input image, but the local information capture of
the image is slightly insufficient. For this reason, we use the main input
block to deal with this limitation. The main input block is composed of five
different kernel sizes and steps convolutional layers, so that the purpose of
stepping down the convolution kernel is to extract the input information
quickly when the output size is large, and then use the small convolution
kernel to extract the local information after the input size becomes smaller.

After taking into account the fast processing and local information processing
of the main input block, the input information is transferred to the
subsequent blocks for subsequent processing. Furthermore, between each block,
we add a UCD layer, which consists of a convolutional layer with 1×1 kernel
size and a downsampling layer which a sampling ratio is 0.5. The UCD layer
allows us to speed up the network without reducing the amount of information
in the input but maintaining the ratio between the information, and the size
of the input is reduced to half of the original size after the UCD layer.
Afterwards, the feature extraction is performed by a network layer composed of
different numbers of SSN blocks, while maintaining the same resolution of the
input. Due to the existence of the self-attention mechanism, SSN can capture
the correlation of different local information features, so as to achieve
mutual dependence between local information and global information. Finally,
the results are output through an average pooling layer and a projection layer
as well as a classifier layer.

Through the historical images of driving trips, we can obtain information such
as position, yaw and environment. SSN is similar to a typical vision network
and can adjust the stride size of the middle layer to obtain feature maps of
different sizes according to the requirements, which can be applied to
downstream tasks with different inputs, such as trajectory prediction and
image classification.

Figure 3: The overview of SSN network structure. RRU is Reinforcement Region
Unit. FMHSA is Fast Multi-Head Self-Attention module. IRU is Information
Refinement Unit.

###  III-B SSN blcok

The proposed SSN module consists of a Reinforcement Region Unit (RRU), a Fast
Multi-Head Self-Attention (FMHSA) module and an Information Refinement Unit
(IRU), as shown in Fig. 3. We will describe these four components in the
following.

Reinforcement region unit. In vision tasks, data augmentation is usually
essential to improve model generalization effectively by training the
augmented data. Common data augmentation methods such as flip, rotation and
scaling, etc, but adding augmented data should not weaken the final
performance of the model.

In other words, a good model should maintain effective operating output for
similar but variant data as well so that the model has better input
acceptability. However, the absolute position encoding used in the common
attention was originally designed to exploit the order of the tokens, but it
breaks the input acceptability because each patch adds a unique position
encoding to it [17]. Moreover, the concatenation between the local information
obtained by the information capture module at the beginning of the model and
the structural information inside the patch [18] is ignored. In order to
maintain input acceptability, the Reinforcement Region Unit (RRU) is designed
to extract the local information from the input to the ”SSN” module, defined
as:

$RRU(X)=Conv(Conv(X)).$ (1)

Fast Multiple Head Self-Attention (FMHSA) module consists of one convolutional
layer, one linear layer and one multi-head self-attention. With this scheme,
we can build up the connection between different local information which
results from RRU. By this way, the collision avoidance task is able to get an
outstanding result since on each frame the trajectory is composed of
continuously predicted positions. Moreover, these positions are sequentially
related which means autonomous driving vehicles must arrive at the first
target position then they can move to the next target position. Our FMHSA
module is suitable to solve this problem because it can transfer local
information between areas.

The Information Refinement Unit (IRU) is used to efficiently extract the local
information obtained by FMHSA, and after processing by this unit, the
extracted local information is fed into the pooling and classifier layers. The
original FFN proposed in ViT consists of two linear layers separated by the
GELU activation [19]. First, expand the input dimension by 4 times, and then
scale down the expanded dimension:

$FFN(X)=GELU(XW1+b1)W2+b2.$ (2)

This has the advantage of using a linear function for forward propagation
before using the GELU() function, which greatly improves the efficiency of the
model operation. However, this strategy leads to a certain performance
sacrifice when the network is propagating fast in this region. Our design
concept can be used to deal with this problem. First, we use the convolutional
layer of a larger convolution kernel to obtain the characteristics of the
input information with a large field of view, then use the linear function
layer to conduct quickly, and finally use the convolutional layer of a small
convolution kernel. The convolutional layer obtains refined information, thus
taking into account both operational efficiency and model performance. The
expression of the information refinement unit (IRU) can be written as

$IRU(X)=Conv(L(Conv(X))),$ (3)

where L(X)=WX+b. After designing the above three unit modules, the SSN block
can be formulated as:

$\begin{split}A&=RRU(X)\\\ B&=FMHSA(A)\\\ C&=IRU(B)+B\end{split}$ (4)

In the experiment part, we will prove the efficiency of SSN network.

##  IV Experiment

In this section, we investigate the effectiveness of the SSN architecture by
conducting experiments on an autonomous driving obstacle avoidance task based
on a driving map as the main input. We compare the proposed SSN with other
popular models before showing in Fig. 2, and then compare the experimental
results to draw an analytical conclusion. We defined three different types of
collisions which are front collision, rear collision and side collision. These
situations are caused by different unsuitable physic parameters and in Fig. 1.

###  IV-A Dataset and description

We use l5kit dataset [20] as our data source which contains over 1,000 hours
of data. This was collected by a fleet of 20 autonomous vehicles along a fixed
route in Palo Alto, California, over a four-month period. It consists of
170,000 scenes, where each scene is 25 seconds long and captures the
perception output of the self-driving system, which encodes the precise
positions and motions of nearby vehicles, cyclists, and pedestrians over time.
On top of this, the dataset contains a high-definition semantic map with
15,242 labeled elements and a high-definition aerial view over the area.

###  IV-B Data preprocessing

The data mainly includes the following main concepts: Scenes, Frames, and
Agents. A scene is identified by the host (i.e. which car was used to collect
it) and a start and end time. It consists of multiple frames (=snapshots at
discretized time intervals). The scene datatype stores reference to its
corresponding frames in terms of the start and end index within the frames
array (described below). The frames in between these indices all correspond to
the scene (including the start index, excluding the end index.

A frame captures all information that was observed at a time. This includes
the timestamp, which the frame describes; data about the ego vehicle itself
such as rotation and position; a reference to the other agents (vehicles,
cyclists and pedestrians) that were captured by the ego’s sensors; a reference
to all traffic light faces (see below) for all visible lanes. An agent is an
observation by the AV of some other detected object. Each entry describes the
object in terms of its attributes such as position and velocity, and gives the
agent a tracking number to track it over multiple frames (but only within the
same scene!) and its most probable label.

The input of this dataset is images of Ego car, which is one of properties of
Ego Dataset. And the output of our model are position and yaw which are
properties of EgoDataset as well. By this way, we can simulate vehicles’
driving as human driving actions. During human driving process, drivers
control accelerator and driving wheels to move vehicles, accelerator is used
for velocity and driving wheel for yaw. The output of our model is also
velocity and yaw. Thus, we use this method to simulate the trajectories of
vehicles so that we can change velocity and yaw to avoid collisions during
driving.

###  IV-C Result

The tables of test results which are processed by four different network
structures are shown in Tab. I. Compared with other transformer-based and
convolution-based counterparts, our model achieved better accuracy and faster
processing speed. In particular, our model achieves 2.6 times on front
collision which is 13.6 times less than RepVGG, 5.8 times less than ViT, and
12.6 times less than ResNet50, indicating the benefit of SSN block for
capturing both local and global information. We can see that SSN consistently
outperforms other models by a large margin.

TABLE I: Different collision times of four models per 10000 frames. Method |  Front |  Side |  Rear
---|---|---|---
L5Kit (ResNet-50) [20] |  15.2 |  20.7 |  8.3
RepVGG [8] |  16.2 |  11.7 |  10.6
Vit [16] |  8.4 |  7.7 |  9.2
SSN (Ours) |  2.6 |  13.3 |  3.5

##  V Conclusion

This paper proposes a novel hybrid architecture named SSN for vision-based
autonomous driving tasks and other vision tasks. The designed SSN
architectures take advantages of both CNNs and self-attention to capture local
and global information, improving the ability of the sequentially related
inputs. Extensive experiments on lykit dataset demonstrate the effectiveness
and superiority of the proposed SSN architecture.

## References

  * [1] K. L. Chung, “Markov chains,” _Springer-Verlag, New York_ , 1967.
  * [2] N. Metropolis and S. Ulam, “The monte carlo method,” _Journal of the American statistical association_ , vol. 44, no. 247, pp. 335–341, 1949.
  * [3] U. Z. A. Hamid, Y. Saito, H. Zamzuri, M. A. A. Rahman, and P. Raksincharoensak, “A review on threat assessment, path planning and path tracking strategies for collision avoidance systems of autonomous vehicles,” _International Journal of Vehicle Autonomous Systems_ , vol. 14, no. 2, pp. 134–169, 2018.
  * [4] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [5] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” California Univ San Diego La Jolla Inst for Cognitive Science, Tech. Rep., 1985.
  * [6] H. Yao, X. Tang, H. Wei, G. Zheng, Y. Yu, and Z. Li, “Modeling spatial-temporal dynamics for traffic prediction,” _arXiv preprint arXiv:1803.01254_ , vol. 1, p. 9, 2018.
  * [7] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International conference on machine learning_. PMLR, 2019, pp. 6105–6114.
  * [8] X. Ding, X. Zhang, N. Ma, J. Han, G. Ding, and J. Sun, “Repvgg: Making vgg-style convnets great again,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 13 733–13 742.
  * [9] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [10] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” _Neural computation_ , vol. 1, no. 4, pp. 541–551, 1989.
  * [11] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [12] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [13] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [14] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan _et al._ , “Searching for mobilenetv3,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 1314–1324.
  * [15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [17] X. Chu, Z. Tian, B. Zhang, X. Wang, X. Wei, H. Xia, and C. Shen, “Conditional positional encodings for vision transformers,” _arXiv preprint arXiv:2102.10882_ , 2021.
  * [18] M. A. Islam, S. Jia, and N. D. Bruce, “How much position information do convolutional neural networks encode?” _arXiv preprint arXiv:2001.08248_ , 2020.
  * [19] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” _arXiv preprint arXiv:1606.08415_ , 2016.
  * [20] J. Houston, G. Zuidhof, L. Bergamini, Y. Ye, L. Chen, A. Jain, S. Omari, V. Iglovikov, and P. Ondruska, “One thousand and one hours: Self-driving motion prediction dataset,” in _Conference on Robot Learning_. PMLR, 2021, pp. 409–418.