* Wang et al. (2021) Jialu Wang, Yang Liu, and Caleb Levy. 2021.  Fair classification with group-dependent label noise.  , 526–536 pages.
  * Wang et al. (2022c) Jialu Wang, Xin Eric Wang, and Yang Liu. 2022c.  Understanding Instance-Level Impact of Fairness Constraints.  , 23114–23130 pages.  https://proceedings.mlr.press/v162/wang22ac.html
  * Wang and Wang (2022) Xiuling Wang and Wendy Hui Wang. 2022.  Providing Item-Side Individual Fairness for Deep Recommender Systems.  , 11 pages.  https://doi.org/10.1145/3531146.3533079
  * Wang et al. (2020) Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. 2020.  Towards fairness in visual recognition: Effective strategies for bias mitigation.  , 8919–8928 pages.
  * Watkins (2020) Elizabeth Anne Watkins. 2020\.  Took a Pic and Got Declined, Vexed and Perplexed: Facial Recognition in Algorithmic Management.  , 6 pages.  https://doi.org/10.1145/3406865.3418383
  * Wei et al. (2021) Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. 2021\.  Learning with noisy labels revisited: A study using real-world human annotations.
  * Wexler et al. (2020) James Wexler, Mahima Pushkarna, Sara Robinson, Tolga Bolukbasi, and Andrew Zaldivar. 2020\.  Probing ML Models for Fairness with the What-If Tool and SHAP: Hands-on Tutorial.  , 1 pages.  https://doi.org/10.1145/3351095.3375662
  * Wick et al. (2019) Michael Wick, swetasudha panda, and Jean-Baptiste Tristan. 2019\.  Unlocking Fairness: a Trade-off Revisited.  https://proceedings.neurips.cc/paper/2019/file/373e4c5d8edfa8b74fd4b6791d0cf6dc-Paper.pdf
  * Wu and He (2022) Ziwei Wu and Jingrui He. 2022.  Fairness-Aware Model-Agnostic Positive and Unlabeled Learning.  , 11 pages.  https://doi.org/10.1145/3531146.3533225
  * Xie et al. (2017) Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. 2017.  Controllable Invariance through Adversarial Feature Learning.  https://proceedings.neurips.cc/paper/2017/file/8cb22bdd0b7ba1ab13d742e22eed8da2-Paper.pdf
  * Xu et al. (2021a) Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. 2021a.  To be robust or to be fair: Towards fairness in adversarial training.  , 11492–11501 pages.
  * Xu et al. (2021b) Xinyi Xu, Lingjuan Lyu, Xingjun Ma, Chenglin Miao, Chuan Sheng Foo, and Bryan Kian Hsiang Low. 2021b.  Gradient Driven Rewards to Guarantee Fairness in Collaborative Machine Learning.  , 16104–16117 pages.  https://proceedings.neurips.cc/paper_files/paper/2021/file/8682cc30db9c025ecd3fee433f8ab54c-Paper.pdf
  * Xu et al. (2022) Yilun Xu, Hao He, Tianxiao Shen, and Tommi S. Jaakkola. 2022\.  Controlling Directions Orthogonal to a Classifier.  https://openreview.net/forum?id=DIjCrlsu6Z
  * Yan and Zhang (2022) Tom Yan and Chicheng Zhang. 2022.  Active fairness auditing.  , 24929–24962 pages.  https://proceedings.mlr.press/v162/yan22c.html
  * Yang and Roberts (2021) Eddie Yang and Margaret E. Roberts. 2021.  Censorship of Online Encyclopedias: Implications for NLP Models.  , 12 pages.  https://doi.org/10.1145/3442188.3445916
  * Yang et al. (2020a) Forest Yang, Mouhamadou Cisse, and Sanmi Koyejo. 2020a.  Fairness with Overlapping Groups; a Probabilistic Perspective.  , 4067–4078 pages.  https://proceedings.neurips.cc/paper%5Ffiles/paper/2020/file/29c0605a3bab4229e46723f89cf59d83-Paper.pdf
  * Yang et al. (2020c) Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. 2020c.  Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy.  , 12 pages.  https://doi.org/10.1145/3351095.3375709
  * Yang et al. (2020b) Wanqian Yang, Lars Lorch, Moritz Graule, Himabindu Lakkaraju, and Finale Doshi-Velez. 2020b.  Incorporating Interpretable Output Constraints in Bayesian Neural Networks.  , 12721–12731 pages.  https://proceedings.neurips.cc/paper%5Ffiles/paper/2020/file/95c7dfc5538e1ce71301cf92a9a96bd0-Paper.pdf
  * Yao et al. (2021) Huihan Yao, Ying Chen, Qinyuan Ye, Xisen Jin, and Xiang Ren. 2021.  Refining Language Models with Compositional Explanations.  , 8954–8967 pages.  https://proceedings.neurips.cc/paper/2021/file/4b26dc4663ccf960c8538d595d0a1d3a-Paper.pdf
  * Yao and Huang (2017) Sirui Yao and Bert Huang. 2017.  Beyond Parity: Fairness Objectives for Collaborative Filtering.  https://proceedings.neurips.cc/paper/2017/file/e6384711491713d29bc63fc5eeb5ba4f-Paper.pdf
  * Yeom et al. (2018a) Samuel Yeom, Anupam Datta, and Matt Fredrikson. 2018a.  Hunting for discriminatory proxies in linear regression models.  , 4568–4578 pages.
  * Yeom et al. (2018b) Samuel Yeom, Anupam Datta, and Matt Fredrikson. 2018b.  Hunting for Discriminatory Proxies in Linear Regression Models.  https://proceedings.neurips.cc/paper/2018/file/6cd9313ed34ef58bad3fdd504355e72c-Paper.pdf
  * Yik et al. (2022) William Yik, Limnanthes Serafini, Timothy Lindsey, and George D Montañez. 2022.  Identifying Bias in Data Using Two-Distribution Hypothesis Tests.  , 831–844 pages.
  * Yona and Rothblum (2018) Gal Yona and Guy Rothblum. 2018.  Probably Approximately Metric-Fair Learning.  , 5680–5688 pages.  https://proceedings.mlr.press/v80/yona18a.html
  * Yurochkin et al. (2020) Mikhail Yurochkin, Amanda Bower, and Yuekai Sun. 2020.  Training individually fair ML models with sensitive subspace robustness.  https://openreview.net/forum?id=B1gdkxHFDH
  * Yurochkin and Sun (2021) Mikhail Yurochkin and Yuekai Sun. 2021.  SenSeI: Sensitive Set Invariance for Enforcing Individual Fairness.  https://openreview.net/forum?id=DktZb97%5FFx
  * Zafar et al. (2017) Muhammad Bilal Zafar, Isabel Valera, Manuel Rodriguez, Krishna Gummadi, and Adrian Weller. 2017\.  From Parity to Preference-based Notions of Fairness in Classification.  https://proceedings.neurips.cc/paper%5Ffiles/paper/2017/file/82161242827b703e6acf9c726942a1e4-Paper.pdf
  * Zhang et al. (2022) Angie Zhang, Alexander Boltz, Chun Wei Wang, and Min Kyung Lee. 2022.  Algorithmic Management Reimagined For Workers and By Workers: Centering Worker Well-Being in Gig Work.  , 20 pages.  https://doi.org/10.1145/3491102.3501866
  * Zhang et al. (2018) Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018\.  Mitigating Unwanted Biases with Adversarial Learning.  , 6 pages.  https://doi.org/10.1145/3278721.3278779
  * Zhang and Davidson (2021) Hongjing Zhang and Ian Davidson. 2021.  Towards Fair Deep Anomaly Detection.  , 11 pages.  https://doi.org/10.1145/3442188.3445878
  * Zhang and Bareinboim (2018) Junzhe Zhang and Elias Bareinboim. 2018.  Equality of Opportunity in Classification: A Causal Approach.  https://proceedings.neurips.cc/paper%5Ffiles/paper/2018/file/ff1418e8cc993fe8abcfe3ce2003e5c5-Paper.pdf
  * Zhang (2022) Marilyn Zhang. 2022\.  Affirmative Algorithms: Relational Equality as Algorithmic Fairness.  , 13 pages.  https://doi.org/10.1145/3531146.3533115
  * Zhang and Long (2021) Yiliang Zhang and Qi Long. 2021.  Assessing Fairness in the Presence of Missing Data.  , 16007–16019 pages.  https://proceedings.neurips.cc/paper/2021/file/85dca1d270f7f9aef00c9d372f114482-Paper.pdf
  * Zhao et al. (2020) Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J. Gordon. 2020\.  Conditional Learning of Fair Representations.  https://openreview.net/forum?id=Hkekl0NFPr
  * Zhu et al. (2022) Zhaowei Zhu, Tianyi Luo, and Yang Liu. 2022.  The Rich Get Richer: Disparate Impact of Semi-Supervised Learning.  https://openreview.net/forum?id=DXPftn5kjQK

Pipeline-Aware Taxonomy of Literature Survey

Stage | Step | Problem Identification | Measurement | Mitigation
---|---|---|---|---
Viability Assessments | Cost/Benefit |  |  |
| General |  (Raji et al., 2022), (Barocas et al., 2020), (Passi and Sengers, 2020) |  (Wang et al., 2022b), (Coston et al., 2022) |
Problem Formulation | Prediction Target |  |  (Black et al., 2022),(Obermeyer et al., 2019), (”N/A”, 2020),
---
(Pastaltzidis et al., 2022)
| (Benami et al., 2021)
| Predictive Attributes | (Grgić-Hlača et al., 2018) |  |
| General |  |  (Robertson et al., 2021), (Zhang et al., 2022), (Navarro and Shaer, 2022), (Nikkhah et al., 2022), (Passi and Barocas, 2019)
---
, (Chouldechova et al., 2018), (Jacobs and Wallach, 2021b), (Cheng et al.,
2021b), (Ilvento et al., 2020), (Lum and Isaac, 2016)
|
Data Collection | Sampling |  |  (Buolamwini and Gebru, 2018), (Yang and Roberts, 2021), (Suresh et al., 2022), (Pahl et al., 2022),
---
, (Cai et al., 2022)
(Mishler, 2019), (Yik et al., 2022) |  |  (Sharma et al., 2020b), (van Breugel et al., 2021), (Beretta et al., 2021), (Tsai et al., 2022), (Shekhar et al., 2021), (Roh et al., 2021b)
---
| Annotation |  |  (Wei et al., 2021), (Wang et al., 2021), (Sachdeva et al., 2022), (Christoforou et al., 2021)
---
(Liu and Wang, 2021), (Wang et al., 2022a), (Birhane et al., 2021) | (Yang et al., 2020c)
| Feature Measurement |  |  (Wan, 2022), (Pandey and Caliskan, 2021), (Frye et al., 2020)
---
(Swinger et al., 2019) |
| Record Linkage |  |  |
| General |  |  (Chen et al., 2021), (Hettiachchi et al., 2021), (Sambasivan et al., 2021), (Park et al., 2021b),
---
(Kallus and Zhou, 2018), (Marda and Narayan, 2020)
|  (Li et al., 2022b), (Ding et al., 2021), (Zhang and Long, 2021), (Park et
al., 2021a), (Ji et al., 2020),
---
(Kallus et al., 2020), (Akpinar et al., 2021), (Bolukbasi et al., 2016),
(Kamiran and Calders, 2012)
(Chen et al., 2018), (Gebru et al., 2018), (Boyd, 2021)
Data Preprocessing | Feature Creation |  |  |
| Feature Selection |  (Belitz et al., 2021b), (Yeom et al., 2018b), (Black et al., 2019), (Leino et al., 2019) |  (Belitz et al., 2021b), (Tan and Celis, 2019) |  (Yeom et al., 2018b), (Fong et al., 2022), (He et al., 2020)
| Data Cleaning (Omission) |  |  (Xie et al., 2017), (Breck et al., 2019) |  (Calmon et al., 2017), (Zhang and Davidson, 2021), (Hynes et al., 2017), (Krishnan et al., 2016)
| General |  |  (Caliskan et al., 2022), (Maity et al., 2021a), (Liang et al., 2021), (Biswas and Rajan, 2021)
---
|  (Cousins, 2022), (Sharaf et al., 2022), (Choi et al., 2020), (Shui et al.,
2022),
---
(Wang et al., 2022c), (Ganev et al., 2022)
|  (Yao and Huang, 2017), (Oneto et al., 2020), (Li et al., 2021c), (Du et
al., 2021),
---
(Sweeney and Najafian, 2020), (Papakyriakopoulos et al., 2020), (Lucchesi et
al., 2022), (Creager et al., 2019),
(Buyl and De Bie, 2020), (Celis et al., 2020), (Tr”̈auble et al., 2021), (Kim
et al., 2022),
Statistical Modeling | Hypothesis Class |  (Black and Fredrikson, 2021b), (Nanda et al., 2021), | (Zhu et al., 2022) | (Li et al., 2022c)
| Optimization Function |  |  (Pujol et al., 2020), (Suriyakumar et al., 2021), (Hashimoto et al., 2018), (Dasgupta et al., 2022),
---
(D’Amour et al., 2020) |  |  (Xu et al., 2021a) , (Russell et al., 2017), (Ali et al., 2019),
---
(Current et al., 2022), (Roh et al., 2021b), (Romano et al., 2020),
(Sharifi-Malvajerdi et al., 2019), (Chzhen et al., 2020b), (Cho et al., 2020),
(Roh et al., 2021a),
(Louppe et al., 2017), (Li et al., 2021a), (Xu et al., 2021b),
| Regularizers |  |  |  (Kancheti et al., 2022), (Shah et al., 2022)
| Hyperparameters | (Matthews et al., 2020) |  |  |  (Venkatesh et al., 2021),
---
(Sikdar et al., 2022),(Schmucker et al., 2020), (Islam et al., 2021),(Perrone
et al., 2021)
| General |  |  (Ma et al., 2021), (Ghazimatin et al., 2022), (Lohaus et al., 2020),
---
(Coston et al., 2021), (Beutel et al., 2019)
(Akpinar et al., 2022b), (Bagdasaryan et al., 2019), (Singh, 2022)
|  (Navon et al., 2021), (Chen et al., 2019), (Kulynych et al., 2020),
---
(Wang and Russakovsky, 2021) , (Ma et al., 2022)
|  (Yao et al., 2021), (Lamy et al., 2019), (Ruoss et al., 2020),
---
(Li et al., 2021b), (Balunovic et al., 2022), (Dwork et al., 2018),
(Kamishima et al., 2018), (Burke et al., 2018), (Mehrotra and Celis,
2021),(Nandy et al., 2022),
(Nabi et al., 2019), (Foster et al., 2022),
(Zhang et al., 2018), (Kim et al., 2019), (Sharma et al., 2021), (Halevy et
al., 2021)
(Lahoti et al., 2020), (Petersen et al., 2021), (Zhao et al., 2020),
(Salvador et al., 2022), (Wang and Wang, 2022), (Mishler and Kennedy, 2022),
(Wu and He, 2022)
(Agarwal et al., 2018), (Kilbertus et al., 2018), (Madras et al., 2018a),
(Chuang and Mroueh, 2021),
(Celis et al., 2019), (Garg et al., 2019), (Komiyama et al., 2018), (Oneto et
al., 2019),
(Ali et al., 2021), (Grabowicz et al., 2022), (Fish and Stark, 2022), (Chen
and Hooker, 2020), (Thomas et al., 2021), (Quadrianto and Sharmanska, 2017),
(Kusner et al., 2017)
(Donini et al., 2018), (Heidari et al., 2018), (Bower et al., 2021),(Vargo et
al., 2021),
(Yurochkin and Sun, 2021), (Xu et al., 2022), (Metevier et al., 2019),
(Narasimhan et al., 2020),
(Cayci et al., 2020), (Singh et al., 2021b), (Elzayn et al., 2019), (Kearns et
al., 2018),
(Yona and Rothblum, 2018), (Cotter et al., 2019), (Goel et al., 2018), (Raff
et al., 2018),
(Liu et al., 2021), (Shah et al., 2021), (Leqi et al., 2019), (Chzhen et al.,
2020a),
(Bechavod et al., 2020), (Do et al., 2021), (Mishler et al., 2021), (Usunier
et al., 2022),
(Hebert-Johnson et al., 2018), (Agarwal et al., 2019), (Wick et al., 2019),
(Farnadi et al., 2018), (Har-Peled and Mahabadi, 2019), (Baharlouei et al.,
2020), (Yurochkin et al., 2020), (Meyer et al., 2021),
Testing and Validation | Train-test split |  |  |
| Evaluation Metrics |  |  (Hiranandani et al., 2020), (Dullerud et al., 2022), (Black et al., 2022), (Zhang, 2022), (Khalili et al., 2021),
---
|  (Michalsky, 2019), (Sharma et al., 2020a), (Diana et al., 2021), (Kim et
al., 2018),
---
(Zhang and Bareinboim, 2018), (Kallus and Zhou, 2019), (Smith et al., 2020),
(Maity et al., 2021b)
(Jiang et al., 2022), (Herington, 2020), (Cousins, 2021), (Coston et al.,
2020), (Garg et al., 2019), (Mutlu et al., 2022), (Awasthi et al., 2021),
(Taskesen et al., 2021), (Roth et al., 2022), (Yan and Zhang, 2022),
|  (Moyer et al., 2018), (Mhasawade and Chunara, 2021), (Zafar et al., 2017),
(Kilbertus et al., 2017), (Canetti et al., 2019), (Chzhen et al., 2019),
---
(Balcan et al., 2019), (Yang et al., 2020a), (Chen et al., 2020), (Yang et
al., 2020b),
(Huang et al., 2021), (Bendekgey and Sudderth, 2021)
| General |  |  (Olofsson et al., 2018), (Si et al., 2021)
---
(Black et al., 2020), (Lum et al., 2022), (Bellamy et al., 2018), (Hardt et al., 2021), (Paullada et al., 2021), (Saleiro et al., 2018), (Kim et al., 2020) |  |  (Coston et al., 2019b), (Dixon et al., 2018),
---
(Cheng et al., 2021a), (Wang et al., 2019),
Deployment and Integration | Human-Computer Handoff |  |  (Jacobs et al., 2021), (Keswani et al., 2021), (Green and Chen, 2019a), (McLaughlin et al., 2022), (Donahue et al., 2022b),
---
(Leung et al., 2020), (Ma and Gajos, 2022), (Watkins, 2020)
(Cheng et al., 2022) | (Anik and Bunt, 2021)
| Maintenance |  |  |
| Oversight |  |  (Albarghouthi and Vinitsky, 2019), |
| General |  |  (Sendak et al., 2020), (Lum et al., 2020), (Akpinar et al., 2022a), (Park et al., 2022),
---
(Madaio et al., 2020), (Green and Chen, 2019b), (Green and Chen, 2019a),
(Cheng et al., 2022), (De-Arteaga et al., 2020)
|  (Ghosh et al., 2022), (Giguere et al., 2022),(Slack et al., 2020), (Reader
et al., 2022),
---
(Hinder et al., 2020), (Biswas and Mukherjee, 2021)
|  (Nasr and Tschantz, 2020), (Singh et al., 2021b), (Almuzaini et al., 2022),
(Noriega-Campero et al., 2019), (Suresh and Guttag, 2021), (Madras et al.,
2018b), (Coston et al., 2019a)
---

Table 1. A taxonomy of the papers surveyed into the various sections of the ML
pipeline they study, and whether they identify, measure, or mitigate a source
of bias. The pink papers correspond to case studies within problem
identification. Yellow colored references denote papers that correspond to
more traditional approaches to fairness, i.e. imposing a fairness constraint
on top of a pre-made modeling process, or introducing a new notion of fairness
in the testing and evaluation section.

##  5\. Classroom Study

Here, we present further details on the population and sampling and study
design of the classroom study in Section 2.2.

Population and sampling.  In total, 37 students participated in the class
activity. Prior completion of at least one Machine Learning course was a
prerequisite for enrollment. So all participants had a non-negligible
background in Machine Learning. Specifically, $\sim 80\%$ characterized the
familiarity with ML as intermediate, $\sim 14\%$ as advances, and $\sim 5\%$
as elementary. All students enrolled in the course took it as an elective.
This fact implies that compared to a random sample of students with ML
knowledge, our participants were likely more aware of ML harms and more
motivated to address them.

Study design. Students were introduced to the ML pipeline through an
approximately 45-minutes long lecture. The lecture focused on the supervised
learning paradigm and broke down the supervised learning pipeline into the
following five stages:

  1. (1)

Problem Formulation corresponding to the choice of features and predictive
targets.

  2. (2)

Data collection and processing corresponding to the choice and processing of
the input data $D=\\{(x_{i},y_{i})\\}_{i=1}^{n}$

  3. (3)

Model specification corresponding to the choice of the hypothesis class, $H$.

  4. (4)

Model fitting/training: choosing a specific loss function, $L$, and optimizing
it via existing optimization techniques to obtain a model $h\in H$

  5. (5)

Deployment in real-world, corresponding to the deployment of $h$ in real-world
practice, and its predictions $\hat{y}$ translating into decisions that can
harm/benefit certain individuals and communities

For each stage, students were presented with multiple examples of how choices
at that stage can lead to harmful outcomes, such as unfairness, at the end of
the pipeline (see Figure 1). They were then asked to team up with 3-4
classmates and pick a societal domain as the focus of their group activity.
Examples offered to them included employment (e.g., hiring employees);
education (e.g., admitting students); medicine (e.g., diagnosing skin cancer);
housing (e.g., allocating limited housing units child welfare (e.g.,
investigating referral calls); criminal justice (e.g., pretrial sentencing);
public safety (e.g., allocating patrol resources in policing e-commerce (e.g.,
advertising; ranking sellers on Amazon); social media (e.g., news
recommendation; content moderation); and transportation (e.g., autonomous
vehicles). One team suggested finance (e.g., credit lending) as their topic.
Third, students were given 30 minutes to discuss the following questions about
their application domain with teammates and submit their written responses
individually:

  * •

Characterizing the specific predictive task their team focused on.

  * •

The type of harm observed.

  * •

Their hypotheses around the sources of this harm in through the ML pipeline.

  * •

Theirhypotheses around potentially effective remedies for addressing those
sources.

We present an overview of the application domains that students discussed, and
how they defined the predictive task in their area. Our findings are discussed
in Section 2.2.

Table 2. An overview of the predictive tasks students chose to analyze through a pipeline-centric view of the ML pipeline. Domain | Students | Predictive Task |
---|---|---|---
Child Welfare | 3 |  | Predicting the risk of a child running away from their foster care within 90 days of being in the child welfare system,
---
based on a combination of demographic and clinical characteristics, and
information about them in the welfare system.
Education | 4 |  | Predicting whether an individual is cheating in an online exam
---
based on visual and auditory cues (e.g., irregular eye/body movement, mouse
movement, facial expression, and noise).
Employment | 6 | Predicting whether a company will hire an applicant based on the information in their resume. |
Finance | 3 | Assessing the credit worthiness of individuals using their demographic & socio-economic data. |
Housing | 3 | Predicting the value of a real-estate properties |
Medicine | 8 |  | 1) Predicting which people will/should receive an organ transplant.
---
2) Predicting prognosis (e.g., mortality) for comatose patients post-cardiac
arrest
using demographic, medical history, and medical screening data (e.g., CT scan,
EEG data)
Public Safety | 4 |  | 1) Allocating police presence to high risk areas;
---
2) Identifying suspicious individuals given a list of wanted criminals.
Social Media | 4 | Predicting whether a news is fake. |
Transportation | 2 | Detecting objects (e.g., human, vehicles, road conditions) for AVs. |

##  6\. Research Agenda

Reflecting on the state of fairness literature from a pipeline-aware
perspective, we offer five key insights to start operationalizing a pipeline-
aware approach to fairness. Namely, we identify (1) the need to investigate
and document choices made along real-world pipelines, including those related
to bias mitigation; (2) the need to bridge the gap across literatures which
_identify_ ways the bias enters ML pipeliens on-the-ground, such as HCI, and
literatures which build operational mitigation techniques, such as FairML; (3)
the need to study interaction effects across decisions made along the
pipeline; (4) the need to address holes in the current research–paying
attention to neglected areas of the pipeline such as viability assessment, and
entire modes of research such as producing _measurement_ techniques to catch
many of the entry points for bias identified along the pipeline; and (5)
finally, the need to produce guidance on how to choose among several biased
building choices.

Investigation of Real-World Pipelines The algorithmic fairness literature has
a dearth of knowledge about what on-the-ground pipelines look like. Although
many papers outline abstracted pipelines (including our own), the actual model
creation steps taken by practitioners for a variety of real-world systems
(e.g. in consumer finance, healthcare, hiring systems) are at the very least
not cataloged in any centralized place. Bias entering from choices along the
pipeline cannot be studied, measured, and mitigated if we do not know what
choices are being made—thus mapping out actual pipeline choices is a necessary
step to operationalize a pipeline-aware approach to fairness.
Action Items: (1) We encourage exploration and documentation of pipeline
decisions made in a variety of machine learning pipelines in different
applications; and (2) centralization of this information so that researchers
can study how pipeline decisions they may have been unaware of impact fairness
behavior.

Disconnect between Problem Identification and Mitigation Bias problems are
often discovered in disciplines on the fringe or outside of machine learning,
such as human-computer interaction literature and database management
(Sambasivan et al., 2021). Since many papers which point to problems along the
AI pipeline, especially in the earlier stages, are structured around
interviews—they may elide more low-level technical sources of pipeline choices
leading to bias, e.g. such as record linkage issues leading to non-IID data
dropping. While they are helpful in the important first step of identifying
problems, they provide little direction for creating operationalizable
solutions to these problems, or often even sufficiently detailed information
about each system studied to be able to understand the mapping between data
collection problems and model behavior.

Symmetrically, many papers in the mainstream algorithmic fairness literature
do not test their techniques on real decision-making systems–or even on
datasets beyond Adult, German Credit, and a few others. Though case studies
may often be disregarded as implementing old methods and thus not novel, it is
crucial that we give them more attention so that we learn about the failure
modes of the techniques that we create. While there are papers about the
overall problems production teams face when implementing fairness goals, e.g.
(Richardson et al., 2021; Madaio et al., 2020; Holstein et al., 2019;
Sambasivan et al., 2021), these do not provide an in-depth catalogue of the
successes and failures of all pipeline intervention techniques in practice.
Proposed methods to intervene on the machine learning pipeline should be
tested in a variety of real-world systems to see where they fail and succeed.
Action Items: (1) We encourage bridging the problems identified by the HCI,
CSCW, and other literatures, with the technical detail of the algorithmic
fairness literature to introduce mitigation techniques for data collection
harms; (2) In particular, we encourage AI Fairness researchers to build off of
tools for addressing generalized data and modeling pipeline issues, i.e. not
specialized to fairness problems, and adapt them for debiasing ML systems–for
example, extending Breck et al. (Breck et al., 2019)’s data validation
pipeline to address fairness concerns; (3) We encourage testing the pipeline-
based bias mitigation methods proposed to date in on-the-ground ML systems.
This is necessary to uncover which perform best under various circumstances,
determine failure modes, and learn how to integrate these techniques into
actual ML systems.

Interaction Effects Most fairness-related papers focus on _one issue_ in the
pipeline: they present a bias problem, then often give a mitigation method,
implicitly assuming that the identified source of bias is the only one of
interest to the model practitioner, and the suggested intervention will not
lead to other effects on the model, including other forms of bias. That is,
much of the literature fails to provide insight into how different biases and
mitigation techniques along the pipeline interact. Does solving each issue in
isolation work, or does a pipeline-aware approach to fairness need to engage
with interaction effects to work in practice? There are a few papers that show
the impacts of the intersection between multiple sources of bias has on the
effectiveness of interventions–e.g., Li et al. (Li et al., 2022a) point to how
active sampling techniques to address sampling bias can make models _more_
unfair when label bias is present. However, any set of choices on the machine
learning pipeline may have interaction effects, suggesting many paths for
exploration. Beyond studying interaction effects, there are very few papers
(with some exceptions (Suresh et al., 2022; Katell et al., 2020)) which engage
with the entire pipeline: taking fairness into account when making every
modeling decision, and testing along the way—which is the end goal of a
pipeline-based approach to fairness.
Action Items: We recommend (1) investigating interaction effects of various
sources of bias, and bias mitigation strategies across the ML pipeline, and
(2) developing tools that allow for such exploration (for example, extending
(Akpinar et al., 2022b).)

Holes in Research: Measurement Methods and Others Within the few papers that
do discuss pipeline-aware approaches to fairness, most are problem
identification or mitigation techniques— only 17%999We calculate this by
taking all of the measurement papers identified (67) and subtracting the
number that simply introduce new fairness metrics (15), and dividing this
number (52) by the total number of pipeline fairness papers identified. of the
papers that we identified are actually providing techniques to measure whether
or not a given pipeline choice will lead to downstream unfairness ex-ante. But
measurement is key because it may allow us to decide when or when not to take
an action. How can we effectively measure algorithmic harms? Relatedly, there
are many proposed solutions of how to solve a problem once it has already
happened–e.g. unrepresentative data, etc.—but how can we develop tests to
_prevent_ choices that lead to algorithmic harms? For example, can we predict
beforehand whether and how fairness problems will result from the way in which
a model is integrated into a given decision structure? Additionally, we find
that research on _how_ bias can enter machine learning decision systems via
choices made along the pipeline is unevenly distributed across the pipeline.
There are some areas of the pipeline that are well-studied in this regard,
such as data collection; but the majority of the pipeline has light-to-no
coverage: e.g. viability assessments, problem formulation, and large parts of
organizational integration. These areas must be prioritized for searching for
fairness failure modes and mitigation techniques. Finally, the majority of the
research to date measures the effect of various pipeline choices on common
definitions, e.g. demographic disparity, accuracy disparity, etc. How can we
map the pipeline in a general enough fashion that we may be able to understand
how more contextualized notions of fairness are impacted by pipeline
decisions?
Action Items (1) We encourage building methods for _measuring_ the harms
introduced by decisions along the machine learning pipeline; (2) Exploring
pipeline decisions that have received little attention; and (3) exploring the
fairness effects of pipeline decisions beyond the most common metrics.

Guidance on Choosing Among Imperfect Design Alternatives ML practitioners
often face choices between imperfect/biased alternatives. It will almost
always be the case that with adequate effort, they can produce fairer, but not
perfectly fair models. Any choice a designer makes will likely lead to some
form of bias, some more and some less problematic in the given decision-making
domain. However, there is little guidance in the literature on deciding when
one alternative is better than another in light of all contextual
considerations, when certain biases are conditionally acceptable, and how the
answers to these questions depend on the context.
Action Items: (1) Developing normative guidelines to weigh a variety of
imperfect/biased design choices against each other is a crucial avenue for
collaboration between ML experts and ethicists; (2) Exploring how different
types of bias are currently traded off in practice when they are discovered;
and (3) Understanding if there are any relationships or couplings of different
sources or forms of bias that often go together, or have opposing
relationships.

##  7\. Pipeline Cards

Here, we introduce _pipeline cards_ , a framework for interrogating the
machine learning development pipeline in order to search for and mitigate
sources of bias.

###  7.1. Viability Assessment

####  7.1.1. Fit: Is a machine learning system a good solution for the
problem?

  * •

What are the policy or business goal(s)?

    * –

What tensions or trade-offs might exist between these goals?

  * •

How can introducing an ML model promote that goal?

    * –

What data is accessible for the project– is there a sufficiently good proxy
available for the desired prediction task? (Raji et al., 2022)

  * •

What alternatives are there to introducing a new ML system? What are the
benefits and downsides of these other approaches when compared to making an ML
system?

####  7.1.2. Capacity: Can the organization build and maintain the system?

  * •

Is there organizational capacity to build and maintain this algorithm (data,
expertise, budget)?

  * •

Does the project have community and organizational buy-in? What is the
evidence to that effect?

###  7.2. Problem Formulation

####  7.2.1. Prediction Task

  * •

What is my numerical proxy that will be used for prediction? How does it
relate to the policy goal?

    * –

Do I have any alternatives? Why have I chosen the one I did?

    * –

Is my proxy valid? (Coston et al., 2022) Some ways to test:

      * *

Is the proxy measuring a facially different outcome or behavior (e.g. arrests
instead of convictions in a pretrial risk assessment)?

      * *

Is there ground truth data about the behavior or outcome that the model is
attempting to predict? Does the proxy correlate well with that information?

      * *

If there are multiple possible prediction proxies available, are they all
highly correlated?

    * –

Is the proxy equally viable and closely related for the entire population the
model will be used over?

####  7.2.2. Input Features

  * •

What input features are being included to predict the proxy?

    * –

Is there reason to believe some of these input features will be of varying
quality across different demographic populations?

####  7.2.3. Problem Universe

  * •

Is this task being performed on the correct population in order to solve the
policy goal? (E.g. is predicting likelihood of dropout for educational
intervention better to do on 10th grade or 11th grade students?)

  * •

Are some groups disproportionately under-represented in the universe
definition relative to others?

###  7.3. Data Collection

####  7.3.1. General

  * •

What is the relevant population for the project (demographically, temporally,
geographically)? Is there access to data over that entire population?

    * –

How might some individuals or groups be left out of this data?

  * •

Does the data have to be i.i.d. (randomly sampled)? How far from a random
sample is the data being collected or used?

    * –

E.g. Is the data only available based on condition? (e.g., only those who
filled out information release form when applying for loan can be a part of
dataset)

####  7.3.2. Collecting Own Data

  * •

Refer to Datasheets for Datasets (Gebru et al., 2018).

####  7.3.3. Using available data

  * •

Does the data have information that is likely to be of higher quality for some
groups (e.g. home address?)

  * •

What are the potential mechanisms that could lead to divergent data quality
across groups?

  * •

How is data being linked across datasets?

    * –

Are the processes for matching individuals across data sources equally
accurate across different populations?

      * *

For instance, married vs maiden names may bias match rates against women,
while inconsistencies in handling of multi-part last names may make matching
less reliable for hispanic individuals.

    * –

A data loading process that drops records with “special characters” might
inadvertently exclude names with accents or tildes.

###  7.4. Data Preprocessing

  * •

How is missing or damaged data being handled?

    * –

Are some groups affected by these data issues more than others?

    * –

Data be imputed instead? If so, how might this process impact different
demographic groups differently?

  * •

What data transformations are being used? Might they have different effects or
accuracy across different groups?

    * –

E.g., Using data transformations with differential error rates across groups
(e.g., using word embeddings trained on an English corpus for user-generated
text that may contain submissions in other languages)

  * •

How are features built from the data? How are they encoded?

    * –

Might some of these definitions or encodings have differential meaning across
groups (e.g., distance-based features relative to a home address that is less
likely to be stable/current for more vulnerable populations)

###  7.5. Statistical Modeling

  * •

What model type (hypothesis class)?

    * –

How might it impact different demographic groups differently—e.g. choosing a
learning rule that over- or under-emphasizes outliers or minority populations
(e.g. differentially private learning rules under-emphasize minority
populations (Bagdasaryan et al., 2019))

  * •

What is the learning rule? Might it have an inductive bias that benefits or
harms a portion of the data population?

  * •

What is the loss function and how well does it reflect the project’s goals?
Should any additional constraints or incentives be added?

###  7.6. Testing and Validation

####  7.6.1. Metrics

  * •

What metrics determine “model performance” in the pipeline? How is one model
chosen over another?

    * –

Is any notion of fairness being considered? (E.g. even differential accuracy?)

    * –

How are hyperparameters chosen– can behaviors beyond accuracy be included as a
tiebreaker?

    * –

How is the robustness of these metrics being ensured? Am I performing trials,
or using cross-validation?

####  7.6.2. Representative Testing Data

  * •

Will performance be tested on data that is from the same distribution as my
deployment population? (e.g. is a credit risk assessment system tested on US
data to be used in the UK?)

####  7.6.3. Representative Testing Environment

  * •

In addition to model-only testing, will the model be tested in the full
decision system into which it will be embedded (i.e. with human components as
well)?

###  7.7. Deployment and Monitoring

####  7.7.1. Deployment

  * •

How will the model be used as a component of the decision system into which it
is embedded?

    * –

Will the model’s predictions directly become the final decision?

    * –

If there is human involvement, where and how will that occur?

    * –

How much discretion do humans have over adhering to model recommendations?

    * –

How are model predictions communicated to decision-makers? Is there reason to
believe that the way results are communicated will result in bias towards a
particular group?

    * –

Is there potential for disparities arising from the downstream actions the
model is informing (e.g., english-only outreach calls, after school programs
that might not be accessible to students with family obligations, etc)?

####  7.7.2. Monitoring

  * •

Will my system be monitored? What conditions will it monitor (accuracy,
fairness)?

    * –

How will errors or issues identified by this monitoring be integrated into
system updates or retraining?

  * •

Under what conditions will the monitoring system trigger a warning, or even
shut the system down? What are the organizational processes for these warnings
and/or shutdowns?

  * •

What avenues do affected individuals have to challenge decisions or
identify/correct errant data?

###  7.8. Other Aspects of the Pipeline

  * •

What other design decision points exist in my pipeline? What are the decisions
I’ve made at those junctures? How might they affect the model’s fairness
behavior?


# Empowering the trustworthiness
of ML-based critical systems
through engineering activities

Juliette MATTIOLI1, Agnès DELABORDE2,3, Souhaiel KHALFAOUI 4,3, Freddy LECUE1,
Henri SOHIER3 and Frédéric JURIE5,3

1Thales, France
2Laboratoire National de métrologie et d’Essais LNE
3IRT SystemX, France
4Valeo, France
5Caen University, France

This paper reviews the entire engineering process of trustworthy Machine
Learning (ML) algorithms designed to equip critical systems with advanced
analytics and decision functions. We start from the fundamental principles of
ML and describe the core elements conditioning its trust, particularly through
its design: namely domain specification, data engineering, design of the ML
algorithms, their implementation, evaluation and deployment. The latter
components are organized in an unique framework for the design of trusted ML
systems.1[] This paper reviews the entire engineering process of trustworthy
Machine Learning (ML) algorithms designed to equip critical systems with
advanced analytics and decision functions. We start from the fundamental
principles of ML and describe the core elements conditioning its trust,
particularly through its design: namely domain specification, data
engineering, design of the ML algorithms, their implementation, evaluation and
deployment. The latter components are organized in an unique framework for the
design of trusted ML systems.1

##  1 Introduction

Machine learning (ML) models are becoming inevitable components of Artificial
Intelligence (AI) systems, including systems that require safety-critical
environmental perception and decision-making. ML engineering [Treveil et al.,
2020, Serban et al., 2021] is a new field leading to new issues and forcing
companies to adapt their engineering practices and processes: 1) classic
considerations on specification, traceability and validation are deeply
challenged [Bosch et al., 2021, Ozkaya, 2020]; 2) processing data in ML
algorithms requires new processes with new best practices [Zinkevich, 2017],
as highlighted by ML Model Operationalization Management (MLOps) approaches;
3) advanced perception and complex decisions of a ML system must present new
assesses trustworthiness through security, privacy, safety, explainability,
etc. and other attributes related to specific concerns such as application
domain concerns. To maximize the trustworthiness of ML-based critical systems,
such attributes – and the methods for concretely assessing their values – must
be clearly identified and mapped onto the ML processes and its lifecycle.

This paper presents ML algorithm engineering as a pipeline of processes and
details the main challenges for reaching trustworthiness in the development
procedures of an industrialized ML component. This paper is a result of the
first year of the Confiance.ai program [Braunschweig et al., 2022, Chiaroni et
al., 2021] which gathers multiple companies and research centers from
different industries working on the development of trustworthy safety- and
business-critical systems at scale.

##  2 ML Algorithm Engineering

Algorithm engineering (AE) refers to the process required to bridge the gap
between algorithm and concrete implementation in a dedicated programming
paradigm to yield efficient, easily usable and well-tested implementations.
This process encompasses a number of topics such as algorithm design,
theoretical analysis, implementation, tuning, debugging evaluation,
validation, and testing. Thus, AE is a methodology that combines theory with
implementation and validation conducted through experimentation.

Figure 1: AE process focusing on problem scoping, algorithm design and
delivery

In essence, MLOps entails a set of practices and tools focused on software and
systems engineering with close collaboration between ML developers, operation
and engineering teams to improve quality of services while ML Engineering is
often portrayed as the creation of a ML model, its fine-tuning, validation and
deployment.

In real-world industrial settings, the ML model is only a small part of the
overall system and significant additional engineering and system
functionalities are required to ensure that the ML model can operate in a
reliable, predictable and scalable way with proper engineering of data and
model pipelines, monitoring and logging, etc. To capture such issues, we
present a ML algorithm engineering pipeline (see fig. 2), where we distinguish
requirements-driven development, safety-driven development and ML-driven
development. At the starting point, data must be available in size but also
minimally prepared, validated and representative of the task at hand for
training.

Figure 2: Machine Learning Algorithm Engineering Pipeline

Main subtasks are encapsulated as a series of steps within the pipeline such
as:

  * •

Problem specification: It results in functional and non-functional
requirements covering every aspect of the ML item: safety, performance,
Operational Design Domain (ODD), etc. The ODD is the description of the
specific operating condition(s) in which a safety-critical function or system
is designed to properly operate as expected, including but not limited to
environmental conditions and other domain constraints [Koopman and Fratrik,
2019]. This usually drives the data collection task.

  * •

Data engineering: A ML model requires large amounts of data, which helps the
model learn how to perform its purpose. Before it can be used, data need to be
cleaned, organized, analyzed and visualized to support feature engineering.
Data acquisition is the process of aggregating data into a homogeneous set.
Among other properties, the collected data need to be sizable, accessible,
understandable, relevant, reliable, and usable. Data preparation, or data
processing, is the process of transforming raw data to make it usable for the
model’s purpose. Thus, one of the key challenges is to establish data sets
that are of sufficient quality for training and inference. The importance of
this task is highlighted by the data-first ML movement [Zhou et al., 2021].

  * •

ML Algorithm Design: The ML algorithm has to be designed or selected from
existing ML libraries. By feeding a training set to the ML algorithm, it can
learn appropriate parameters and features. Once training is complete, the
model will be refined by using the validation dataset. This may involve
modifying or discarding variables and includes a process of tweaking model-
specific settings (hyperparameters) until an acceptable accuracy level is
reached. Different models can be employed, validated and fine-tuned.

  * •

Implementation: To develop a ML component, one has to decide on the targeted
hardware and system platform, the IDE (Integrated Development Environment) and
the language for development. These choices can typically impact the time-
behavior or the power consumption. Embedded systems can be highly constrained.

  * •

Evaluation and verification: Finally, after an acceptable set of
hyperparameters is found and the model accuracy is optimized, the model is
tested on a data set and/or assessed through formal verification. The
evaluation can go beyond functional performance (such as accuracy) and
encompass metrics relative to any other expected performance criteria. Example
of such metrics are: explainability, interpretability, biais. Based on the
feedback, one may return to training the model to adjust performance through
output settings, or deploy the model as needed.

  * •

Model Deployment: The integration of the ML model as a component in the
overall system requires a tuning to the system characteristics, or indirectly
to the environment of deployment; this adaptation can imply additional
iterations of specification, development and testing. In addition, one should
ensure that the model, once deployed, is monitored, that maintenance tasks can
be performed, and that the model can be adapted to the evolution of the
environment of deployment.

It is crucial to consider the evolution of the data, and its model for
continuous deployment, as data might change overnight, then impacting the
performance of models. Continuous integration, development, deployment and
testing need to be always in the loop for MLOps.

The following paragraphs further detail the important subtasks of the ML
algorithm engineering pipeline. However, we do not describe in more details
the implementation of the models nor their deployments, as these tasks are
very dependent on the computing targets.

##  3 Problem specification

The first step of ML algorithm engineering process is to state the problem
precisely. Operational requirements at the system level (i.e. where the system
is considered as a black box) are derived into functional and physical
requirements at the ML component level. The resulting requirements can for
example be related to the component’s functional performance, safety,
integrability or maintainability. They can also for example combine customer
requirements, operational constraints, regulatory restrictions, or
implementation realities. In all cases, the functions should be evaluated for
their safety related attributes.

Ideally, the design of a component should be based on a clear input domain and
a clear input-to-output relation. However, in ML, the traditional programming
paradigm is no longer suitable: 1) the environment’s complexity is often
difficult to reduce to a clear input domain; 2) instead of hard coding a clear
input-to-output relation, one provides examples of inputs and outputs to a
machine to generate the algorithms.

The requirements have to be refined and completed up to the point where they
allow the development of the ML-based component. In particular, in the case of
supervised machine learning, data could be considered as detailed requirements
of the intended behavior of the ML-based component. Similarly, the structure
of the ML model, its parameters and hyper-parameters could also be considered
as detailed requirements for the ML-based component. The idea of the ODD
[Gyllenhammar et al., 2020] can be used to indicate where ML-based critical
systems can operate safely. In the automotive domain, an example of ODD are
the closed roads, weather conditions, and presence of pedestrians or animals,
etc.

Furthermore, ML Model requirements should express the expected properties of
the ML Model with their acceptable tolerances. The ML Model specification
activity steps are:

  * •

ML Data Requirements are developed from the analysis of the subsystem
requirements (including the ODD).

  * •

ML Model requirements that specify nonfunctional requirements like performance
objectives are stated in quantitative terms with tolerances where applicable.

  * •

ML model requirements should define the ML Model outputs properties (e.g.
boundaries, normalization).

  * •

ML Model requirements should define the expected ML Model response to
robustness or generalization issues (e.g. specification of adversarial
attacks) .

  * •

Derived ML Model Requirements and the reason for their existence are defined.

  * •

Derived ML Model Requirements, if any, are provided to the system processes,
including the system safety assessment process.

  * •

Each subsystem requirement allocated to ML Item should be covered by either ML
Model requirement(s) or ML Data requirement(s) or both.

  * •

ML Model requirements should be consistent with ML Data requirements.

  * •

ML Data Requirements conform to the Requirements Standards, should be
verifiable and consistent.

##  4 Data Engineering

Data Engineering (DE) is a discipline that aims to organize, structure, trace
and select data in such a way that its quality, availability, relevance and
traceability can be guaranteed throughout the life cycle of the data. DE is
then grouping all the engineering aspects of systems, processing, models and
management of data, including but not limited to big data. [ISO/IEC
25024:2015, 2015], in the SQuaRE series of normative references for system and
software quality, sets requirements and methods for the evaluation of the
quality of data. In particular, this SQuaRE standard highlights the need that
quality characteristics be “specified, measured, and evaluated whenever
possible using validated or widely accepted measures and measurement methods”.
DE is known that running ML end-to-end requires a large amount of time
dedicated to preparing data, which includes acquiring, cleaning, organizing,
analyzing, visualizing, and feature engineering.

The goal of data acquisition is to find data sets that can be used to train ML
models. This activity faces the following issue: how to search valuable data
for the downstream task?

After gathering the data from relevant sources we need to move forward to data
engineering which aims at improving the quality of a data set. This stage
helps us gain a better understanding of the data and prepares it for further
evaluation.

Data processing is the cornerstone of ML, as it will shape the input where
data is the raw facts and figures, which could be structured and unstructured
and acquisition means acquiring data for the given task at hand. Specifically,
data sets tend to be unbalanced, have a high degree of heterogeneity, lack
labels, tend to drift over time, contain implicit dependencies and generally
require vast amounts of pre-processing effort before they are usable.

###  4.1 ODD-based data set specification

Data sets should sufficiently cover the input domain. To reach this objective,
descriptive attributes are used to characterize each data sample. These
attributes correspond to explicit and interpretable operating parameters
associated with the complex input space. This could come from the system
requirements where operational design domain is specified. Thus, a data set
specification (DSS) specifies a group of data elements and the conditions
under which this group is collected. A DSS can define the sequence in which
data elements are included, whether they are mandatory, what verification
rules should be employed and the characteristics of the collection (e.g. its
scope). Then, DSS consists in mapping required diversity to fully cover the
operational design domain. The mapping produces an exhaustive list of
attributes, which correspond to the dimensions that will be explored and
sampled to achieve the targeted diversity and completeness.

These attributes are linked to the expected scenarios and conditions as well
as semantic intra-class variability. The data sets should also be highly
representative and complete, particularly regarding the coverage of corner
case inputs.

###  4.2 Data Segregation

The fundamental goal of a supervised ML system is to use an accurate model
based on the quality of its pattern prediction for data that it has not been
trained on. As such, existing labeled data is used as a proxy for
future/unseen data, this data segregation task involves breaking processed
data into three independent data sets —- train, validation, and test:

  * •

Training set is used to initially train the algorithm and teach it how to
process information. This set defines model classifications through
parameters, establishing the behavior of the machine learning model.

  * •

Validation set is used to tune some hyperparameters of a model (e.g. number of
hidden layers, learning rate, number of neurons per layer for neural netwoks),
to anticipate some learning issues (overfitting, underfitting, etc.) and to
estimate the accuracy of the model.

  * •

Test set is used to assess the accuracy and performance of the models. This
set is meant to expose any issues or mistraining in the model.

###  4.3 Data Characterization and scoping

[Nazabal et al., 2020] have recently proposed to look at data engineering
problem looking through the prism of the Data Organization, Data Quality
issues, and Feature Engineering reading grid.

  * •

Data Organization: The first issue we face is producing a representation of
the data that is well suited to the task at hand. The first step is to
structure the raw data so that it can be read correctly (data parsing). In a
second step, a basic exploration of the data produces metadata for all
elements (data dictionary). Then, data from several sources are combined into
one extended table (data integration). In the last step, data is transformed
from the original desired raw format.

  * •

Data Quality: Any problems in the data should be diagnosed, repaired or even
removed. Data quality problems relate to the data itself but not to the
underlying structure of the data obtained earlier, which must be qualified
independently. Common data cleaning operations include normalizing the data
(canonicalisation), resolving missing entries (missing data), correcting
errors or abnormal values (anomalies).

  * •

Feature Engineering: Once the data is organized and cleaned, the next question
is how to represent it in forms suitable for processing. Feature engineering
specifically applies to ML algorithms. Its processes can be based on some
transformations of the raw data or even dedicated to creating new features
from the raw data. This process usually relies on expert knowledge and is
domain specific.

ML is generating renewed interest in data quality [Mattioli et al., 2022]. One
understands that data qualification is a broad topic, that encompasses both
the data itself and its relation to the ML algorithm, as well as a
qualification of all the processes involved in the creation of the data set.
Nevertheless, there is no consensus on what comprises the data quality
dimensions.

For example, [Batini et al., 2016] proposed a classification framework where
dimensions are included in the same cluster according to the similarity of the
characteristics they measure. They end up with eight categories named after
their representative dimension:

  * •

Accuracy, correctness, validity and precision focus on the adherence to a
given reality of interest.

  * •

Completeness, pertinence and relevance refer to the capability of representing
all and only the relevant aspects of the reality of interest.

  * •

Redundancy, minimality, compactness and conciseness refer to the capability of
representing the aspects of the reality of interest with the minimal use of
informative resources.

  * •

Readability, comprehensibility, clarity and simplicity refer to ease of
understanding and fruition of data by users.

  * •

Accessibility and availability are related to the ability of the user to
access information from his or her culture, physical status/functions and
technologies available.

  * •

Consistency, cohesion and coherence refer to the capability of data to comply
without contradictions to all properties of the reality of interest, as
specified in terms of integrity constraints, data edits, business rules and
other formalisms.

  * •

Usefulness, related to the advantage the user gains from the use of
information.

  * •

Trust, including believability, reliability and reputation, catching how much
information derives from an authoritative source. The trust cluster
encompasses also issues related to security.

##  5 ML Algorithm Design

This phase requires model technique selection and application, model training,
model hyperparameter setting and adjustment, model validation, ensemble model
development and testing, algorithm selection, and model optimization. Thus,
this phase decides first the model type, variant and, where applicable, the
structure of the model to be produced in the Model Learning stage. The process
of adaptation is called training, in which samples of input data are provided
along with desired outcomes. The algorithm then optimally configures itself so
that it can not only produce the desired outcome when presented with the
training inputs, but can generalize to produce the desired outcome from new,
previously unseen data. This training is the “learning” part of ML.

Numerous types of ML techniques are available, including multiple types of
classification models (to identify the category that the input belongs to) and
regression models (to predict a continuous-valued attribute) for supervised
tasks, clustering models (to group similar items into sets) for unsupervised
tasks, and reinforcement learning models (to provide an optimal set of
actions).

A common question is “Which ML architecture should I use?”. The DEEL project
establishes the following table [Delseny et al., 2021] which gives a short
summary of the most common ML techniques, and indicates their main
applications. Each kind of ML technique will rely on one or several hypothesis
function space(s), and one or several exploration algorithms (not listed in
this document) to minimize a loss function on the training dataset.

Techniques |  Applications
---|---
Linear models: Linear & logistic regressions, SVM |  Classification, Regression
Neighbourhood models: KNN, K means, Kernel density |  Classification, Regression, Clustering, Density estimation
Trees: decision trees, regression trees |  Classification, Regression
Graphical models: Bayesian network, Conditional Random Fields |  Classification, Density estimation
Combination of models: Random Forest, Adaboost, XGboost |  Classification, Regression, Clustering, Density estimation
Neural networks, Deep Learning |  Classification, Regression

After choosing the model, among the various algorithms present, one needs to
tune the hyper parameters of each model to achieve the desired performance.

  * •

Select the right algorithm based on the learning objective and data
requirements.

  * •

Configure and tune hyperparameters for optimal performance and determine a
method of iteration to attain the best hyperparameters.

  * •

Identify the features that provide the best results.

  * •

Determine whether model explainability or interpretability is required.

  * •

Develop ensemble models for improved performance.

  * •

Test different model versions for performance.

  * •

Identify requirements for the model’s operation and deployment.

The resulting model can then be evaluated to determine whether it meets the
business and operational requirements.

##  6 Evaluation and Verification

Ensuring that a safety-critical system will perform adequately in their
intended operational environment is a mandatory part of overall system
validation. Traditional software validation includes traceability from
requirements to system level tests. However, the use of ML techniques
frustrates this approach due to the use of training data rather than a
traditional design process. In addition, software validation should be based
on tests that show a level of performance that is adapted to the criticality
of the risks, and performed on a data set that is fully representative of the
factors of influence of the model. As previously mentioned however, the
specification of the functional characteristics of the model and of the
environment of operation may lead to the multiplicity of the factors of
influence, and a valid demonstration of the performance of the model would
imply relying on testing data sets of huge volume (in the worst case, millions
of sample). Verification through formal methods or by simulation are
interesting tracks to fulfill this goal, but they are still at an early stage
of research.

Verification therefore requires at least ensuring that training data and
testing data cover all relevant operational conditions. Making this problem
tractable in practice is generally accomplished by constraining the
operational environment to a subset of all possible situations that could be
dealt with by a human operator. That approach to limiting the operational
needs of the system is known as adopting an ODD [Koopman and Fratrik, 2019].

The testing of an ML component aims at detecting gaps between achieved and
intended (targeted) behaviors of ML models. Formally, ML testing refers to any
activity designed to reveal ML bugs, where a ML bug refers to any imperfection
in a ML item that causes a discordance between the output of the model and the
output of reference. Examples of gaps could be due to shift in training and
testing data distribution, or wrong assessment of data fit to the task at
hand, therefore data is usually the cause of wrong or unexpected errors.

This definition underlines three preliminary challenges to overcome. First, ML
system may have different types of ‘required conditions’, i.e. properties that
should be verified – we may classify them into basic functional requirements
(e.g. correctness and model relevance) and non-functional requirements (e.g.
efficiency, robustness, fairness, interpretability). The verification of such
properties requires the use of different methods and metrics, which means that
the selection of the best tools for the verification of the component must be
preceded by a definition of the required conditions: "What do we want to prove
through testing?". Secondly, an ML bug may exist in the data, the learning
program, or the framework. Here again, this means that the testing strategy
should either address the component itself, or question other "sub-component",
which may make the testing more complex since establishing a causal link
between the bug and its source may be difficult, and the definition of a
testing protocol allowing the distinction of independent and dependent
variables is not trivial in an ML pipeline. Finally, the notion of testing
activity may encompass several radically different approaches for testing.
This may include test input generation, test oracle identification, test
adequacy evaluation, and bug triage. The selection of the approach must be
based on a trade-off between the technical feasibility of performing such test
on the ML component and the required conditions initially formalized.

###  6.1 Quality Control

Quality control (QC) is an essential part of the verification and validation
of the ML component. QC may be performed through an estimation of the success
of the task solved by the component. Traditional metrics for regression
problems include Mean Squared Error (MSE) or Mean Absolute Error (MAE), while
classification problems can be evaluated through precision, accuracy and
recall. In classification problems, a confusion matrix (depicting the
distribution of true/false negatives/positives for each class) is a practical
tool for visualizing of the errors, and allows the computation of most metrics
(precision, recall, sensitivity, specificity, F1 score, ROC curve, etc.).

The most common evaluation protocol consist in maintaining a hold-out
validation set. This consists on setting apart some portion of the data as the
test set. The process would be to train the model with the remaining fraction
of the data, tuning its parameters with the validation set and finally
evaluating its performance on the test set. The reason to split data in three
parts is to avoid information leaks. The main inconvenient of this method is
that if there is small amount of data available, the validation and test sets
will contain so few samples that the tuning and evaluation processes of the
model will not be effective. An alternative is k-Fold, which consists in
splitting the data into k partitions of equal size.

Another interesting approach is: Iterated k-fold validation with shuffling.
This technique is relevant when having little data available and it is needed
to evaluate models as precisely as possible.

Functional performance evaluation has its own challenges. The selection of the
most adapted metrics to reflect the desired level of performance, as well as
the selection of a suitable protocol for testing, require careful work.
However, the notion of QC should go further beyond a simple estimation of
functional performance.

First, we note that the recourse to a validation set, in this context, is part
of the ML algorithm design step. Here the focus is made on the technical
validity of the algorithm design. This means that there are only few links
between this testing activity and, for example, the operational constraints
established in the specification phase. In the same idea, the influence of the
training data is ignored at this stage, since traditional protocols do not
necessarily take into account the informational value of the data points in
each set (hence, a risk for representativeness, or a risk of ignoring corner
case values either in training or testing). QC should then encompass more than
a simple evaluation procedure of the ML algorithm: QC procedures should be
formalized and deployed ideally at each stage of the ML pipeline, with
different objectives and verification strategy for each stage, but with one
overarching objective in my mind: "How can I ensure the quality of all the
processes involved in the development of the ML component".

Although each domain has their own traditional ways of performing
qualification (for example, data qualification has its own procedures), their
link with the particularities and the constraints of ML components is not
always well established. In addition, some aspects of the verification and
validation strategies are underestimated, or at least not part of the routine,
in ML engineering. For example, information emanating from data engineering
about the limits and constraints of the data should reflect in the overall
strategy of the evaluation of the model. The system in which the ML component
is intended must also provide its own set of constraints with which to check
the compliance of the component.

This means that all brick of the ML pipeline should include specific QC
procedures, and the information should propagate to the relevant bricks of the
pipeline and condition the overall evaluation of the quality of the component.

###  6.2 Algorithm / Component Characterization

Figure 3: Idealized Workflow of ML testing

ML software includes many different components (data, learning algorithms,
etc.). The testing phase must therefore include steps to verify each ones of
these components.

The behaviors of ML-based systems depend on the data used to train them. Any
problems in the data affect the quality of the resulting model, and by cascade
effect, may lead to other problems in the operation of the model.
Consideration should be given to the ability of the data to train or evaluate
a particular model (completeness of the data), whether the data is
representative of the data that the system will have to process, whether the
data contains a lot of noise (particularly on labels), whether there is a bias
between training and test data, whether there is any poisoning of the data or
data containing adversarial noise. There are also different methodologies that
can be used to search for such bugs.

ML, especially the phase for training the model, is computationally intensive.
Deep learning frameworks (e.g., pytorch, tensorflow, etc.) simplify the
writing of learning programs, making it easier for developers. Therefore, they
play a more important role in ML development than in traditional software
development. For this reason, it is important to test these frameworks and
check that they do not contain any bugs. Various authors have shown that the
most widely used frameworks are far from being bug-free, and have proposed
methodologies for testing these frameworks: [Xiao et al., 2018, Guo et al.,
2018, Sun et al., 2017].

It is also necessary to detect bugs that may occur in the model training
software. These software have two components: the optimization algorithm
designed by the developer or taken from a framework, and the actual training
code that developers write to use, deploy or configure the optimization
algorithm. A bug in the training phase can come from the design of the
optimizer, a misconfiguration, or a misuse of the optimizer, as well as from
errors in the code using it. We can cite, as an example, [Schaul et al.,
2013], who proposed unit tests designed for stochastic optimization. They can
be used to test learning algorithms in order to detect bugs as early as
possible. [Zhang et al., 2020] proposed an idealized workflow of ML testing,
detailing the different component, which we reproduce in Figure 3.

###  6.3 Certification and Assurance Case

Certification standards should impose neither a specific model nor a specific
training technique. The focus should rather be on the properties, such as
explainability and robustness that the model must possess after training.
Other properties such as maintainability, auditability, etc. could also be
checked at this stage. The depth of demonstration of these properties can vary
depending on the requirements. If these properties are required for the
overall safety demonstration, then in-depth demonstration is necessary. To
illustrate that purpose, the Federal Aviation Administration (FAA) launched in
2016 an initiative called “Overarching Properties”. The objective of this
initiative is to develop a minimum set of properties such that if a product is
shown to possess all these properties, then it can be certified. As of 2019,
the three overarching properties retained are:

  * •

Intent. The defined intended functions are correct and complete with respect
to the desired system behavior.

  * •

Correctness. The implementation is correct with respect to its defined
intended functions, under foreseeable operating conditions.

  * •

Innocuousness. Any part of the implementation that is not required by the
defined intended behavior has no unacceptable safety impact.

These properties are, by construction, too abstract to constitute an
actionable and complete means of compliance for certification. In practice,
they shall be refined to be applicable, leaving an opportunity to establish a
specific set of methods for the implementation and verification of these
properties for the certification of ML systems.

However, following the FAA Initiative, if no requirement stems from the safety
assessment and component specification, then the ML model could remain a
“black box”, without explainability and/or robustness demonstration. Some
verification activities can be performed directly on the model, before
implementation. If it is the case, it should be demonstrated that the results
of these verification activities are preserved after implementation.

The assurance of a system is typically communicated in the form of an
assurance case, capturing “a reasoned and compelling argument, supported by a
body of evidence, that a system, service or organization will operate as
intended for a defined application in a defined environment”.

###  6.4 Explainability

A first way to assess the explainability of results produced by ML is through
a human evaluation. This is, to date, the most reliable approach to assessing
explainability. [Doshi-Velez and Kim, 2017] have proposed a taxonomy of
explainability assessment methods. This human-based approach uses the results
of human evaluation on simplified tasks. The second one, based on function,
does not require human experiments but uses a quantitative metric as a proxy
for the quality of the explanation (e.g. through the depth of a decision
tree).

Automatic assessment of explainability allows for more objective procedures
and easier scalability than human assessment. However, it requires the
definition of metrics, which are not easy to establish and may depend on the
application domains. As an illustration, we can cite [Cheng et al., 2018] who
analyzed the impact of object masking in the image domain, [Zhou et al., 2018]
defined the concepts of metamorphic relationship models useful to help end-
users understand the operation of an ML system.

##  7 Conclusion

As any critical system, a critical system which embeds ML needs to have well
defined development methods from its design to its deployment and
qualification. This requires a complete tool chain ensuring trust at all
stages, as:

  1. 1.

Specification, knowledge and data management;

  2. 2.

Algorithm and system architecture design;

  3. 3.

Characterization, verification and validation of ML functions;

  4. 4.

Deployment, particularly on embedded architecture;

  5. 5.

Qualification, certification from a system perspective.

To guarantee a trustworthy algorithmic design (robust, reliable…), we
presented how algorithm engineering can integrate the ML paradigms and
specific challenges that arise. We noted that the ML engineering pipeline
requires several specific activities meant to ensure the overall
trustworthiness, in terms of design and evaluation. Depending on the stage of
the ML engineering cycle, several properties should be assessed, and the
approaches should leverage knowledge and best practices for various
disciplines.

In addition, the safety and security of critical systems which embed ML
require the demonstration of the following four properties:

  * •

Validity: to guarantee that an AI-based system will do what it is meant to do
– everything that it is meant to do and just what it is meant to do.

  * •

Security: to ensure robustness and resilience to adversarial conditions, such
as decoying and cyberattacks.

  * •

Explainability: to be able to provide human-level, understandable and context-
relevant justifications and explanations.

  * •

Responsibility: to be compliant with ethical, legal and regulatory frameworks.

Here, the robustness characterizes its ability to provide correct answers in
the face of unknown situations or maliciousness. However, this property is
harder to prove than accuracy. Indeed, a non-accurate system cannot be robust.
But more importantly, an accurate system may not be robust. This is the case
of a learning-based system that has memorized the training data and will make
wrong decisions in the future based on new data. This phenomenon is called
overfitting.

Moreover, ML remains vulnerable, and if one is not careful, particularly
sensitive to so-called "adversarial" attacks, attacks that take advantage of
the functioning of the underlying algorithms to generate small perturbations
in the analyzed data and force the AI to return an incorrect result. Many
defenses have been proposed in the last few years by the scientific community
but are sometimes refuted with new attacks making them obsolete. This is why
it is necessary to develop methods and tools to design robust algorithms and
at least characterize their robustness.

It is also necessary to prove that ML-based critical systems are controllable,
i.e. well-founded or consistent, if it can be proved that they only do what is
expected of them. The questions related to the problems of robustness and
consistency are beginning to be the subject of work related to formal proofs.
The latter aim at providing a priori guarantees on the reliability of a
system, contrary to validation methodologies by direct experimentation which
aim at providing a posteriori guarantees. Finally, understanding AI and its
reasoning is necessary to determine how much we can trust it.

For ML approaches, data are therefore crucial for learning, testing and
validation. It is not enough to have a lot of data, it must be of "good
quality" and representative of the domain of use of the system concerned,
without which these approaches give poor results. New methodologies need to be
defined for a better control of data acquisition, exploration, enrichment,
annotation and preparation stages.

An algorithm engineering based approach for ML-based safety critical system
allows a sound definition of each separate steps to conduct the development of
an ML component. This allows in particular the identification of the variety
of tasks, activities and fields of expertise involved in the development, and
helps spotting the properties of trustworthiness that need to be checked. The
development of a trustworthy ML component still requires an important amount
of research and practice towards a comprehensive framework providing all the
necessary guarantees of compliance.

### Acknowledgment

This work has been supported by the French government under the "France 2030”
program, as part of the SystemX Technological Research Institute Research
Institute

## References

  * [Batini et al., 2016] Batini, C., Scannapieco, M., et al. (2016).  Data and information quality.  Cham, Springer.
  * [Bosch et al., 2021] Bosch, J., Olsson, H. H., and Crnkovic, I. (2021).  Engineering AI systems: A research agenda.  In Artificial Intelligence Paradigms for Smart Cyber-Physical Systems, pages 1–19. IGI Global.
  * [Braunschweig et al., 2022] Braunschweig, B., Gelin, R., and Terrier, F. (2022).  The wall of safety for ai: approaches in the confiance. ai program.  In Safeai@ aaai.
  * [Cheng et al., 2018] Cheng, C.-H., Huang, C.-H., Ruess, H., Yasuoka, H., et al. (2018).  Towards dependability metrics for neural networks.  In 2018 16th ACM/IEEE Int.Conference on Formal Methods and Models for System Design, pages 1–4. IEEE.
  * [Chiaroni et al., 2021] Chiaroni, J., Zillner, S., Bertels, N., Bezombes, P., Bonhomme, Y., Amadou-Boubacar, H., Cantat, L., Cattaneo, G., Cordesse, L., Curry, E., et al. (2021).  Franco-German position paper on" Speeding up industrial AI and trustworthiness".  PhD thesis, Secrétariat général pour l’investissement.
  * [Delseny et al., 2021] Delseny, H., Gabreau, C., Gauffriau, A., Beaudouin, B., Ponsolle, L., Alecu, L., Bonnin, H., Beltran, B., Duchel, D., Ginestet, J.-B., et al. (2021).  White paper machine learning in certified systems.  arXiv preprint arXiv:2103.10529.
  * [Doshi-Velez and Kim, 2017] Doshi-Velez, F. and Kim, B. (2017).  Towards a rigorous science of interpretable machine learning.  arXiv preprint arXiv:1702.08608.
  * [Guo et al., 2018] Guo, Q., Xie, X., Ma, L., Hu, Q., Feng, R., and All (2018).  An orchestrated empirical study on deep learning frameworks and platforms.  arXiv preprint arXiv:1811.05187.
  * [Gyllenhammar et al., 2020] Gyllenhammar, M., Johansson, R., Warg, F., Chen, D., Heyn, H.-M., Sanfridson, M., Söderberg, J., Thorsén, A., and Ursing, S. (2020).  Towards an operational design domain that supports the safety argumentation of an automated driving system.  In 10th European Congress on Embedded Real Time Systems (ERTS 2020).
  * [ISO/IEC 25024:2015, 2015] ISO/IEC 25024:2015 (2015).  Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Measurement of data quality.
  * [Koopman and Fratrik, 2019] Koopman, P. and Fratrik, F. (2019).  How many operational design domains, objects, and events?  In Safeai@ aaai.
  * [Mattioli et al., 2022] Mattioli, J., Robic, P.-O., and Jesson, E. (2022).  Information Quality: the cornerstone for AI-based Industry 4.0.  Procedia Computer Science 201C, pages 453–460.
  * [Nazabal et al., 2020] Nazabal, A., Williams, C. K., Colavizza, G., Smith, C. R., and Williams, A. (2020).  Data engineering for data analytics: a classification of the issues, and case studies.  arXiv preprint arXiv:2004.12929.
  * [Ozkaya, 2020] Ozkaya, I. (2020).  What is really different in engineering ai-enabled systems?  IEEE Software, 37(4):3–6.
  * [Schaul et al., 2013] Schaul, T., Antonoglou, I., and Silver, D. (2013).  Unit tests for stochastic optimization.  arXiv preprint arXiv:1312.6055.
  * [Serban et al., 2021] Serban, A., van der Blom, K., Hoos, H., and Visser, J. (2021).  Practices for engineering trustworthy machine learning applications.  In 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN), pages 97–100. IEEE.
  * [Sun et al., 2017] Sun, X., Zhou, T., Li, G., Hu, J., Yang, H., and Li, B. (2017).  An empirical study on real bugs for machine learning programs.  In 2017 24th Asia-Pacific Software Engineering Conference (APSEC), pages 348–357. IEEE.
  * [Treveil et al., 2020] Treveil, M., Omont, N., Stenac, C., Lefevre, K., Phan, D., Zentici, J., Lavoillotte, A., Miyazaki, M., and Heidmann, L. (2020).  Introducing MLOps.  O’Reilly Media.
  * [Xiao et al., 2018] Xiao, Q., Li, K., Zhang, D., and Xu, W. (2018).  Security risks in deep learning implementations.  In 2018 IEEE Security and privacy workshops (SPW), pages 123–128. IEEE.
  * [Zhang et al., 2020] Zhang, J. M., Harman, M., Ma, L., and Liu, Y. (2020).  Machine learning testing: Survey, landscapes and horizons.  IEEE Transactions on Software Engineering.
  * [Zhou et al., 2021] Zhou, S., Zhang, J., Jiang, H., Lundh, T., and Ng, A. Y. (2021).  Data augmentation with mobius transformations.  Mach. Learn. Sci. Technol., 2(2):25016.
  * [Zhou et al., 2018] Zhou, Z. Q., Sun, L., Chen, T. Y., and Towey, D. (2018).  Metamorphic relations for enhancing system understanding and use.  IEEE Transactions on Software Engineering, 46(10):1120–1154.
  * [Zinkevich, 2017] Zinkevich, M. (2017).  Rules of machine learning: Best practices for ml engineering.  URL: https://developers. google. com/machine-learning/guides/rules-of-ml.