# A FEW-SHOT ATTENTION RECURRENT RESIDUAL U-NET FOR CRACK SEGMENTATION

###### Abstract

Recent studies indicate that deep learning plays a crucial role in the
automated visual inspection of road infrastructures. However, current learning
schemes are static, implying no dynamic adaptation to users’ feedback. To
address this drawback, we present a few-shot learning paradigm for the
automated segmentation of road cracks, which is based on a U-Net architecture
with recurrent residual and attention modules (R2AU-Net). The retraining
strategy dynamically fine-tunes the weights of the U-Net as a few new
rectified samples are being fed into the classifier. Extensive experiments
show that the proposed few-shot R2AU-Net framework outperforms other state-of-
the-art networks in terms of Dice and IoU metrics, on a new dataset, named
CrackMap, which is made publicly available at
https://github.com/ikatsamenis/CrackMap.

Index Terms—  Semantic segmentation, U-Net, attention, recurrent residual
convolutional unit, road cracks

##  1 Introduction

The development of cracks on the road surface is a frequently occurring defect
and can constitute a safety hazard for road users. Cracking in its various
types (longitudinal, oblique, alligator cracks, etc.) affects the traffic flow
and safety, resulting in poor performance of the road infrastructure,
accidents, as well as increased $CO_{2}$ emissions, fuel costs, and time
delays. Indicatively, for 2006, the comprehensive cost of traffic crashes
where road conditions contributed to crash occurrence or severity, in the
United States alone, is estimated at $217.5 billion, which corresponds to
43.6% of the total crash costs [1]. More recent evidence highlights that
approximately $400 billion is invested globally each year in pavement
construction and maintenance [2]. Therefore, the adoption of effective
monitoring strategies can lead to enormous economic, social, and environmental
benefits to the community.

Recently, there is a great research interest in the automatic visual
inspection of road distress, by analyzing visual data. Generally, deteriorated
pavement produces rough surfaces, which entails that various image processing
methods such as thresholding [3], edge detection [4], and mathematical
morphology [5] can be used to localize crack regions. The core idea behind
these approaches is that cracked regions tend to demonstrate non-uniformity,
while on the other hand, the color and textural characteristics of the non-
deteriorated road surface are more consistent and smoother. However, even
though such techniques are computationally efficient, they are susceptible to
image noise and fail to generalize the differentiation between the defect and
the surface background [6].

Current developments in deep learning and artificial intelligence technology
have led Convolutional Neural Networks (CNNs) to be an effective tool for the
automatic visual inspection of road infrastructures [7, 8, 9]. The main asset
of such deep architectures, compared to the aforementioned conventional image
processing methods, is the fact that they leverage throughout the learning
procedure annotated ground truth data [10]. Thereby, these algorithms
demonstrate high identification accuracy by effectively learning the essential
features needed to classify a given pixel as defective or not.

Usually, Fully Convolutional Networks (FCNs), or their variants U-Nets, are
considered for providing a precise pixel-based segmentation of damaged areas
from RGB images of road infrastructures [6, 11, 12]. This is mainly due to the
fact that FCNs have emerged as powerful segmentation tools, especially for
performing accurate pixel-based classification tasks for challenging problems
(e.g., biomedical imaging problems with data expressed either in 2D or 3D [13,
14], as well as crack segmentation [15]). To enhance the performance of the
original U-Net, numerous elements have been introduced, such as residual
convolutional units [16, 17] and attention gates instead of the typical skip
connection [18, 19].

(a) The proposed U-Net architecture with recurrent residual and attention
modules.

(b) The proposed few-shot retraining strategy.

Fig. 1: A schematic representation of the proposed (a) R2AU-Net and (b) few-
shot learning scheme for road crack segmentation.

###  1.1 Current limitations and our contribution

Inspired by the above research work, we present R2AU-Net, which is a deep
U-Net structure with recurrent residual and attention modules (see Fig. 1a).
Compared to the standard U-Net, R2AU-Net incorporates recurrent residual
convolutional layers (R2CL) that ensure better feature representation for the
segmentation task and attention gates to highlight salient features which are
passed through the skip connections [17, 18].

Still, however, the most crucial hindrance of the aforementioned typical deep
learning frameworks is that they treat the segmentation task as a static
procedure. More specifically, they leverage knowledge derived from labeled
data, but, nevertheless, it is not possible to further refine their outputs by
exploiting user’s interaction, especially in cases where the deep network
underperforms [14]. To this end, we introduce a few-shot refinement scheme
which is a semi-supervised learning paradigm, based on the R2AU-Net, that is
able to adapt the model’s behavior and weights according to user’s feedback,
to further increase the segmentation performance (see Fig. 1b).

##  2 Proposed Architecture

###  2.1 R2AU-Net architecture for road crack segmentation

In this section, we present R2AU-Net, whose architecture is illustrated in
Fig. 1a and is a combination of Recurrent Residual U-Net [17] and Attention
U-Net [18]. It is emphasized that the model was designed for segmenting cracks
in RGB images and, hence, provides meticulous information on a variety of
metrics and properties that are critical for the automated and robotic-driven
maintenance process, such as geometry, type, orientation, length, density, and
shape of cracks.

In particular, the operations within the Recurrent Convolutional Layers (RCL)
in R2CL (see Fig. 1a) are carried out based on the discrete time steps which
are expressed according to the RCNN [20]. Suppose we have an input at layer
$l$ within an R2CL block, and a pixel located at $(i,j)$ within an input on
the feature map $k$ in the RCL. We denote $\mathcal{Y}^{l}_{ijk}(t)$ the
output of the model at time step $t$, which can be expressed as:

$\mathcal{Y}^{l}_{ijk}(t)=(w_{k}^{\varrho})^{T}\cdot
x_{l}^{\varrho(i,j)}(t)+(w_{k}^{r})^{T}\cdot x_{l}^{r(i,j)}(t-1)+\beta_{k}$
(1)

where $x_{l}^{\varrho(i,j)}(t)$ and $x_{l}^{r(i,j)}(t-1)$ represent
respectively the inputs to the standard convolution layers and $l^{th}$ RCL.
In parallel, $w_{k}^{\varrho}$ and $w_{k}^{r}$ denote respectively the weights
of the standard convolutional layer and RCL that correspond to the $k^{th}$
feature map, whereas $\beta_{k}$ symbolizes the bias. The RCL’s output is
activated by the ReLU function $\varrho$ as follows:

$\mathcal{R}(x_{l},w_{l})=\varrho(\mathcal{Y}^{l}_{ijk}(t))=max(0,\mathcal{Y}^{l}_{ijk}(t))$
(2)

Let $x_{l}$ an input sample of the R2CL unit, then, the R2CL’s output
$x_{l+1}$, in both the downsampling layer in the encoding path and the
upsampling layer in the decoding path of the R2AU-Net, can be calculated using
the following equation:

$x_{l+1}=x_{l}+\mathcal{R}(x_{l},w_{l})$ (3)

In parallel, as one can observe in Fig. 1a we integrate into R2AU-Net an
attention gate mechanism to focus on points and shapes of interest (i.e., road
cracks) [18]. More specifically, for each pixel $i$, the attention
coefficients $\alpha_{i}\in[0,1]$ tend to yield higher values in target crack
regions and lower values in background road areas. We obtain the output of the
attention gate in layer $l$ by multiplying element-wise the input feature maps
and attention coefficients: $\hat{x}_{i}^{l}=x_{i}^{l}\cdot\alpha_{i}^{l}$.
Attention values are calculated for each pixel vector
$x_{i}^{l}\in\mathbb{R}^{F_{l}}$, where $F_{l}$ denotes the number of feature
maps in layer $l$. Also, a gating vector $g_{i}\in\mathbb{R}^{F_{g}}$ is
utilized in order to determine the focus area per pixel. To achieve greater
performance the attention coefficient is derived by leveraging the additive
attention:

$Q^{l}_{\alpha}=\psi^{T}(\varrho(W_{x}^{T}x_{i}^{l}+W_{g}^{T}g_{i}+\beta_{g}))+\beta_{\psi}\textrm{,}\quad
a_{i}^{l}=\sigma(Q^{l}_{\alpha})$ (4)

where $\sigma$ corresponds to the sigmoid activation function,
$W_{x}\in\mathbb{R}^{F_{l}\times F_{int}}$ and
$W_{g}\in\mathbb{R}^{F_{g}\times F_{int}}$ are linear transformations that are
calculated by utilizing channel-wise 1×1×1 convolutions for the input tensors,
and, lastly, $\beta_{g}\in\mathbb{R}^{F_{int}}$ and
$\beta_{\psi}\in\mathbb{R}$ denote the bias.

###  2.2 Few-shot learning for segmentation refinement

As shown in Fig. 1b, we hereby propose a dynamic rectification scheme that
leverages expert users’ feedback on a small part of the data in order to
improve the overall performance of the aforementioned R2AU-Net. The proposed
retraining strategy dynamically updates the weights of the model, so that (a)
the refined incoming samples are trusted as much as possible, while
simultaneously (b) a minimal degradation of the already gained knowledge is
achieved. To this end, let us denote $p_{ij}$ the soft label value of a pixel
that is located in position $(i,j)$ of a given image. Then, for each input $n$
we calculate the average image confidence score $I_{n}$, defined as:

$I_{n}=\frac{1}{\sum_{\forall
i,j}{\zeta_{ij}}}\cdot\sum_{i=1}^{R}\sum_{j=1}^{C}\zeta_{ij}\cdot p_{ij}$ (5)

where C and R correspond to the image’s columns and rows respectively. In
parallel, $\zeta_{ij}\in\\{0,1\\}$ equals 1 when $p_{ij}>\vartheta$ and 0
otherwise, where $\vartheta$ is the detection acceptance threshold, which is
set to 0.5. As such, the confidence score considers only the cracked regions
over the image $n$, provided by the deep classifier. Subsequently, we rank the
images according to $I_{n}$ scores. The 5% of the lower ranked images are
provided to an engineer expert, who rectifies the model’s segmentation
outputs. Lastly, the refined few-shot annotated data are fed back to the
network for updating the model’s weights.

##  3 EXPERIMENTAL SETUP AND RESULTS

###  3.1 Dataset description

For the training procedure five datasets, consisting of 4,717 images in total,
that depict crack defects, were utilized (see Table 1). During the data
preprocessing step, the RGB data were resized to a resolution of 256×256
pixels. Lastly, 80% of the data was used for training the models (3,774
images), while the rest 20% was used for validation (943 images).

Set | Name | Number of RGB samples
---|---|---
| CFD | 118
| CRACK500 | 3,363
Train and | Cracktree200 | 206
Validation | DeepCrack | 521
| GAPS384 | 509
| Total | 4,717
Test | CrackMap | 120 – 6
Table 1: Utilized datasets for training and evaluation tasks.

For the data collection process of the CrackMap dataset that constitutes the
test dataset for the current study, a GoPro HERO9 Black was used. During the
data acquisition process, the optical sensor was mounted on an inspection
vehicle (see Fig. 2a). It is emphasized that the acquired data are RGB images
with an aspect ratio of 4:3 and, in particular, with a pixel resolution of
5,184×3,888 (see Fig. 2b). Moreover, the RGB sensor was set to shoot at a high
frame rate and, more specifically, at 50 frames per second in order to ensure
sufficient data acquisition regarding both the positive (road surface with
cracks) and negative (non-deteriorated road surface) events.

(a) Vehicle-mounted sensor

(b) Acquired RGB data

Fig. 2: Experimental setup for acquiring RGB road images.

It is also highlighted that in order to deal with the severe class imbalance
problem from the acquired RGB data, image patches with a resolution of 256×256
were manually extracted and then annotated. The patches were segmented and
verified by engineer experts, within the framework of the H2020 HERON project
[21]. As presented in Table 1, the CrackMap dataset contains 120 annotated
images with a resolution of 256×256. We evaluate the comparative models that
perform the crack segmentation task on the CrackMap data, minus the 6
extracted images that correspond to the 5% of the lower ranked images and were
eventually utilized for the refinement process (see Section 2.2). CrackMap has
been made available to the scientific community, for verifying the results and
further research, at https://github.com/ikatsamenis/CrackMap.

###  3.2 Comparative algorithms and training configuration

The validation of the proposed methodology for the crack segmentation task is
based on examining its performance against other state-of-the-art approaches
that perform crack recognition and precise localization in a different way. In
particular, we compare the proposed static and dynamically refined R2AU-Net
models on the CrackMap dataset (minus the 6 extracted images) with the
following segmentation algorithms: (i) U-Net [12], (ii) V-Net [22], (iii)
ResU-Net [16], (iv) R2U-Net [17], (v) Attention U-Net [18], and (vi) ResUNet-a
[23].

The aforementioned deep models were developed using Keras and TensorFlow
libraries in Python. In parallel, they were trained and evaluated on NVIDIA
Tesla T4 GPU provided by Google Colab. We trained the neural networks for 100
epochs with early stopping criteria set to 10 epochs in order to avoid
overfitting, using mini-batches of size 8. The training processes started from
scratch by randomly initializing the networks’ weights. The models are trained
end-to-end using the Adam algorithm to optimize the dice loss function, in
order to deal with class imbalance problems. It is noted in parallel that the
optimizer is set to its default parameters ($\beta_{1}=0.9$ and
$\beta_{2}=0.999$), with an initial learning rate of $10^{-3}$ that is decayed
by a factor of 10 each time there was no improvement in the validation loss,
for 5 consecutive epochs. Lastly, for the rectification mechanism, we fine-
tune the proposed R2AU-Net by retraining it for 5 epochs, with a learning rate
reduced by a factor of 10, to avoid damaging its weights.

Input image

Ground truth

U-Net [12]

V-Net [22]

ResU-Net [16]

R2U-Net [17]

AttU-Net [18]

ResUNet-a [23]

R2AU-Net

Fig. 3: Visual comparison of the static deep models’ segmentation outputs.

###  3.3 Experiments and comparisons

Fig. 3 depicts a visual comparison of the output masks generated by our method
and the aforementioned comparative models. For a quantitative analysis of the
experimental results, the performance of the implemented models is evaluated
in terms of the Dice coefficient and IoU metric. In particular, we compute the
aforementioned metrics for every RGB image of the CrackMap dataset (minus the
6 extracted images) and, thereby, we report the average values across all 114
images of the test set with a confidence level of 95%. As can be observed in
Table 2 the proposed R2AU-Net outperforms the various state-of-the-art
algorithms by at least 1.69% and 1.89% in terms of Dice and IoU scores
respectively.

Model | Avg. Dice | Avg. IoU
---|---|---
U-Net [12] |  49.73% $\pm$ 3.75% |  35.54% $\pm$ 3.36%
V-Net [22] |  38.04% $\pm$ 4.65% |  26.57% $\pm$ 3.68%
ResU-Net [16] |  60.37% $\pm$ 2.50% |  44.53% $\pm$ 2.48%
R2U-Net [17] |  70.74% $\pm$ 1.59% |  55.39% $\pm$ 1.84%
AttU-Net [18] |  52.64% $\pm$ 3.70% |  38.06% $\pm$ 3.21%
ResUNet-a [23] |  63.28% $\pm$ 2.47% |  47.60% $\pm$ 2.49%
R2AU-Net | 72.43% $\pm$ 1.36% | 57.28% $\pm$ 1.61%
FS R2AU-Net | 77.06% $\pm$ 1.17% | 63.11% $\pm$ 1.50%
Table 2: Performance evaluation and comparisons.

Finally, Fig. 4 illustrates indicative segmentation outputs of the R2AU-Net
before and after applying the proposed few-shot rectification mechanism. As
shown in Table 2, the rectified R2AU-Net model demonstrated increased
performance of 4.63% and 5.83% in terms of Dice and IoU respectively, after
the few-shot refinement procedure. To investigate whether this improvement is
statistically significant, we exploit the Wilcoxon signed-rank test on the
obtained scores of the two models, which is a nonparametric statistical test
that compares two paired groups [24]. The obtained p-values for both metrics
are lower than .001 and, thus, we can reject the null hypothesis, which
entails that there is a statistically significant difference in the
comparative results of the R2AU-Net, before and after the proposed few-shot
refinement process.

Input image

Ground truth

Original

Rectified

Fig. 4: Output masks produced by the original R2AU-Net against the proposed
rectified few-shot learning paradigm.

##  4 CONCLUSION

This paper presents a few-shot learning strategy for road crack segmentation.
The scheme is based on R2AU-Net which exploits recurrent residual and
attention mechanisms to capture richer global context information and local
semantic features. Also, the adopted few-shot refinement process, through
which the network weights are dynamically updated as a few incoming rectified
samples are fed into the algorithm, led to state-of-the-art performance on a
new publicly available dataset.

## References

  * [1] E. Zaloshnja and T. R. Miller,  “Cost of crashes related to road conditions, united states, 2006,”  in Annals of Advances in Automotive Medicine/Annual Scientific Conference. Association for the Advancement of Automotive Medicine, 2009, vol. 53, p. 141.
  * [2] C. Torres-Machí et al.,  “Sustainable pavement management: Integrating economic, technical, and environmental aspects in decision making,”  Transportation Research Record, vol. 2523, no. 1, pp. 56–63, 2015\.
  * [3] H. Oliveira and P. L. Correia,  “Automatic road crack segmentation using entropy and image dynamic thresholding,”  in 2009 17th European Signal Processing Conference, 2009, pp. 622–626.
  * [4] H. Zhao, G. Qin, and X. Wang,  “Improvement of canny algorithm based on pavement edge detection,”  in 2010 3rd International Congress on Image and Signal Processing, 2010, vol. 2, pp. 964–967.
  * [5] N. Tanaka and K. Uematsu,  “A crack detection method in road surface images using morphology.,”  MVA, vol. 98, pp. 17–19, 1998.
  * [6] S. L. Lau et al.,  “Automated pavement crack segmentation using u-net-based convolutional neural network,”  IEEE Access, vol. 8, pp. 114892–114899, 2020.
  * [7] T. U. Ahmed et al.,  “An integrated cnn-rnn framework to assess road crack,”  in 2019 22nd International Conference on Computer and Information Technology (ICCIT), 2019, pp. 1–6.
  * [8] N. Ogawa et al.,  “Distress level classification of road infrastructures via cnn generating attention map,”  in 2020 IEEE 2nd Global Conference on Life Sciences and Technologies (LifeTech), 2020, pp. 97–98.
  * [9] A. K. Pandey et al.,  “Convolution neural networks for pothole detection of critical road infrastructure,”  Comp. and Electrical Engineering, vol. 99, pp. 107725, 2022.
  * [10] I. Katsamenis et al.,  “Simultaneous precise localization and classification of metal rust defects for robotic-driven maintenance and prefabrication using residual attention u-net,”  Automation in Construction, vol. 137, pp. 104182, 2022.
  * [11] J. Long, E. Shelhamer, and T. Darrell,  “Fully convolutional networks for semantic segmentation,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
  * [12] O. Ronneberger, P. Fischer, and T. Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Cham, 2015, pp. 234–241, Springer International Publishing.
  * [13] H. Huang et al.,  “Unet 3+: A full-scale connected unet for medical image segmentation,”  in 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 1055–1059.
  * [14] A. Voulodimos et al.,  “A few-shot u-net deep learning model for covid-19 infected area segmentation in ct images,”  Sensors, vol. 21, no. 6, 2021.
  * [15] M. D. Jenkins et al.,  “A deep convolutional neural network for semantic pixel-wise segmentation of road and pavement surface cracks,”  in 2018 26th European Signal Processing Conference (EUSIPCO). IEEE, 2018, pp. 2120–2124.
  * [16] Z. Zhang, Q. Liu, and Y. Wang,  “Road extraction by deep residual u-net,”  IEEE Geoscience and Remote Sensing Letters, vol. 15, no. 5, pp. 749–753, 2018.
  * [17] M. Z. Alom et al.,  “Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation,”  arXiv preprint:1802.06955, 2018.
  * [18] O. Oktay et al.,  “Attention u-net: Learning where to look for the pancreas,”  arXiv preprint:1804.03999, 2018.
  * [19] J. König et al.,  “A convolutional neural network for pavement surface crack segmentation using residual connections and attention gating,”  in 2019 IEEE International Conference on Image Processing (ICIP), 2019, pp. 1460–1464.
  * [20] M. Liang and X. Hu,  “Recurrent convolutional neural network for object recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3367–3375.
  * [21] I. Katsamenis et al.,  “Robotic maintenance of road infrastructures: The heron project,”  in Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments, New York, NY, USA, 2022, PETRA ’22, p. 628–635, Association for Computing Machinery.
  * [22] F. Milletari, N. Navab, and S. A. Ahmadi,  “V-net: Fully convolutional neural networks for volumetric medical image segmentation,”  in 2016 fourth international conference on 3D vision (3DV). Ieee, 2016, pp. 565–571.
  * [23] F. I. Diakogiannis et al.,  “Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data,”  ISPRS Journal of Photogrammetry and Remote Sensing, vol. 162, pp. 94–114, 2020.
  * [24] F. Wilcoxon,  Individual comparisons by ranking methods,  Springer, 1992.


#  Assessing Perceived Fairness from Machine Learning Developer’s Perspective

Anoop Mishra
University of Nebraska
6001 Dodge Street
Omaha, Nebraska
USA - 68182
<EMAIL_ADDRESS>
Deepak Khazanchi
University of Nebraska
6001 Dodge Street
Omaha, Nebraska
USA - 68182
<EMAIL_ADDRESS>

###### Abstract

Fairness in machine learning (ML) applications is an important practice for
developers in research and industry. In ML applications, unfairness is
triggered due to bias in the data, curation process, erroneous assumptions,
and implicit bias rendered within the algorithmic development process. As ML
applications come into broader use developing fair ML applications is
critical. Literature suggests multiple views on how fairness in ML is
described from the user’s perspective and students as future developers. In
particular, ML developers have not been the focus of research relating to
perceived fairness. This paper reports on a pilot investigation of ML
developers’ perception of fairness. In describing the perception of fairness,
the paper performs an exploratory pilot study to assess the attributes of this
construct using a systematic focus group of developers. In the focus group, we
asked participants to discuss three questions\- 1) What are the
characteristics of fairness in ML? 2) What factors influence developer’s
belief about the fairness of ML? and 3) What practices and tools are utilized
for fairness in ML development?

The findings of this exploratory work from the focus group show that to assess
fairness developers generally focus on the overall ML application design and
development, i.e., business-specific requirements, data collection, pre-
processing, in-processing, and post-processing. Thus, we conclude that the
procedural aspects of organizational justice theory can explain developer’s
perception of fairness. The findings of this study can be utilized further to
assist development teams in integrating fairness in the ML application
development lifecycle. It will also motivate ML developers and organizations
to develop best practices for assessing the fairness of ML-based applications.

_K_ eywords machine learning $\cdot$ perceived fairness $\cdot$ perception
$\cdot$ procedural justice theory $\cdot$ ML developers

##  1 Introduction

Machine learning (ML) is practiced in the research, industry, and education
sectors to make complex decisions and assist humans [1, 2, 3, 4, 5]. Machine
learning in natural language processing and computer vision are employed in
developing decision-making systems and decision-support systems [2, 3].
Measuring fairness and developing fair ML applications has become a widespread
practice in research, industry, and academia. However, literature suggest that
fairness in ML is a very subjective term and possess many definitions [6, 7,
8]. Mehrabi et al. 2021 defines fairness in ML as deficiency of favoritism
toward an individual or a group based on acquired characteristics [7].
Similarly, Pessach et al. 2022 describes different notions of fairness like
individual and group fairness. The subjective characteristic in fairness is
caused due to introduction of bias. An unfair model in ML is triggered mainly
due to bias in the data and erroneous assumptions rendered within the
algorithmic development process [4]. Literature suggests that bias in data has
many shapes and forms. Thus, algorithmic bias and algorithmic fairness are
discussed in literature [6, 7, 9]. Algorithmic fairness is an area of research
applied to mitigate bias and explain fairness in AI systems [7, 4].
Researchers have multiple views and descriptions for algorithmic fairness,
thus lacks a rigid definition. Mehrabi et al. 2021 define a large class of
biases by representing a feedback loop relationship between data, algorithm,
and users [7]. They argue that most of the definitions and work on fairness
are developed in the West. When these are applied to different problem types,
it introduces historical bias, contextual bias, and representation bias [7,
4]. It may lead to an unfair AI decision-making system. However, few
researchers see fairness in ML systems as multi-dimensional aspects like
psychology, political science, and economics [10, 4].

Investigating notions of fairness is important because, if not considered,
incorrect outcomes or perceptions may lead to severe societal and business
concerns. The literature includes scenarios discussing the severe effect on
society: Amazon’s AI-based recruiting system reported bias against women in
the recruitment process; Apple’s credit card approval process biased against
women; and Stanford’s COVID-19 vaccination algorithms were biased towards a
specific group [11, 12, 13, 4].

Researchers tried to understand fairness from the sociotechnical domain by
conducting studies on the human perception of fairness. The literature
describes that different stakeholder has distinct interpretations of fairness
concerning the same ML model [14, 4]. These studies are conducted to
understand the relationship between human perceptions of fairness and proposed
notions of fairness in literature [8, 14, 15, 16, 17, 18]. Harrison et al.
2022 conducted an empirical user study to investigate the trade-off between
competing notions of fairness and human perception of fairness such that ML
models can embed these trade-offs to build fair ML applications [19]. Based on
the literature, perceived fairness in ML is described as human perception and
interpretation of ML models based on outcomes predicted by the ML model.

Prior studies are conducted to investigate user’s perception of fairness. Most
of the studies performed a randomized between-subject experiment on Amazon’s
Mechanical Turk. The validity of research on Amazon’s Mechanical Turk has been
questioned in the past [20, 4]. Kasinidou et al. 2021 and Kleanthous et al.
2022 investigated student’s (as future developers) perception of fairness and
justice in algorithmic decision-making in three separate scenarios. However,
the potential gap lies within the understanding of the human perception of
fairness which has to do with the ML developer’s perception of fairness. ML
developers are crucial actors to investigate perceived fairness because they
are responsible for designing, developing, and evaluating ML models in ML
development process. We conducted virtual focus groups to explore and assess
characteristics and factors that influence ML developer’s perceived fairness.
The goal of this research study is to assess the characteristics of perceived
fairness from developer’s perspective. Thus, we asked three questions from ML
developers in the systematic focus groups:

  1. 1.

How would you describe the fairness of ML applications from your (developer’s)
perspective?

  2. 2.

What are the factors that influence your (developer’s) belief about the
perceived fairness of ML applications?

  3. 3.

What practices or tools do you utilize to practice fairness?

Inductive thematic analysis and LDA-based topic modeling are utilized to
accomplish the research objective of assessing developer’s perceived fairness.
Section 3.2 discusses more about this approach. Our findings assist to
understand the relation between actual ML developer’s perceptions and proposed
notions of fairness in the literature. Researchers explain that notions of
fairness are associated with distributed fairness from organization justice
theory, which means fairness measures based on outcomes [21, 22, 6]. The
present trend in fairness advocates for developing procedural notions of
fairness, i.e., procedural fairness that explain fairness based on the process
[23, 21, 24]. This research study’s findings conclude that the developer’s
perceived fairness relates to procedural fairness of organizational justice
theory in the decision-making process. Based on the findings, our research
contributions are-

  1. 1.

Developed attributes (themes) of ML developer’s perceived fairness,

  2. 2.

Investigated the association between ML developer’s perceived fairness with
procedural fairness,

  3. 3.

Proposed definition of perceived fairness from ML developer’s perspective

This research study’s findings will help researchers and motivate ML
developers and practitioners to understand perceived fairness from a multi-
dimensional perspective. This research will also help organizations understand
perceived fairness and provide insight into how fairness is addressed when
interpreting ML applications. The latter section includes the related work
discussing prior research on fairness, practice and tools in fairness, and
human perception (users and students) of fairness. The related sections also
discuss the gaps in existing methods. Section LABEL:Methods shows the approach
of inductive thematic analysis. Further sections include findings and
discussions, association of procedural fairness and perceived fairness,
definition of perceived fairness, and conclusions.

##  2 Background

###  2.1 Algorithmic fairness in decision-making

In artificial intelligence (AI), machine learning (ML) is described as data-
driven process that learns with experiencing the data without explicitly
programmed [3, 1]. The advancement in ML approaches allows models to assist
humans in decision-making tasks. Useful representations are learned from data
that evaluates the ML model’s performance. However, if the data consist biases
then models from data-driven process may inherits the bias. Since, ML models
like neural networks are black box that causes intermediate process to be
opaque, it becomes difficult to assess whether the decisions are justified or
biased [4, 7]. Prior research indicates critical concern on data collection
methods, because flawed data include bias which can result in unfair decisions
[7, 25, 6]. Thus, it is essential to understand the definition of a fairness
in ML. Literature suggests that ML model is unfair if it produces unfavorable
treatment to people based on specific demography [26, 27, 28, 4]. Fairness in
ML is a popular and multi-dimensional concept that depends on cultures,
objectives, contexts and problem definition [10, 14, 29, 30]. Black box nature
of ML models that lacks explainability can lead to harmful consequences.
Techniques utilizing computational and mathematical frameworks are utilized to
practice interpretability, bias identification, that improves algorithmic
fairness, such as IBM’s AI360, Tensorflow constrained optimization framework,
fairlearn, etc. [31, 32, 33, 34, 4]. Deng et al. 2022 empirically explored the
practice of ML toolkits from ML practitioners. They concluded that
practitioners need toolkits such that they can contextualize, collaborate and
communicate in explainability with non-technical peers for ML fairness [34].

###  2.2 Human perception of fairness

Prior research advocates that for developing a fair decision-making system
consult with subject matters, ethical checks, planning, and human checks must
be considered [12, 25, 2, 4]. Lee et al. 2017 finds that different
stakeholders perceive distinct interpretations of fairness with same ML model
[14]. For example, an online experiment conducted by Wang et al. focusing
AI/ML fairness conclude that if algorithmic outcomes are inclined towards an
individual, then it was rated as fair by the users [10, 4]. Researchers made
efforts to understand fairness from the sociotechnical domain by conducting
studies to investigate human’s perception of fairness. These studies are
organized to understand the relationship between human’s perception of
fairness and proposed notions of fairness in literature [8, 14, 15, 16, 17,
18]. Woodruff et al. 2018 explored the perceived fairness of users from a
marginal population, they found that ML fairness cooperates with user’s trust
[18]. Berkel et al. 2021 performed an online crowdsourcing study to
investigate how information presentation influences human’s perceived fairness
[35]. Srivastava et al. 2019 conducted user’s study to investigate the
relationship between mathematical notions and human perceived fairness [8].
Leeet al. 2021 performed user’s experimentsfocusing on people’s (Black
American) perception of fairness in AI healthcare for skin cancer diagnosis.
This research seeks to examine individual-level differences targeting
trustworthiness in human decision-making in contrast to AI-based algorithmic
decision-making [17].

###  2.3 Fairness from organization justice theory

Piero et al. 2014 explains that fairness is concerned with social norms and
governing rules [36, 4]. They discuss four forms of fairness considering the
fourfold model of justice theory: distributive justice, procedural justice,
interpersonal justice, and informational justice. They claim that perceived
fairness is highly correlated with psychological well-being and distress.
Algorithmic fairness is broadly explored with distributed (justice) fairness.
Few attempts are made to explore procedural fairness [23, 21]. Distributive
Fairness is defined as perceived fairness is the process for distribution of
rewards across group members [37, 38, 39, 40, 4]. Whereas procedural justice
is the perceived fairness of rules and decision processes used to determine
outcomes [38, 23, 41, 39, 22].

####  2.3.1 Procedural Fairness

Few studies explain the advantage of procedural fairness over distributed
fairness. Morse et al. 2021 and Rueda 2022 suggest that procedural fairness in
ML augments explainability and transparency in the ML model [24, 21]. Biran et
al. 2017 claims that bias and fairness are highly related to the
interpretation and explainability [29, 4]. In machine learning, the
explanation is complex due to its black-box nature. Doshi-Velez et al. 2017
explain the relation between interpretability with reliability and fairness by
discussing real-world scenarios [30, 4]. The definition of fairness,
explainability, and interpretability is motivated by multiple theories of
lens, including psychology, philosophy, cognitive science, and ethics[30, 1,
2, 4]. However, in this study, we are targeting organization justice theory.
We identified components of procedural fairness from the literature to
accomplish our objective. Lee et al. 2019 describes procedural fairness using
transparency, control, and principle [23]. Rueda 2022 explains procedural
fairness as avoidance of bias, accountability, and transparency [24]. Morse et
al. 2021 discuss the components of procedural fairness from Leventhal 1980 as
bias impression, consistency, representativeness, correctability, accuracy,
and ethicality [41, 21]. These components are utilized and discussed in detail
in section 4.2.

Most of the studies discussed in section 2.2 include participants as users and
students. Few studies utilized Amazon Mechanical Turk for user’s study.
However, the validity of research on Amazon’s Mechanical Turk has been
questioned in the past [20, 4]. There are no studies that explored ML
developer’s perception of fairness. Literature suggests that consulting with
subject matter experts, ethical checks, planning, and human checks must be
considered for developing a fair decision-making model [25, 2, 4]. Studying
perceived fairness from organization justice theory as a theory of lens will
help define and describe attributes of perceived fairness and develop a
conceptualization of the factors that influence the beliefs of ML developers.
Thus, it is essential to understand the characteristics of perceived fairness
from ML developer’s perspective.

##  3 Methods

###  3.1 Virtual Focus Group

A focus group in research is a group discussion of people with similar
characteristics where they share experiences and discuss in order to generate
data [42, 43]. Focus group discussions are utilized as a qualitative approach
to assess in-depth understanding of social issues [44, 43]. In this research
study, focus groups are used to explore ML developer’s perceptions of
fairness. The participants targeted for the focus groups are ML developers,
data scientists, and ML engineers from the industry who participate in the
design and development of ML applications.

####  3.1.1 Focus group participants

Motivated by literature and to explore ML developer’s attributes of perceived
fairness, we conducted three virtual focus groups on Zoom. Anonymity is
ensured for all companies including participants, companies, and intermediate
representatives. Table 1 shows the participation details for each company. .

Table 1: Participants in the focus groups Company ID |  Type of Company |  Participants committed to participate |  Participants who actually participated
---|---|---|---
1 |  Life insurance, finance, medicare |  5 |  3
2 |  Railroad |  11 |  3
3 |  Transportation and logistics |  Unknown |  3

Three companies located in United States participated in the virtual focus
groups. The company size is more-than-1000-employees for all three companies.
The number of participants who actually participated are less with
participants committed to participate. In total, nine participants from three
distinct companies participated in three virtual focus groups. The diversity
of the companies are shown in table 1. The job-role portfolio of the
participants includes data scientists, senior data scientist, and software
developers. Out of 9 participants, 3 participants are females and 6 are male.
Out of all, $75\%$ were extremely familiar with ML concepts including
professional ML experience of 3 to 5 years, and took the college-level course
of ML. $25\%$ participants are experts and have more than 5 years of
professional experience in ML development.

####  3.1.2 Study Design

The research has Institutional Review Board (IRB) approval for conducting the
focus groups. The virtual focus group was designed to develop an understanding
of perceived fairness and it’s attribute among developers. This research study
utilized MIRO as a brainstorming tool and Zoom to conduct the focus groups.
MIRO is an online visual platform where teams can connect, collaborate,
create, and brainstorm together (see https://miro.com/). All sessions are
conducted synchronously. Each company along with its participants has one
focus group. Each focus group is given a 75-minutes window to participate.
Participants from each company were provided with their own unique session ID
on MIRO. Participants were notified by email along with MIRO and Zoom web
links in advance of the session, so they could plan for their participation.
Reminder e-mails were sent during the session to encourage more participation.
Each company is assigned one focus group. After logging in at Zoom, each
company with participants receives an introductory session about the focus
group agenda. Each participant is asked to first fill out a questionnaire for
demographic details. The second step on the agenda is brainstorming questions,
and the participant is asked to enter as many ideas as possible on MIRO during
the 75-minutes window, as well as comment and discuss fellow participant’s
ideas. The participants recorded their ideas and discussion in the form of
notes at MIRO. These notes include phrases, short sentences, and long
sentences. The final step on the agenda is a closing discussion based on the
brainstorming session. Table 2 shows the agenda and instructions for each
focus group. The table 2 was repeated for each company as seen in table 1.

Table 2: Focus group agenda and instructions for participants Activity |  Instructions
---|---
Introductory session |  Welcoming the participants; Making participants aware of Institutional Review Board (IRB) approval for conducting the focus group; Introduction to the focus group with agendas; Introduction of the topic and research objective; Tools introduction and demo; Request to fill out demographic questionnaire;
Beginning Questionnaire |  Please answer all questions to the best of your knowledge. All questions are voluntary. After you have answered all questions, "Notify" the host, and you will be taken back to the agenda. When you are back at the agenda, go to Brainstorming Question 1 and you can start discussing and entering ideas.
Brainstorming Question 1 |  How would you describe the fairness of ML applications from your perspective? Alternatively, what are the characteristics of the perceived fairness of ML applications? Think broadly to include individual behaviors and processes based on your past experiences like projects and team meetings. Enter 5 to 10 separate ideas. Comment on ideas other people have entered and/or enter more of your own ideas. Feel free to expand on other people’s ideas.
Brainstorming Question 2 |  What are the factors that influence your belief about the perceived fairness of ML applications? Think broadly to include individual behaviors and processes based on your past experiences like projects and team meetings. Enter 5 to 10 separate ideas. Comment on ideas other people have entered and/or enter more of your own ideas. Feel free to expand on other people’s ideas.
Brainstorming Question 3 |  What practices or tools do you utilize to mitigate bias and practice fairness in ML application development? Think broadly to include individual behaviors, processes, technologies, and tools based on your past experiences like projects and team meetings. Enter 5 to 10 separate ideas. Comment on ideas other people have entered and/or enter more of your own ideas. Feel free to expand on other people’s ideas.
Closing Discussion |  Summarizing all participant’s ideas and requesting their agreement for closing the agenda.

###  3.2 Developing attributes/themes from focus group’s data

####  3.2.1 Thematic analysis

The focus group data collected from the participant’s brainstorming are
qualitative in nature. Thus, the data are further analyzed utilizing thematic
analysis. Thematic analysis is broadly used for the analysis of qualitative
data for recognizing different patterns and allows researchers to formulate
rich, detailed, and transparent meanings [45]. Braun et al. [45] explain that
thematic analysis uses familiarization, code formulation, generation of
themes, themes review, defining and naming themes, and report formation. In
this research study, the thematic analysis is used to identify emerging themes
to formulate and assess the perceived fairness of ML developers.

####  3.2.2 Topic Modeling

Topic modeling is a statistical modeling approach for discovering abstract
“topics” in a collection of documents like newspapers and digital corpus [46].
Latent Dirichlet Allocation (LDA) approach is used for conducting the topic
modeling on focus group data. LDA is a popular topic modeling technique to
extract topics from the collection of documents [47]. The details of LDA-based
topic modeling can be found at [47].

An inductive approach is used to derive the themes by coding qualitative data
into clusters of similar entities and conceptual categories. The theme
derivation is done by integrating thematic analysis and LDA-based topic
modeling approach. The themes helped to formulate the theoretical explanation
and definition of ML developer’s perceived fairness in machine learning.

##  4 Findings and Discussions

###  4.1 Results and findings

In this section, we present the findings from the focus groups data collected
from ML developers. The focus groups suggest that participant’s ideas and
discussions are influenced by their personal experience, knowledge base, and
practice gained through developing ML applications. Inductive approach using
thematic analysis and topic modeling using LDA assisted to derive themes from
the developer’s discussion on focus groups. Table 3 describes the derived
themes including their attributes which describe the sub-themes derived from
the focus group data. The supportive evidence from focus groups in table 3
shows the transcripts from focus group discussions. These themes are bias
mitigation, data, model design, model validity, business rules, and users
interaction, which describe the attributes of the developer’s perceived
fairness. All the themes, based on ML developer’s discussion are described
below.

Table 3: Themes derived from thematic analysis and LDA topic modeling with transcripts as supporting evidence Themes |  Attributes |  Supporting evidence from focus groups
---|---|---
Bias Mitigation |  Historical bias, asymmetric bias, selection bias, unintended bias, human bias, implicit bias, outliers |  1) "Data wrangling and derivations are not done in such a way as to be cherry-picking data or unduly biasing the results based on human desire".
Data |  Demographics, population, data source, sampling, protected category, balanced/unbalanced data, representation, data review, data diversity, data collection, anonymity, features |  1) "if the data is coming into the model is skewed towards a certain group, the results will reflect it".
2) "I know there is a belief that machine learning models are biased. To me if
the coming data is unbalanced, but the model isn’t doing anything to skew the
output results, it is fair".
Model Design |  Algorithmic selection, Adaptability, Blank Slate, Model structure, hyperparameters, auto/manual design, active design |  1) "active design changes should be considered for the sake of fairness, as unfair".
2) "choose the appropriate ML algorithm for the data such that if data is
balanced vs unbalanced Data in any stratification factors, use a ML Model
appropriate for that design".
Model validity |  Residual analysis, performance metrics, human feedback, explainability, risk assumption, output measures, human choices, boundary conditions |  "Microsoft Azure studio is utilized as a tool for practicing fairness, as the developers stated that for outcome analysis it has a feature to control fairness by ensuring the accuracy/recall is similar across the protected groups."
Business Rules |  Project requirements, business constraints, user’s feedback, requirements, construct, user’s usability, use-case analysis, goal-specific selection, target objective, explainability, ethics, privacy |  "training data used to construct the model must precisely represent the requirements of the business product."
User interaction |  user’s feedback, user’s usability, explainability to users, case dependent |  "one more concern with data insufficiency is when populating the data with median or average for the crucial features, results are not acceptable by the users. So, data MUST be collected appropriately by the application. In Freight Acquisition model (FAM) which is currently deprecated due to data inconsistencies, users believed that model would be giving exact yes/no to call a customer. It took some time to explain the process".

####  4.1.1 Bias mitigation

: An unfair model in ML is affected due to bias in the data [6, 7, 9]. The
findings implies that developers are concerned with training the ML model
which is a true representative of the population. Key bias form that involved
in the discussion are historical, unintended (bias due to location), implicit
bias, and human annotated bias. The bias mitigation techniques used by the
developers include hold-out sampling, rigorous residual analyses, chi-square
tests on residuals (to ensure no statistically significant patterns exist in
residual distributions), implicit bias by the protected classification, and
differences in the model predictions across groups.

####  4.1.2 Data

: This theme explain data and its properties that highly depend on how data is
collected. The developer’s perception suggests that data collection (sampling)
is important to target true representatives of the population. ML developers
discussed how data is stored, processed, and transmitted in ML development
process. Data representation in input data and project requirements is
important to build an ML model. Feature engineering and data wrangling
techniques are used widely to understand the data. They discuss that feature
engineering helps them to analyze how certain variables are being weighted
based on the historical understanding of the data/problem. Key data
representation practices discussed are data transformation, cross-validation,
data wrangling, and dimensionality reduction.

####  4.1.3 Model design

: Model design reveals the model development process incorporating algorithmic
selection, parameters selection, and training of the ML model. ML Developer’s
discussed that developers must be a "blank slate" while performing the
quantitative evaluation of the model with proper metrics such that no bias
through the developer’s action can be logged.

####  4.1.4 Model validity

: This theme illustrates whether the ML model accomplishes its intended
business objective or not. The developer’s discussion enlightens that
developing explainable models is their key aspect of practicing fairness. Key
practices discussed are risk assumptions from use-case testing, evaluating
fairness from the true objectives of the scenario, peer review of code from
fellow developers, marginal analysis, analyzing true predictions based on
demographics, human-in-loop, and evaluating model performance with multiple
metrics over time.

####  4.1.5 Business rules

: Business rules are framing the ML problem by defining constraints, rules,
ethics, privacy, and stakeholder’s goals. The developer’s discussion explains
that the true objectives and goals of the ML must be set such that evaluations
can be done within the boundaries of business rules, not on human choices.
They explained that the development should not include features that violate
privacy.

####  4.1.6 Users Interaction

: The findings suggest that developers are users oriented. The developers
discuss that the business directly deals with people. Thus, a fair ML model
should be beneficial to end-users, not biased towards certain groups, and
being explainable to users. One of the important aspects of their perceived
fairness was explaining the ML flow process to users and domain experts. They
further explained that user’s feedback is recorded and then integrated with
the ML process such that ML applications are built with the appropriate
context.

Developers discussion on "fairness": Based on the findings of our study and
the discussion above, we conclude that the developer’s perceived fairness
comprises the complete ML process including privacy, ethics, the intention of
ML development, business constraints and goals, explainability to users, and
user’s usability. Interestingly, one of the developers claims that fairness in
machine learning is a subjective term and evaluation of ML models must include
ML-pipeline process.

###  4.2 Defining Perceived Fairness

In section 2.3, we reviewed the components of procedural fairness from
organization justice theory, discovered from literature. Lee et al. 2019
describes procedural fairness using transparency, control, and principle [23].
As per Lee et al. 2019, transparency is the rules of the decision-maker that
are perceived as fair and warranted including an explanation of decision
outcomes and information representativeness. Control is described as the
degree of control over the decision that individuals receive, and principle is
defined as demonstrations of consistency, competency, benevolence, and voice.
Rueda 2022 explains procedural fairness as avoidance of bias, accountability,
and transparency in medical scenarios [24]. Rueda 2022 defines transparency as
the procedure that explains ML algorithms working and processing that lead to
the outcome. Accountability is also related to the robustness of the model and
avoidance of bias describes not including attributes that can cause
unfavorable decisions [24]. Morse et al. 2021 discuss the components of
procedural fairness proposed by Leventhal 1980 as bias impression,
consistency, representativeness, correctability, accuracy, and ethicality [41,
21]. Consistency defines the uniformity of decision procedures across people
and time, accuracy is the measure of validity and high-quality information,
ethicality describes practicing moral standards and values, representativeness
describes proper population representation, bias suppression subjects to
prevent favoritism by the decision maker, and lastly, correctability are
approaches to correct flawed decisions [41, 21].

Table 4 shows an association of themes describing developer’s perceived
fairness with the components of procedural fairness proposed by Lee et al.
2019, Rueda 2022, and Leventhal 1980. These associations are proposed by the
union of the procedural fairness component’s description from the literature
discussed and the ML developer’s discussion in the focus groups. For example,
the theme "Data" describes the properties and characteristics of data in ML
development process, as discussed in Sections 4.1. This theme aligns with
Lee’s et al. 2019 transparency and control because of information
representativeness and its impact on decisions in the data-driven process.
Rueda’s 2022 avoidance of bias for fair decision-making, and Leventhal’s 1980
consistency and representativeness for uniform decision-making across people
and time. Thus, we conclude that the association in table 4 illustrates that
procedural aspects of organizational justice theory, i.e., procedural fairness
can explain the developer’s perception of ML fairness.

Definition of perceived fairness\- In section 4.1, we discussed the
characteristics of perceived fairness from the developer’s perspective. Based
on the findings of this study, a developer’s perception of fairness relates to
aspects of data, user characteristics, understanding of ML model design and
validity, and understanding of business rules that impacts their behaviors and
ability to build ML systems that are free from bias. This implies that the ML
systems are designed and built to be fair in processes, transparent in actions
(explainable), have the opportunity for multiple voices to be integrated into
their development, and are impartial to all users in their outcomes.

Table 4: Relationship between themes of fairness and procedural (justice) fairness components from literature Themes |  Lee et al. 2019 |  Rueda 2022 |  Leventhal 1980; Morse et al 2020
---|---|---|---
Bias Mitigation |  Control |  Avoidance of bias, Accountability |  Bias suppression
Data |  Control, Transparency |  Avoidance of bias |  Consistency, Representiveness
Model Design |  Transparency |  Transparency |  Correctability
Model Validity |  Transparency, Control, Principle |  Transparency, Accountability |  Accuracy, Correctability
Business Rules |  Transparency, |  Avoidance of bias, Accountability |  Ethicality
Users Interaction |  Transparency, Control, Principle |  Accountability |  Ethicality

##  5 Implications and Future Works

###  5.1 Implications

Table 5: Explaining relationships between attributes of perceived fairness of developers and ML development process Themes |  Relationship with ML process
---|---
Bias Mitigation |  Define and plan, pre-processing
Data |  Define and plan, pre-processing
Model Design |  In-processing
Model Validity |  Post-processing
Business Rules |  Define and plan
Users Interaction |  Define and plan, post-processing

In this research study, we acknowledge the relationship between themes
mentioned in table 3 with the ML development process utilized to develop ML
models and applications. The relationship shown in table 5 is validated by the
definitions discussed in the literature [48]. In literature, the ML process is
divided broadly into define-and-plan, pre-processing, in-processing, and post-
processing [3, 49, 50]. All these ML processes have distinct objectives for
any framed ML problem [48]. The findings will help and motivate ML
practitioners, researchers, and organizations to develop and further explore
the research for formulating fairness in ML. This research study will also
help researchers to understand the perceived fairness in ML in a realistic
setting and provide insights into how perceived fairness is addressed while
evaluating ML applications. It will also motivate other ML developers and
organizations to develop and practice ethical and fair ML decision-making
models to benefit society and businesses.

###  5.2 Future Works

This is a work-in-progress article. As observed, there are only 9
participants, and a focus group is organized just to perform a pilot study of
the existing research. However, the results advocate some promising future
directions including performing a large survey study.

##  6 Conclusion

In this pilot research study, we explored the ML developer’s perception of
fairness using pilot investigation through focus groups. Three companies along
with nine ML developers participated in focus groups. An inductive approach,
integrating thematic analysis and LDA-topic modeling is utilized to derive
themes that describe attributes of ML developers’ perceived fairness. The
findings of the study conclude two major arguments- 1) developer’s perceived
fairness generally focuses on the overall ML application design and
development, i.e., business-specific requirements, pre-processing, in-
processing, and post-processing. 2) the procedural aspects of organizational
justice theory can explain developer’s perception of fairness. Finally, we
proposed the definition of perceived fairness from ML developer’s perspective.

##  7 Acknowledgement

This pilot research study acknowledges and thanks all the companies and their
participants along with their managers whose participation made this study
possible. This article acknowledges the GRACA 2021 grant under the title
Perceived Fairness from Developer’s Perspective in Artificial Intelligent
Systems. The writings of this research paper were included in the GRACA grant
application and oral presentation at the research fair presentation of the
Office of Research and Creativity Activity under the title Perceived Fairness
from Developer’s Perspective in Artificial Intelligent Systems at the
University of Nebraska at Omaha (UNO) in Spring 2022. The virtual focus group
for this research study has been approved by the Institutional Review
Board(IRB 263-21-EX).

## References

  * [1] Anoop Mishra.  Intelligent human-aware decision making for semi-autonomous human rehabilitation assistance using modular robots.  Master’s thesis, University of Nebraska at Omaha, 2019.
  * [2] Anoop Mishra and Harvey Siy.  An interdisciplinary approach for teaching artificial intelligence to computer science students.  In Proceedings of the 21st Annual Conference on Information Technology Education, pages 344–344, 2020.
  * [3] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning.  MIT press, 2016.
  * [4] Anoop Mishra.  Perceived fairness from user’s and developer’s perspective in artificial intelligent systems, Ph.D. Preproposal.  In University of Nebraska at Omaha, 2022.
  * [5] Anoop Mishra.  Perceived fairness from developer’s perspective in artificial intelligent systems (oral presentation).  In GRACA, Office of Research and Creative Activity at the University of Nebraska at Omaha, 2022.
  * [6] Dana Pessach and Erez Shmueli.  A review on fairness in machine learning.  ACM Computing Surveys (CSUR), 55(3):1–44, 2022.
  * [7] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  ACM Computing Surveys (CSUR), 54(6):1–35, 2021.
  * [8] Megha Srivastava, Hoda Heidari, and Andreas Krause.  Mathematical notions vs. human perception of fairness: A descriptive approach to fairness for machine learning.  In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2459–2468, 2019.
  * [9] Heinrich Jiang and Ofir Nachum.  Identifying and correcting label bias in machine learning.  In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 702–712. PMLR, 26–28 Aug 2020.
  * [10] Ruotong Wang, F Maxwell Harper, and Haiyi Zhu.  Factors influencing perceived fairness in algorithmic decision-making: Algorithm outcomes, development procedures, and individual differences.  In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–14, 2020.
  * [11] Jeffrey Dastin.  Amazon scraps secret AI recruiting tool that showed bias against women.  Reuters, October 2018.
  * [12] Caitlin Hudon.  On Stanford’s COVID-19 Vaccination Algorithm, December 2020\.
  * [13] Taylor Telford.  Apple Card algorithm sparks gender bias inquiry - The Washington Post, nov 2019.
  * [14] Min Kyung Lee, Ji Tae Kim, and Leah Lizarondo.  A human-centered approach to algorithmic services: Considerations for fair and motivating smart community service management that allocates donations to non-profit organizations.  In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 3365–3376, 2017.
  * [15] Min Kyung Lee.  Understanding perception of algorithmic decisions: Fairness, trust, and emotion in response to algorithmic management.  Big Data & Society, 5(1):2053951718756684, 2018.
  * [16] Min Kyung Lee and Su Baykal.  Algorithmic mediation in group decisions: Fairness perceptions of algorithmically mediated vs. discussion-based social division.  In Proceedings of the 2017 acm conference on computer supported cooperative work and social computing, pages 1035–1048, 2017.
  * [17] Min Kyung Lee and Katherine Rich.  Who is included in human perceptions of ai?: Trust and perceived fairness around healthcare ai and cultural mistrust.  In Proceedings of the 2021 CHI conference on human factors in computing systems, pages 1–14, 2021.
  * [18] Allison Woodruff, Sarah E Fox, Steven Rousso-Schindler, and Jeffrey Warshaw.  A qualitative exploration of perceptions of algorithmic fairness.  In Proceedings of the 2018 chi conference on human factors in computing systems, pages 1–14, 2018.
  * [19] Galen Harrison, Julia Hanson, Christine Jacinto, Julio Ramirez, and Blase Ur.  An empirical study on the perceived fairness of realistic, imperfect machine learning models.  In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 392–402, 2020.
  * [20] Kyle A Thomas and Scott Clifford.  Validity and mechanical turk: An assessment of exclusion methods and interactive experiments.  Computers in Human Behavior, 77:184–197, 2017.
  * [21] Lily Morse, Mike Horia M Teodorescu, Yazeed Awwad, and Gerald C Kane.  Do the ends justify the means? variation in the distributive and procedural fairness of machine learning algorithms.  Journal of Business Ethics, pages 1–13, 2021.
  * [22] Nina Grgić-Hlača, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller.  Beyond distributive fairness in algorithmic decision making: Feature selection for procedurally fair learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [23] Min Kyung Lee, Anuraag Jain, Hea Jin Cha, Shashank Ojha, and Daniel Kusbit.  Procedural justice in algorithmic fairness: Leveraging transparency and outcome control for fair algorithmic mediation.  Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–26, 2019.
  * [24] Jon Rueda, Janet Delgado Rodríguez, Iris Parra Jounou, Joaquín Hortal-Carmona, Txetxu Ausín, and David Rodríguez-Arias.  “just” accuracy? procedural fairness demands explainability in ai-based medical resource allocations.  AI & society, pages 1–12, 2022.
  * [25] Google Brain.  People + AI Research, September 2021.
  * [26] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel.  Fairness through awareness.  In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214–226, 2012.
  * [27] Matt J Kusner, Joshua R Loftus, Chris Russell, and Ricardo Silva.  Counterfactual fairness.  arXiv preprint arXiv:1703.06856, 2017.
  * [28] Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Allison Woodruff, Christine Luu, Pierre Kreitmann, Jonathan Bischof, and Ed H Chi.  Putting fairness principles into practice: Challenges, metrics, and improvements.  In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 453–459, 2019.
  * [29] Or Biran and Courtenay Cotton.  Explanation and justification in machine learning: A survey.  In IJCAI-17 workshop on explainable AI (XAI), volume 8, pages 8–13, 2017.
  * [30] Finale Doshi-Velez and Been Kim.  Towards a rigorous science of interpretable machine learning.  arXiv preprint arXiv:1702.08608, 2017.
  * [31] Eustasio del Barrio, Paula Gordaliza, and Jean-Michel Loubes.  Review of mathematical frameworks for fairness in machine learning.  arXiv preprint arXiv:2005.13755, 2020.
  * [32] Sriram Vasudevan and Krishnaram Kenthapadi.  Lift: A scalable framework for measuring fairness in ml applications.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2773–2780, 2020.
  * [33] Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, et al.  Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias.  arXiv preprint arXiv:1810.01943, 2018.
  * [34] Wesley Hanwen Deng, Manish Nagireddy, Michelle Seng Ah Lee, Jatinder Singh, Zhiwei Steven Wu, Kenneth Holstein, and Haiyi Zhu.  Exploring how machine learning practitioners (try to) use fairness toolkits.  In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 473–484, 2022.
  * [35] Niels Van Berkel, Jorge Goncalves, Daniel Russo, Simo Hosio, and Mikael B Skov.  Effect of information presentation on fairness perceptions of machine learning predictors.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–13, 2021.
  * [36] Jose Maria Peiro, Vicente Martinez-Tur, and Carolina Moliner.  Perceived Fairness, pages 4693–4696.  Springer Netherlands, 2014.
  * [37] Ronald L Cohen.  Distributive justice: Theory and research.  Social justice research, 1(1):19–40, 1987.
  * [38] Dean B McFarlin and Paul D Sweeney.  Distributive and procedural justice as predictors of satisfaction with personal and organizational outcomes.  Academy of management Journal, 35(3):626–637, 1992.
  * [39] Jerald Greenberg.  A taxonomy of organizational justice theories.  Academy of Management review, 12(1):9–22, 1987.
  * [40] Nina Grgic-Hlaca, Elissa M Redmiles, Krishna P Gummadi, and Adrian Weller.  Human perceptions of fairness in algorithmic decision making: A case study of criminal risk prediction.  In Proceedings of the 2018 world wide web conference, pages 903–912, 2018.
  * [41] Gerald S. Leventhal.  What Should Be Done with Equity Theory?, pages 27–55.  Springer US, Boston, MA, 1980.
  * [42] Jenny Kitzinger.  Qualitative research: introducing focus groups.  Bmj, 311(7000):299–302, 1995.
  * [43] Deepak Khazanchi and Ilze Zigurs.  Patterns for effective management of virtual projects: Theory and evidence.  International Journal of e-Collaboration (IJeC), 2(3):25–49, 2006\.
  * [44] Tobias O. Nyumba, Kerrie Wilson, Christina J Derrick, and Nibedita Mukherjee.  The use of focus group discussion methodology: Insights from two decades of application in conservation.  Methods in Ecology and evolution, 9(1):20–32, 2018.
  * [45] Virginia Braun and Victoria Clarke.  Using thematic analysis in psychology.  Qualitative research in psychology, 3(2):77–101, 2006.
  * [46] Xin Li and Lei Lei.  A bibliometric analysis of topic modelling studies (2000–2017).  Journal of Information Science, 47(2):161–175, 2021.
  * [47] David M Blei, Andrew Y Ng, and Michael I Jordan.  Latent dirichlet allocation.  Journal of machine Learning research, 3(Jan):993–1022, 2003.
  * [48] AMaZON AWS.  Amazon machine learning documentation, 2023.
  * [49] Christian Janiesch, Patrick Zschech, and Kai Heinrich.  Machine learning and deep learning.  Electronic Markets, 31(3):685–695, 2021.
  * [50] Zhiqiang Ge, Zhihuan Song, Steven X Ding, and Biao Huang.  Data mining and analytics in the process industry: The role of machine learning.  Ieee Access, 5:20590–20616, 2017.