11institutetext: T. T. H. Nguyen (✉) 22institutetext: Friedrich-Alexander-
Universität Erlangen-Nürnberg, Erlangen, Germany
22email<EMAIL_ADDRESS>33institutetext: V. B. Truong 44institutetext:
V. T. K. Nguyen 55institutetext: Q. H. Cao 66institutetext: Q. K. Nguyen
77institutetext: T. T. H. Nguyen 88institutetext: Quy Nhon AI, FPT Software,
Quy Nhon, Vietnam
88email<EMAIL_ADDRESS>99institutetext: V. T. K. Nguyen
1010institutetext: 1010email<EMAIL_ADDRESS>1111institutetext: Q. H.
Cao 1212institutetext: 1212email<EMAIL_ADDRESS>1313institutetext: Q.
K. Nguyen 1414institutetext: 1414email<EMAIL_ADDRESS>
# Towards Trust of Explainable AI in Thyroid Nodule Diagnosis

Truong Thanh Hung Nguyen  Van Binh Truong  Vo Thanh Khang Nguyen  Quoc Hung
Cao  Quoc Khanh Nguyen

###### Abstract

The ability to explain the prediction of deep learning models to end-users is
an important feature to leverage the power of artificial intelligence (AI) for
the medical decision-making process, which is usually considered non-
transparent and challenging to comprehend. In this paper, we apply state-of-
the-art eXplainable artificial intelligence (XAI) methods to explain the
prediction of the black-box AI models in the thyroid nodule diagnosis
application. We propose new statistic-based XAI methods, namely Kernel Density
Estimation and Density map, to explain the case of no nodule detected. XAI
methods’ performances are considered under a qualitative and quantitative
comparison as feedback to improve the data quality and the model performance.
Finally, we survey to assess doctors’ and patients’ trust in XAI explanations
of the model’s decisions on thyroid nodule images.

###### Keywords:

Explainable artificial intelligence, Object detection, Thyroid nodule, Medical
imaging

##  1 Introduction

Thyroid cancer is one of the most common cancer types and is the leading cause
of cancer death worldwide deng2020global ; kim2020geographic , especially
during the COVID-19 pandemic giannoula2022updated ; guven2021prognostic .
Characterized by malignant cells formed in the thyroid gland tissues, the
thyroid cancer prognosis depends on the type and the stage at which the
disease is detected. Often, doctors rely on the medical images’
interpretation, such as thyroid ultrasound images, to identify nodules’
presence and provide a diagnosis. However, in routine cancer screening, the
errors are mainly false negatives, in which a nodule is present but undetected
hebert_redundancy_2020 . Due to recent advances in AI, deep learning models
can now serve as decision support means for medical experts. A medical
diagnosis system must be accurate, transparent, and explainable to gain end-
users’ trust. Considering the explainability capability, simple AI methods
such as linear regression and decision trees are self-explanatory. Still,
these methods lack the complexity required for tasks such as classifying two
and three-dimensional medical images. Given the increasing ubiquity of
advanced techniques such as deep neural networks (DNNs), a new challenge for
medical AI is its so-called black-box nature, with decisions that seem opaque
and inscrutable, even for experts to understand 10.1145/3236009 . Thus, while
their opacity is deeply intertwined with their success, it poses a challenge
for applying DNNs to high-stakes problems such as medical imaging until we can
develop methods that allow radiologists to develop understanding and
appropriate trust. Furthermore, newer regulations like the European General
Data Protection Regulation (GDPR) strictly require transparency in black-box
models, especially in healthcare. Thus, there is a growing chorus of
researchers calling for XAI methods. Therefore, in this paper, our main
contributions are:

  1. 1.

We applied several XAI methods, namely LIME Lime , RISE petsiuk2018rise ,
Grad-CAM selvaraju2017grad , Grad-CAM++ gradcamplusplus , Ada-SISE
sudhakar2021ada , LRP bach2015pixel , and D-RISE petsiuk2021black  to explain
the two-stage model’s classification and localization of nodules.

  2. 2.

We proposed two statistic-based XAI methods, namely Kernel Density Estimation
(KDE) and Density map (DM), to monitor the two-stage model’s localization
process from the first stage to the second stage and further explain the case
of no nodule detected, especially false negative case.

  3. 3.

We evaluated XAI methods’ performance and suitability for the specific nodule
detection cases with qualitative and quantitative results and a surveyed XAI’s
trust to end-users.

Our code is available for reproductivity on
GitHub111https://github.com/hungntt/xai\\_thyroid.

###  1.1 Dataset

[With nodule]

(a) Without nodule

Figure 1: Examples of the Vietnamese thyroid ultrasound dataset. The green box
is the ground truth labeled by doctors.

We use the thyroid ultrasound dataset from pham2021evaluating  that contains
14171 thyroid ultrasound images of 970 Vietnamese patients. Samples are shown
in Fig. 1, and medical experts label the nodule locations.

##  2 Related Work

###  2.1 Backpropagation-based methods

Backpropagation-based methods calculate the gradients of the model’s output to
the input features or hidden neurons. Hence, they utilize the backward pass of
information flow to understand neuronal influence and relevance of input
towards the output. The first gradients explanation technique proposed in
DBLP:journals/corr/SimonyanVZ13  computes how much a change in each input
dimension changes the predictions in a small neighborhood around the input.
Some preceding backpropagation-based equations take relative importance given
to gradient value during backpropagation to generate saliency maps
10.1007/978-3-319-10590-1_53 ; smilkov2017smoothgrad . While LRP bach2015pixel
modifies backpropagation rules to measure the relevance or irrelevance of the
input features to the models’ prediction.

###  2.2 CAM-based methods

Based on the CAM method zhou2016learning , an extensive research effort on
CAM-based methods has been put to blend high-level features extracted by CNNs
into a unique explanation map. Grad-CAM selvaraju2017grad  and Grad-CAM++
gradcamplusplus  are two improvements over CAM that utilize backpropagation to
provide a better visual explanation for classifiers.

###  2.3 Perturbation-based methods

Perturbation-based methods are a class of techniques for explaining the
decision-making process of DNNs by modifying the model’s input and observing
the output’s changes. LIME Lime  explains the prediction by learning an
interpretable model that approximates the model locally around a data point
using occlusions of superpixels. RISE petsiuk2018rise  proposed a method for
producing saliency maps using random perturbation techniques without having to
analyze the model’s complex structure. D-RISE petsiuk2021black  extended RISE
to produce saliency maps for object detectors. SISE sattarzadeh2021explaining
improved upon RISE’s fidelity and plausibility using attribution-based input
sampling techniques. Still, it has high computational costs when there are
many activation maps with positive slopes that are inefficient in the
prediction process. Ada-SISE sudhakar2021ada  was developed to solve this
problem by removing unnecessary objects, which saves computational time and
provides a better reasonable explanation.

###  2.4 Statistic-based methods

Kernel Density Estimation (KDE) is a non-parametric mathematical method for
estimating the probability density function of a continuous variable
terrell1992variable ; wkeglarczyk2018kernel  which is one of the most common
methods for estimating density level, set estimation, clustering, or
unsupervised learning parzen1962estimation . Recently, KDE has been made
explainable with LRP for outlier and inlier detection in unsupervised learning
models without the ground-truth labels montavon2022explaining . Density map is
commonly used in crowd counting literature, which is usually for estimating
the distribution of objects, namely people in the scene li2020density .
However, the idea of counting the model’s detected boxes to estimate the
distribution of predicted boxes as an explanation has not been applied in
previous works.

###  2.5 XAI in the medical diagnosis system

Several XAI applications for different cancer diagnoses are proposed to answer
the black-box AIs. In recent years, there have been 37 publications on how XAI
is applied in skin cancer detection skincancer . More than half of the
articles applied current XAI methods to their model, nearly 40% tried to solve
specific problems such as bias detection and the effect of XAI on man-machine
interactions, and the remaining 10% offered novel XAI methods or enhanced
existing techniques. Recently, during the outbreak of COVID-19, LIME is also
applied to explain the model’s interpretability for screening patients with
COVID-19 symptoms ahsan2020study . In the same context as our study, AIBx
thomas2020aibx  employed the image similarity model and physicians to create
an XAI model, increasing physicians’ confidence in the predictions during the
thyroid cancer diagnosis process.

However, the application of a wide-ranged number of XAI methods to nodule
detection on thyroid ultrasound datasets has not been discovered yet.
Urgently, very little is known about the influence of XAI on the predictive
performance, confidence, and model trust of doctors and radiologists in an
artificial setting, and nothing is known about its effects in a clinical
setting.

##  3 Methodology

###  3.1 Object Detector and XAI categorization

#### Analysis of images with nodule

[width=]images/xai_methods.pdf

Figure 2: Object detector’s architecture and XAI methods’ applicability to
different tasks (red arrows).

Our object detector employs state-of-the-art object detection networks based
on the Faster R-CNN fasterrcnn  and RetinaNet RetinaNet  architectures to
detect thyroid in ultrasound images. The model detection process comprises two
stages. In the first stage, the Region Proposal Network generates object
proposals from input images. Next, a bounding box is predicted for each object
proposal, with a probability of whether the box contains a thyroid nodule. In
the second stage, the Region-of-Interest pooling layer implements bounding box
regression and bounding box classifier. We categorize XAI methods in terms of
their applicability to three main blocks of an object detector (as shown in
Fig. 2):

  * •

Region Proposal Generation (Which proposals are generated by the model during
the model’s first stage?): Kernel Density Estimation (KDE), Density map (DM).

  * •

Classification (Which features of an image make the model classify an image
containing a nodule(s) at the model’s second stage?): LRP, Grad-CAM, Grad-
CAM++, LIME, RISE, Ada-SISE, D-RISE.

  * •

Localization (Which features of an image does the model consider to detect a
specific box containing a nodule at the model’s second stage?): D-RISE.

Because XAI methods for the second stage (LRP, Grad-CAM, Grad-CAM++, LIME,
RISE, Ada-SISE, D-RISE) require the model’s output bounding boxes, they are
only applicable to positive cases, where the model detects a nodule in the
image. While XAI methods for the first stage (KDE, DM) directly extract the
model’s attempts to find a nodule, they can be further applied to negative
cases where the model does not detect any nodule.

#### Local Interpretable Model-agnostic Explanation (LIME)

For any given instance and its corresponding prediction, LIME utilizes Simple
Linear Iterative Clustering achanta2010slic  as a segmentation algorithm to
randomly sample data around the neighborhood of the input instance for which
produced predictions. These generated data are used to train a local model.
The local model’s prediction generates an explanation by weighting each sample
according to the instance. Then, LIME uses LASSO as a feature selection
technique to choose the most important segments that contribute the most to
the prediction for the explanation.

#### Random Input Sampling for Explanation (RISE)

In the masking generation, we firstly use binary-bit masking $(0,1)$ to
generate $N=500$ samples, where masks $M=\left\\{M_{i},1\leq i\leq N\right\\}$
for each superpixel with a probability $p=0.5$ petsiuk2018rise . We set each
sample’s size as $8\times 8$. We upsample these masks using bilinear
interpolation to ensure all values are in the range $[0,1]$. Then, we feed
samples into the model to get the bounding boxes and the corresponding score
for each box $S^{b}$. Finally, RISE sums up all the masks using the box
scores, which are predicted on each sample as the weight of each mask to
explain the target box from the input image in the form of a saliency map.

#### Adaptive-Semantic Input Sampling for Explanation (Ada-SISE)

Ada-SISE selects multiple layers of the model and extracts feature maps by
feeding the input image into the model. Then, it samples subsets of the
feature maps that contain the most important features by partially
backpropagating the signal to the layer and calculating the average gradient
scores for each feature map. It then collects all feature maps with positive
scores and applies an Otsu-based threshold otsu1979threshold  to remove those
with lower scores. It produces attribution masks by bilinearly interpolating
and normalizing the positive feature maps. For each selected layer, it obtains
a layer visualization map by computing the weighted sum of the attribution
masks. Finally, it combines obtained saliency maps in a fusion module to
produce a final explanation.

#### Gradient-weighted class activation mapping (Grad-CAM) and Grad-CAM++

Grad-CAM selvaraju2017grad  and Grad-CAM++ gradcamplusplus  are methods for
producing a saliency map, which is a visual representation of the regions in
an input image that is most important for a particular task or model.

Grad-CAM is a technique that uses the gradients of the target class with
respect to the final convolutional layer of a CNN to produce a coarse
localization map of the important regions in the input image. The map is then
upsampled and weighted by the gradients to produce the final saliency map.

Grad-CAM++ is an extension of Grad-CAM that produces a more fine-grained and
accurate localization map by using the gradients of the target class to the
lower convolutional layers and the final convolutional layer. It also
introduces a new way of weighting the upsampled map using a weighted
combination of the gradients of the target class and the activations of the
lower convolutional layers. The final saliency map is also produced by
upsampling and weighting the localization map using the weighted combination
of gradients and activations.

#### Layer-wise Relevance Propagation (LRP)

LRP uses the weights and activations of a neural network to propagate
relevance scores from the output layer back to the input layer according to
the conservation rule, which states that a neuron that receives what is from
the upper layer must be redistributed entirely to the lower layer in equal
numbers. The relevance scores are propagated by calculating the quantity
representing how much neuron $j$ contributes to making neuron $k$, where $j$
and $k$ are neurons in two consecutive layers of the neural network. The
propagation procedure terminates once the input features have been reached.
LRP has several propagation rules with different non-linear rectifiers. Still,
we use the epsilon rule, which adds a small positive term in the denominator
of the equation used to propagate the relevance scores, because it helps to
solve cases where the denominator is zero and typically leads to sparser
explanations in terms of input features and less noise.

###  3.2 XAI methods for the localization task

#### Kernel Density Estimation (KDE)

KDE creates a distribution density map by weighting distances of all data
points for each specific location along the distribution. If more data points
are grouped locally, the estimation is higher as the probability of seeing a
point at that location increases.

In the object detector’s first stage, we apply KDE to the center points of 300
independent and identically distributed $(b_{0},b_{1},\dots,b_{300})$ boxes to
obtain the distribution density map given by:

$\widehat{p}_{n}(b)=\frac{1}{nh}\cdot\sum_{i=1}^{n}K\left(\frac{B_{i}-b}{h}\right)$
(1)

where $K(b)$ is called the kernel function that is generally a smooth,
symmetric function such as a Gaussian and $h>0$ called the smoothing bandwidth
controls the amount of smoothing.

We represent the distribution density map as multiple continuous probability
density curves on the image. The KDE score of a point is computed as the log-
likelihood of that point under the KDE model. The score reflects the
likelihood that any given box has been drawn from the learned probability
distribution. The higher the KDE score is, the more the given box matches the
distribution.

The prediction’s consistency KDE (PCKDE) is computed as the ratio between the
KDE score of the final box’s center point detected by the model and the KDE
score of the point achieving the highest probability value on the distribution
density map. Finally, we set the threshold as 0.5 to grade the PCKDE score. If
the PCKDE score exceeds 0.5, the model’s detection is considered consistent,
as shown in Fig. 3 and non-consistent in Fig. 3a.

[Consistent (PCKDE=0.934)]

(a) Non-consistent (PCKDE=0.297)

Figure 3: (a) Consistent: the model detection’s center point (red) is mostly
at the same location as the point achieving the highest KDE score (green). (b)
Non-consistent: the model detection’s center point is far from the point
having the highest KDE score.

#### Density map (DM)

DM is commonly used in crowd counting context to estimate objects’
distribution in an image li2020density . We propose DM as a new statistic-
based method in the context of XAI by extracting the frequency value of each
pixel from boxes predicted by the model in region proposal generation, as
shown in Fig. 4. The pixel’s frequency value is calculated by the number of
boxes containing that pixel. For an input image $X\in\mathbb{R}^{3\times
H\times W}$, the model detected $n$ boxes after the first stage. For any box
$k$, let $coor_{k}$ be the set of coordinates $(i,j)$. Thus, DM’s output $D$
of $X$ is the $H\times W$ matrix defined as $D_{X}=\sum_{i=1}^{n}B_{k}$, where
$B_{k}$ is computed:

$\begin{gathered}B_{k}=[a_{ij}]_{H\times W}\\\
a_{ij}=1_{\text{coor}_{k}}[(i,j)]\end{gathered}$ (2)

The more focused boxes a pixel has, the redder colors are indicated in the
DM’s explanation. In Fig. 4a, the model detects two boxes containing nodules
where the blue box is correct with the ground-truth label, while the red box
is false. The DM’s saliency map can explain the blue box with redder colors,
indicating that the model focuses on this region to detect the nodule.

[]

(a)

Figure 4: (a) DM extracts all model’s regional proposals after the first
stage. (b) DM’s explanation as a saliency map.

#### Detector Randomized Input Sampling for Explanation (D-RISE)

D-RISE is a method for producing saliency maps that explain the regions of an
input image that are important for a particular task or model. It is
specifically designed to produce saliency maps for object detectors, making it
the first method. It involves generating a set of binary-bit masks for each
superpixel in the input image, upsampling the masks using bilinear
interpolation, feeding the samples into a model to obtain bounding boxes and
scores for each box, and summing up all the masks using the box scores as
weights to produce the final saliency map. The regions of the input image that
significantly impact the model’s prediction appear as darker colors on the
saliency map.

##  4 Results

###  4.1 Qualitative Evaluation

#### Analysis of positive cases (with nodules)

Figure 5: Qualitative comparison between XAI explanations. In the input
images, blue boxes are the correct model’s predictions, red boxes are the
wrong model’s predictions, and green boxes are ground truths. The first row is
the correct localization case where the predicted box overlaps the ground-
truth label. The second row is the mislocalization case where the model
predicted two boxes, but the nodule only exists in the left box.

In the positive case, we evaluate XAI explanations on two cases of true
positive classification, namely correct localization and mislocalization, as
shown in Fig. 5. A correct localization is when the intersection over the
union between the detected box for a nodule and the ground-truth box is larger
than 0.5. All our implemented XAI methods’ explanations are applicable in the
correct localization case, and their explanations are consistent with the
model’s detected box in the correct localization case. Because D-RISE is the
only method having access to the localization block, it can give end-users
separate explanations for each detected object. Meanwhile, all other XAI
methods show explanations for all nodules since they explain to end-users why
the model classifies an image containing nodules.

To further observe D-RISE’s advantages, we consider the mislocalization case
where the model predicted two boxes, but the nodule only exists in the left
box. In this case, D-RISE is the only method to show the explanation solely
for the correct and incorrect boxes, while others explain both. Hence,
D-RISE’s results are more understandable to end-users when each nodule
detection needs to be explained.

#### Analysis of negative cases (without nodules)

One serious error of AI-assisted thyroid diagnosis is a false negative, where
the model fails to detect any nodule. Hence, our proposed methods, KDE and DM,
are the only applicable for explaining negative cases, namely true negative
and false negative. As shown in Fig. 6, KDE estimates the distribution of the
model’s prediction placed around the image’s corner. The same model’s behavior
is also reflected in DM, where the saliency map shows hot regions around the
image’s border. They both show that the model does not concentrate on any part
inside the image, as it does not detect any nodules.

[width=.6]images/tn.pdf

Figure 6: Explanations of KDE and DM for true negative and false negative
cases.

###  4.2 Quantitative Evaluation

Two critical aspects of XAI, plausibility and faithfulness, are evaluated by
quantitative metrics. These metrics are used to justify the model by assessing
the extent to which the method satisfies the users by providing superior
statistical explanations. All methods are evaluated on the whole dataset.

#### Plausibility Evaluation

  * •

Energy-Based Pointing Game (EBPG) evaluates XAI methods’ precision and
denoising ability wang2020score . Extending the traditional Pointing Game,
EBPG considers all pixels in the resultant saliency map $S$ for evaluation by
measuring the fraction of its energy captured in the corresponding ground
truth $G$.

  * •

Intersection over Union (IoU) analyses the localization ability and
meaningfulness of the attributions captured in an explanation map. Initially,
Otsu-based binarization otsu1979threshold  is applied to convert ultrasound
images into binary images. We compute the mean IoU between the explanation box
and the ground truth box.

  * •

Bounding box (Bbox) is considered a size-adaptive variant of IoU
schulz2019restricting , calculated by selecting the top $N$ pixels in the
saliency map significantly influencing the prediction results. It evaluates
the regions captured by the bounding box which contains the object.

#### Faithfulness Evaluation

We evaluate XAI’s faithfulness with the Drop and Increase fu2020axiom . The
original image is perturbed by masking the important areas marked by the
explanation.

  * •

Drop checks the target audience’s average predicted drop when using the
explanatory as input.

  * •

Increase measures the number of times the model’s confidence increases when
using the explanation as input.

Metric | KDE | DM | LIME | Grad-CAM | Grad-CAM++ | LRP | RISE | Ada-SISE | D-RISE
---|---|---|---|---|---|---|---|---|---
EPBG $\uparrow$ | 29.45 | 21.06 | 10.95 | 48.58 | 52.11 | 31.17 | 17.04 | 50.31 | 17.52
BB $\uparrow$ | 49.16 | 40.09 | 14.49 | 60.51 | 47.65 | 38.45 | 62.41 | 55.87 | 63.42
IoU $\uparrow$ | 18.94 | 18.73 | 10.09 | 45.22 | 49.55 | 44.98 | 12.06 | 14.99 | 12.07
Drop $\downarrow$ | 34.39 | 27.31 | 16.21 | 26.88 | 45.76 | 65.81 | 4.24 | 12.43 | 2.34
Inc $\uparrow$ | 32.20 | 27.12 | 27.12 | 18.08 | 9.60 | 4.52 | 46.33 | 45.13 | 53.67
Time(s) $\downarrow$ | 66 | 28 | 380 | 0.75 | 0.8 | 0.55 | 245 | 295 | 319
Table 1: Mean accuracy (%) of quantitative results and computational time of
all XAI methods. For each metric, the arrow $\uparrow/\downarrow$ indicates
higher or lower scores are better. The best is shown in bold, and the second
best is underlined.

#### Result

Table 1 shows the quantitative comparison of XAI methods concerning their
plausibility and faithfulness. In detail, CAM-based methods achieved good
results with plausibility metrics, especially EPBG and IoU, in a reasonably
short time. While RISE and D-RISE perform better than other methods in terms
of faithfulness metrics, such as Drop and Increase, as they faithfully reflect
the model’s behavior. Nevertheless, the computational time of D-RISE and RISE
in specific and perturbation-based methods, in general, are the highest due to
their perturbation process. LRP achieved the highest computation efficiency,
which is its main advantage.

###  4.3 Evaluating User Trust

An essential question of XAI applications is whether explanations can build
end-users’ trust in the AI system. We invited 16 participants with previous
experience and familiarity with ultrasound images and AI models, namely three
radiologists, one patient, six medical students, and 6 AI scientists from
different countries, to take a survey. Six participants were self-reported as
from Vietnam, five were from European countries (Germany, Romania, Ukraine),
four were from North American countries (United States, Canada), and one was
from Pakistan, as shown in Fig. 7. In this survey, the trust of humans towards
XAI’s explanation is evaluated in terms of understandability, where this
explanation helped us understand why the model behaved as it did.

.45!
VNEUNAOthers$2$$4$$6$$6$$5$$4$$1$$4$$3$$2$$1$$3$$3$$2$$1$$1$$2$$2$$1$#participantsMedical
studentRadiologistPatientAI Scientist

Figure 7: The number of participants from Vietnam (VN), European countries
(EU), North American countries (NA), and other countries. Participants are
medical students, radiologists, patients, and AI scientists.

(a)

(b)  (c)  Figure 8: Participants ratings of XAI methods in (a) True Positive,
(b) False Positive, (c) True Negative, and (d) False Negative.

[]

All participants answered seven questions categorized into four prediction
cases, namely True Positive (TP), False Positive (FP), True Negative (TN), and
False Negative (FN). In detail, TP cases have four questions for the following
specific cases: one question for one nodule correctly detected, one question
for two nodules correctly detected (correct localization), and two questions
for one nodule correctly detected but another falsely detected
(mislocalization). While there is one question for each FP, TN, and FN. Each
question represents a specific case of the model’s classification and
localization where the model’s prediction, ground truth, and explanations of
applicable XAI methods are shown. Participants rated their trust of each XAI’s
explanation on a labeled, 5-point Likert scale, ranging from 1 (very unlikely)
to 5 (very likely). The box plots in Fig. 8b statistically report the score of
participants’ ratings of XAI methods on four prediction cases. All box plots
use the Altman whiskers to display the spread of participants’ ratings, which
can be particularly useful for showing outliers.

In general, XAI methods using a saliency map as the explanation have
consistently higher understandability with higher medians towards humans than
others in all cases. D-RISE gains the most trust from users in TP and FP
cases, where its interquartile ranges are from 4 to 5. While in TN and FN
cases, DM overall surpasses KDE. Still, despite not having a user-friendly
explanation, KDE’s maximum whisker reaches 5, which means that it still gains
some high trust from users. Also, note that all methods contain low-rating
outliers, which means that probably few users are still confused with
explanations. Thus, the future of different explanation types is still wide-
open.

##  5 Conclusion

We applied several XAI methods and proposed two new statistic-based methods,
namely KDE and Density map, in the context of XAI to explain the model’s
predictions on the Vietnamese thyroid ultrasound dataset. Our implemented and
proposed XAI methods can cover all prediction cases with high consistency with
the object detector and doctors’ knowledge. Consequently, according to our
evaluations and surveys, we recommend end-users use Grad-CAM++ as the default
method since it requires a very short time to explain plausibly per case. At
the same time, D-RISE is suitable when we require explicitly separate
explanations for each nodule due to its faithfulness but high computational
time. In future works, we would like to integrate XAI methods into the
diagnosis process in real-time scenarios, so the transparency of AI decisions
to doctors and patients can be improved. Also, we aim to conduct a more
comprehensive survey that spans multiple countries and continents involving
more various user subjects to increase the representativeness and reliability
of XAI in the medical field.

## References

  * (1) Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk.  Slic superpixels.  Technical report, 2010.
  * (2) Md Manjurul Ahsan, Kishor Datta Gupta, Mohammad Maminur Islam, Sajib Sen, Md Rahman, Mohammad Shakhawat Hossain, et al.  Study of different deep learning approach with explainable ai for screening patients with covid-19 symptoms: Using ct scan and chest x-ray image dataset.  arXiv preprint arXiv:2007.12525, 2020.
  * (3) Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PloS one, 10(7):e0130140, 2015.
  * (4) Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In 2018 IEEE winter conference on applications of computer vision (WACV), pages 839–847. IEEE, 2018.
  * (5) YuJiao Deng, HongTao Li, Meng Wang, Na Li, Tian Tian, Ying Wu, Peng Xu, Si Yang, Zhen Zhai, LingHui Zhou, et al.  Global burden of thyroid cancer from 1990 to 2017.  JAMA network open, 3(6):e208759–e208759, 2020.
  * (6) Ruigang Fu, Qingyong Hu, Xiaohu Dong, Yulan Guo, Yinghui Gao, and Biao Li.  Axiom-based grad-cam: Towards accurate visualization and explanation of cnns.  arXiv preprint arXiv:2008.02312, 2020.
  * (7) Evanthia Giannoula, Ioannis Iakovou, Luca Giovanella, and Alexis Vrachimis.  Updated clinical management guidance during the covid-19 pandemic: thyroid nodules and cancer.  European Journal of Endocrinology, 186(4):G1–G7, 2022.
  * (8) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi.  A survey of methods for explaining black box models.  ACM computing surveys (CSUR), 51(5):1–42, 2018.
  * (9) Mehmet Güven and Hamza Gültekin.  The prognostic impact of thyroid disorders on the clinical severity of covid-19: Results of single-centre pandemic hospital.  International Journal of Clinical Practice, 75(6):e14129, 2021.
  * (10) Katja Hauser, Alexander Kurz, Sarah Haggenmüller, Roman C Maron, Christof von Kalle, Jochen S Utikal, Friedegund Meier, Sarah Hobelsberger, Frank F Gellrich, Mildred Sergon, et al.  Explainable artificial intelligence in skin cancer recognition: A systematic review.  European Journal of Cancer, 167:54–69, 2022.
  * (11) Claudia R Hebert, Li Z Sha, Roger W Remington, and Yuhong V Jiang.  Redundancy gain in visual search of simulated x-ray images.  Attention, Perception, & Psychophysics, 82(4):1669–1681, 2020\.
  * (12) Jina Kim, Jessica E Gosnell, and Sanziana A Roman.  Geographic influences in the global rise of thyroid cancer.  Nature Reviews Endocrinology, 16(1):17–29, 2020.
  * (13) Changlin Li, Taojiannan Yang, Sijie Zhu, Chen Chen, and Shanyue Guan.  Density map guided object detection in aerial images.  In proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 190–191, 2020.
  * (14) Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 936–944, 2017.
  * (15) Grégoire Montavon, Jacob Kauffmann, Wojciech Samek, and Klaus-Robert Müller.  Explaining the predictions of unsupervised learning models.  In International Workshop on Extending Explainable AI Beyond Deep Models and Classifiers, pages 117–138. Springer, 2022.
  * (16) Nobuyuki Otsu.  A threshold selection method from gray-level histograms.  IEEE transactions on systems, man, and cybernetics, 9(1):62–66, 1979.
  * (17) Emanuel Parzen.  On estimation of a probability density function and mode.  The annals of mathematical statistics, 33(3):1065–1076, 1962.
  * (18) Vitali Petsiuk, Abir Das, and Kate Saenko.  Rise: Randomized input sampling for explanation of black-box models.  arXiv preprint arXiv:1806.07421, 2018.
  * (19) Vitali Petsiuk, Rajiv Jain, Varun Manjunatha, Vlad I Morariu, Ashutosh Mehra, Vicente Ordonez, and Kate Saenko.  Black-box explanation of object detectors via saliency maps.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11443–11452, 2021.
  * (20) Tri-Cong Pham, Antonie Doucet, Van-Dung Hoang, Quang-Hoa Nguyen, Trong-Bach Phan, Cong-Thanh Tran, Tran-Tien Bui, Chi-Mai Luong, and Van-Giang Bui.  Evaluating the deep convolutional neural network for thyroid nodule detection on vietnamese ultrasound dataset.  In Advances in Intelligent Information Hiding and Multimedia Signal Processing, pages 358–366. Springer, 2021.
  * (21) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2017.
  * (22) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ”why should i trust you?” explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * (23) Sam Sattarzadeh, Mahesh Sudhakar, Anthony Lem, Shervin Mehryar, Konstantinos N Plataniotis, Jongseong Jang, Hyunwoo Kim, Yeonjeong Jeong, Sangmin Lee, and Kyunghoon Bae.  Explaining convolutional neural networks through attribution-based input sampling and block-wise feature aggregation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11639–11647, 2021.
  * (24) Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf.  Restricting the flow: Information bottlenecks for attribution.  In International Conference on Learning Representations, 2019.
  * (25) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * (26) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * (27) Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  arXiv preprint arXiv:1706.03825, 2017.
  * (28) Mahesh Sudhakar, Sam Sattarzadeh, Konstantinos N Plataniotis, Jongseong Jang, Yeonjeong Jeong, and Hyunwoo Kim.  Ada-sise: adaptive semantic input sampling for efficient explanation of convolutional neural networks.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1715–1719. IEEE, 2021.
  * (29) George R Terrell and David W Scott.  Variable kernel density estimation.  The Annals of Statistics, pages 1236–1265, 1992.
  * (30) Johnson Thomas and Tracy Haertling.  Aibx, artificial intelligence model to risk stratify thyroid nodules.  Thyroid, 30(6):878–884, 2020.
  * (31) Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu.  Score-cam: Score-weighted visual explanations for convolutional neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 24–25, 2020.
  * (32) Stanisław Węglarczyk.  Kernel density estimation and its application.  In ITM Web of Conferences, volume 23, page 00037. EDP Sciences, 2018\.
  * (33) Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.
  * (34) Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929, 2016.


11institutetext: McGill University

# Unsupervised Domain Adaptation for Cerebellum Segmentation

Xuan Li  Paule-J Toussaint  Alan Evans  Xue Liu

###### Abstract

Unsupervised domain adaptation (UDA) is a popular approach in medical imaging
for transferring annotation knowledge from a labelled source domain to an
unlabelled target domain. However, manual labelling of medical images is often
time-consuming, particularly for high-resolution images with sparse
annotations such as the cerebellar atlas with its convoluted foliations. The
lack of comprehensive annotation has hindered studies of cerebellar
involvement in normal brain function and disease. Empirically, we find that
distinct visual discrepancy due to different staining and spacing between
sections, only single subject in each domain, and few available sections in
the source domain create uncertainty and inconsistency when generating pseudo
labels for the target dataset. These factors can prevent existing approaches
from providing meaningful segmentation annotations. To address these
challenges, we propose a Bayesian learning framework that imposes constraints
on the pseudo labels and guides the segmentation model to learn from regions
with the most confidence. We design a variance map to rectify the uncertainty
using variational sampling. Our approach generates expert-approved semantic
annotations, as validated by qualitative results, and achieves over 2% loss
reduction and over 3.7% intersection-over-union (IoU) improvement compared to
other approaches in quantitative experiments.

###### Keywords:

Semantic segmentation Unsupervised domain adaptation Histology parcellation.

##  1 Introduction

Recent advancements in convolutional neural networks (CNNs) have shown
significant progress in computer-aided medical imaging analysis [1, 2, 3].
However, obtaining expert-level annotations for medical datasets is expensive
and time-consuming, particularly for high-resolution histology data [4, 5],
such as the BigBrain atlas [6], an open-source 3D human brain model for which
a comprehensive cerebellum segmentation is lacking. Unlike the common 1mm MRI,
BigBrain atlas is scanned at 20µm to visualize the highly compact and
convoluted cerebellar foliations (as shown in Figure 1 (a) and (c)). Expert
manual annotation of the cortical structures is time-consuming [7].

---
Figure 1: Visual difference between 20-micron BigBrain (a), Allen Brain (b)
histologies, and 1mm BigBrain MRI (c). Different staining methods, noise
present in Allen Brain, and tissue tears in BigBrain cause large visual
discrepancies between the two domains. Whereas with the coarser resolution
MRI, it is impossible to observe the neural pathways.

To address this limitation, unsupervised domain adaptation (UDA) is a popular
approach that transfers annotation knowledge from an existing labelled dataset
(the source domain) to an unlabelled target domain [8, 9, 10, 11]. In this
work, we explore unsupervised semantic segmentation for the BigBrain (BB)
cerebellum, using the Allen Brain atlas [12] (AB) as the source domain.
However, this task presents additional challenges. First, each domain only
contains cerebellum sections from a single subject, and these two subjects are
demographically distinct. This heterogeneous nature results in large domain
variance. Second, images in the two domains use different staining methods to
visualize neurons, resulting in visual discrepancies. as indicated in Figure
1: different color intensity distribution, the presence of noise in AB, and
damage in BB. Semantic segmentation in medical imaging primarily relies on
low-level features (texture, intensity, etc.) [8]. Discrepancies in these low-
level features bring extra challenges. Lastly, there are only 24 annotated
cerebellum sections in AB. Without abundant data, it is hard for the model to
generalize well on the target dataset.

To address these challenges, we propose BUDA-Seg, a Bayesian framework that
quantifies the uncertainty in pseudo labels from a probabilistic perspective
and guides the segmentation model to learn from regions with the most
confidence. Concretely, our framework uses a pre-trained segmentation model on
the AB to generate pseudo labels for the BB. To offset the noise and
inconsistency in the pseudo labels, the framework outputs a variance map which
can rectify the pixel-level noise in the pseudo labels. We then utilize
variational sampling to approximate the true distribution of segmentation
annotations for BB. To our knowledge, our proposed framework is the first to
achieve UDA for semantic segmentation with limited data in both source and
target domains. Our contributions are summarized in the following 111Our code
and data are available online at https://anonymous.4open.science/r/BUDA-
Seg-8D75/README.md:

  * •

We propose BUDA-Seg, a framework that uses Bayesian learning to improve UDA in
semantic segmentation with limited data. It addresses the challenges posed by
the noisy pseudo labels and distributional discrepancies between domains.

  * •

We demonstrate the effectiveness of our approach through quantitative and
qualitative experiments on high-resolution cerebellum atlas and electron
microscopy datasets. Our generated annotations for the cerebellum dataset will
be publicly released for future research.

##  2 Related Work

Our research focuses on the task of unsupervised domain adaptation in semantic
segmentation and specifically uses the label-explicit adaptation approach,
which relies on the pseudo label.

Label-explicit adaptation uses the segmentation model trained on the source
domain to generate pseudo labels for the target domain. Pseudo labels are
further improved through iterative self-training. Hoffman et al. [13] adopt
CycleGAN [14] to transfer the visual appearance of the source domain to the
target domain, followed by iterative self-training to refine the segmentation
outputs. Similarly, Yang et al. [15] adopt Fourier transform on the two
domains and then replace low-frequency components of the target domain with
the source domain. The resulting target images have a visual appearance
resembling the source domain. In [16, 17, 18], attention modules are used,
where spatial and context information is used to locate and capture relevant
semantics between two domains. Their works primarily concern UDA in natural
scene images where the semantic information is on a high level (car, road,
etc.). However, in histological data, the semantic information is at a low
level (color, intensity, curvature, etc.), and using pseudo labels directly
for training does not preserve the low-level features well, as we will show in
our experimental results. Wu et al. [19] studied UDA in microscopy images by
inserting dropout layers [20] to the UNet [21] to generate pseudo labels with
increased diversity. However, their approach requires multiple forward passes
during both training and testing, imposing a heavy computational burden and
limiting scalability to larger medical datasets. In comparison, our approach
increases the diversity by sampling in the output predictions, requiring only
one forward pass per data point and offering advantages in terms of run time
and GPU consumption.

---
Figure 2: Overview of our Bayesian uncertainty learning framework. For BB
input, Seg Net generates an output map, along with an uncertainty map.
Repeated variational sampling is performed to adjust the uncertainty map and
add it back to the output. Dice loss is calculated between each sampling and
the pseudo label. The final loss is obtained by averaging over all Dice
losses.

##  3 Method

We begin by pre-training a segmentation network on AB in a supervised manner.
We then use the pre-trained network to generate pseudo labels for BB. Since
the domain gap between the two datasets is large and the amount of available
data is limited, the resulting pseudo labels are error-prone. To address this
issue, we propose a Bayesian uncertainty learning approach, as shown in Figure
2. Specifically, we have the $SegNet$ output an uncertainty map to identify
pixels that might have large losses between the noisy pseudo labels and
predictions.

To implement this approach, we feed a BB image $x\in\mathbb{R}^{W\times H}$
into $Seg_{B}$ to obtain an output prediction $O\in\mathbb{R}^{W\times H\times
C}$ and an uncertainty map $V\in\mathbb{R}^{W\times H\times C}$, where $W,H$
are the spatial resolution and $C$ is the number of classes. We then perform a
Gaussian sampling $\hat{O}_{u}\sim\mathcal{N}(O_{u},V^{2}_{u})$ for each pixel
$u\in W\times H\times C$ to obtain an uncertainty-guided output. To ensure
that gradients for both the output and uncertainty map could be back-
propagated for training, We implement the sampling as follows:

$\hat{O}_{k,u}=O_{u}+\epsilon_{k}V_{u}\ ,$ (1)

where $\epsilon_{k}\sim\mathcal{N}(0,1)$, and $k$ is the $k$-th sampling. This
stochastic sampling does not require extra forward passes to the neural net,
and it can be implemented in parallel on the GPU for all pixels and all images
in a batch, making it computationally efficient and using only a fraction of
computing resource. We then calculate the probability $P_{k,u}$ using softmax
of $\hat{O}_{k,u}$, and compute the Dice loss:

$Dice\ Loss_{k}=1-\frac{2\sum_{u\in W\times H}P_{k,u}\hat{Y}_{u}}{\sum_{u\in
W\times H}P^{2}_{k,u}+\sum_{u\in W\times H}\hat{Y}^{2}_{u}}\ ,$ (2)

where $\hat{Y}_{u}$ is the pseudo label at pixel location $u$. This sampling
is repeated for a total of $K$ times. The final loss is averaged over all
samplings:

$Loss_{final}=\frac{1}{K}\sum_{k=1}^{K}Dice\ Loss_{k}\ .$ (3)

To leverage iterative self-training, we use the current predictions of BB as
the new pseudo labels for the next training iteration. We repeat a total of
$T$ iterations.

Discussion: The rationale for our proposed uncertainty learning approach is
that pseudo labels for BB are error-prone, and forcing the model to learn from
these labels would aggravate the noise. In contrast, the introduced
uncertainty map allows the model to rectify pixels with high segmentation loss
that might be due to the noisy pseudo label, and thus assigns higher variance
to offset the segmentation loss. Note that, under these conditions, variance
is analogous to the absolute value of uncertainty. We therefore use the two
terms interchangeably. Our segmentation network is modelled in a Bayesian way,
in order to provide calibrated uncertainty of the pseudo labels, hence named
BUDA-Seg.

To provide an intuitive explanation of our Bayesian uncertainty training, We
provide a simple example in Figure 3. Suppose we have a target input $x=1$,
feeding into a model $f(\cdot)$ to obtain an output $o=2$. Considering we have
a noisy pseudo label $\hat{y}=6$, whereas the unknown ground truth $y=4$.
Forcing the model to minimize the false distance between $\hat{y}$ and $o$
would not help training converge. Therefore, we propose an uncertainty
indicator $v$ generated by $f(\cdot)$ to offset the large loss. For a concrete
example, $v=1$, and we have the new output $o^{\prime}=3$. The reduced loss
$3=6-(2+1)$ could help $f(\cdot)$ generate output closer to the ground truth
and thus stabilize the training.

---
Figure 3: An intuitive example of regular training (top row) and uncertainty
learning (bottom row). The uncertainty indicator can offset a large loss
between output and noisy pseudo label

##  4 Experiments

In this section, we describe the datasets used for our experiments. We include
the setup for training, metrics used for evaluation, and choices of baseline
approaches in the Appendix.

Datasets The Allen Brain (AB) adult human brain atlas used here, consists of
668 Nissl-stained coronal sections of the left hemisphere at 200µm spacing,
scanned at $1\times 1$µm per pixel, with 106 annotated cerebral slices
(specimen 708084, 34 y.o. female, Modified Brodmann labels) [12]. We
downloaded 24 slices containing the cerebellum and created patches of
$1000\times 1000$ pixel resolution. A total of 1006 patches are created to
form the source domain data. We release these data online.

The BigBrain (BB) is an open-source 3D model of a whole human brain at 20µm
isotropic resolution, reconstructed from high-resolution scans of 7,404
histological sections marked for cell bodies with Merker (silver) stain (65
y.o. male) [6]. Each section is about $6500\times 5700$ pixel resolution. The
differences in staining (contrast) and in the original resolution of the
images add complexity to the model. Given the high visual similarity between
adjacent 20µm -thick slices containing the cerebellum, we kept 22 sections
with large differences and created patches of $1000\times 1000$ pixel
resolution. A total of 600 patches are created to form the target domain data.
Expert annotation on 3 independent sections are split into 75 patches, which
are used for evaluation only.

---
Figure 4: Segmentation results comparison. (0) original BB inputs; (1) our
approach; (2) Wu et al. [19], (3) Yang et al. [16], (4) Hoffman et al. [13],
(5) Chung et al. [22], (6) Kamnitsas et al. [23], (7) Zou et al. [17]. Gray =
background; blue = molecular layer; red = Purkinje cell layer; green =
granular layer; white = white matter. Additional results are in Appendix and
code repo.

##  5 Results

In this section, we present our main segmentation results and compare them
with other baseline approaches. We also report an ablation study, a
qualitative assessment, and an extensive experiment on another medical dataset
to demonstrate the effectiveness of our framework.

Table 1: Quantitative comparisons. Mean Dice loss, and mean IoU, with standard deviations (%). Our proposed BUDA-Seg achieves the lowest Dice loss and highest IoU. | Dice loss $\downarrow$ | IoU $\uparrow$
---|---|---
BUDA-Seg (Ours) | 55.7 $\pm$ 0.7 | 73.22 $\pm$ 1.0
Wu et al. [19] | 56.8 $\pm$ 1.1 | 70.54 $\pm$ 1.6
Yang et al. [15] | 57.9 $\pm$ 1.3 | 67.42 $\pm$ 1.9
Hoffman et al. [13] | 60.2 $\pm$ 2.0 | 67.16 $\pm$ 2.5
Chung et al. [22] | 57.7 $\pm$ 1.4 | 65.9 $\pm$ 2.1
Kamnistas et al. [23] | 61.3 $\pm$ 2.2 | 65.9 $\pm$ 2.4
Zou et al. [17] | 60.7 $\pm$ 1.5 | 67.1 $\pm$ 1.9

###  5.1 Main Results

We display our segmentation results on BB in Figure 4. BUDA-Seg (column (1))
provides the most stable predictions compared to other baselines. For example,
in row 2, the input BB patch has a large portion of tears and artifacts caused
by sectioning. BUDA-Seg shows adaptation to these regions while still
producing a meaningful semantic segmentation. When data have elongated
structures (long stripes such as row 1), BUDA-Seg provides consistent
segmentation results, whereas other approaches generate discontinuous
predictions. Our approach also faithfully preserves the Purkinje cell layer
(indicated in red), which is a thin and sparse layer between the granular
layer (green) and the molecular layer (blue). Other approaches either miss or
incorrectly place the Purkinje cell layer. The cerebellum has a highly
convoluted structure, and being able to distinguish the boundaries is crucial
for understanding cerebellar structure and function. BUDA-Seg achieves overall
better fine-grained segmentation results by preserving the space and gaps and
thus having less connected regions, such as the space between curvatures in
rows 3 and 4. We further provide the mean Dice loss and IoU in Table 1. BUDA-
Seg achieves the lowest loss by 2.0% loss reduction compared with Wu et al.
[19]. Our approach also obtains the highest IoU with more than 3.7%
improvement. In addition, our approach requires one forward pass to quantify
the uncertainty, thus resulting in over 60% GPU memory efficiency and over 50%
time efficiency compared with Wu et al. [19], as is shown in Table 2.

Table 2: GPU usage in gigabytes (GB) and training speed in seconds (s) comparison on a batch size of 1 image with $K=4$. Our approach is over 60% memory efficient, and 50% training time speedup. Method | GPU usage(GB)$\downarrow$ | Training speed(s)$\downarrow$
---|---|---
BUDA-Seg | 3.06 | 0.371
Wu et al. | 8.01 | 0.708

Figure 5: Boxplot of mean Dice loss comparison between our approach and
without uncertainty learning at each iteration.

Figure 6: Dice loss comparison when transferring from MitoEM [24](source
domain) to the FAFB [25](target domain) in the upper figure, and vice versa in
the lower figure.

###  5.2 Effectiveness of Bayesian learning

To validate the effectiveness of our proposed Bayesian learning strategy, we
compute the Dice loss at each training iteration and compare our approach
against a baseline approach without uncertainty learning. As shown in Figure
6, in the first iteration, our approach effectively mitigates regions with
large segmentation loss and trains the model to offset the inconsistency
between the input data and pseudo label with an uncertainty map. In contrast,
the baseline approach does not offer a flexible way to rectify the
inconsistency, resulting in a larger Dice loss.

Furthermore, with iterative training, our approach enables models to learn
from uncertainty, resulting in a steadily decreasing Dice loss after each
iteration. In contrast, the baseline approach cannot adapt the model to
steadily learn between the input and pseudo label after each iteration,
causing an increased loss after the third iteration. These results demonstrate
the effectiveness of our proposed Bayesian learning strategy in improving
segmentation performance and mitigating inconsistency.

###  5.3 Qualitative Visual Assessment

We randomly sampled 20 BB input images with corresponding segmentation results
from 4 competitive approaches (BUDA-Seg, [19], [16], [22]). An independent
expert rater assigned a ranking score of 1 to 4 to each segmentation result
based on its quality, with 4 indicating the highest quality and 1 the lowest.
Our approach received the highest overall score of 63, while the other
approaches obtained scores of 60, 43, and 34, respectively. These results
demonstrate the superior performance of our approach in generating high-
quality semantic masks.

###  5.4 Extensive Experiments

To further validate the effectiveness of our proposed framework on different
datasets, we apply BUDA-Seg to two publically available datasets MitoEM [24]
and FAFB [25]. The MitoEM [24] dataset consists of 500 sections of a rat
brain, with each section having a resolution of 4096$\times$4096\. The FAFB
[25] dataset is a serial section of a fly brain scanned at 40$nm$, and we use
500 cropped sections with a resolution of 4000$\times$4000\. We conduct two
experiments: one where we transfer the annotation from the source domain
MitoEM [24] to the target domain FAFB [25] (shown in the upper plot of Figure
6), and another where we transfer the annotation from FAFB [25] to MitoEM [24]
(shown in the bottom plot of Figure 6). In both experiments, BUDA-Seg achieves
the lowest Dice loss, demonstrating its effectiveness and generalizability
across different datasets.

##  6 Conclusion

This paper introduces BUDA-Seg, a Bayesian learning framework for UDA in
semantic segmentation. Our approach addresses the challenge of distributional
discrepancy between the source and target domains by estimating prediction
uncertainty and adapting the training process from noisy pseudo labels. We
validate our method on a high-resolution cerebellum atlas dataset and two
electron microscopy images, demonstrating its effectiveness in achieving
state-of-the-art segmentation results in UDA.

## References

  * [1] J. D. Schmahmann, J. Doyon, D. MacDonald, C. J. Holmes, K. Lavoie, A. S. Hurwitz, N. Kabani, A. W. Toga, A. C. Evans, and M. Petrides, “Three-dimensional mri atlas of the human cerebellum in proportional stereotaxic space,” NeuroImage, vol. 10, pp. 233–260, 1999.
  * [2] J. D. Schmahmann, J. Doyon, A. W. Toga, A. C. Evans, and M. Petrides, MRI atlas of the human cerebellum.  Academic Press, 2000.
  * [3] M. I. Sereno, J. Diedrichsen, M. Tachrount, G. Testa-Silva, H. d’Arceuil, and C. De Zeeuw, “The human cerebellum has almost 80% of the surface area of the neocortex,” Proceedings of the National Academy of Sciences, vol. 117, no. 32, pp. 19538–19543, 2020.
  * [4] C. Mottolese, N. Richard, S. Harquel, A. Szathmari, A. Sirigu, and M. Desmurget, “Mapping motor representations in the human cerebellum,” Brain, vol. 136, no. 1, pp. 330–342, 2013.
  * [5] J. D. Schmahmann, X. Guell, C. J. Stoodley, and M. A. Halko, “The theory and neuroscience of cerebellar cognition,” Annual review of neuroscience, vol. 42, pp. 337–364, 2019.
  * [6] K. Amunts, C. Lepage, L. Borgeat, H. Mohlberg, T. Dickscheid, M.-É. Rousseau, S. Bludau, P.-L. Bazin, L. B. Lewis, A.-M. Oros-Peusquens, et al., “Bigbrain: an ultrahigh-resolution 3d human brain model,” Science, vol. 340, no. 6139, pp. 1472–1475, 2013.
  * [7] Y.-R. Van Eycke, A. Foucart, and C. Decaestecker, “Strategies to reduce the expert supervision required for deep learning-based segmentation of histopathological images,” Frontiers in medicine, p. 222, 2019.
  * [8] H. Guan and M. Liu, “Domain adaptation for medical image analysis: a survey,” IEEE Transactions on Biomedical Engineering, vol. 69, no. 3, pp. 1173–1185, 2021.
  * [9] J. Ren, I. Hacihaliloglu, E. A. Singer, D. J. Foran, and X. Qi, “Unsupervised domain adaptation for classification of histopathology whole-slide images,” Frontiers in bioengineering and biotechnology, vol. 7, p. 102, 2019.
  * [10] X. Fang, H. Bai, Z. Guo, B. Shen, S. Hoi, and Z. Xu, “Dart: domain-adversarial residual-transfer networks for unsupervised cross-domain image classification,” Neural Networks, vol. 127, pp. 182–192, 2020.
  * [11] D. Tellez, G. Litjens, P. Bándi, W. Bulten, J.-M. Bokhorst, F. Ciompi, and J. Van Der Laak, “Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology,” Medical image analysis, vol. 58, p. 101544, 2019.
  * [12] S.-L. Ding, J. J. Royall, S. M. Sunkin, L. Ng, B. A. Facer, P. Lesnar, A. Guillozet-Bongaarts, B. McMurray, A. Szafer, T. A. Dolbeare, et al., “Comprehensive cellular-resolution atlas of the adult human brain,” Journal of Comparative Neurology, vol. 524, no. 16, pp. 3127–3481, 2016.
  * [13] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell, “Cycada: Cycle-consistent adversarial domain adaptation,” in International conference on machine learning, pp. 1989–1998, PMLR, 2018\.
  * [14] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, pp. 2223–2232, 2017.
  * [15] Y. Yang and S. Soatto, “Fda: Fourier domain adaptation for semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4085–4095, 2020.
  * [16] J. Yang, W. An, C. Yan, P. Zhao, and J. Huang, “Context-aware domain adaptation in semantic segmentation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 514–524, 2021.
  * [17] Y. Zou, Z. Yu, B. Kumar, and J. Wang, “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training,” in Proceedings of the European conference on computer vision (ECCV), pp. 289–305, 2018.
  * [18] W. Yan, Y. Wang, M. Xia, and Q. Tao, “Edge-guided output adaptor: Highly efficient adaptation module for cross-vendor medical image segmentation,” IEEE Signal Processing Letters, vol. 26, no. 11, pp. 1593–1597, 2019.
  * [19] S. Wu, C. Chen, Z. Xiong, X. Chen, and X. Sun, “Uncertainty-aware label rectification for domain adaptive mitochondria segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 191–200, Springer, 2021.
  * [20] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in international conference on machine learning, pp. 1050–1059, PMLR, 2016.
  * [21] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention, pp. 234–241, Springer, 2015\.
  * [22] I. Chung, D. Kim, and N. Kwak, “Maximizing cosine similarity between spatial features for unsupervised domain adaptation in semantic segmentation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1351–1360, 2022.
  * [23] K. Kamnitsas, C. Baumgartner, C. Ledig, V. Newcombe, J. Simpson, A. Kane, D. Menon, A. Nori, A. Criminisi, D. Rueckert, et al., “Unsupervised domain adaptation in brain lesion segmentation with adversarial networks,” in International conference on information processing in medical imaging, pp. 597–609, Springer, 2017.
  * [24] D. Wei, Z. Lin, D. Franco-Barranco, N. Wendt, X. Liu, W. Yin, X. Huang, A. Gupta, W.-D. Jang, X. Wang, et al., “Mitoem dataset: large-scale 3d mitochondria instance segmentation from em images,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 66–76, Springer, 2020.
  * [25] Z. Zheng, J. S. Lauritzen, E. Perlman, C. G. Robinson, M. Nichols, D. Milkie, O. Torrens, J. Price, C. B. Fisher, N. Sharifi, et al., “A complete electron microscopy volume of the brain of adult drosophila melanogaster,” Cell, vol. 174, no. 3, pp. 730–743, 2018.


# Towards Transfer Unlearning: Empirical Evidence of Cross-Domain Bias
Mitigation

Huimin Lu1, Masaru Isonuma1,2, Junichiro Mori1, 3, Ichiro Sakata 1

1The University of Tokyo, 2The University of Edinburgh, 3RIKEN AIP
Correspondence<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) often inherit biases from vast amounts of
training corpora. Traditional debiasing methods, while effective to some
extent, do not completely eliminate memorized biases and toxicity in LLMs. In
this paper, we study an _unlearning_ -based approach to debiasing in LLMs by
performing gradient ascent on hate speech against minority groups, i.e.,
minimizing the likelihood of biased or toxic content. Specifically, we propose
a _mask language modeling unlearning_ technique, which unlearns the harmful
part of the text. This method enables LLMs to selectively forget and
disassociate from biased and harmful content. Experimental results demonstrate
the effectiveness of our approach in diminishing bias while maintaining the
language modeling abilities. Surprisingly, the results also unveil an
unexpected potential for cross-domain _transfer unlearning_ : debiasing in one
bias form (e.g. gender) may contribute to mitigating others (e.g. race and
religion).

Towards Transfer Unlearning: Empirical Evidence of Cross-Domain Bias
Mitigation

Huimin Lu1, Masaru Isonuma1,2, Junichiro Mori1, 3, Ichiro Sakata 1 1The
University of Tokyo, 2The University of Edinburgh, 3RIKEN AIP Correspondence:
<EMAIL_ADDRESS>

##  1 Introduction

In recent years, the natural language processing (NLP) field has experienced a
transformative shift with the introduction of Large Language Models (LLMs).
The remarkable capabilities of LLMs are largely attributed to scaling laws
Kaplan et al. (2020), which suggest their capability heavily depends on the
model size and training dataset size. However, training on massive corpus
often results in LLMs inadvertently acquiring social biases present in their
training datasets Webster et al. (2021); Nangia et al. (2020); Nadeem et al.
(2021). Therefore, addressing these biases is crucial for the development of
fair and responsible LLMs.

Numerous studies have been conducted to mitigate the bias and toxicity
inherent in LLMs Zhao et al. (2018); Barikeri et al. (2021); Liang et al.
(2020); Ravfogel et al. (2020); Schick et al. (2021). However, recent studies
Meade et al. (2022) empirically show that these approaches are effective in
reducing bias while compromising the language modeling performance, as
indicated by increased perplexity on unbiased text. Notably, existing post-hoc
techniques manage to retain language modeling performance, but they fail to
detect more subtle and implicit toxic content.

To overcome this challenge, our study explores an unlearning-based approach
that makes LLMs _forget_ biased and toxic content. By running gradient ascent
on biased text, our method minimizes the likelihood of biased content while
minimizing the degradation of language modeling capabilities. Our debiasing
method is inspired by the successes of prior work Chen and Yang (2023); Jang
et al. (2023), which demonstrated the technique’s efficacy in unlearning
privacy-sensitive data.

This study explores _Mask Language Modeling (MLM) unlearning_ , which
selectively unlearns harmful content within the text by forgetting only toxic
or biased tokens. Through empirical investigation on gender-biased text, we
discovered that _MLM unlearning_ effectively reduces gender bias without
significantly deteriorating language modeling performance. Furthermore,
experimental results demonstrated that unlearning gender-biased text also
contributes to mitigating other types of bias, such as race and religion. This
finding suggests that unlearning-based methods can potentially eliminate a
wide range of bias and toxicity.

In summary, our contributions are twofold: we propose an unlearning-based
debiasing technique that unlearns toxic content while preserving language
modeling ability. We also present that our methods potentially have cross-
domain applicability in mitigating bias and toxicity in LLMs.

##  2 Related Work

This section reviews existing debiasing techniques specifically for language
models, highlighting their contributions and limitations.

#### Counterfactual Data Augmentation (CDA)

Counterfactual Data Augmentation (CDA) Zhao et al. (2018); Barikeri et al.
(2021) attempts to mitigate bias within training datasets by altering bias-
associated terms, such as gender pronouns and religious identifiers. This
technique generates a more balanced corpus by creating counterfactual
examples, where terms indicative of bias attributes are systematically swapped
to diversify representation (e.g., _he/she, jews/christians/muslims_).
However, CDA risks erasing or misrepresenting minority groups, such as those
identifying with non-binary gender categories.

#### SentenceDebias

SentenceDebias Liang et al. (2020) adopts a sophisticated approach by
identifying and neutralizing bias at the sentence level. This method first
computes sentence embeddings to detect a bias subspace. Debiasing is then
achieved by projecting sentences onto the bias subspace and subtracting this
projection from the original embedding, thereby neutralizing gender bias.
However, this approach can inadvertently remove essentially related concepts,
as seen in sentences like “She gave birth to me,” where gender specificity is
crucial.

#### Iterative Nullspace Projection (INLP)

Iterative Nullspace Projection (INLP) Ravfogel et al. (2020) follows a similar
route, removing specific attributes from representations by training a linear
classifier to predict the attribute and then projecting the representations
into the nullspace of the classifier’s weight matrix. Like SentenceDebias,
INLP risks removing pertinent information. Also, both SentenceDebias and INLP,
which involve altering the model’s internal representations, will result in a
degradation in the model’s language modeling abilities Meade et al. (2022).

#### Self-Debias

Self-Debias Schick et al. (2021) represents a novel, post-hoc strategy for
bias mitigation, leveraging the model’s internal knowledge for self-debiasing.
This method first prompts the model to generate both biased and unbiased
versions of text through specific pre-prompting like “The following text
discriminates against people because of their gender”, and then compares token
generation probabilities to distill non-discriminatory outputs. While
innovative, its efficiency is hampered by the necessity of generating dual
outputs for each instance, and its reliance on external tools like the
Perspective API for bias detection has been critiqued for missing subtle and
implicit biases Gehman et al. (2020).

A critical limitation of the aforementioned methods is that they focus on
mitigating specific types of bias, often overlooking the interconnected and
multifaceted nature of bias across domains such as gender, race, and religion.
Our proposed debiasing technique distinguishes itself through its empirical
validation of cross-domain generalizability. By demonstrating that unlearning
gender-based biases also reduces biases in other domains, our approach offers
a promising direction for comprehensive and universal debiasing solutions.

##  3 Bias and Toxicity Unlearning

In general, LLMs are trained to minimize a loss function through gradient
descent, which updates the model parameters to maximize the likelihood of the
actual data. Conversely, unlearning employs _gradient ascent_ to maximize the
loss function for specific undesirable patterns in the training data, thereby
minimizing the likelihood of such patterns.

###  3.1 Masked Language Unlearning

Building upon these precedents, we propose _Masked Language Modeling (MLM)
Unlearning_. This approach selectively unlearns harmful content within the
text by forgetting only toxic or biased tokens, while preserving the language
modeling capabilities. The loss function for _MLM Unlearning_ is formulated as
Equation 1:

$\mathcal{L}_{MLM\,Unlearning}=\log{P(x_{i}|X\setminus x_{i})},$ (1)

where $x_{i}$ denotes the toxic or biased token to be unlearned.
$P(x_{i}|X\setminus x_{i})$ represents the probability of the masked $i$-th
word given the surrounding context $X\setminus x_{i}$, where $X$ is the set of
tokens in the text. By minimizing the above loss function, models are trained
to dissociate the masked word from a specific given context. Therefore, in the
context of _unlearning_ bias and toxicity, models are expected to effectively
_unlearn_ the associations between an attribute term (e.g. _women_) and
harmful content. This method allows for unlearning only the problematic
knowledge inherent in models, without significantly harming language modeling
capabilities.

##  4 Experiment

This section contains content that may be offensive.

###  4.1 Experimental Setup

#### Unlearning Dataset

Our study uses the Dynamically Generated Hate Speech Dataset (DGHS; Vidgen et
al., 2021) for unlearning. The examples of DGHS are collected through a
combination of human and automated efforts to encompass a wide range of hate
speech instances targeting various groups. We specifically extracted the
subset that contains hate speech targeting women. The following examples
illustrate the range of toxic content we seek to mitigate:

  1. 1.

“Women have no backbones”

  2. 2.

“It was dead the moment you let the women vote, simple”

To implement an _MLM Unlearning_ , we utilized GPT-4 to identify words that
could be considered toxic or indicative of bias within the given context.
These identified words were then masked as follows.

  1. 1.

“Women have no [MASK]”

  2. 2.

“It was dead the moment you let the [MASK] vote, simple”

#### Implementation Details

Recent work Jang et al. (2023) pointed out that unlearning many examples at
once might result in a large degradation of language modeling performance.
Therefore, in the experiment, we set the batch size as $8$ to let the model
unlearn only a few examples at once. We run unlearning for 50 training steps
with a learning rate of $1e-5$ on GPT-2 Radford et al.
(2019)111https://huggingface.co/openai-community/gpt2.

###  4.2 Evaluation Setup

To quantitatively evaluate our method, we used the following datasets that are
not used for unlearning.

#### Wikitext-2

To assess the impact of our debiasing approach on the model’s language
capabilities, we employed the Wikitext-2 dataset Merity et al. (2016),
consisting of high-quality Wikipedia articles. Perplexity served as our
primary metric for evaluating language modeling performance.

#### CrowS-Pairs

We used CrowS-Pairs Nangia et al. (2020), a crowd-sourced collection designed
to evaluate model biases across a variety of dimensions including gender,
race, and religion. Each entry in CrowS-Pairs features a pair of sentences:
one stereotypically biased and the other anti-stereotypical regarding the same
context. See Appendix B.1 for an example.

The bias score is calculated based on the model’s preference rate for
stereotypical sentences over their anti-stereotypical counterparts. An ideally
unbiased model would exhibit a 50% preference rate, indicating no bias between
the pair of sentences.

#### StereoSet

StereoSet Nadeem et al. (2021) extends the methodology for bias measurement by
incorporating a triadic structure for each evaluation instance: a
stereotypical sentence, an anti-stereotypical sentence, and an unrelated
sentence. This structure allows for a fine-grained assessment of stereotypical
bias within language models. See Appendix B.2 for an example.

The bias score is calculated based on the model’s preference for stereotypical
over anti-stereotypical responses, aiming for an ideal score close to 50%,
indicating no bias. Additionally, the language modeling ability is assessed by
the model’s ability to choose semantically meaningful sentences (stereotypical
or anti-stereotypical) over unrelated ones, aiming for scores approaching 100%
to indicate strong language modeling capability.

###  4.3 Results and Discussions

In this section, we present the experimental results of our proposed method,
and compare them with the debiasing results of CDA, SentenceDeias, INLP and
Self-Debias as re-implemented and reported by previous study Meade et al.
(2022). Note that our analysis relies on their reported results without
independently re-implementing these methods. Therefore, due to variations in
the baseline performances of the original GPT-2 model across different
studies, we recommend referring to the relative improvement in scores rather
than the absolute scores for a fair comparison.

#### Language Modeling Performance

Table 1 shows the perplexity on Wikitext-2 and the language model scores from
StereoSet for our proposed method across increasing unlearning steps,
alongside the results of previous methods. Our _MLM Unlearning_ approach
maintains language modeling performance comparable to, if not exceeding, that
of previous methods. This finding highlights the balance our method achieves
between debiasing effectiveness and preservation of language modeling
abilities. Further analysis of perplexity results can be found in Appendix
B.3.

Model |  Perplexity ($\downarrow$) |  LM Score ($\uparrow)$
---|---|---
GPT2 (Ours) |  29.94 |  91.24
+10steps |  $\uparrow$0.07 30.01 |  $\downarrow$0.29 90.95
+20steps |  $\uparrow$0.29 30.23 |  $\downarrow$0.76 90.48
+30steps |  $\uparrow$0.50 30.44 |  $\downarrow$1.02 90.22
+40steps |  $\uparrow$0.66 30.60 |  $\downarrow$1.28 89.96
+50steps |  $\uparrow$0.72 30.66 |  $\downarrow$1.35 89.89
GPT2 Meade et al. (2022) |  30.16 |  91.01
+CDA |  $\uparrow$5.19 35.34 |  $\downarrow$0.65 90.36
+SentenceDebias |  $\uparrow$35.34 65.49 |  $\downarrow$3.59 87.43
+INLP |  $\uparrow$12.48 42.53 |  $\uparrow$0.60 91.62
+Self-Debias |  $\uparrow$1.75 31.91 |  $\downarrow$1.94 89.07
Table 1: Perplexities and StereoSet language modeling scores (LM Score) for gender debiased GPT-2 models. Model |  Bias Score (%) CrowS-Pairs |  Bias Score (%) StereoSet
---|---|---
Gender
GPT2 (Ours) |  58.40 |  63.14
+10steps |  $\downarrow$0.77 57.63 |  $\downarrow$0.39 62.75
+20steps |  $\downarrow$1.91 56.49 |  $\downarrow$0.39 62.75
+30steps |  $\downarrow$2.29 56.11 |  $\downarrow$1.57 61.57
+40steps |  $\downarrow$1.91 56.49 |  $\downarrow$2.36 60.78
+50steps |  $\downarrow$2.29 56.11 |  $\downarrow$1.96 61.18
GPT2 Meade et al. (2022) |  56.87 |  62.65
+CDA |  56.87 |  $\uparrow$1.37 64.02
+SentenceDebias |  $\downarrow$0.76 56.11 |  $\downarrow$6.59 56.05
+INLP |  $\downarrow$3.43 53.44 |  $\downarrow$2.48 60.17
+Self-Debias |  $\downarrow$0.76 56.11 |  $\downarrow$1.81 60.84
Race
GPT2 (Ours) |  57.75 |  59.98
+10steps |  $\downarrow$1.55 56.20 |  $\downarrow$1.04 58.94
+20steps |  $\downarrow$2.71 55.04 |  $\downarrow$1.66 58.32
+30steps |  $\downarrow$2.91 54.84 |  $\downarrow$2.18 57.80
+40steps |  $\downarrow$2.91 54.84 |  $\downarrow$2.18 57.80
+50steps |  $\downarrow$2.71 55.04 |  $\downarrow$2.60 57.38
GPT2 Meade et al. (2022) |  59.69 |  58.90
+CDA |  $\uparrow$0.97 60.66 |  $\downarrow$1.59 57.31
+SentenceDebias |  $\downarrow$4.26 55.43 |  $\downarrow$2.47 56.43
+INLP |  56.69 |  $\uparrow$0.06 58.96
+Self-Debias |  $\downarrow$6.40 53.29 |  $\downarrow$1.58 57.33
Religion
GPT2 (Ours) |  67.62 |  56.96
+10steps |  67.62 |  $\uparrow$1.27 58.23
+20steps |  $\downarrow$0.95 66.67 |  $\downarrow$2.53 54.43
+30steps |  $\downarrow$4.76 62.86 |  $\downarrow$2.53 54.43
+40steps |  $\downarrow$3.81 63.81 |  $\downarrow$3.80 53.16
+50steps |  $\downarrow$3.81 63.81 |  $\downarrow$3.80 53.16
GPT2 Meade et al. (2022) |  62.86 |  63.26
+CDA |  $\downarrow$11.43 51.43 |  $\uparrow$0.29 63.55
+SentenceDebias |  $\downarrow$27.62 35.24 |  $\downarrow$3.64 59.62
+INLP |  $\downarrow$0.96 61.90 |  $\uparrow$0.69 63.95
+Self-Debias |  $\downarrow$4.76 58.10 |  $\downarrow$2.81 60.45
Table 2: Bias scores for debiased models. Note that our proposed unlearning
method only debiased the model on gender, while previous methods debiased on
gender, race, and religion.

#### Bias Score

Table 2 shows the bias score of our method and existing debiasing techniques
on the CrowS-Pairs and StereoSet benchmarks. Note that our unlearning efforts
were exclusively focused on gender bias, in contrast to the broader scope of
bias (encompassing gender, race, and religion) addressed by the other methods.
Despite this narrower focus, our approach has demonstrated parity in debiasing
effectiveness across all bias domains. Specifically, our gender-specific
debiasing efforts have shown unexpected generalization to other domains of
bias, such as race and religion. This phenomenon suggests a potential for
_transfer unlearning_ , where debiasing in one area may inadvertently benefit
others, a promising aspect that underscores the adaptability and broad
applicability of our unlearning methodology. One plausible explanation for
this _transfer unlearning_ is that the embeddings of words like "women" and
"Black" may occupy nearby regions in the embedding space due to shared
contexts in the training data (i.e., both can be subjects of discrimination).
Therefore, when the model minimizes the likelihood of generating "women" in a
biased context via gradient ascent, it adjusts the embedding space in a way
that also affects similarly biased terms like "Black."

##  5 Conclusion

In this study, we explored a _Mask Language Modeling Unlearning_ technique
aimed at reducing biases in LLMs by minimizing the connection between certain
bias attributes and harmful content. Our experiments demonstrated that this
method effectively mitigates bias without compromising the model’s language
modeling capabilities, as evidenced by performance on various benchmarks.
Interestingly, we observed that debiasing in one specific domain (e.g.,
gender) inadvertently led to debiasing in others (e.g., race and religion),
indicating a potential for _transfer unlearning_. This finding invites further
investigation into questions like which bias domains are most transferable and
whether previous methods like CDA exhibit similar cross-domain debiasing
effects.

For discussions of the major limitations of our study, please refer to Section
A.

## References

  * Barikeri et al. (2021) Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021.  RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1941–1955, Online. Association for Computational Linguistics.
  * Chen and Yang (2023) Jiaao Chen and Diyi Yang. 2023.  Unlearn what you want to forget: Efficient unlearning for LLMs.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12041–12052, Singapore. Association for Computational Linguistics.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020.  RealToxicityPrompts: Evaluating neural toxic degeneration in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3356–3369, Online. Association for Computational Linguistics.
  * Jang et al. (2023) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023.  Knowledge unlearning for mitigating privacy risks in language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14389–14408, Toronto, Canada. Association for Computational Linguistics.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _Preprint_ , arXiv:2001.08361.
  * Liang et al. (2020) Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020.  Towards debiasing sentence representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5502–5515, Online. Association for Computational Linguistics.
  * Meade et al. (2022) Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022.  An empirical survey of the effectiveness of debiasing techniques for pre-trained language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1878–1898, Dublin, Ireland. Association for Computational Linguistics.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.  Pointer sentinel mixture models.  _Preprint_ , arXiv:1609.07843.
  * Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021.  StereoSet: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5356–5371, Online. Association for Computational Linguistics.
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020.  CrowS-pairs: A challenge dataset for measuring social biases in masked language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1953–1967, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020.  Null it out: Guarding protected attributes by iterative nullspace projection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7237–7256, Online. Association for Computational Linguistics.
  * Schick et al. (2021) Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021.  Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP.  _Transactions of the Association for Computational Linguistics_ , 9:1408–1424.
  * Vidgen et al. (2021) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1667–1682, Online. Association for Computational Linguistics.
  * Webster et al. (2021) Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. 2021.  Measuring and reducing gendered correlations in pre-trained models.  _Preprint_ , arXiv:2010.06032.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics.

##  Appendix A Limitations

Our study has two significant limitations: the reproducibility of the mask
rules set by GPT-4 and the invalid unlearning of tokens following the mask.

#### Mask Rules

As discussed in Section 4.1, we used GPT-4 to identify and mask bias-related
keywords. This approach raises concerns about reproducibility and the
sensitivity of experimental results to the specific words chosen for masking.
To address this, future work should establish explicit mask rules, such as
consistently masking specific bias-attribute words, to reduce reliance on
heuristics and enhance reproducibility.

#### Invalid Unlearning

The target model in our experiment is GPT-2, a causal language model that
predicts and generates tokens sequentially based on previous tokens. In our
masked language unlearning settings, any words following the masked token are
ignored. For example, in the sentence "It was dead the moment you let the
women vote, simple", our method effectively removes the association between
"It was dead the moment you let the" and "women". However, the remaining words
after the mask become irrelevant in this context. This presents a major
challenge that we aim to address in future research.

##  Appendix B Appendix

###  B.1 CrowS-Pairs Example

A data entry in the CrowS-Pairs dataset Nangia et al. (2020) is illustrated
below:

  1. 1.

Stereotypical: “The man was highly skilled in CAD engineering. He was
considered one of the best engineers in his field”

  2. 2.

Anti-stereotypical: “The woman was highly skilled in CAD engineering. She was
considered one of the best engineers in her field”

###  B.2 StereoSet

A data entry in the StereoSet dataset Nadeem et al. (2021) is shown below.
Given a context with a blank to fill, the model is tested by measuring the
likelihood of three choices:

  1. 1.

Context: “A mother is [BLANK]”

  2. 2.

Choice 1 (Stereotypical): “A mother is caring”

  3. 3.

Choice 2 (Anti-stereotypical): “A mother is mean”

  4. 4.

Choice 3 (Unrelated): “A mother is ape”

###  B.3 Perplexity Results

Figure 1: Perplexity Results across Different Unlearning Steps

Figure 1 shows the perplexity on Wikitext-2 and DGHS after applying our
proposed method across increasing unlearning steps. Specifically, the DGHS-
train comprises texts directly used for unlearning, whereas the DGHS-test set
contains texts that were not seen by the model during unlearning. The results
indicate a slight increase in perplexity on the Wikitext-2 dataset, affirming
our method’s efficacy in preserving the language modeling performance.
Conversely, a substantial rise in perplexity for both the DGHS training and
testing segments underscores the model’s reduced inclination to replicate
toxic and biased content against women. This difference highlights the
effectiveness of our approach in reducing undesirable bias without undermining
the model’s overall language modeling abilities.