# Counterfactual Multi-Token Fairness in Text Classification

Pranay Lohia<EMAIL_ADDRESS>IBM Research AIIndia<EMAIL_ADDRESS>
(2022)

###### Abstract.

The counterfactual token generation has been limited to perturbing only a
single token in texts that are generally short and single sentences. These
tokens are often associated with one of many sensitive attributes. With
limited counterfactuals generated, the goal to achieve invariant nature for
machine learning classification models towards any sensitive attribute gets
bounded, and the formulation of Counterfactual Fairness gets narrowed. In this
paper, we overcome these limitations by solving root problems and opening
bigger domains for understanding. We have curated a resource of sensitive
tokens and their corresponding perturbation tokens, even extending the support
beyond traditionally used sensitive attributes like Age, Gender, and Race to
Nationality, Disability, and Religion. The concept of Counterfactual
Generation has been extended to multi-token support valid over all forms of
texts and documents. We define the method of generating counterfactuals by
perturbing multiple sensitive tokens as Counterfactual Multi-token Generation.
The method has been conceptualized to showcase significant performance
improvement over single-token methods and validated over multiple benchmark
datasets. The emendation in counterfactual generation propagates in achieving
improved Counterfactual Multi-token Fairness.

††copyright: none††copyright: acmcopyright††journalyear: 2022††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation email; June 21–24, 2022; Seoul, Korea

##  1\. Introduction

In current times when there is a growing demand to uncover measures and
overcome discrimination in the real world, humans are re-analyzing and re-
designing their algorithms to mitigate the unfairness present in their
machine-learning classification models. An important aspect that arises for
these models is Counterfactual Fairness, which requires similar model
predictions for identical texts even though referencing different sensitive
groups. These references are mainly in terms of tokens present in texts, which
we refer to as sensitive tokens. To solve this problem, we generate
counterfactuals by perturbing these sensitive tokens present in original texts
and check whether prediction changes or not. By performing these tests, we can
identify original texts and their counterfactuals where unfairness is present.
These counterfactuals are then tagged the same labels as original texts and
augmented with originals for re-training the model. This is done to make the
classification model invariant of the sensitive tokens and achieve fairness.
We have used the notion of Counterfactual given by (garg2019counterfactual, )
where counterfactual doesn’t necessarily result in changing the label by the
classifier.

Past works have failed to make much progress in this area because of the
limitations of resources involving sensitive tokens and their corresponding
perturbations. Only a tip of the iceberg set of sensitive attributes are
studied. These sensitive attributes like age, gender, and race, etc. although
addressed a lot in literature, don’t have a resource of tokens and their
perturbations to make their study more standardized. Sensitive attributes like
Disability and Religion which demands much importance, have just been skimmed
through in some literature. Algorithms and methods developed to address
Counterfactual Fairness have validated their applications on shorter texts or
texts involving only one sensitive token. Examples are: 1. Some people are
gay, 2. I saw a black person (garg2019counterfactual, ) (disabilitywords, ).
We can find only a single sensitive token i.e., gay and black present in each
text respectively. However, much of real-world texts are not such small or
single sentences. They are big sentences and paragraphs involving multi-token
i.e., multiple sensitive tokens of the same sensitive attribute or tokens
spread across multiple sensitive attributes.

Example: She is a Christian, who visits Church. Sensitive tokens are She,
Christian, and Church.

Apart from perturbing them individually using single-token methods, we can
also generate more counterfactuals by perturbing all three tokens together or
in pairs e.g., He is an obedient Jew, who visits Synagogue daily and She is an
obedient Hindu, who visits Temple daily. We can generate more counterfactuals
by perturbing multiple tokens. A higher flip-rate (black2020fliptest, ) will
signify that we can identify more unfairness present in the model. We
formulate this method of generating counterfactuals by perturbing multiple
sensitive tokens in texts as Counterfactual Multi-token Generation. Using the
augmented data generated, we can make the model achieve better fairness, which
we define as Counterfactual Multi-token Fairness.

We make the following four major contributions:

  1. (1)

Data curation of words and identification of their corresponding perturbations
for sensitive attributes age, disability, race, nationality, gender, and
religion.

  2. (2)

Identification of sensitive attributes multi-tokens in texts and proposing a
better performing method of generating counterfactuals i.e., Counterfactual
Multi-token Generation than the single-token perturbation method.

  3. (3)

Employing the usage of Explainability in improving the performance of the
counterfactual generation and fairness method.

  4. (4)

With the application of the Counterfactual Multi-token generation method in
data augmentation based bias-mitigation, we define a significantly high-
performing Counterfactual Fairness method i.e., Counterfactual Multi-token
Fairness.

The remainder of the paper is organized as follows. First, we provide a study
of related works in Sec.2. In Sec.3, we provide the definition of
Counterfactual Multi-token Generation followed by the algorithmic details in
Sec.4. In Sec.5, we propose Counterfactual Multi-token Fairness. In Sec.6, we
provide a detailed resource of sensitive words/tokens and their corresponding
perturbation tokens. In Sec.8.1, we showcase performance improvements of the
multi-token method over the standard single-token method. The usage of
explainability in improving fairness is addressed in Sec.8.2. In Sec.8.3, we
validate how our generation method helps in achieving overall improved
Counterfactual Multi-token Fairness. Finally, we conclude the paper in Sec.9
with future work propositions.

##  2\. Related Work

Fairness in Text. Fairness and Ethics literature has a deep focus on
classification model fairness whether detection or mitigation.
(DBLP:journals/corr/abs-1810-01943, ) articulates various metrics and
algorithms to handle fairness in machine learning models. Training data can
contain biases because of historical prejudices, and these can propagate to
machine learning models. With the advancement in AI and much of the
technologies driven towards using AI models, the presence of unwanted
prejudices can hamper the reputation both socially and economically. In terms
of modalities, structured data and text is being studied thoroughly in
fairness literature. (bolukbasi2016man, ), (dwork2012fairness, ),
(hardt2016equality, ), (louizos2015variational, ), (zemel2013learning, ) lays
down foundational areas for studying Fairness in Text.
(bolukbasi2016quantifying, ), (netzer2019words, ), (zhao2018learning, ),
(gonen2019lipstick, ), (zhao2019gender, ), and (yang2020causal, ) provides
methods for removing unfairness from word representations and embeddings.
(shandilya2018fairness, ) talks about fairness in text summarization.

Counterfactual Multi-token Generation. Sensitive attributes based debiasing
has been studied, but are limited to small sentences involving single-token
perturbation handling. (prost2019debiasing, ), (bordia2019identifying, ),
(sun2019mitigating, ), (font2019equalizing, ), and (qian2019reducing, )
highlights mitigating gender bias in machine translations and natural language
processing. (manzini2019black, ) talks about unfairness in AI models when
announcing judgments on grounds of persons’ race. Table 1 lists the
approximate number of papers or studies associated with various sensitive
attributes. (prabhakaran2019perturbation, ) talks about how sensitive tokens
can be perturbed to identify and mitigate unfairness in machine learning
models. Removal of unintended bias from sentiment analysis and toxicity
detection models has been studied in (caliskan2017semantics, ),
(webster2018mind, ), (agelist, ), and (de2019bias, ).
(kiritchenko2018examining, ), (may2019measuring, ), and (gonen2019lipstick, )
talks about how counterfactuals or perturbation generation can help in removal
of unfairness from unstructured texts. (kusner2017counterfactual, ),
(ribeiro2020beyond, ), and (kilbertus2017avoiding, ) forms the basis of our
work on Counterfactual Multi-token Generation and Fairness. They provide a
formal definition of counterfactual fairness and counterfactual token
fairness. (garg2019counterfactual, ) highlights how model prediction changes
when identity terms involving sensitive tokens, in it predominately gender and
sexual orientation tokens (such as gay, lesbian, transgender etc.), are
perturbed.

attribute #papers associated age 33 disability 5 race 35 gender 42 religion 3

Table 1.  Approximate number of papers or studies associated with sensitive
attributes

Counterfactual Multi-token Fairness. Social scientists have raised concerns
over the entire concept of Counterfactual Fairness (kohler2018eddie, )
(krieger2014causal, ). However, with the growing misconduct against minorities
in the real-world, the need for changing the ever-existing social-construct
and prejudice need to be broken. (kusner2017counterfactual, ) points out that
fairness is achieved only when original and counterfactuals have the same
prediction under some constraints (dwork2012fairness, ).
(wachter2017counterfactual, ) and (grari2020adversarial, ) characterize
adversarial examples similar to counterfactuals, where similar output is
expected as in original texts. (DBLP:conf/ijcai/Wu0W19, ),
(kilbertus2020sensitivity, ), and (di2020counterfactual, ) validates how
counterfactuals fairness is important in achieving model fairness.
Explainability methods like LIME (ribeiro2016should, ) and Anchors
(ribeiro2018anchors, ) suggest that explainability token(s) anchors the
predictions locally. These tokens are responsible for the model’s prediction.
Since perturbations to these tokens can alter model predictions, they can be
combined with sensitive tokens to form the multi-token setup which can be
perturbed to their corresponding perturbations to generate counterfactuals.
Machine learning literature proposes multiple methods to achieve fairness,
however, data augmentation involving perturbation of tokens to generate
counterfactuals is the most viable in achieving Counterfactual Fairness
(park2018reducing, ) when handling multiple sensitive attributes. These
flipped counterfactuals (garg2019counterfactual, ) are augmented with the
original data and are used to re-train the model. This makes the model
invariant to the sensitive tokens and their corresponding perturbations,
making the model counterfactually fair (garg2019counterfactual, ).

##  3\. Problem Definition

In (kusner2017counterfactual, ) and (garg2019counterfactual, ),
counterfactuals are generated but limited to perturbing single token in the
text. We term this type of counterfactuals generation as Counterfactual
Single-token Generation. E.g., Original Text: {Who would read a book by a
woman.} will be perturbed to, Perturbed Text: {Who would read a book by a
man.}
There are four limitations to the above technique:

  1. (1)

Single token counterfactuals miss the contextual information leading to
unrealistic sentences (boleda2016lambada, ). e.g., Original Text: The statue
of Christ is magnificent in the church. will be perturbed to Either: The
statue of Vishnu is magnificent in the church., Or: The statue of Christ is
magnificent in the temple. Just from the observation, we can say that these
are contextually unrealistic and semantically wrong perturbations.

  2. (2)

It doesn’t generate correct counterfactuals when the text has multiple
token(s) of a particular sensitive attribute. e.g., He and his friends are too
jovial. Here, one needs to choose between He and his for performing
counterfactual single-token generation, which not only misses other sensitive
words/tokens for perturbation but makes the perturb text grammatically and
syntactically wrong.

  3. (3)

It doesn’t have support for counterfactuals generation involving a combination
of multiple sensitive attributes. e.g., History of our nation only started
when the white man arrived. Based on single token generation, we have to
either choose between white, which belong to race attribute and to be
perturbed to black, or man, which belong to gender attribute and to be
perturbed to woman. While these are realistic counterfactuals, but one misses
the third counterfactual where multiple tokens are perturbed together. Multi-
token Counterfactual: History of our nation only started when the black woman
arrived.

  4. (4)

Since we are missing important counterfactuals, our analysis of Counterfactual
fairness (kusner2017counterfactual, ) gets flawed. There can be many texts
whose multi-token perturbation counterfactuals will get a different label than
their original, and will not be covered in the analysis. Past counterfactuals
generation methods don’t take into consideration whether the counterfactuals
generated are factually correct or not.

###  3.1. Counterfactual Multi-token Generation

To overcome these limitations and attain superior performance, we propose a
multi-token(s) based counterfactual generation method as following:

Notation 3.1. For a text input $x$ $\epsilon$ $X$, where $X$ is the text data
and $x$ is a sequence of tokens [$x_{1}$, $x_{2}$, $x_{3}$,.., $x_{n}$]. Using
a classifier $f$, we can predict $f(x)$. (garg2019counterfactual, ) defines
set of counterfactuals associated with $x$ as $\Phi(x)$. We denote set of
sensitive attributes as $S$ = $A,B,C,D,..,etc$. Representation of any
sensitive attribute, for e.g., $A$, in the text input is done by its
associated words which we term as sensitive multi-token(s), i.e.,
$\\{a_{1},a_{2},..,a_{n}\\}$. Counterfactuals are generated by perturbing
these sensitive multi-token(s), for e.g., $\\{a_{1},a_{2},..,a_{n}\\}$, to
their corresponding perturbations, i.e.,
$\\{a_{1}^{\prime},a_{2}^{\prime},..,a_{n}^{\prime}\\}$. Hence, sensitive
multi-token(s) belonging to these sensitive attributes are denoted as $s$ =
$[\\{a_{1},a_{2},..,a_{n}\\},\\{b_{1},b_{2},..,b_{n}\\},...etc.]$, and their
corresponding perturbations as:
$s^{\prime}$ =
$[\\{a_{1}^{\prime},a_{2}^{\prime},....,a_{n}^{\prime}\\},\\{b_{1}^{\prime},b_{2}^{\prime},....,b_{n}^{\prime}\\},...etc.]$.

###### Definition 3.1.

Generation of counterfactuals by perturbing every sensitive token(s) present
in the text both individually and in various combinations. Using the above
notations, we formulate Counterfactual Multi-token Generation as:

(1) $\Phi_{S}(x)=\bigcup_{s\neq s^{\prime}\epsilon S}\Phi_{s,s^{\prime}}(x)$

The above formulation is preceded by identification of all sensitive token(s)
present in the text. We denote identified sensitive token(s) as $x_{s}$, where

(2) $x_{s}=\bigcup_{x_{i}\epsilon s}x_{i},\forall x_{i}\epsilon x$

After identification, sensitive token(s) $x_{s}$ are replaced with their
corresponding perturbations $x_{s}^{\prime}$ in the text $x$ using the method
proposed in (bolukbasi2016man, ). We get a direct mapping between sensitive
multi-token(s) $s$ and their corresponding perturbations $s^{\prime}$. This
helps in generating semantically correct counterfactuals. The perturbations
are performed both individually and in various combinations of the sensitive
token(s) to generate multiple counterfactuals $\Phi_{S}(x)$ using the multi-
token generation formulation as in the equation 1.

##  4\. Algorithm

We present the algorithmic details associated with generating counterfactuals
using the multi-token generation method. It involves the identification of
sensitive token(s), followed by their replacement with perturbation token(s)
in various combinations keeping in mind semantic, syntactic, and dependency
information.

###  4.1. Identification of sensitive attributes’ words or token(s) in text

The presence of words or token(s) in each text related to age, disability,
race, nationality, gender, and religion is noted. Table 2 showcases certain
examples, where each text can have multiple sensitive attributes’ multi-token
either belonging to a particular sensitive attribute or of multiple sensitive
attributes combined i.e., age, disability, race, nationality, gender, and
religion.

Text Multi-token age disability race nat gender religion Love it! Alas,
though, there are no fun non-derogatory nouns for the male equivalent of
tomboy. My son’s choices will be relegated to ”sissy”/”mama’s boy” or
”variant”/”nonconforming”. Yes, he’s a ”boy” and ”human” – but as you noted,
the power of self-identification as something cool and different can be
magical. boy, male, son, he Interesting take on religion Jeff, thanks. I
remember all those ”devout, born-agains” just loving young Dubya because he
was–like them–a ”good Christian man.” Who laughed as he sent a mentally-ill
white woman to the gas chamber as governor of Texas. A lying master of dirty
tricks, who fled the Catholic church, in Texas, for one with more voters. Most
Europeans have gotten past tying religion and politics. If only we could. God
may not be dead, but she certainly has better things to do than hang out in
the voting booth. young mentally, ill white his, man, she, he, woman
christian, church, god He has said in public about Mexicans, Muslims, women,
blacks, handicapped, and so on. I don’t hate him, but I do seriously object to
his candidacy. Furthermore, hating an individual can’t be bigotry. Bigotry is
about hating groups and collections of people for their characteristics,
gender, background, color, religion, preferences, age, etc. Disliking a
person, even hating that person, for what he or she says, believes, or does,
rather than that person’s classification, is not bigotry. handicapped black
mexicans his, women, she, him, he muslims

Table 2. Sensitive attributes’ multi-token(s) present in text

###  4.2. Counterfactual Multi-token Generation method

Counterfactuals are generated by replacing identified sensitive token(s) with
their corresponding perturbation token(s) individually and in various
combinations. From the definition 3.1, it is clear that multi-token method of
counterfactual generation is super-set of single-token generation method
(kusner2017counterfactual, ). Popular grammar checker (grammar, ) and language
models (peters2018deep, ; devlin2018bert, ; brown2020language, ) are used to
filter out unrealistic counterfactuals. Table 3 showcases some examples based
on Counterfactual Multi-token Generation.

Original Counterfactuals She is going to church. He is going to church. She is
going to temple. He is going to temple. He and his friends are amazing. She
and her friends are amazing. He has said in public about women. She has said
in public about women. He has said in public about men. She has said in public
about men.

Table 3. Examples of Counterfactual Multi-token generation

####  4.2.1. Dependency parsing

Dependency parsing is needed to segregate non-dependent multi-token during
counterfactuals generation. It is only valid for multi-token setup. Method:

  1. (1)

Dependency tree in each text is identified using spaCy (spaCy, ) based
dependency parser.

  2. (2)

Token(s) present in separate branches of the dependency tree are put in
separate lists.

  3. (3)

Based on these lists, multiple counterfactuals are created for a particular
text separately.

For e.g., She is going to church, a white guy will be there too. Based on
dependency parsing, She is going to church and a white guy will be there too
belongs to two different dependency trees. Since, both the clauses are
independent, counterfactuals involving perturbations happening across both
clauses together are redundant for the classification task.
Hence, the useful counterfactuals are:

  1. (1)

He is going to church, a white guy will be there too.

  2. (2)

She is going to temple, a white guy will be there too.

  3. (3)

He is going to temple, a white guy will be there too.

  4. (4)

She is going to church, a black guy will be there too.

  5. (5)

She is going to church, a white girl will be there too.

  6. (6)

She is going to church, a black girl will be there too.

Dependency parsing, Syntactical, and Semantic check are part of the post-
processing steps performed after counterfactuals generation.

##  5\. Counterfactual Multi-token Fairness

Counterfactual Multi-token Generation is followed by Counterfactual Bias
detection, method for improving the performance of the generation method using
Explainability, and finally data augmentation based bias-mitigation to achieve
Counterfactual Multi-token Fairness.

###  5.1. Counterfactual Bias detection

Counterfactual Bias detection (kusner2017counterfactual, ) involves
identifying the original texts whose counterfactuals, generated after
perturbation, produces a different label (or the label flips) than its
original label. Machine-learning models are trained on the original
samples/texts. For a particular dataset, the trained model is used to predict
labels for counterfactuals. Counterfactuals are labeled as flipped if their
predicted label is different from the predicted label of original texts. For
analyzing Counterfactual Bias, we define:

###### Definition 5.1.

flip-rate as the ratio of the number of original texts whose at least one
counterfactual flipped and the total number of original texts.

Higher flip-rate signifies that the method is able to identify more original
texts which has Counterfactual Bias. Counterfactuals of original texts having
Counterfactual Bias are then tagged labels of original texts. After tagging,
all counterfactuals are concatenated together to form Counterfactual Data.
Original and counterfactual data are together used in achieving Counterfactual
Multi-token Fairness. An important inference to draw from it will be how a
larger size of Counterfactual Data, i.e., higher flip-rate, helps in achieving
better fairness.

###  5.2. Performance improvement using Explainability

Based on the method articulated in section 4, there is a possibility that in
many cases, labels of counterfactuals may not flip because words identified
may not be sufficient in contributing towards a particular label. This will
bring us to a situation where we are deprived of an ample number of flipped
counterfactuals. Also, in our task to achieve fairness, we are losing out on
identifying several original texts which have Counterfactual Bias.
Counterfactuals of texts can be used in achieving a higher percentage of
fairness. To solve this deficiency, we used standard model explanation methods
like LIME (ribeiro2016should, ) and Anchors (ribeiro2018anchors, ). These
methods help in identifying token(s) that are contributing to the class label,
and perturbing them will change the class label. These token(s) will also be
indirectly dependent or related to the sensitive token(s). Hence, to generate
more efficient counterfactuals and to achieve a higher percentage of
Counterfactual Multi-token Fairness, we need to combine explainability words
with the sensitive token(s). With sensitive words as the default tokens to be
used for perturbation, we append explainability words to the list if there is
an intersection between sensitive words and explainability words. In
counterfactuals, antonyms of explainability words are used as their perturbing
words. For e.g., Original Text: He is a polite white man.. Sensitive token(s)
are : He, white, man, Explainability token(s) are: polite, man. Hence, new
counterfactuals generated are: 1\. She is a rude black woman. 2\. He is a rude
black man. 3\. He is a rude white man. 4\. She is a rude white woman.
Counterfactuals generated using the multi-token method are tagged the same
label as the original and are termed as Counterfactual Data. They are
augmented with the original data to form the model re-training set. This is
done to make the model invariant of the counterfactuals. A significant
decrement in flip-rate on the test data using the re-trained model showcases
the model becoming fairer and invariant to the counterfactuals. We define this
method of achieving fairness based on generating multi-token counterfactuals
as Counterfactual Multi-token Fairness.

##  6\. Sensitive Attributes’ Words and Perturbations

Counterfactuals are generated by perturbing sensitive words in the original
text with their corresponding perturbations. Hence, there is a need for a
resource of these words and perturbations for sensitive attributes. We have
curated words and corresponding perturbations for the following sensitive
attributes. The perturbations are identified by querying the antonyms of
sensitive words from Merriam-Webster’s Thesaurus (merriam1978merriam, ).

###  6.1. AGE

####  6.1.1. Words Curation and Perturbations Identification

To identify age-related words, we curate a list of age-related terms, as
mentioned in (agelist, ). It categorizes words into two divisions i.e., old
and young. Extending the list, we consider seed words in each division and
extend them using distance in word embedding space. In Table 4 complete
resource of age-related words and their corresponding perturbations have been
articulated.

Words Perturbations Words Perturbations parents children elders kids elderly
kiddish adult youth old teenage elder teenager parent child old young older
younger oldest youngest young old children parents kids elders kiddish elderly
teens elders youngster adult teenage old teenager elder teenagers elders child
parent younger older youngest oldest

Table 4. Words and their corresponding perturbations for sensitive attribute
age

###  6.2. DISABILITY

####  6.2.1. Words Curation and Perturbations Identification

Social Biases as barriers for a person with disabilities have been studied in
(disabilitywords, ). It highlights terms or words used in texts when
representing disability. Disability-related words are queried, and their
antonyms are identified as corresponding perturbations. Table 5 provides a
complete resource for disability-related words and their corresponding
perturbations.

Words Perturbations Words Perturbations mental sane mentally intellectually
illness healthy blind sighted deaf hearing sight-deficient sighted depression
cheerfulness insane sane dyslexia unimpaired slow-learner fast-leaner losers
winners loser winner slow fast depressed cheerful disorder in-order ill
healthy problems solutions issues solutions help no-help treatment no-
treatment care neglect medication hindrance therapy inattention treated
mistreated counseling discourage meds diseases medications hindrance homeless
settled drugs diseases homelessness settleness drug disease alcohol non-
intoxicant police civilian addicts non-addicts

Table 5. Words and their corresponding perturbations for sensitive attribute
disability

###  6.3. RACE

####  6.3.1. Words Curation and Perturbations Identification

As a sensitive attribute, Race has been studied thoroughly in standard
datasets like UCI (Dua:2019, ) Adult (an income dataset based on a 1994 US
Census database; sensitive attributes: sex, race), and ProPublica COMPAS (a
prison recidivism dataset; sensitive attributes: sex, race). We have compiled
race-related words from these two datasets in the following list: [black,
white, asian, caucasian, hispanic, asian-pac-islander, amer-indian-eskimo,
arabic, and oriental]. Perturbations for Race can be any of the other words
present in their respective lists. Table 6 shows a subset of race related
words and their corresponding perturbations from which one word can be
randomly chosen as its perturbation for generating counterfactuals.

Topic words | Perturbation words
---|---
black | white,asian,caucasian,
| hispanic,asian-pac-islander,
| amer-indian-eskimo,arabic,oriental
white | black,asian,caucasian,
| hispanic,asian-pac-islander,
| amer-indian-eskimo,arabic,oriental
asian | black,white,caucasian,
| hispanic,asian-pac-islander,
| amer-indian-eskimo,arabic,oriental
British | Canadians,Indians
| Americans,Chinese,other nationalities
Chinese | Canadians,Indians
| Americans,Russians,other nationalities
Americans | Canadians,Indians
| Chinese,Russians,other nationalities
Table 6. Words and their corresponding perturbations for sensitive attribute
race

###  6.4. NATIONALITY

####  6.4.1. Words Curation and Perturbations Identification

Nationality: We have curated 222 nationality-related words from the Wikipedia
list of nationalities (wiki:NAT, ). Perturbations for Nationality can be any
of the other words present in their respective lists. Table 7 shows a subset
of nationality related words and their corresponding perturbations from which
one word can be randomly chosen as its perturbation for generating
counterfactuals.

Words | Perturbations
---|---
British | Canadians,Indians
| Americans,Chinese,other nationalities
Chinese | Canadians,Indians
| Americans,Russians,other nationalities
Americans | Canadians,Indians
| Chinese,Russians,other nationalities
Table 7. Words and their corresponding perturbations for sensitive attribute
nat

###  6.5. GENDER

####  6.5.1. Words Curation and Perturbations Identification

To identify gender words, we take the list described in (bolukbasi2016man, ).
We have considered gender-specific keywords extracted using word embeddings
described in (bolukbasi2016man, ) totaling 165-word replacements. These word
replacements are perturbations words. The subset resource containing gender
words and their corresponding perturbations are articulated in Table 8. The
complete resource is detailed in the Appendix.

Words Perturbations Words Perturbations he she he’s she’s his her him her man
woman men women spokesman spokeswoman wife husband himself herself mother
father chairman chairwoman daughter son guy girl boys girls brother sister
female male dad mom actress actor girlfriend boyfriend lady gentleman
businessman businesswoman grandfather grandmother nephew niece gays lesbians
maiden master bisexual heterosexual bachelor bachelorette prince princess
monks nuns lad lass fiancee fiance maternal paternal widows widowers
motherhood fatherhood witch wizard monk nun queens kings housewives
househusbands patriarch matriarch pa ma

Table 8. Words and their corresponding perturbations for sensitive attribute
gender

###  6.6. RELIGION

####  6.6.1. Words Curation and Perturbations Identification

Till now religion as a sensitive attribute hasn’t been analyzed in depth. One
of the novel aspects of this paper is the identification of religion-related
words and their corresponding perturbations. Kaggle’s Religious and
Philosophical texts dataset (kagglereligion, ) is used as a corpus to identify
the religion words. This dataset consists of five texts taken from Project
Gutenberg, which is an online archive of religious books. The five textbooks
are 1. The King James Bible, 2. The Quran, 3. The Book Of Mormon, 4. The
Gospel of Buddha, and 5. Meditations, by Marcus Aurelius. The data consists of
59722 text documents. All documents as a chunk are lemmatized using nltk’s
WordNet Lemmatizer followed by stemming using Snowball Stemmer. To generate
the vocabulary, 100 most frequent words are identified followed by removing
words with length less than 3. We end up with 54 religion-related words.
Perturbations were manually curated by analyzing and contextualizing each word
from Wikipedia’s page on Religion (wiki:religion, ) and then, identifying
corresponding perturbations. Table 9 provides a subset-resource for religion-
related words and their corresponding perturbations curated from Project
Gutenberg’s religious books. For generating counterfactuals, religious words
can be perturbed to any of its corresponding perturbations chosen randomly.
The complete resource is detailed in the Appendix.

Words | Perturbations
---|---
christians | hindus,muslims,sikhs,jews
muslims | hindus,sikhs,jews,christians
hindus | muslims,sikhs,jews,christians
jews | hindus,muslims,sikhs,christians
church | mosque,temple,synagogue
mosque | church,temple,gurudwara,synagogue,monastery
temple | church,mosque,gurudwara,synagogue,monastery
bible | quran,gita,tipitaka,granth-sahib,torah
gita | bible,quran,tipitaka,granth-sahib,torah
god | devil
vishnu | jesus,muhammad,moses,buddha
buddha | jesus,muhammad,moses,vishnu
rome | mecca,israel,varanasi
holy | unholy
Table 9. Words and their corresponding perturbations for sensitive attribute
religion

##  7\. Datasets

We have established our propositions and validated them on three widely used
datasets in the area of Fairness in Text.

####  7.0.1. Jigsaw Unintended Bias in Toxicity Classification

It is the Wikipedia Talk Page Dataset (borkan2019nuanced, ) that is publicly
available at Kaggle (kagglejigsaw, ) and prepared by Jigsaw
(dixon2018measuring, ). The data consists of instances of 226235 online
comments. These comments are manually labeled into various levels of TOXICITY.
Apart from descriptive multi-label toxicity labels, there is another target
column with binary class which signify where a comment text is TOXIC or NON-
TOXIC.

####  7.0.2. IMDB Dataset of 50K Movie Reviews

It is the Large Movie Review Dataset (maas-EtAl:2011:ACL-HLT2011, ) with
50,000 movie reviews labeled with the binary sentiment of positive and
negative. The dataset contains an even number of positive and negative
reviews. It is publicly available at Kaggle (kaggleimdb, ).

####  7.0.3. SMS Spam Collection Dataset

It is a set of SMS tagged messages that contain a set of 5,574 SMS messages in
English tagged accordingly being ham (legitimate) or spam
(almeida2011contributions, ; hidalgo2012validity, ; almeida2013towards, ;
webspam, ). It is publicly available at Kaggle (kagglespam, ).

##  8\. Experiment

The details of the models used in the experiments are: 1) Logistic Regression:
sklearn linear model is used with lbfgs solver, L2 penalty, while rest are
default parameters. 2) Neural Network: The sequential model consists of three
hidden layers with units = [12, 8, 6]. The input dimension is 32 and relu
activation function is used. 3) Naïve-Bayes: sklearn Gaussian Naïve Bayes
model is used with all parameters set to default.

###  8.1. Counterfactual Bias detection

We perform Counterfactual Bias detection on the three datasets using commonly
used classification models and compare results between counterfactual single-
token and multi-token generation methods. The experiments are performed for
each sensitive attribute individually and as one case altogether.
Classification models used include an embedding layer.

####  8.1.1. Jigsaw Dataset

Model: Logistic regression. Results: Table 10. Accuracy: 92.31%

attribute single-token multi-token age 3.25 4.56 disability 2.56 3.98 race
4.47 6.83 nat 2.37 5.62 gender 0.94 1.38 religion 1.93 2.33 all 3.11 4.38

Table 10. Flip-rate comparison on jigsaw dataset

####  8.1.2. IMDB Dataset

Model: Neural Network. Results: Table 11. Accuracy: 90.78%

attribute single-token multi-token age 3.13 4.62 disability 2.46 4.31 race
4.38 7.2 nat 4.22 6.7 gender 0.82 1.42 religion 1.74 2.4 all 2.9 4.73

Table 11. Flip-rate comparison on imdb dataset

####  8.1.3. SPAM Dataset

Model: Naive-Bayes. Results: Table 12. Accuracy: 89.46%

attribute single-token multi-token age 0.2 0.25 disability 0.4 1.1 race 0 0.16
nat 0 0.2 gender 0.11 0.23 religion 0 0.15 all 0.36 0.65

Table 12. Flip-rate comparison on spam dataset

Based on the results, the multi-token generation outperforms the single-token
generation method.

###  8.2. Performance improvement using Explainability

The experimental results using the two explainability methods on the datasets
are in Table 13, 14, and 15.

attribute multi-token multi-token multi-token and LIME and Anchors age 4.56
4.58 4.63 disability 3.98 4.03 4.08 race 6.83 7.05 7.25 nat 5.62 7.15 7.24
gender 1.38 1.45 1.50 religion 2.33 2.35 2.41 all 4.38 4.41 4.52

Table 13. Flip-rate comparison on jigsaw dataset using explainability

attribute multi-token multi-token multi-token and LIME and Anchors age 4.62
4.64 4.71 disability 4.31 4.33 4.48 race 7.2 7.32 7.74 nat 6.7 7.2 7.4 gender
1.42 1.44 1.52 religion 2.4 2.41 2.45 all 4.73 4.75 4.82

Table 14. Flip-rate comparison on imdb dataset using explainability

attribute multi-token multi-token multi-token and LIME and Anchors age 0.25
0.25 0.4 disability 1.1 1.15 1.23 race 0.16 0.16 0.2 nat 0.2 0.21 0.24 gender
0.23 0.23 0.25 religion 0.15 0.16 0.23 all 0.65 0.67 0.7

Table 15. Flip-rate comparison on spam dataset using explainability

From the results, we can infer that ability to generate flipped
counterfactuals are higher when we use explainability token(s) along with
sensitive token(s). Also, out of the two explainability methods, we can
observe that Anchors is performing better than LIME. As mentioned in
(ribeiro2018anchors, ), Anchors generate high precision and complete coverage
explanation. Anchors lead to higher human precision than linear explanations.
Using Anchors token(s) along with sensitive token(s), we can generate a higher
number of flipped counterfactuals. Hence, for experimentation purposes in the
following sub-section, we will be using anchors token(s) along with sensitive
token(s) in the multi-token setup.

###  8.3. Counterfactual Bias Mitigation

Following is the experimental method for data augmentation based bias-
mitigation to achieve Counterfactual Multi-token Fairness:

  1. (1)

Each sensitive attribute-specific data is split into an 80:20 ratio for
training and testing, respectively. The same process is followed for the
entire data for performing the experiment combining all sensitive attributes.

  2. (2)

Sensitive attribute-specific data includes only those texts in which that
particular sensitive attribute token(s) are present.

  3. (3)

Now, the following steps will be performed for both single-token and multi-
token setup (on entire data and each attribute specific data) and, results
will be compared.

  4. (4)

We also have one setup where after re-training based on the multi-token
method, the flip-rate is calculated on the test data using the single-token
method. This will signify how the augmentation method using the multi-token
setup can improve the overall fairness proposition.

    1. (a)

The model is trained on the training data.

    2. (b)

The trained data is perturbed to generate counterfactuals using the sensitive
token(s) and their corresponding perturbation token(s).

    3. (c)

Flipped counterfactuals are identified.

    4. (d)

Using the trained model, flip-rate is calculated on the test data.

    5. (e)

The flipped samples are tagged the same label as original texts and is
augmented with the training data.

    6. (f)

The combined data is used to re-train the model.

    7. (g)

Using the re-trained model, the flip-rate in re-calculated on the test data.

    8. (h)

Here, we define Counterfactual Fairness Increment (CFI) (in %) equal to flip-
rate decrement (in %).

    9. (i)

Calculation of flip-rate using the single-token method on the multi-token
method re-trained model is also performed.

  5. (5)

We compare Counterfactual Fairness Increment (CFI) (in %) between the single-
token and multi-token method for the entire data and each attribute. Along-
with, we analyse the Fairness-Utility trade-off and identify the Accuracy Drop
(AD in %) post re-training.

Based on the above experimental method, results are obtained for all three
datasets.

attribute single-token fr% multi-token fr% single-token fr% on multi-token
pre-retraining post-retraining AD CFI pre-retraining post-retraining AD CFI
re-training method age 3.34 2.98 2.4 10.78 4.63 3.0 2.4 35.2 2.64 disability
2.91 2.12 3.1 27.1 4.07 2.41 2.8 40.8 1.74 race 4.53 3.83 2.3 15.4 7.31 4.27
2.3 41.6 3.2 nat 4.3 3.4 2.4 20.9 7.11 4.17 2.2 41.4 3.0 gender 0.99 0.67 2.4
32.3 1.52 0.7 2.5 54 0.53 religion 2.07 1.3 2.5 37.2 2.42 1.5 2.6 38 0.9 all
3.13 2.75 3.4 12.14 4.55 2.77 3.2 39.12 2.4

Table 16. Counterfactual Fairness Increment comparison on jigsaw dataset

attribute single-token fr% multi-token fr% single-token fr% on multi-token
pre-retraining post-retraining AD CFI pre-retraining post-retraining AD CFI
re-training method age 3.15 3.1 1.2 1.6 4.8 3.52 1.8 26.67 2.85 disability 2.5
2.3 2.1 8 4.5 2.47 2 45.11 2.15 race 4.3 3.5 2.1 18.6 7.8 3.8 2.1 51.28 3.4
nat 4.1 3.4 2 17.1 7.2 3.6 2 50 3.2 gender 1 0.8 2 20 1.5 0.85 2.1 43.33 0.6
religion 1.95 1.62 2.4 16.92 2.5 1.75 2.3 30 1.5 all 3 2.5 2.6 16.67 4.9 3.2
2.7 34.7 2.3

Table 17. Counterfactual Fairness Increment comparison on imdb dataset

attribute single-token fr% multi-token fr% single-token fr% on multi-token
pre-retraining post-retraining AD CFI pre-retraining post-retraining AD CFI
re-training method age 0.2 0.18 1.1 10 0.42 0.24 1.2 42.86 0.15 disability 0.5
0.4 1.5 20 1.25 0.71 1.4 43.2 0.36 race 0.1 0.1 1.4 0 0.2 0.1 1.6 50 0 nat 0.1
0.1 2.1 0 0.3 0.1 2.1 66.67 0 gender 0.13 0.1 1.1 23.08 0.27 0.12 1.6 55.56
0.07 religion 0.2 0.08 2.1 60 0.25 0.1 1.9 60 0 all 0.34 0.3 2.1 11.76 0.8 0.4
2.3 50 0.25

Table 18. Counterfactual Fairness Increment comparison on spam dataset

We can draw the following inferences from tables 16, 17, and 18:

  1. (1)

Counterfactual Fairness Increment (in %) is significantly higher in the multi-
token method than the single-token method. Improvement in the CFI across all
datasets and sensitive attributes show that the multi-token method works
better than the single-token method.

  2. (2)

We observe the least flip-rate in the overall experiment setup when the flip-
rate calculation is done using the single-token method on the multi-token
method re-trained model. This signifies that the Counterfactual Multi-token
Fairness performs best in both direct and indirect setup.

  3. (3)

A max accuracy drop of around 3% is noted post retraining while an average
fairness rise of 30-40%, and even 60% in some cases is observed. So, one has
to handle a trade-off of 3% fall in accuracy to gain a high percentage
fairness increment. Also, accuracy drop difference between single-token and
multi-token method is negligible.

##  9\. Conclusion and Future Work

This work presents a novel data collection and curation of sensitive tokens
and identification of their corresponding perturbation token(s) across
multiple sensitive attributes like age, disability, race, nationality, gender,
and religion. We present a resource of sensitive words along with their
corresponding perturbations that eventually help in counterfactuals
generation. We present a detailed data curation for Religion as a sensitive
attribute by coming up with its words and corresponding perturbations. We have
showcased a method for identifying multiple sensitive tokens in texts which
extend beyond small and single sentence analysis. These multiple tokens can
belong to a particular sensitive attribute or can be distributed across
multiple sensitive attributes. Using the resource of sensitive tokens and
their perturbations, we have proposed a novel multi-token counterfactuals
generation method. This method took into consideration contextual, dependency
information, and grammatical understanding while generating counterfactuals.
As part of this work, we have used third-party contextual and grammar checking
solutions. However, as part of the future work, we aim to cover the full or
semi-contextual understanding of texts while generating counterfactuals with
the multi-token setup. Some work has already been done in this field in
(ebrahimi2017hotflip, ) and (ribeiro2020beyond, ), but they are limited to
small and single sentences. With the multi-token setup proposed by us in
place, a new dimension of future work opens up. The proposition of methods for
generating factually correct counterfactuals will be taken in future work. We
have showcased Fairness-Utility trade-off as well. To improve the performance
of counterfactuals generation, we have used standard explainability methods.
Our proposition gets justified by the improvement in flip-rate scores over
single-token and naive multi-token methods. Finally, we perform bias-
mitigation to achieve counterfactual multi-token fairness and have shown that
fairness achieved using the multi-token method is much higher than the single-
token method.

## References

  * [1] Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel.  Counterfactual fairness in text classification through robustness.  In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 219–226, 2019.
  * [2] Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl.  Social biases in nlp models as barriers for persons with disabilities.  pages 5491–5501, 01 2020.
  * [3] Emily Black, Samuel Yeom, and Matt Fredrikson.  Fliptest: fairness testing via optimal transport.  In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 111–121, 2020.
  * [4] Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John T. Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang.  AI fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias.  CoRR, abs/1810.01943, 2018.
  * [5] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In Advances in neural information processing systems, pages 4349–4357, 2016.
  * [6] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel.  Fairness through awareness.  In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214–226, 2012.
  * [7] Moritz Hardt, Eric Price, and Nati Srebro.  Equality of opportunity in supervised learning.  In Advances in neural information processing systems, pages 3315–3323, 2016.
  * [8] Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel.  The variational fair autoencoder.  arXiv preprint arXiv:1511.00830, 2015.
  * [9] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork.  Learning fair representations.  In International Conference on Machine Learning, pages 325–333, 2013.
  * [10] Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai.  Quantifying and reducing stereotypes in word embeddings.  arXiv preprint arXiv:1606.06121, 2016.
  * [11] Oded Netzer, Alain Lemaire, and Michal Herzenstein.  When words sweat: Identifying signals for loan default in the text of loan applications.  Journal of Marketing Research, 56(6):960–980, 2019.
  * [12] Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang.  Learning gender-neutral word embeddings.  arXiv preprint arXiv:1809.01496, 2018.
  * [13] Hila Gonen and Yoav Goldberg.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  arXiv preprint arXiv:1903.03862, 2019.
  * [14] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang.  Gender bias in contextualized word embeddings.  arXiv preprint arXiv:1904.03310, 2019.
  * [15] Zekun Yang and Juan Feng.  A causal inference method for reducing gender bias in word embedding relations.  In AAAI, pages 9434–9441, 2020.
  * [16] Anurag Shandilya, Kripabandhu Ghosh, and Saptarshi Ghosh.  Fairness of extractive text summarization.  In Companion Proceedings of the The Web Conference 2018, pages 97–98, 2018.
  * [17] Flavien Prost, Nithum Thain, and Tolga Bolukbasi.  Debiasing embeddings for reduced gender bias in text classification.  arXiv preprint arXiv:1908.02810, 2019.
  * [18] Shikha Bordia and Samuel R Bowman.  Identifying and reducing gender bias in word-level language models.  arXiv preprint arXiv:1904.03035, 2019.
  * [19] Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang.  Mitigating gender bias in natural language processing: Literature review.  arXiv preprint arXiv:1906.08976, 2019.
  * [20] Joel Escudé Font and Marta R Costa-Jussa.  Equalizing gender biases in neural machine translation with word embeddings techniques.  arXiv preprint arXiv:1901.03116, 2019.
  * [21] Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun.  Reducing gender bias in word-level language models with a gender-equalizing loss function.  arXiv preprint arXiv:1905.12801, 2019.
  * [22] Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and Alan W Black.  Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings.  arXiv preprint arXiv:1904.04047, 2019.
  * [23] Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell.  Perturbation sensitivity analysis to detect unintended model biases.  arXiv preprint arXiv:1910.04210, 2019.
  * [24] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan.  Semantics derived automatically from language corpora contain human-like biases.  Science, 356(6334):183–186, 2017.
  * [25] Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge.  Mind the gap: A balanced corpus of gendered ambiguous pronouns.  Transactions of the Association for Computational Linguistics, 6:605–617, 2018.
  * [26] Mark Diaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle.  Addressing age-related bias in sentiment analysis.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, page 1–14, New York, NY, USA, 2018. Association for Computing Machinery.
  * [27] Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai.  Bias in bios: A case study of semantic representation bias in a high-stakes setting.  In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120–128, 2019.
  * [28] Svetlana Kiritchenko and Saif M Mohammad.  Examining gender and race bias in two hundred sentiment analysis systems.  arXiv preprint arXiv:1805.04508, 2018.
  * [29] Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger.  On measuring social biases in sentence encoders.  arXiv preprint arXiv:1903.10561, 2019.
  * [30] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva.  Counterfactual fairness.  In Advances in Neural Information Processing Systems, pages 4066–4076, 2017.
  * [31] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh.  Beyond accuracy: Behavioral testing of nlp models with checklist.  arXiv preprint arXiv:2005.04118, 2020.
  * [32] Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf.  Avoiding discrimination through causal reasoning.  In Advances in Neural Information Processing Systems, pages 656–666, 2017.
  * [33] Issa Kohler-Hausmann.  Eddie murphy and the dangers of counterfactual causal thinking about detecting racial discrimination.  Nw. UL Rev., 113:1163, 2018.
  * [34] Nancy Krieger.  On the causal interpretation of race.  Epidemiology, 25(6):937, 2014.
  * [35] Sandra Wachter, Brent Mittelstadt, and Chris Russell.  Counterfactual explanations without opening the black box: Automated decisions and the gdpr.  Harv. JL & Tech., 31:841, 2017.
  * [36] Vincent Grari, Sylvain Lamprier, and Marcin Detyniecki.  Adversarial learning for counterfactual fairness.  arXiv preprint arXiv:2008.13122, 2020.
  * [37] Yongkai Wu, Lu Zhang, and Xintao Wu.  Counterfactual fairness: Unidentification, bound and algorithm.  In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 1438–1444. ijcai.org, 2019.
  * [38] Niki Kilbertus, Philip J Ball, Matt J Kusner, Adrian Weller, and Ricardo Silva.  The sensitivity of counterfactual fairness to unmeasured confounding.  In Uncertainty in Artificial Intelligence, pages 616–626. PMLR, 2020.
  * [39] Pietro G Di Stefano, James M Hickey, and Vlasios Vasileiou.  Counterfactual fairness: removing direct effects through regularization.  arXiv preprint arXiv:2002.10774, 2020.
  * [40] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ” why should i trust you?” explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * [41] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  Anchors: High-precision model-agnostic explanations.  In AAAI, volume 18, pages 1527–1535, 2018.
  * [42] Ji Ho Park, Jamin Shin, and Pascale Fung.  Reducing gender bias in abusive language detection.  arXiv preprint arXiv:1808.07231, 2018.
  * [43] Gemma Boleda, Denis Paperno, German Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, and Raquel Fernandez.  The lambada dataset: word prediction requiring a broad discourse context.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin: Association for Computational Linguistics; 2016. p. 1525-1534. ACL (Association for Computational Linguistics), 2016.
  * [44] Mark Fullmer.  grammark.  https://github.com/markfullmer/grammark, 2014.
  * [45] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  arXiv preprint arXiv:1802.05365, 2018.
  * [46] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [47] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  2020\.
  * [48] Explosion.  spacy.  https://github.com/explosion/spaCy, 2017.
  * [49] Merriam-Webster.  The Merriam-Webster Thesaurus.  Pocket Books, 1978.
  * [50] Dheeru Dua and Casey Graff.  UCI machine learning repository, 2017.
  * [51] Wikipedia contributors.  Lists of people by nationality — wikipedia-the free encyclopedia, 2004\.  [Online; accessed 22-July-2004].
  * [52] Kaggle.  Religious and philosophical texts, 2019.
  * [53] Wikipedia contributors.  Religion — wikipedia-the free encyclopedia, 2004.  [Online; accessed 22-July-2004].
  * [54] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman.  Nuanced metrics for measuring unintended bias with real data for text classification.  In Companion Proceedings of The 2019 World Wide Web Conference, pages 491–500, 2019.
  * [55] Kaggle.  Jigsaw unintended bias in toxicity classification, 2018.
  * [56] Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman.  Measuring and mitigating unintended bias in text classification.  In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73, 2018.
  * [57] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.  Learning word vectors for sentiment analysis.  In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
  * [58] Kaggle.  Imdb dataset of 50k movie reviews, 2018.
  * [59] Tiago A Almeida, José María G Hidalgo, and Akebo Yamakami.  Contributions to the study of sms spam filtering: new collection and results.  In Proceedings of the 11th ACM symposium on Document engineering, pages 259–262, 2011.
  * [60] Jose Maria Gomez Hidalgo, Tiago A Almeida, and Akebo Yamakami.  On the validity of a new sms spam collection.  In 2012 11th International Conference on Machine Learning and Applications, volume 2, pages 240–245. IEEE, 2012.
  * [61] Tiago Almeida, José María Gómez Hidalgo, and Tiago Pasqualini Silva.  Towards sms spam filtering: Results under a new dataset.  International Journal of Information Security Science, 2(1):1–18, 2013.
  * [62] Tiago A. Almeida and José María Gómez Hidalgo.  ”webpage: Sms spam collection dataset - http://www.dt.fee.unicamp.br/ tiago/smsspamcollection/”, 2011.
  * [63] Kaggle.  Sms spam collection dataset, 2016.
  * [64] Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou.  Hotflip: White-box adversarial examples for text classification.  arXiv preprint arXiv:1712.06751, 2017.

##  Appendix A Sensitive Attributes’ Words and Perturbations

Counterfactuals are generated by perturbing sensitive words in the original
text with their corresponding perturbations. Hence, there is a need for a
resource of these words and perturbations for sensitive attributes. We have
curated words and corresponding perturbations for the following sensitive
attributes. The perturbations are identified by querying the antonyms of
sensitive words from Merriam-Webster’s Thesaurus [49].

###  A.1. AGE

To identify age-related words, we curate a list of age-related terms, as
mentioned in [26]. It categorizes words into two divisions i.e., old and
young. Extending the list, we consider seed words in each division and extend
them using distance in word embedding space. In Table 19 complete resource of
age-related words and their corresponding perturbations have been articulated.

Topic words | Perturbation words
---|---
parents | children
elders | kids,teens,teenagers
elderly | kiddish
adult | youngster,youth
old | teenage
elder | teenager
parent | child
old | young,younger,youngest
older | younger
oldest | youngest
young | old,older,oldest
children | parents
kids | elders
kiddish | elderly
teens | elders
youngster | adult
teenage | old
teenager | elder
teenagers | elders
child | parent
youth | adult
younger | older
youngest | oldest
Table 19. Topic words and their corresponding perturbation words for sensitive
attribute age

###  A.2. DISABILITY

Social Biases as barriers for a person with disabilities have been studied in
[2]. It highlights terms or words used in texts when representing disability.
Disability-related words are queried, and their antonyms are identified as
corresponding perturbations. Table 20 provides a complete resource for
disability-related words and their corresponding perturbations.

Topic words | Perturbation words
---|---
mental | sane
mentally | intellectually
illness | healthy
blind | sighted
deaf | hearing
sight-deficient | sighted
depression | cheerfulness
insane | sane
dyslexia | unimpaired
slow-learner | fast-leaner
losers | winners
loser | winner
slow | fast
depressed | cheerful
disorder | in-order
ill | healthy
problems | solutions
issues | solutions
help | no-help
treatment | no-treatment
care | neglect
medication | hindrance
therapy | inattention
treated | mistreated
counseling | discourage
meds | diseases
medications | hindrance
homeless | settled
drugs | diseases
homelessness | settleness
drug | disease
alcohol | non-intoxicant
police | civilian
addicts | non-addicts
Table 20. Topic words and their corresponding perturbation words for sensitive
attribute disability

###  A.3. RACE

As a sensitive attribute, Race has been studied thoroughly in standard
datasets like UCI [50] Adult (an income dataset based on a 1994 US Census
database; sensitive attributes: sex, race), and ProPublica COMPAS (a prison
recidivism dataset; sensitive attributes: sex, race). We have compiled race-
related words from these two datasets in the following list: [black, white,
asian, caucasian, hispanic, asian-pac-islander, amer-indian-eskimo, arabic,
and oriental]. Perturbations for Race can be any of the other words present in
their respective lists. Table 21 shows a subset of race related words and
their corresponding perturbations from which one word can be randomly chosen
as its perturbation for generating counterfactuals.

Topic words | Perturbation words
---|---
black | white,asian,caucasian,
| hispanic,asian-pac-islander,
| amer-indian-eskimo,arabic,oriental
white | black,asian,caucasian,
| hispanic,asian-pac-islander,
| amer-indian-eskimo,arabic,oriental
asian | black,white,caucasian,
| hispanic,asian-pac-islander,
| amer-indian-eskimo,arabic,oriental
British | Canadians,Indians
| Americans,Chinese,other nationalities
Chinese | Canadians,Indians
| Americans,Russians,other nationalities
Americans | Canadians,Indians
| Chinese,Russians,other nationalities
Table 21. Words and their corresponding perturbations for sensitive attribute
race

###  A.4. NATIONALITY

Nationality: We have curated 222 nationality-related words from the Wikipedia
list of nationalities [51]. Perturbations for Nationality can be any of the
other words present in their respective lists. Table 22 shows a subset of
nationality related words and their corresponding perturbations from which one
word can be randomly chosen as its perturbation for generating
counterfactuals.

Words | Perturbations
---|---
British | Canadians,Indians
| Americans,Chinese,other nationalities
Chinese | Canadians,Indians
| Americans,Russians,other nationalities
Americans | Canadians,Indians
| Chinese,Russians,other nationalities
Table 22. Words and their corresponding perturbations for sensitive attribute
nat

###  A.5. GENDER

To identify gender words, we take the list described in [5]. We have
considered gender-specific keywords extracted using word embeddings described
in [5] totaling 165-word replacements. These word replacements are
perturbations words. The complete resource containing gender words and their
corresponding perturbations are articulated in Table 23.

Topic words Perturbation words Topic words Perturbation words Topic words
Perturbation words he she he’s she’s He’s She’s his her her his she he she’s
he’s She’s He’s him her man woman women men men women woman man spokesman
spokeswoman wife husband himself herself son daughter mother father father
mother chairman chairwoman daughter son husband wife guy girl girls guys girl
guy boy girl boys girls brother sister spokeswoman spokesman female male
sister brother male female herself himself brothers sisters dad mom actress
actor mom dad sons daughters girlfriend boyfriend daughters sons lady
gentleman boyfriend girlfriend sisters brothers mothers fathers king queen
businessman businesswoman grandmother grandfather grandfather grandmother
ladies gentlemen uncle aunty males females congressman congresswoman grandson
granddaugther queen king businessmen businesswomen wives husbands widow
widower nephew niece bride bridegroom females males aunt uncle gay lesbian
gays lesbians lesbian gay chairwoman chairman fathers mothers moms dads maiden
master bisexual heterosexual granddaughter grandson younger-brother younger-
sister lads lass lion lioness bisexuals heterosexuals gentleman gentlewoman
homosexual heterosexual homosexuals heterosexuals bachelor bachelorette niece
nephew husbands wives prince princess salesman saleswoman hers hims dude
dudette princess prince lesbians gays councilman councilwoman actresses actors
gentlemen gentlewomen stepfather stepmother monks nuns ex-girlfriend ex-
boyfriend lad lass nephews nieces maid man-servant daddy mommy fiance fiancee
fiancee fiance kings queens dads moms waitress waiter maternal paternal
heroine hero nieces nephews girlfriends boyfriends sir madam mistress master
grandma grandpa widows widowers diva divus teenage-boy teenage-girl nuns monks
countrymen countrywomen teenage-girl teenage-boy nun monk brides bridegrooms
housewife househusband spokesmen spokeswomen suitors female-suitors motherhood
fatherhood stepmother stepfather hostess host schoolboy schoolgirl brotherhood
sisterhood stepson stepdaughter congresswoman congressman uncles aunties witch
wizard monk nun paternity maternity suitor female-suitor businesswoman
businessman gal guy statesman stateswoman schoolgirl schoolboy fathered
mothered goddess god stepdaughter stepson grandsons granddaughters godfather
godmother mommy daddy boyhood girlhood grandmothers grandfathers grandpa
grandma boyfriends girlfriends queens kings witches wizards aunts uncles
granddaughters grandsons heterosexual homosexual heterosexuals homosexuals
widower widow salesmen saleswomen maids man-servants gals guys housewives
househusbands fatherhood motherhood princes princesses matriarch patriarch
patriarch matriarch ma pa pa ma councilmen councilwomen mr mrs mrs mr

Table 23. Topic words and their corresponding perturbation words for sensitive
attribute gender

###  A.6. RELIGION

Till now religion as a sensitive attribute hasn’t been analyzed in depth. One
of the novel aspects of this paper is the identification of religion-related
words and their corresponding perturbations. Kaggle’s Religious and
Philosophical texts dataset [52] is used as a corpus to identify the religion
words. This dataset consists of five texts taken from Project Gutenberg, which
is an online archive of religious books. The five textbooks are 1. The King
James Bible, 2. The Quran, 3. The Book Of Mormon, 4. The Gospel of Buddha, and
5. Meditations, by Marcus Aurelius. The data consists of 59722 text documents.
All documents as a chunk are lemmatized using nltk’s WordNet Lemmatizer
followed by stemming using Snowball Stemmer. To generate the vocabulary, 100
most frequent words are identified followed by removing words with length less
than 3. We end up with 54 religion-related words. Perturbations were manually
curated by analyzing and contextualizing each word from Wikipedia’s page on
Religion [53] and then, identifying corresponding perturbations. Table 24
provides a complete resource for religion-related words and their
corresponding perturbations curated from Project Gutenberg’s religious books.
For generating counterfactuals, religious words can be perturbed to any of its
corresponding perturbations chosen randomly.

Topic words | Perturbation words
---|---
muslims | hindus,sikhs,jews,christians,buddhists,atheists
hindus | muslims,sikhs,jews,christians,buddhists,atheists
sikhs | hindus,muslims,jews,christians,buddhists,atheists
jews | hindus,muslims,sikhs,christians,buddhists,atheists
christians | hindus,muslims,sikhs,jews,buddhists,atheists
buddhists | hindus,muslims,sikhs,jews,christians,atheists
atheists | hindus,muslims,sikhs,jews,christians,buddhists
church | mosque,temple,gurudwara,synagogue,monastery
cathedral | mosque,temple,gurudwara,synagogue,monastery
basilica | mosque,temple,gurudwara,synagogue,monastery
chapel | mosque,temple,gurudwara,synagogue,monastery
mosque | church,temple,gurudwara,synagogue,monastery
masjid | church,temple,gurudwara,synagogue,monastery
temple | church,mosque,gurudwara,synagogue,monastery
gurudwara | church,temple,mosque,synagogue,monastery
synagogue | church,temple,gurudwara,mosque,monastery
monastery | church,temple,gurudwara,synagogue,mosque
bible | quran,gita,tipitaka,granth-sahib,torah
quran | bible,gita,tipitaka,granth-sahib,torah
gita | bible,quran,tipitaka,granth-sahib,torah
ramayana | bible,quran,tipitaka,granth-sahib,torah
mahabharata | bible,quran,tipitaka,granth-sahib,torah
tipitaka | bible,quran,gita,granth-sahib,torah
granth-sahib | bible,quran,gita,tipitaka,torah
torah | bible,quran,gita,tipitaka,granth-sahib
vedas | bible,quran,tipitaka,granth-sahib,torah
testament | quran,gita,tipitaka,granth-sahib
talmud | bible,quran,gita,tipitaka,granth-sahib
god | devil
devil | god,lord,allah,prophet
lord | devil
allah | devil
jesus | muhammad,moses,vishnu,buddha
prophet | devil
moses | jesus,muhammad,vishnu,buddha
muhammad | jesus,moses,vishnu,buddha
peter | muhammad,moses,vishnu,buddha
john | muhammad,moses,vishnu,buddha
jacob | jesus,muhammad,vishnu,buddha
samuel | jesus,muhammad,vishnu,buddha
christ | muhammad,moses,vishnu,buddha
vishnu | jesus,muhammad,moses,buddha
buddha | jesus,muhammad,moses,vishnu
gospel | devil
israel | mecca,rome,varanasi,bodh-gaya
mecca | israel,rome,varanasi,bodh-gaya
rome | mecca,israel,varanasi,bodh-gaya
medina | israel,rome,varanasi,bodh-gaya
jerusalem | varanasi,bodh-gaya
arabia | israel,rome,varanasi,bodh-gaya
varanasi | israel,rome,mecca,bodh-gaya
bodh-gaya | israel,rome,varanasi,mecca
holy | unholy
unholy | holy
Table 24. Topic words and their corresponding perturbation words for sensitive
attribute religion