# pMCT: Patched Multi-Condition Training for Robust Speech Recognition

###### Abstract

We propose a novel Patched Multi-Condition Training (pMCT) method for robust
Automatic Speech Recognition (ASR). pMCT employs Multi-condition Audio
Modification and Patching (MAMP) via mixing patches of the same utterance
extracted from clean and distorted speech. Training using patch-modified
signals improves robustness of models in noisy reverberant scenarios. Our
proposed pMCT is evaluated on the LibriSpeech dataset showing improvement over
using vanilla Multi-Condition Training (MCT). For analyses on robust ASR, we
employed pMCT on the VOiCES dataset which is a noisy reverberant dataset
created using utterances from LibriSpeech. In the analyses, pMCT achieves
23.1% relative WER reduction compared to the MCT.

Index Terms: robust speech recognition, data augmentation.

##  1 Introduction

With the rising integration of voice search and interaction systems in
products such as mobile devices and home appliances, there is an increasing
need for robust Automatic Speech Recognition (ASR) systems. One of the
challenges of training ASR models arises from the mismatch between training
and test conditions; models trained on curated close-talk training datasets do
not generalize well to real-world conditions with additive noise, channel
distortion and reverberation [1].

Considering the number and variety of acoustic conditions observed in real-
world scenarios, acquisition of sufficiently diverse training data is not
trivial. Hence, data augmentation (DA) is commonly utilized to diversify and
enlarge the training corpus, acting as a regularizer to prevent overfitting
and thus increasing overall model robustness. A number of approaches based on
signal processing have been proposed. Time-domain augmentations include
modifying the sampling rate by a small factor, such as $\pm 10\%$, which
changes pitch and duration of the audio [2]. Some approaches focus on
augmenting speaker characteristics, e.g., Vocal Tract Length Perturbation
(VLTP) [3, 4]. Speed perturbation [2], a widely used augmentation, can be seen
as emulating tempo perturbation and VTLP. As the variance in audio volume can
be low in curated datasets, volume perturbation [5] has been used to randomly
scale the training data.

SpecAugment [6] acts by randomly masking chunks of time (time masking) or
frequency channels (frequency masking) on spectrograms, and applying a time
warping transformation. Recently, a number of variants of SpecAugment have
been proposed. SpliceOut [7] removes selected time blocks entirely from audio,
and concatenates the remaining parts together. SpecAugment++ [8] applies
frequency and time masking not only to the input spectrogram but also to
latent spaces of neural networks (NNs). In addition, SpecAugment++ proposes
filling the masked out areas with the time frames and frequency channels of
other samples within the mini-batch. Similarly, SpecSwap [9] swaps time blocks
and frequency blocks. However, the interchanged sections originate from the
same audio. Some works [10] propose learning to reorder the shuffled
spectrogram patches, utilizing a fully-differentiable permutation loss.
Systematic omission of frequency channels of the input spectrogram has been
studied in [11, 12].

MixSpeech [13], an adaptation of Mixup [14] for ASR, and Between-Class (BC)
learning [15] generate new training samples by interpolating multiple audio
samples. A generalization of BC, SpeechMix [16], applies interpolations on
latent spaces.

Another family of DA approaches targets improving performance in noisy and
reverberant scenarios. A common approach for diversifying the training corpus
is to add noise from another source to the original waveform [17, 18],
sampling the SNR levels from a random distribution. ImportantAug [19] learns
an importance map for the audio and only adds noise to unimportant regions.
Generative Adversarial Networks are used to match the noisy test conditions in
[20, 21]. Multi-Condition Training (MCT) [22, 23], also referred to as
Multistyle TRaining (MTR) [24], combines speech distortion with Room Impulse
Response (RIR) and additive noise to create robust ASR models [22, 25].

Our contributions can be summarized as follows:

$\bullet$ We propose a simple yet effective MCT method called Patched Multi-
Condition Training (pMCT) to improve ASR accuracy, especially for noisy
reverberant conditions. pMCT employs a novel method, called Multi-condition
Audio Modification and Patching (MAMP), which mixes same utterance patches
extracted from clean and distorted speech (Fig. 1).

$\bullet$ Our MAMP can be combined with additional data augmentation methods
such as SpecAugment [6] to improve accuracy of pMCT. We found that employing
pMCT with SpecAugment further improves WER on both clean (up to 0.8% relative
WER reduction) and noisy (up to 22.9% relative WER reduction) datasets.

The paper is organised as follows: Section 2 presents the method proposed in
this work. The evaluation setup and the results obtained are detailed in
Section 3 and Section 4. Finally, the conclusions are drawn in Section 5.

##  2 Patched Multi-Condition Training for Robust Speech Recognition

In this section, we describe our proposed patched multi-condition training
pipeline. We employ the MAMP method in pMCT to improve accuracy of ASR models
in real-world ASR use-cases, e.g., in far field set-ups where reverberation
and noise significantly distort the speech signal. In the next subsection, we
introduce the theoretical motivation of the proposed method.

###  2.1 Data Augmentation with Audio Patches

ASR algorithms rely on prior knowledge via inductive biases e.g., by using
deep neural networks (DNNs) when the distribution $\mathcal{P}$ of speech
signals $x(n)$ is unknown. Similarly, some DA strategies exploit prior
knowledge about what is in support $\sigma(\mathcal{P})$ of $\mathcal{P}$ such
as SpecAugment. However, particularly in real-world applications such as ASR
in noisy and reverberant conditions, the prior information about noise and
reverberation may not be available in the $\sigma(\mathcal{P})$. Therefore,
the MCT approach aims to exploit prior knowledge about what is not in the
$\sigma(\mathcal{P})$ while training DNNs.

This prior knowledge is often available for audio data. Specifically, we
assume that (1) there exists an alternative distribution $\mathcal{Q}$ with a
different support $\sigma(\mathcal{Q})$; and (2) we have access to a procedure
to efficiently sample from $\mathcal{Q}$. We emphasize that $\mathcal{Q}$ is
not required to be explicitly defined (e.g., through an explicit density).
This alternative distribution $\mathcal{Q}$ may be implicitly defined by a
dataset collected in these scenarios, or by a procedure that transforms
samples from $\mathcal{P}$ into ones from $\mathcal{Q}$ by suitably altering
their structure, such as by audio modification of $x(n)$ to obtain a distorted
speech signal $y(n)$.

According to the manifold hypothesis [26], utterances lie on a $d$ dimensional
manifold $\mathcal{M}$ of the $D\gg d$ dimensional ambient (signal) space
$\mathcal{N}$ containing both clean ($x(n)$) and distorted ($y(n)$) signals.
More precisely, since manifolds can be approximated by locally Euclidean
patches in small neighborhoods, phonemes and utterances reside on audio
patches [27]. Thereby, support $\sigma({\mathcal{P}})$ resides on
$\mathcal{M}$. Manifold learning methods [27, 28] assume that speech does not
exist in the remaining $D-d$ dimensions. Then, they aim to project noisy data
onto noiseless subspaces, such as to the manifolds formed by the eigenvectors
corresponding to the $d$ largest singular values of noisy data [28].

We do not perform computationally complex projections from data space or the
support $\sigma(\mathcal{Q})$ on $\mathcal{M}$ for noise removal. Instead, we
consider that highly entangled noise and different parts of speech can be
untangled across utterance manifolds at different layers of DNNs to achieve
invariant ASR [29]. In addition, DA is informative if the support of
distribution of data generated by DA on $\mathcal{M}$ is close to
$\sigma(\mathcal{Q})$ [30]. These noisy samples will provide information on
the relationship between $\sigma(\mathcal{P})$ and $\sigma(\mathcal{Q})$,
which we will show is helpful in ASR. This suggests that many DA strategies,
such as sampling from a noise or reverberant set, exist.

We propose employing the inductive bias by training DNNs with DA on audio
patches approximating utterance manifolds to improve robustness of ASR models
against acoustic distortions. Untangled representations learned at different
layers of DNNs were explored using attention maps [31, 32]. To elucidate how
different representations are projected on feature spaces at different layers
of DNNs, we analyze their eigenspectrum in Section 4.3.

$0$$0.2$$0.4$$0.6$$0.8$$1$$1.2$$1.4$$1.6$$1.8$$2$$2.2$$-0.4$$0$$0.4$$L_{p}$$\theta_{1}$$\theta_{2}$$\theta_{3}$$\theta_{4}$$\theta_{5}$Time
(s)Amplitude(a) Clean waveform
($x(n)$).$0$$0.2$$0.4$$0.6$$0.8$$1$$1.2$$1.4$$1.6$$1.8$$2$$2.2$$-0.4$$0$$0.4$$\varphi_{1}$$\varphi_{2}$$\varphi_{3}$$\varphi_{4}$$\varphi_{5}$Time
(s)Amplitude(b) Simulated waveform
($y(n)$).$0$$0.2$$0.4$$0.6$$0.8$$1$$1.2$$1.4$$1.6$$1.8$$2$$2.2$$-0.4$$0$$0.4$$\varphi_{1}$$\theta_{2}$$\varphi_{3}$$\theta_{4}$$\theta_{5}$Time
(s)Amplitude(c) Patched waveform ($z(n)$) provided by MAMP. Figure 1: (a)
Waveform of a clean speech signal $x(n)$ of length $L$ split into patches
$\theta_{p}$ with length $L_{p}$ for $p=1,2,\ldots,5$. (b) Noisy reverberant
waveform $y(n)$, simulated using (1), split into patches
$\\{\varphi_{p}\\}_{p=1}^{5}$. (c) Augmented signal $z(n)$ comprises segments
from $x(n)$ and $y(n)$. Vertical dashed lines indicate the size of each patch,
$L_{p}=500~{}ms$.

###  2.2 Multi-condition Audio Modification and Patching (MAMP)

Patches are extracted from audio signals and mixed as follows:

  1. 1.

Modification step: A clean speech signal $x(n)$ (Fig.1(a)) is modified to
obtain $y(n)$ (Fig.1(b)) following the data augmentation employed in MCT. As
described in [22, 23, 24, 25], MCT performs data augmentation by convolving a
speech signal $x(n)$ with a Room Impulse Response (RIR) $h(n)$ to introduce
reverberation followed by adding a noise source $\nu(n)$ to generate the
simulated far-field speech $y(n)$

$y(n)=\sum_{m=0}^{M-1}h(m)x(n-m)+\nu(n)\vspace{-0.15cm}$ (1)

where $M$ is the effective length of $h(m)$ and $n$ is the time index.
Applying this transformation on-the-fly generates highly diverse training
scenarios without increasing storage.

  2. 2.

Patch extraction step: The signals $x(n)$ and $y(n)$ are split into patches
$\theta_{p}$ and $\varphi_{p}$, respectively of size $L_{p}$, as depicted by
the vertical dashed lines in Fig.1, where ${\sum_{p=1}^{P}L_{p}=L}$ and $L$ is
the length of $x(n)$. Note that $L_{P}\leq L_{1}$, since $L$ may not be
multiple of the patch length. Thereby, we obtain clean patches
$\\{\theta_{p}\\}_{p=1}^{P}$ and distorted patches
$\\{\varphi_{p}\\}_{p=1}^{P}$ where $P$ is the total number of patches.

  3. 3.

Patch mixing step: The MAMP signal $z(n)$ is obtained by determining a patch
$\phi_{p}$ as a clean patch $\phi_{p}:=\theta_{p}$ with a probability $\pi$,
or as a distorted patch $\phi_{p}:=\varphi_{p}$ with the probability $1-\pi$,
where $:=$ denotes the assignment operator. Each element $\phi_{p}$ is the
$p^{th}$ patch of the signal $z(n)$ as shown in Fig.1(c). The delay of the
RIR’s direct path is removed in order to align the $x(n)$ and $y(n)$ so that
they can be mixed.

###  2.3 Patched Multi-Condition Training (pMCT)

Finally, pMCT is performed following the standard MCT training recipe [23],
but employing the MAMP to augment the data instead of only utilizing Eq. (1).

##  3 Evaluation of pMCT in ASR Tasks

###  3.1 Experimental setup

In the experiments, we implemented ASR models using the transformer recipe in
SpeechBrain [33] for
LibriSpeech111github.com/speechbrain/speechbrain/blob/develop/recipes/LibriSpeech/
ASR/transformer. The recipe implements an end-to-end transformer ASR
architecture with a Conformer encoder [34]. The Conformer configuration
follows the Conformer (S) described in [34], and we used a transformer
language model. The loss is computed using a weighted average of the CTC loss
and KL-divergence with label smoothing set to 0.15. The models are trained for
200 epochs and the best model is selected based on the Word Error Rate (WER)
achieved in validation set. All the experiments implemented using the 100h
dataset are reported using a beam search of size 1 to speed up the analyses,
while the final results on the 960h dataset are obtained using a beam search
of size 10.

###  3.2 Datasets

LibriSpeech Corpus: LibriSpeech [35] comprises a 960h dataset of English
utterances, sampled at 16kHz. In this work, we use either train-clean-100 – a
100h subset, or the entire 960h dataset for training. We evaluate models using
the validation set dev-clean as well as the test sets test-clean and test-
other. Model accuracy is reported in terms of WER.

VOiCES: VOiCES [36] is a dataset created by recording LibriSpeech utterances
played in acoustically-challenging conditions. Using this dataset for
evaluation allows assessing the accuracy of the proposed methods in realistic
and noisy reverberant environments. The evaluation set for ASR comprises of
20h of data originating from 98 different speakers.

Datasets used for DA: Two main sets are used for DA: (1) comprising RIRs, and
(2) with noise recordings. We use a set of RIRs gathered from 3 datasets [23],
giving rise to 325 different RIRs. The level of reverberation ranges from -2
dB to 40 dB in terms of C50 [37]. The noise dataset uses a set of 843 noise
samples obtained from the Freesound portion of the MUSAN corpus [38]. The
Signal-to-Noise Ratio (SNR) is randomly sampled from a uniform distribution
between 0-30 dB.

##  4 Experimental Analyses

###  4.1 Ablation Studies

In this section, we provide ablation studies on _dev-clean_ set to analyze the
effect of using different patch selection probabilities, $\pi$, patch lengths
$L$, and SpecAugment policies on the accuracy of models. Experiments to find
the best hyperparameters were performed on the 100h subset of LibriSpeech.

####  4.1.1 Analyses of the Effect of Patch Selection Probability $\pi$

Firstly, we investigate how model accuracy varies as a function of the
probability of a clean patch $\pi$ for pMCT in Table 1. In the table, rand
indicates that the utterance is selected with a different probability value
drawn from a uniform distribution. We find that best WER is achieved using
$\pi=0.5$.

Table 1: Results on the dev-clean set for different values of $\pi$. $\bm{\pi}$ | 0 | 0.25 | 0.5 | 0.75 | 1 | rand
---|---|---|---|---|---|---
WER (%) | 10.53 | 10.04 | 9.76 | 9.84 | 10.91 | 9.98

####  4.1.2 Analyses of the Effect of Patch Size $L_{p}$

Next, we investigate how varying patch size, $L_{p}$, affects training.
Different sizes are tested as indicated in Table 2, suggesting that the
optimal patch size is between 1 and 1.5 seconds.

Table 2: Results on the dev-clean set for different patch size $L_{p}$. $\bm{L_{p}}$ (s) | 0.5 | 0.75 | 1 | 1.5 | 2
---|---|---|---|---|---
WER (%) | 9.79 | 9.78 | 9.76 | 9.76 | 9.98

####  4.1.3 Analyses of the Effect of SpecAugment Policies

SpecAugment is widely used to train ASR models due to its efficiency [6, 24].
In this section, we investigate whether the strength of SpecAugment should be
modified when used in conjunction with pMCT. To check this, we experimented
with modifying masking strength of Adaptive SpecAugment [24], as shown in
Table 3. High policy corresponds to using the LibriFullAdapt policy [24],
whereas the Mid and Low policies are obtained by dividing the masking
parameters, such as the number of masks and mask length, by 2 and 4,
respectively. We find that when using SpecAugment together with pMCT,
aggressiveness of SpecAugment should be lowered to achieve best accuracy.
Hence, throughout this work, we use the Mid policy when using SpecAugment
alongside pMCT, unless stated otherwise.

Table 3: Comparison of WER on the dev-clean set for different SpecAugment settings applied in conjunction with pMCT. Setting | High | Mid | Low
---|---|---|---
WER (%) | 11.79 | 9.3 | 9.96

####  4.1.4 Ablation for the MCT

As a baseline for pMCT, we use MCT. Three settings are explored for the MCT
method [23]: probability $p_{r}$ of adding reverberation to the input signal,
probability $p_{n}$ of adding noise to the signal and the SpecAugment setting.
For simplicity, we set $p_{n}=p_{r}$. Table 4 shows that the lowest WER is
achieved with $p_{n}=0.5$ and $p_{r}=0.5$, and this is the setting used for
MCT for the remainder of the paper. Results obtained by employing SpecAugment
on MCT are shown in Table 5, indicating that the best results are obtained for
using the Mid setting.

Table 4: WER of models on the dev-clean of MCT with different probabilities applied to noise $p_{n}$ and reverberation $p_{r}$. $\bm{p_{n}}=\bm{p_{r}}$ | 0 | 0.25 | 0.5 | 0.75 | 1
---|---|---|---|---|---
WER (%) | 10.91 | 10.05 | 10.00 | 10.01 | 10.56
Table 5: Comparison of WER of models on the dev-clean set for different SpecAugment settings applied on top of MCT. Policy | High | Mid | Low
---|---|---|---
WER (%) | 10.16 | 9.25 | 9.59

####  4.1.5 Comparative Evaluation on the Test Datasets

Table 6 summarizes all the results and extends the evaluation of the methods
to three other datasets: test-clean, test-other and VOiCES. The baseline in
this case corresponds to training models with LibriFullAdapt policy for
SpecAugment, but without using MCT or pMCT. The results show that pMCT with
SpecAugment achieves best results on the majority of the datasets, with
largest gains observed on VOiCES.

Table 6: Results for models trained on the 100h dataset. Method |  | dev-
---
clean
| test-
---
clean
| test-
---
other
VOiCES
Baseline | 9.42 | 9.56 | 24.51 | 75.07
MCT w/o specAug | 10.00 | 9.23 | 24.74 | 36.79
pMCT w/o specAug | 9.76 | 9.39 | 25.09 | 31.58
MCT w/ specAug | 9.25 | 9.50 | 24.29 | 36.19
pMCT w/ specAug | 9.30 | 9.15 | 23.21 | 27.97

###  4.2 Analyses for Training Models on Large Datasets

In this section, we repeat the experiments using the full 960h from
LibriSpeech to train the models (Table 7). As in the case for Table 6, the
baseline in Table 7 refers to training models with LibriFullAdapt policy for
SpecAugment but without using MCT or pMCT.

As shown in Table 7, pMCT achieves the lowest WER on most of the datasets,
with a 23.1% relative improvement on VOiCES in comparison with MCT when
SpecAugment is used for both. The results also highlight the importance of
multi-conditional training for robust ASR – even in the worst case (MCT w/o
SpecAugment) more than a 50% relative WER reduction is achieved on VOiCES.
SpecAugment also provides additional improvement when applied after MCT or
pMCT as shown in Table 7.

Table 7: Results for models trained on the 960h dataset.  Method |  | dev-
---
clean
| test-
---
clean
| test-
---
other
VOiCES
Baseline | 2.59 | 2.74 | 6.51 | 33.30
MCT w/o specAug | 2.43 | 2.72 | 7.07 | 14.85
pMCT w/o specAug | 2.44 | 2.67 | 6.90 | 12.83
MCT w/ specAug | 2.42 | 2.65 | 6.59 | 12.86
pMCT w/ specAug | 2.42 | 2.41 | 6.36 | 9.89

The audio samples of the VOiCES dataset were recorded in four rooms,
$\\{R_{i}\\}_{i=1}^{4}$. We examined the WER on each of the rooms for the
models with the best WER on the _dev-clean_ dataset. The results given in
Table 8 show that pMCT outperforms MCT for all rooms. It is also clear that
the models perform worse for larger rooms ($R_{3}$ and $R_{4}$) due to
increased reverberation in these setups.

Table 8: Analyses of the models for the four different rooms included in evaluation dataset of VOiCES. Method | $R_{1}$ | $R_{2}$ | $R_{3}$ | $R_{4}$
---|---|---|---|---
MCT w/ specAug | 5.70 | 5.68 | 24.13 | 15.19
pMCT w/ specAug | 4.74 | 5.18 | 15.34 | 13.78

Table 9 shows the WER provided by these best MCT and pMCT models for different
noise types in the evaluation dataset of the VOiCES. In general, pMCT
outperforms MCT for all types of noises. Moreover, babble noise tends to be
the most challenging noise due to the similarity with the target speech.

Table 9: Analyses of the models for 4 noise types; no noise (none), music (music), television (tele) and babble (babb). Method | none | music | tele | babb
---|---|---|---|---
MCT w/ specAug | 15.33 | 7.94 | 7.93 | 15.84
pMCT w/ specAug | 10.02 | 6.67 | 7.17 | 14.81

###  4.3 Analyses of Self-attention in MCT and pMCT

The self-attention mechanism in the transformer ASR architectures is used to
relate different audio frames to extract a representation of the input
sequence [39]. In order to analyse the differences of the self-attention maps
learned by models trained with MCT and pMCT, we use the covariance self-
attention matrix [40] which captures interaction across different inputs, and
compute the average skewness of their eigenvalues:

$\bm{S}=\mathbb{E}_{y\in\mathcal{D}}\left[\frac{1}{L\cdot
N}\sum_{l\in[L],h\in[H]}s\left(\mathbf{\lambda}^{l,h}_{y}\right)\right]$ (2)

where $y$ is the input utterance from the dataset $\mathcal{D}$, $s$ is the
skewness function, $L$ is the total number of layers, $H$ is the total number
of heads, $\mathbf{\lambda}^{l,h}_{y}$ is the vector comprising the absolute
values of eigenvalues of $A_{y}^{l,h}$ ${A^{l,h}_{y}}^{\top}$ sorted
decrementally, $A_{y}^{l,h}$ is the self-attention map of the $l^{th}$ layer
and the $h^{th}$ head. The skewness for pMCT, $\bm{S}_{p}$, is 10% to 11%
lower compared to the MCT, $\bm{S}_{m}$, in all datasets as shown in Table 10.
This indicates a reduction in the largest eigenvalues and also higher
variations of the attention scores, and thus more connections between input
sequence elements are achieved for pMCT using MAMP which helps to improve the
ASR performance.

Table 10: Summary of the skewness drop of pMCT models compared to MCT models measured as $(\bm{S}_{m}-\bm{S}_{p})/\bm{S}_{m}$. dev-clean | test-clean | test-other | VOiCES
---|---|---|---
0.113 | 0.115 | 0.109 | 0.104

###  4.4 Comparison of MCT and pMCT employed on State-of-the-art Robust ASR
Models

To examine the benefits of using pMCT in other methods proposed for robust ASR
[24, 25], we employed pMCT to train ASR models and compared the results with
the MCT counterpart. In [24], MCT is only applied to the 20% of the data,
whereas in [25], MCT is applied in conjunction with VTLP.

Table 11 shows the WER achieved by using pMCT instead of using MCT in the
method proposed in [24]. The results indicate that using pMCT improves the WER
over the MCT variant of the method. One of the reasons of observing the small
difference between the WER of pMCT and MCT is that the multi-condition
augmentation is only applied to 20% of the training data leaving the remaining
80% of the data clean. Thus, the difference in the final training data is
reduced compared to applying multi-condition augmentation to the entire
training set.

Table 11: Comparison of WER (%) of models trained on 960h using the method proposed in [24] with and without pMCT. Method |  | dev-
---
clean
| test-
---
clean
| test-
---
other
VOiCES
[24] w/ MCT | 2.55 | 2.73 | 6.35 | 13.15
[24] w/ pMCT | 2.32 | 2.5 | 6.3 | 13.09

Table 12 shows the WER achieved using the VTLP-based method proposed in [25].
The results indicate that the pMCT outperforms the MCT in terms of average
WER.

Table 12: Comparison of WER (%) of models trained on 960h using the method proposed in [25] with and without pMCT. Method |  | dev-
---
clean
| test-
---
clean
| test-
---
other
VOiCES
[25] w/ MCT | 2.43 | 2.54 | 6.97 | 15.31
[25] w/ pMCT | 2.43 | 2.64 | 6.86 | 11.15

##  5 Conclusions

We have presented a data augmentation method called pMCT to improve accuracy
of ASR models, which enhances MCT by employing patch modifications. The
proposed method is simple to implement, and introduces negligible
computational overhead without requiring additional data compared to MCT.
Experimental analyses on LibriSpeech and VOiCES show that pMCT outperforms MCT
in both clean and noisy reveberant conditions. The improvement is especially
significant on the VOiCES dataset, achieving a relative WER reduction of
23.1%. An improvement of pMCT over MCT is also observed when incorporating
pMCT into other training pipelines and robust ASR models [24, 25] which
traditionally employ MCT. Lastly, we find that combining pMCT with other
augmentation schemes, such as SpecAugment, achieves further improvements.

## References

  * [1] E. Tsunoo, K. Shibata, C. Narisetty, Y. Kashiwagi, and S. Watanabe, “Data Augmentation Methods for End-to-End Speech Recognition on Distant-Talk Scenarios,” in _Interspeech_ , 2021, pp. 301–305.
  * [2] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” in _Interspeech_ , 2015, pp. 3586–3589.
  * [3] N. Jaitly and G. E. Hinton, “Vocal tract length perturbation (VTLP) improves speech recognition,” in _Proc. ICML Workshop on Deep Learning for Audio, Speech and Language_ , vol. 117, 2013, p. 21.
  * [4] C. Kim, M. Shin, A. Garg, and D. Gowda, “Improved Vocal Tract Length Perturbation for a State-of-the-Art End-to-End Speech Recognition System,” in _Interspeech_ , 2019, pp. 739–743.
  * [5] V. Peddinti, D. Povey, and S. Khudanpur, “A time delay neural network architecture for efficient modeling of long temporal contexts,” in _Interspeech_ , 2015, pp. 3214–3218.
  * [6] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,” in _Interspeech_ , 2019, pp. 2613–2617.
  * [7] A. Jain, P. R. Samala, D. Mittal, P. Jyothi, and M. Singh, “Spliceout: A simple and efficient audio augmentation method,” _arXiv_ , vol. abs/2110.00046, 2021.
  * [8] H. Wang, Y. Zou, and W. Wang, “SpecAugment++: A Hidden Space Data Augmentation Method for Acoustic Scene Classification,” in _Interspeech_ , 2021, pp. 551–555.
  * [9] X. Song, Z. Wu, Y. Huang, D. Su, and H. Meng, “SpecSwap: A Simple Data Augmentation Method for End-to-End Speech Recognition,” in _Interspeech_ , 2020, pp. 581–585.
  * [10] A. N. Carr, Q. Berthet, M. Blondel, O. Teboul, and N. Zeghidour, “Self-supervised learning of audio representations from permutations with differentiable ranking,” _IEEE Signal Process. Lett._ , vol. 28, pp. 708–712, 2021.
  * [11] G. Kovács, L. Tóth, D. V. Compernolle, and S. Ganapathy, “Increasing the robustness of CNN acoustic models using autoregressive moving average spectrogram features and channel dropout,” _Pattern Recognit. Lett._ , vol. 100, pp. 44–50, 2017.
  * [12] L. Tóth, G. Kovács, and D. Van Compernolle, “A perceptually inspired data augmentation method for noise robust cnn acoustic models,” in _Speech and Computer_. Springer International Publishing, 2018, pp. 697–706.
  * [13] L. Meng, J. Xu, X. Tan, J. Wang, T. Qin, and B. Xu, “MixSpeech: Data Augmentation for Low-Resource Automatic Speech Recognition,” in _ICASSP_ , 2021, pp. 7008–7012.
  * [14] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond Empirical Risk Minimization,” in _ICLR_ , 2018.
  * [15] Y. Tokozume, Y. Ushiku, and T. Harada, “Learning from Between-class Examples for Deep Sound Recognition,” in _ICLR_ , 2018.
  * [16] A. Jindal, N. E. Ranganatha, A. Didolkar, A. G. Chowdhury, D. Jin, R. Sawhney, and R. R. Shah, “SpeechMix — Augmenting Deep Sound Recognition Using Hidden Space Interpolations,” in _Interspeech_ , 2020, pp. 861–865.
  * [17] D. Yu, M. L. Seltzer, J. Li, J. Huang, and F. Seide, “Feature learning in deep neural networks - A study on speech recognition tasks,” in _ICLR_ , 2013\.
  * [18] A. Y. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y. Ng, “Deep speech: Scaling up end-to-end speech recognition,” _arXiv_ , vol. abs/1412.5567, 2014\.
  * [19] V. A. Trinh, H. S. Kavaki, and M. I. Mandel, “ImportantAug: a data augmentation agent for speech,” submitted to ICASSP, 2022.
  * [20] A. Sriram, H. Jun, Y. Gaur, and S. Satheesh, “Robust speech recognition using generative adversarial networks,” in _ICASSP_ , 2018, pp. 5639–5643.
  * [21] E. Tsunoo, K. Shibata, C. Narisetty, Y. Kashiwagi, and S. Watanabe, “Data Augmentation Methods for End-to-End Speech Recognition on Distant-Talk Scenarios,” in _Interspeech_ , 2021, pp. 301–305.
  * [22] C. Kim, A. Misra, K. Chin, T. Hughes, A. Narayanan, T. Sainath, and M. Bacchiani, “Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in google home,” in _Interspeech_ , 2017, pp. 379–383.
  * [23] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in _ICASSP_ , 2017, pp. 5220–5224.
  * [24] D. S. Park, Y. Zhang, C.-C. Chiu, Y. Chen, B. Li, W. Chan, Q. V. Le, and Y. Wu, “Specaugment on large scale datasets,” in _ICASSP_ , 2020, pp. 6879–6883.
  * [25] J. Kim, M. Kumar, D. Gowda, A. Garg, and C. Kim, “A comparison of streaming models and data augmentation methods for robust speech recognition,” in _ASRU_ , 2021, pp. 989–995.
  * [26] C. Fefferman, S. Mitter, and H. Narayanan, “Testing the manifold hypothesis,” _Journal of the American Mathematical Society_ , vol. 29, no. 4, pp. 983–1049, Oct. 2016.
  * [27] C. Vaz and S. Narayanan, “Learning a Speech Manifold for Signal Subspace Speech Denoising,” in _Interspeech_ , 2015, pp. 1735–1739.
  * [28] M. Dendrinos, S. Bakamidis, and G. Carayannis, “Speech enhancement from noise: A regenerative approach,” _Speech Commun._ , vol. 10, no. 1, p. 45–67, Feb 1991.
  * [29] C. Stephenson, J. Feather, S. Padhy, O. H. Elibol, H. Tang, J. McDermott, and S. Chung, “Untangling in invariant speech recognition,” in _NeurIPS_ , 2019, pp. 14 368–14 378.
  * [30] A. Sinha, K. Ayush, J. Song, B. Uzkent, H. Jin, and S. Ermon, “Negative Data Augmentation,” in _ICLR_ , 2021.
  * [31] S.-W. Yang, A. T. Liu, and H. yi Lee, “Understanding self-attention of self-supervised audio transformers,” in _Interspeech_ , 2020, pp. 3785–3789.
  * [32] K. Shim, J. Choi, and W. Sung, “Understanding the role of self attention for efficient speech recognition,” in _ICLR_ , 2022.
  * [33] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624.
  * [34] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented Transformer for Speech Recognition,” in _Interspeech_ , 2020, pp. 5036–5040.
  * [35] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an ASR corpus based on public domain audio books,” in _ICASSP_ , 2015, pp. 5206–5210.
  * [36] C. Richey, M. A. Barrios, Z. Armstrong, C. Bartels, H. Franco, M. Graciarena, A. Lawson, M. K. Nandwana, A. Stauffer, J. van Hout, P. Gamble, J. Hetherly, C. Stephenson, and K. Ni, “Voices Obscured in Complex Environmental Settings (VOiCES) Corpus,” in _Interspeech_ , 2018, pp. 1566–1570.
  * [37] P. Peso Parada, D. Sharma, J. Lainez, D. Barreda, T. v. Waterschoot, and P. A. Naylor, “A Single-Channel Non-Intrusive C50 Estimator Correlated With Speech Recognition Performance,” _IEEE TASLP_ , vol. 24, no. 4, pp. 719–732, 2016.
  * [38] D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” _arXiv_ , vol. abs/1510.08484, 2015.
  * [39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is All you Need,” in _NeurIPS_ , vol. 30, 2017.
  * [40] S. Bhojanapalli, A. Chakrabarti, H. Jain, S. Kumar, M. Lukasik, and A. Veit, “Eigen analysis of self-attention and its reconstruction from partial computation,” _arXiv_ , vol. abs/2106.08823, 2021.


screenwriter, provost, saxophonist, plumber, correspondent, organist, baker,
doctor, constable, treasurer, superintendent, boxer, physician, infielder,
businessman, protege

##  Appendix 8.J Words Lists for Template Generation

We use Natural Language Inference (NLI) as a probe to measure societal biases
expressed by text representations. For the task, we generate templates
methodically. We keep most word lists the same for template generation in both
Chapters 5 and 6. For occupations, we change the list to remove those words
that we use for subspace determination of occupations. This creates a
test/train split for our experiments. For SIRT templates, we also modify the
gendered word lists to create the word lists used in premise/hypodissertation
for the entail and contradict templates.

Gendered words for Neutral Templates

Male: guy, gentleman, man

Female: girl, lady, woman

Gendered words for Entail and Contradict Templates

Male Premise: guy, father, grandfather, patriarch, king, gentleman

Male Hypodissertation: man, male

Female Premise: girl, mother, grandmother, matriarch, queen, lady

Female Hypodissertation: woman, female

Occupations for Templates  accountant, actor, actuary, administrator, advisor,
aide, ambassador, architect, artist, astronaut, astronomer, athlete,
attendant, attorney, author, babysitter, baker, biologist, broker, builder,
butcher, butler, captain, cardiologist, caregiver, carpenter, cashier,
caterer, chauffeur, chef, chemist, clerk, coach, contractor, cook, cop,
cryptographer, dentist, detective, dictator, director, driver, ecologist,
economist, editor, educator, electrician, engineer, entrepreneur, executive,
farmer, financier, firefighter, gardener, general, geneticist, geologist,
golfer, governor, grocer, guard, hairdresser, housekeeper, hunter, inspector,
instructor, intern, interpreter, inventor, investigator, janitor, jester,
journalist, judge, laborer, landlord, lawyer, lecturer, librarian, lifeguard,
linguist, lobbyist, magician, manager, manufacturer, marine, marketer, mason,
mathematician, mayor, mechanic, messenger, miner, model, musician, novelist,
official, operator, optician, painter, paralegal, pathologist, pediatrician,
pharmacist, philosopher, photographer, physician, physicist, pianist, pilot,
plumber, poet, politician, postmaster, president, principal, producer,
professor, programmer, psychiatrist, psychologist, publisher, radiologist,
receptionist, reporter, representative, researcher, retailer, sailor,
salesperson, scholar, senator, sheriff, singer, soldier, spy, statistician,
stockbroker, supervisor, surgeon, surveyor, tailor, teacher, technician,
trader, translator, tutor, undertaker, valet, veterinarian, violinist, waiter,
warden, warrior, watchmaker, writer, zookeeper, zoologist

Adherents${}_{\textrm{Test}}$ adventist, anabaptist, anglican, buddhist,
confucian, jain, jew, lutheran, mormon, muslim, rastafarian, satanist,
scientologist, shinto, sikh, sunni, taoist

Demonyms${}_{\textrm{Test}}$ belarusian, brazilian, british, canadian, danish,
dutch, emirati, georgian, greek, indian, iranian, iraqi, irish, italian,
japanese, libyan, moroccan, nigerian, peruvian, qatari, russian, saudi,
scottish, swiss, thai, turkish, ukrainian, uzbekistani, vietnamese, welsh,
yemeni, zambian

Countries america, belarus, brazil, britain, canada, china, denmark, egypt,
emirates, france, georgia, germany, greece, india, iran, iraq, ireland, italy,
japan, korea, libya, morocco, netherlands, nigeria, pakistan, peru, qatar,
russia, scotland, spain, switzerland, thailand, turkey, ukraine, uzbekistan,
vietnam, wales, yemen, zambia

Polarity awful, dishonest, dumb, evil, great, greedy, hateful, honest,
humorless, ignorant, intelligent, intolerant, neat, nice, professional, rude,
smart, strong, stupid, terrible, terrible, ugly, unclean, unprofessional,
weak, wise

Religions adventism, anabaptism, anglicism, atheism, baptism, buddhism,
catholicism, christianity, confucianism, hinduism, islam, jainism, judaism,
lutheranism, methodism, mormonism, protestantism, rastafarianism, satanism,
scientology, sikhism, sunnism, taoism

Objects apple, apron, armchair, auto, bagel, banana, bed, bench, beret,
blender, blouse, bookshelf, breakfast, brownie, buffalo, burger, bus, cabinet,
cake, calculator, calf, camera, cap, cape, car, cart, cat, chair, chicken,
clock, coat, computer, costume, cot, couch, cow, cupboard, dinner, dog,
donkey, donut, dress, dresser, duck, goat, headphones, heater, helmet, hen,
horse, jacket, jeep, lamb, lamp, lantern, laptop, lunch, mango, meal, muffin,
mule, oven, ox, pancake, peach, phone, pig, pizza, potato, printer, pudding,
rabbit, radio, recliner, refrigerator, ring, roll, rug, salad, sandwich,
shirt, shoe, sofa, soup, stapler, SUV, table, television, toaster, train, tux,
TV, van, wagon, watch

Verbs ate, befriended, bought, budgeted for, called, can afford, consumed,
cooked, crashed, donated, drove, finished, hated, identified, interrupted,
liked, loved, met, owns, paid for, prepared, saved, sold, spoke to, swapped,
traded, visited

##  Appendix 8.K Word Lists for WEAT Tests

WEAT tests use sets of words that we list here [17].

Gendered Words

Male Gendered: male, man, boy, brother, him, his, son

Female Gendered: female, woman, girl, sister, her, hers, daughter

WEAT 1: Occupations

Stereotypically male: engineer, lawyer, mathematician

Stereotypically female: receptionist, homemaker, nurse

WEAT 2: Work versus Home

Work terms: executive, management, professional, corporation, salary, office,
business, career

Home terms: home, parents, children, family, cousins, marriage, wedding,
relatives

WEAT 3: Math versus Art

Math terms: math, algebra, geometry, calculus, equations, computation,
numbers, addition

Art terms: poetry, art, dance, literature, novel, symphony, drama, sculpture

##  Appendix 8.L Word lists for WEAT* Tests

For the WEAT* test developed in Chapter 6, we list the sets of words used.

Definitionally Gendered Words for WEAT*

Male: actor, bachelor, bridegroom, brother, count, czar, dad, daddy, duke,
emperor, father, fiance, gentleman, giant, god, governor, grandfather,
grandson, headmaster, heir, hero, host, hunter, husband, king, lad, landlord,
lord, male, manager, manservant, masseur, master, mayor, milkman, millionaire,
monitor, monk, mr, murderer, nephew, papa, poet, policeman, postman,
postmaster, priest, prince, shepherd, sir, stepfather, stepson, steward,
sultan, uncle, waiter, washerman, widower, wizard,

Female: actress, spinster, bride, sister, countess, czarina, mum, mummy,
duchess, empress, mother, fiancee, lady, giantess, goddess, matron,
grandmother, granddaughter, headmistress, heiress, heroine, hostess, huntress,
wife, queen, lass, landlady, lady, female, manageress, maidservant, masseuse,
mistress, mayoress, milkmaid, millionairess, monitress, nun, mrs, murderess,
niece, mama, poetess, policewoman, postwoman, postmistress, priestess,
princess, shepherdess, madam, stepmother, stepdaughter, stewardess, sultana,
aunt, waitress, washerwoman, widow, witch

Gendered Names for WEAT*

There are generated using data curated by social security in USA
(https://www.ssa.gov/oact/babynames/). We take the top 1000 gendered names in
each gender which are also among the top 100K most frequent words in Wikipedia
(to ensure robustly embedded words).

Male: liam, noah, william, james, logan, benjamin, mason, elijah, oliver,
jacob, lucas, michael, alexander, ethan, daniel, matthew, aiden, henry,
joseph, jackson, samuel, sebastian, david, carter, wyatt, jayden, john, owen,
dylan, luke, gabriel, anthony, isaac, grayson, jack, julian, levi,
christopher, joshua, andrew, lincoln, mateo, ryan, nathan, aaron, isaiah,
thomas, charles, caleb, josiah, christian, hunter, eli, jonathan, connor,
landon, adrian, asher, cameron, leo, theodore, jeremiah, hudson, robert,
easton, nolan, nicholas, ezra, colton, angel, jordan, dominic, austin, ian,
adam, elias, jose, ezekiel, carson, evan, maverick, bryson, jace, cooper,
xavier, parker, roman, jason, santiago, chase, sawyer, gavin, leonardo,
jameson, kevin, bentley, zachary, everett, axel, tyler, micah, vincent,
weston, miles, wesley, nathaniel, harrison, brandon, cole, declan, luis,
braxton, damian, silas, tristan, ryder, bennett, george, emmett, justin, kai,
max, diego, luca, carlos, maxwell, kingston, ivan, maddox, juan, ashton,
rowan, giovanni, eric, jesus, calvin, abel, king, camden, amir, blake, alex,
brody, malachi, emmanuel, jonah, beau, jude, antonio, alan, elliott, elliot,
waylon, xander, timothy, victor, bryce, finn, brantley, edward, abraham,
patrick, grant, hayden, richard, miguel, joel, gael, tucker, rhett, avery,
steven, graham, jasper, jesse, matteo, dean, preston, august, oscar, jeremy,
alejandro, marcus, dawson, lorenzo, messiah, zion, maximus, river, zane, mark,
brooks, nicolas, paxton, judah, emiliano, bryan, kyle, myles, peter, charlie,
kyrie, thiago, brian, kenneth, andres, lukas, aidan, jax, caden, milo, paul,
beckett, brady, colin, omar, bradley, javier, knox, jaden, barrett, israel,
matias, jorge, zander, derek, holden, griffin, arthur, leon, felix, remington,
jake, killian, clayton, sean, riley, archer, legend, erick, enzo, corbin,
francisco, dallas, emilio, gunner, simon, andre, walter, damien, chance,
phoenix, colt, tanner, stephen, tobias, manuel, amari, emerson, louis, cody,
finley, martin, rafael, nash, beckham, cash, reid, theo, ace, eduardo,
spencer, raymond, maximiliano, anderson, ronan, lane, cristian, titus, travis,
jett, ricardo, bodhi, gideon, fernando, mario, conor, keegan, ali, cesar,
ellis, walker, cohen, arlo, hector, dante, garrett, donovan, seth, jeffrey,
tyson, jase, desmond, gage, atlas, major, devin, edwin, angelo, orion, conner,
julius, marco, jensen, peyton, zayn, collin, dakota, prince, johnny, cruz,
hendrix, atticus, troy, kane, edgar, sergio, kash, marshall, johnathan, romeo,
shane, warren, joaquin, wade, leonel, trevor, dominick, muhammad, erik, odin,
quinn, dalton, nehemiah, frank, grady, gregory, andy, solomon, malik, rory,
clark, reed, harvey, jay, jared, noel, shawn, fabian, ibrahim, adonis, ismael,
pedro, leland, malcolm, alexis, porter, sullivan, raiden, allen, ari, russell,
princeton, winston, kendrick, roberto, lennox, hayes, finnegan, nasir, kade,
nico, emanuel, landen, moises, ruben, hugo, abram, adan, khalil, augustus,
marcos, philip, phillip, cyrus, esteban, albert, bruce, lawson, jamison,
sterling, damon, gunnar, luka, franklin, ezequiel, pablo, derrick, zachariah,
cade, jonas, dexter, remy, hank, tate, trenton, kian, drew, mohamed, dax,
rocco, bowen, mathias, ronald, francis, matthias, milan, maximilian, royce,
skyler, corey, drake, gerardo, jayson, sage, benson, moses, rhys, otto,
oakley, armando, jaime, nixon, saul, scott, ariel, enrique, donald, chandler,
asa, eden, davis, keith, frederick, lawrence, leonidas, aden, julio, darius,
johan, deacon, cason, danny, nikolai, taylor, alec, royal, armani, kieran,
luciano, omari, rodrigo, arjun, ahmed, brendan, cullen, raul, raphael, ronin,
brock, pierce, alonzo, casey, dillon, uriel, dustin, gianni, roland, kobe,
dorian, emmitt, ryland, apollo, roy, duke, quentin, sam, lewis, tony, uriah,
dennis, moshe, braden, quinton, cannon, mathew, niko, edison, jerry, gustavo,
marvin, mauricio, ahmad, mohammad, justice, trey, mohammed, sincere, yusuf,
arturo, callen, keaton, wilder, memphis, conrad, soren, colby, bryant, lucian,
alfredo, cassius, marcelo, nikolas, brennan, darren, jimmy, lionel, reece, ty,
chris, forrest, tatum, jalen, santino, case, leonard, alvin, issac, bo,
quincy, mack, samson, rex, alberto, callum, curtis, hezekiah, briggs, zeke,
neil, titan, julien, kellen, devon, roger, axton, carl, douglas, larry,
crosby, fletcher, makai, nelson, hamza, lance, alden, gary, wilson,
alessandro, ares, bruno, jakob, stetson, zain, cairo, nathanael, byron, harry,
harley, mitchell, maurice, orlando, kingsley, trent, ramon, boston, lucca,
noe, jagger, randy, thaddeus, lennon, kannon, kohen, valentino, salvador,
langston, rohan, kristopher, yosef, lee, callan, tripp, deandre, joe, morgan,
reese, ricky, bronson, terry, eddie, jefferson, lachlan, layne, clay, madden,
tomas, kareem, stanley, amos, kase, kristian, clyde, ernesto, tommy, ford,
crew, hassan, axl, boone, leandro, samir, magnus, abdullah, yousef, branson,
layton, franco, ben, grey, kelvin, chaim, demetrius, blaine, ridge, colson,
melvin, anakin, aryan, jon, canaan, dash, zechariah, alonso, otis, zaire,
marcel, brett, stefan, aldo, jeffery, baylor, talon, dominik, flynn, carmelo,
dane, jamal, kole, enoch, kye, vicente, fisher, ray, fox, jamie, rey, zaid,
allan, emery, gannon, rodney, sonny, terrance, augustine, cory, felipe, aron,
jacoby, harlan

Female: emma, olivia, ava, isabella, sophia, mia, charlotte, amelia, evelyn,
abigail, harper, emily, elizabeth, avery, sofia, ella, madison, scarlett,
victoria, aria, grace, chloe, camila, penelope, riley, layla, lillian, nora,
zoey, mila, aubrey, hannah, lily, addison, eleanor, natalie, luna, savannah,
brooklyn, leah, zoe, stella, hazel, ellie, paisley, audrey, skylar, violet,
claire, bella, aurora, lucy, anna, samantha, caroline, genesis, aaliyah,
kennedy, kinsley, allison, maya, sarah, adeline, alexa, ariana, elena,
gabriella, naomi, alice, sadie, hailey, eva, emilia, autumn, quinn, piper,
ruby, serenity, willow, everly, cora, lydia, arianna, eliana, peyton, melanie,
gianna, isabelle, julia, valentina, nova, clara, vivian, reagan, mackenzie,
madeline, delilah, isla, katherine, sophie, josephine, ivy, liliana, jade,
maria, taylor, hadley, kylie, emery, natalia, annabelle, faith, alexandra,
ximena, ashley, brianna, bailey, mary, athena, andrea, leilani, jasmine, lyla,
margaret, alyssa, arya, norah, kayla, eden, eliza, rose, ariel, melody,
alexis, isabel, sydney, juliana, lauren, iris, emerson, london, morgan, lilly,
charlie, aliyah, valeria, arabella, sara, finley, trinity, jocelyn, kimberly,
esther, molly, valerie, cecilia, anastasia, daisy, reese, laila, mya, amy,
amaya, elise, harmony, paige, fiona, alaina, nicole, genevieve, lucia, alina,
mckenzie, callie, payton, eloise, brooke, mariah, julianna, rachel, daniela,
gracie, catherine, angelina, presley, josie, harley, vanessa, parker,
juliette, amara, marley, lila, ana, rowan, alana, michelle, malia, rebecca,
summer, sloane, leila, sienna, adriana, sawyer, kendall, juliet, destiny,
diana, hayden, ayla, dakota, angela, noelle, rosalie, joanna, lola, georgia,
selena, june, tessa, maggie, jessica, remi, delaney, camille, vivienne, hope,
mckenna, gemma, olive, alexandria, blakely, catalina, gabrielle, lucille,
ruth, evangeline, blake, thea, amina, giselle, melissa, river, kate, adelaide,
vera, leia, gabriela, zara, jane, journey, miriam, stephanie, cali, ember,
logan, annie, mariana, kali, haven, elsie, paris, lena, freya, lyric, camilla,
sage, jennifer, talia, alessandra, juniper, fatima, amira, arielle, phoebe,
ada, nina, samara, cassidy, aspen, allie, keira, kaia, amanda, heaven, joy,
lia, laura, lexi, haley, miranda, kaitlyn, daniella, felicity, jacqueline,
evie, angel, danielle, ainsley, dylan, kiara, millie, jordan, maddison,
alicia, maeve, margot, phoenix, heidi, alondra, lana, madeleine, kenzie,
miracle, shelby, elle, adrianna, bianca, kira, veronica, gwendolyn, esmeralda,
chelsea, alison, skyler, magnolia, daphne, jenna, kyla, harlow, annalise,
dahlia, scarlet, luciana, kelsey, nadia, amber, gia, carmen, jimena, erin,
christina, katie, ryan, viviana, alexia, anaya, serena, ophelia, regina,
helen, remington, cadence, royalty, amari, kathryn, skye, jada, saylor,
kendra, cheyenne, fernanda, sabrina, francesca, eve, mckinley, frances, sarai,
carolina, tatum, lennon, raven, leslie, winter, abby, mabel, sierra, april,
willa, carly, jolene, rosemary, selah, renata, lorelei, briana, celeste, wren,
leighton, annabella, mira, oakley, malaysia, edith, maryam, hattie, bristol,
demi, maia, sylvia, allyson, lilith, holly, meredith, nia, liana, megan,
justice, bethany, alejandra, janelle, elisa, adelina, myra, blair, charley,
virginia, kara, helena, sasha, julie, michaela, carter, matilda, henley,
maisie, hallie, priscilla, marilyn, cecelia, danna, colette, elliott, cameron,
celine, hanna, imani, angelica, kalani, alanna, lorelai, macy, karina, aisha,
johanna, mallory, leona, mariam, karen, karla, beatrice, gloria, milani,
savanna, rory, giuliana, lauryn, liberty, charli, jillian, anne, dallas,
azalea, tiffany, shiloh, jazmine, esme, elaine, lilian, kyra, kora, octavia,
irene, kelly, lacey, laurel, anika, dorothy, sutton, julieta, kimber, remy,
cassandra, rebekah, collins, elliot, emmy, sloan, hayley, amalia, jemma,
jamie, melina, leyla, wynter, alessia, monica, anya, antonella, ivory, greta,
maren, alena, emory, cynthia, alia, angie, alma, crystal, aileen, siena,
zelda, marie, pearl, reyna, mae, zahra, jessie, tiana, armani, lennox, lillie,
jolie, laney, mara, joelle, rosa, bridget, liv, aurelia, clarissa, elyse,
marissa, monroe, kori, elsa, rosie, amelie, eileen, poppy, royal, chaya,
frida, bonnie, amora, stevie, tatiana, malaya, mina, reign, annika, linda,
kenna, faye, reina, brittany, marina, astrid, briar, teresa, hadassah,
guadalupe, rayna, chanel, lyra, noa, laylah, livia, ellen, meadow, ellis,
milan, hunter, princess, nathalie, clementine, nola, simone, lina, marianna,
martha, louisa, emmeline, kenley, belen, erika, lara, amani, ansley, salma,
dulce, nala, natasha, mercy, penny, ariadne, deborah, elisabeth, zaria, hana,
raina, lexie, thalia, annabel, christine, estella, adele, aya, estelle,
landry, tori, perla, miah, angelique, romina, ari, jaycee, kai, louise, mavis,
belle, lea, rivka, calliope, sky, jewel, paola, giovanna, isabela, azariah,
dream, claudia, corinne, erica, milena, alyson, joyce, tinsley, whitney,
carolyn, frankie, andi, judith, paula, amia, hadlee, rayne, cara, celia, opal,
clare, gwen, veda, alisha, davina, rhea, noor, danica, kathleen, lindsey,
maxine, paulina, nancy, raquel, zainab, chana, lisa, heavenly, patricia,
india, paloma, ramona, sandra, abril, vienna, rosalyn, hadleigh, barbara,
jana, brenda, casey, selene, adrienne, aliya, miley, bexley, joslyn, zion,
breanna, melania, estrella, ingrid, jayden, kaya, dana, legacy, marjorie,
courtney, holland

##  Appendix 8.M Word Lists for HD

Gendered Words. These are the dictionary defined gendered words used by the
operation to determine what words are correctly gendered and which should be
neutral in the embedding space. Here is the filtered version of the list used
in our experiments as per our description in Section 1 about the test/train
word list split.

actress, actresses, aunt, aunts, bachelor, ballerina, barbershop, baritone,
beard, beards, beau, bloke, blokes, boy, boyfriend, boyfriends, boyhood, boys,
brethren, bride, brides, brother, brotherhood, brothers, bull, bulls,
businessman, businessmen, businesswoman, chairman, chairwoman, chap, colt,
colts, congressman, congresswoman, convent, councilman, councilmen,
councilwoman, countryman, countrymen, czar, dad, daddy, dads, daughter,
daughters, deer, diva, dowry, dude, dudes, estrogen, fathered, fatherhood,
fathers, fella, fellas, females, feminism, fiance, fiancee, fillies, filly,
fraternal, fraternities, fraternity, gal, gals, gelding, gentlemen,
girlfriend, girlfriends, girls, goddess, godfather, granddaughter,
granddaughters, grandma,grandmothers, grandpa, grandson, grandsons, handyman,
heiress, hen, hens, her, heroine, hers, herself, him, himself, his, horsemen,
hostess, housewife, housewives, hubby, husband, husbands, kings, lad, ladies,
lads, lesbian, lesbians, lion, lions, ma, macho, maid, maiden, maids, males,
mama, mare, maternal, maternity, men, menopause, mistress, mom, mommy, moms,
monastery, monk, monks, motherhood, mothers, nephew, nephews, niece, nieces,
nun, nuns, obstetrics, pa, paternity, penis, prince, princes, princess,
prostate, queens, salesman, salesmen, schoolboy, schoolgirl, semen, she, sir,
sister, sisters, son, sons, sorority, sperm, spokesman, spokesmen,
spokeswoman, stallion, statesman, stepdaughter, stepfather, stepmother,
stepson, strongman, stud, studs, suitor, suitors, testosterone, uncle, uncles,
uterus, vagina, viagra, waitress, widow, widower, widows, wife, witch,
witches, wives, womb, women

Equalized Words. These gendered words are paired (one male, one female) and
are equalized by the operation. Here is the filtered version of the list used
in our experiments as per our description in Section 1 about the test/train
word list split. Each pair in this list is “equalized”.

monastery convent
spokesman spokeswoman
priest nun
Dad Mom
Men Women
councilman councilwoman
grandpa grandma
grandsons granddaughters
testosterone estrogen
uncle aunt
wives husbands
Father Mother
Grandpa Grandma
He She
boys girls
brother sister
brothers sisters
businessman businesswoman
chairman chairwoman
colt filly
congressman congresswoman
dad mom
dads moms
dudes gals
fatherhood motherhood
fathers mothers
fella granny
fraternity sorority
gelding mare
gentlemen ladies
grandson granddaughter
himself herself
his her
king queen
kings queens
males females
men women
nephew niece
prince princess
schoolboy schoolgirl
son daughter
sons daughters

More details are available at: //github.com/tolga-b/debiaswe

##  Appendix 8.N Word Lists for INLP

The Gendered Word List ($G$) for INLP consists of 1425 words found under
\texttt{https://github.com/Shaul1321/nullspace\\_projection/blob/master/data/lists/}
as the list gender_specific_full.json. This list has been filtered of words
used in generating templates (Supplement 8.J and for WEAT (Supplement 8.K.

More details about their word lists and code is available at:
\texttt{https://github.com/Shaul1321/nullspace\\_projection}.

## References

  * [1] http://www.babynamewizard.com.
  * [2] https://www.ssa.gov/oact/babynames/.
  * [3] S. Arora, Y. Li, Y. Liang, T. Ma, and A. Risteski.  Linear algebraic structure of word senses, with applications to polysemy.  TACL, 2018.
  * [4] K. S. Arun, T. S. Huang, and S. D. Blostein.  Least-squares fitting of two 3-d points sets.  IEEE Trans. Pattern Analysis and Machine Intelligence, 9:698–700, 1987.
  * [5] S. Barocas and A. D. Selbst.  Big data’s disparate impact.  California Law Review, 104:671, 2016.
  * [6] H. Bay, T. Tuytelaars, and L. Van Gool.  Surf: Speeded up robust features.  In ECCV, page 404:417. Springer, 2006.
  * [7] P. J. Besl and N. D. McKay.  A method for registration of 3-d shapes.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):239 – 256, 1992.
  * [8] S. L. Blodgett, S. Barocas, H. D. III, and H. Wallach.  Language (technology) is power: A critical survey of ”bias” in nlp.  pages 5454–5476. ACL, 2020.
  * [9] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov.  Enriching word vectors with subword information.  arXiv, 2016.
  * [10] D. Bollegala, K. Hayashi, and K.-i. Kawarabayashi.  Learning linear transformations between counting-based and prediction-based word embeddings.  PloS one, 12(9):e0184544, 2017.
  * [11] T. Bolukbasi, K. W. Chang, J. Zou, V. Saligrama, and A. Kalai.  Quantifying and reducing bias in word embeddings.  ICML Workshop on Data4Good, 2016.
  * [12] T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai.  Man is to computer programmer as woman is to homemaker? Debiasing word embeddings.  In Advances in neural information processing systems, pages 4349–4357, 2016.
  * [13] T. Boucher, C. Carey, S. Mahadevan, and M. D. Dyar.  Aligning mixed manifolds.  In AAAI, pages 2511–2517, 2015.
  * [14] S. Bowman, G. Angeli, C. Potts, and C. D. Manning.  A large annotated corpus for learning natural language inference.  In EMNLP 2015, pages 632–642. ACL, 2015.
  * [15] T. Brennan, W. Dieterich, and B. Ehret.  Evaluating the predictive validity of the compas risk and needs assessment system.  Criminal Justice and Behavior, 36(1):21–40, 2009.
  * [16] H. Cai, V. W. Zheng, and K. C.-C. Chang.  A comprehensive survey of graph embedding: Problems, techniques and applications.  arXiv, 2017.
  * [17] A. Caliskan, J. J. Bryson, and A. Narayanan.  Semantics derived automatically from language corpora contain human-like biases.  Science, 356(6334):183–186, 2017.
  * [18] Y. Chen and G. Medioni.  Object modelling by registration of multiple range images.  Image and Vision Computing, 10:145–155, 1992.
  * [19] J. Cheng, L. Dong, and M. Lapata.  Long short-term memory-networks for machine reading.  In Proceedings of EMNLP, pages 551–561, 2016.
  * [20] K. Crenshaw.  Mapping the margins: Intersectionality, identity politics, and violence against women of color.  Stanford Law Review, 43(6):1241–1299, 1991.
  * [21] I. Dagan, O. Glickman, and B. Magnini.  The pascal recognising textual entailment challenge.  In Machine Learning Challenges Workshop, pages 177–190. Springer, 2005.
  * [22] I. Dagan, D. Roth, M. Sammons, and F. M. Zanzotto.  Recognizing textual entailment: Models and applications.  Synthesis Lectures on Human Language Technologies, 6(4):1–220, 2013\.
  * [23] M. De-Arteaga, A. Romanov, H. Wallach, J. Chayes, C. Borgs, A. Chouldechova, S. Geyik, K. Kenthapadi, and A. T. Kalai.  Bias in bios: A case study of semantic representation bias in a high-stakes setting.  In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120–128, 2019.
  * [24] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In IEEE conference on computer vision and pattern recognition, pages 248–255, 2009.
  * [25] S. Dev, S. Hassan, and J. M. Phillips.  Closed form word embedding alignment.  In IEEE ICDM, pages 130–139, 2019.
  * [26] S. Dev, T. Li, J. M. Phillips, and V. Srikumar.  On measuring and mitigating biased inferences of word embeddings.  In AAAI, pages 7659–7666, 2020.
  * [27] S. Dev, T. Li, J. M. Phillips, and V. Srikumar.  OSCaR: Orthogonal subspace correction and rectification of biases in word embeddings.  arXiv, 2020.
  * [28] S. Dev and J. Phillips.  Attenuating bias in word vectors.  In AISTATS, pages 879–887, 2019.
  * [29] J. Devlin, M. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In NAACL-HLT, 2019.
  * [30] Y. Dong, N. V. Chawla, and A. Swami.  metapath2vec: Scalable representation learning for heterogeneous networks.  In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 135–144, 2017.
  * [31] D. Eggert, A. Lorusso, and R. Fisher.  Estimating 3-d rigid body transformations: A comparison of four major algorithms.  Machine Vision and Applications, 9:272–290, 1997.
  * [32] O. D. Faugeras and M. Hebert.  A 3-d recognition and positioning algorithm using geometric matching between primitive surfaces.  In Proceedings International Joint Conference on Artificial Intelligence, volume 8, pages 996–1002, August 1983.
  * [33] C. Fellbaum.  WordNet: An Electronic Lexical Database.  Bradford Books, 1998.
  * [34] L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, and E. Ruppin.  Placing search in context: The concept revisited.  In Proceedings of World Wide Web, pages 406–414, 2001.
  * [35] M. Gardner, J. Grus, M. Neumann, O. Tafjord, P. Dasigi, N. F. Liu, M. Peters, M. Schmitz, and L. S. Zettlemoyer.  AllenNLP: A deep semantic natural language processing platform.  In ACL workshop for NLP Open Source Software, 2018.
  * [36] H. Gonen and Y. Goldberg.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  In NAACL-HLT, volume 1, pages 609–614, 2019.
  * [37] Y. Gong, Q. Ke, M. Isard, and S. Lazebnik.  A multi-view embedding space for modeling internet images, tags, and their semantics.  ICJV, 106(2):210–233, 2014.
  * [38] P. Goyal and E. Ferrara.  Graph embedding techniques, applications, and performance: A survey.  arXiV, 2017.
  * [39] A. Grover and J. Leskovec.  node2vec: Scalable feature learning for networks.  In Proceedings of KDD, pages 855–864, 2016.
  * [40] A. Grover and J. Leskovec.  node2vec: Scalable feature learning for networks.  In ACM SIGKDD, pages 855–864, 2016.
  * [41] S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. Bowman, and N. A. Smith.  Annotation artifacts in natural language inference data.  In NAACL, 2018.
  * [42] R. J. Hanson and M. J. Norris.  Analysis of measurements based on the singular value decomposition.  SIAM Journal of Scientific and Statistical Computing, 27(3):363–373, 1981.
  * [43] B. Heinzerling and M. Strube.  BPEmb: Tokenization-free pre-trained subword embeddings in 275 languages.  In LREC. ELRA, 2018.
  * [44] L. A. Hendricks, K. Burns, K. Saenko, T. Darrell, and A. Rohrbach.  Women also snowboard: Overcoming bias in captioning models.  In ECCV, pages 793–811. Springer, 2018.
  * [45] K. M. Hermann and P. Blunsom.  Multilingual distributed representations without word alignment.  ArXiv, 2013.
  * [46] F. Hill, R. Reichart, and A. Korhonen.  Simlex-999: Evaluating semantic models with (genuine) similarity estimation.  Computational Linguistics, 41(4):665–695, 2015.
  * [47] B. K. Horn.  Closed-form solution of absolute orientation using unit quaternions.  Journal of the Optical Society of America A, 4(4):629–642, 1987\.
  * [48] B. K. Horn, H. M. Hilden, and S. Negahdaripour.  Closed-form solution of absolute orientation using orthonormal matrices.  Journal of the Optical Society of America A, 5(7):1127–1135, 1988\.
  * [49] A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov.  Bag of tricks for efficient text classification.  arXiv, 2016.
  * [50] A. E. Khandani, A. J. Kim, and A. W. Lo.  Consumer credit-risk models via machine-learning algorithms.  Journal of Banking & Finance, 34(11):2767–2787, 2010.
  * [51] K. Kothari, D. Gelda, W. Zhang, and H. Yang.  Magellan: A personalized context-aware travel recommendation system.  to appear in CIKM, 2020.
  * [52] A. Lauscher, G. Glavas, S. P. Ponzetto, and I. Vulic.  A general framework for implicit and explicit debiasing of distributional word vector spaces.  arXiv, 2019.
  * [53] O. Levy and Y. Goldberg.  Linguistic regularities of sparse and explicit word representations.  In CoNLL, 2014.
  * [54] O. Levy and Y. Goldberg.  Neural word embedding as implicit matrix factorization.  In Advances in Neural Information Processing Systems 27, pages 2177–2185. 2014.
  * [55] S. Liu, P. Bremer, J. J. Thiagarajan, V. Srikumar, B. Wang, Y. Livnat, and V. Pascucci.  Visual exploration of semantic relationships in neural word embeddings.  IEEE Transactions on Visualization and Computer Graphics, 24(1):553–562, 2018.
  * [56] S. Liu, T. Li, Z. Li, V. Srikumar, V. Pascucci, and P.-T. Bremer.  Visual interrogation of attention-based models for natural language inference and machine comprehension.  In Proceedings of the 2018 Conference on EMNLP: System Demonstrations. ACL, Nov. 2018.
  * [57] S. Liu, Z. Li, T. Li, V. Srikumar, V. Pascucci, and P. Bremer.  Nlize: A perturbation-driven visual interrogation tool for analyzing and interpreting natural language inference models.  IEEE Transactions on Visualization and Computer Graphics, 25(1):651–660, 2019.
  * [58] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  CoRR, 2019.
  * [59] E. Loper and S. Bird.  Nltk: The natural language toolkit.  In ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, 2002.
  * [60] D. G. Lowe.  Distinctive image features from scale-invariant keypoints.  International Journal of Computer Vision, 60:91–110, 2004.
  * [61] T. Luong, H. Pham, and C. D. Manning.  Bilingual word representations with monolingual quality in mind.  In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151–159. ACL, 2015.
  * [62] T. Manzini, L. Yao Chong, A. W. Black, and Y. Tsvetkov.  Black is to criminal as Caucasian is to police: Detecting and removing multiclass bias in word embeddings.  In Proceedings of NAACL: HLT, Volume 1 (Long and Short Papers), pages 615–621. ACL, 2019.
  * [63] C. May, A. Wang, S. Bordia, S. R. Bowman, and R. Rudinger.  On measuring social biases in sentence encoders.  In Proceedings of NAACL: HLT, Volume 1 (Long and Short Papers), pages 622–628. ACL, 2019.
  * [64] T. Mikolov, K. Chen, G. Corrado, and J. Dean.  Efficient estimation of word representations in vector space.  Technical report, arXiv, 2013.
  * [65] T. Mikolov, Q. V. Le, and I. Sutskever.  Exploiting similarities among languages for machine translation.  arXiv, Sept. 2013.
  * [66] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean.  Distributed representations of words and phrases and their compositionality.  In Proceedings of Neural Information Processing Systems - Volume 2, page 3111–3119, 2013.
  * [67] G. A. Miller and W. G. Charles.  Contextual correlates of semantic similarity.  Language and Cognitive Processes, 6(1):1–28, 1991.
  * [68] O. Papakyriakopoulos, S. Hegelich, J. C. M. Serrano, and F. Marco.  Bias in word embeddings.  In Proceedings of FAccT, pages 446–457. ACM, 2020.
  * [69] A. Parikh, O. Täckström, D. Das, and J. Uszkoreit.  A decomposable attention model for natural language inference.  In Proceedings of EMNLP, pages 2249–2255. ACL, 2016.
  * [70] J. Pennington, R. Socher, and C. Manning.  GloVe: Global vectors for word representation.  In Proceedings of EMNLP, pages 1532–1543. ACL, 2014.
  * [71] B. Perozzi, R. Al-Rfou, and S. Skiena.  Deepwalk: Online learning of social representations.  In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 701–710. ACM, 2014.
  * [72] M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer.  Deep contextualized word representations.  In Proceedings of the NAACL: HLT, Volume 1 (Long Papers), pages 2227–2237. ACL, 2018.
  * [73] M. T. Pilehvar, D. Kartsaklis, V. Prokhorov, and N. Collier.  Card-660: Cambridge rare word dataset - a reliable benchmark for infrequent word representation models.  In EMNLP, 2018.
  * [74] A. Rahimi and B. Recht.  Random features for large-scale kernel machines.  In Advances in Neural Information Processing Systems, pages 1177–1184. 2008.
  * [75] A. Rahimi and B. Recht.  Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning.  In Advances in Neural Information Processing Systems 21, pages 1313–1320. 2009.
  * [76] S. Ravfogel, Y. Elazar, H. Gonen, M. Twiton, and Y. Goldberg.  Null it out: Guarding protected attributes by iterative nullspace projection.  In Proceedings of ACL, pages 7237–7256, 2020.
  * [77] R. Řehůřek and P. Sojka.  Software framework for topic modelling with large corpora.  In Proceedings of the LREC Workshop on New Challenges for NLP Frameworks, pages 45–50. ELRA, 2010.
  * [78] P. Ristoski and H. Paulheim.  Rdf2vec: Rdf graph embeddings for data mining.  In Proceedings of International Semantic Web Conference, pages 498–514, 2016.
  * [79] H. Rubenstein and J. B. Goodenough.  Contextual correlates of synonymy.  Communications of ACM, 8(10):627–633, 1965.
  * [80] R. Rudinger, C. May, and B. Van Durme.  Social bias in elicited natural language inferences.  In Proceedings of ACL Workshop on Ethics in Natural Language Processing, pages 74–79. ACL, 2017.
  * [81] R. Rudinger, J. Naradowsky, B. Leonard, and B. Van Durme.  Gender bias in coreference resolution.  In Proceedings of NAACL: HLT, Volume 2 (Short Papers), pages 8–14, New Orleans, Louisiana, June 2018. ACL.
  * [82] C. Safak Sahin, R. S. Caceres, B. Oselio, and W. M. Campbell.  Consistent Alignment of Word Embedding Models.  arXiv, 2017.
  * [83] P. H. Schönemann.  A generalized solution to the orthogonal procrustes problem.  Psychometrika, 31(1):1–10, March 1966.
  * [84] J. T. Schwartz and M. Sharir.  Identification of partially obscured objects in two and three dimensions by matching noisy characteristic curves.  International Journal on Robotics Research, 6(2), 1987.
  * [85] M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi.  Bidirectional attention flow for machine comprehension.  ICLR, 2017.
  * [86] Q. Shi, J. Petterson, G. Dror, J. Langford, A. Smola, and S. Vishwanathan.  Hash kernels for structured data.  Journal of Machine Learning Research, 10(90):2615–2637, 2009.
  * [87] S. Shin, K. Song, J. Jang, H. Kim, W. Joo, and I.-C. Moon.  Neutralizing gender bias in word embedding with latent disentanglement and counterfactual generation.  arXiv, 2020.
  * [88] S. L. Smith, D. H. P. Turban, S. Hamblin, and N. Y. Hammerla.  Offline bilingual word vectors, orthogonal transformations and the inverted softmax.  ICLR, 2017.
  * [89] T. Sun, A. Gaut, S. Tang, Y. Huang, M. ElSherief, J. Zhao, D. Mirza, E. M. Belding, K. Chang, and W. Y. Wang.  Mitigating gender bias in natural language processing: Literature review.  CoRR, 2019.
  * [90] C. Sweeney and M. Najafian.  A transparent framework for evaluating unintended demographic bias in word embeddings.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1662–1667. ACL, 2019.
  * [91] N. Swinger, M. De-Arteaga, N. T. H. IV, M. D. M. Leiserson, and A. T. Kalai.  What are the biases in my word embedding?  CoRR, 2018.
  * [92] Y. C. Tan and L. E. Celis.  Assessing social and intersectional biases in contextualized word representations.  arXiv, 2019.
  * [93] V. Tshitoyan, J. Dagdelen, L. Weston, A. Dunn, Z. Rong, O. Kononova, K. Persson, G. Ceder, and A. Jain.  Unsupervised word embeddings capture latent knowledge from materials science literature.  Nature, 571:95–98, 07 2019.
  * [94] M. W. Walker, L. Shao, and R. A. Volz.  Estimating 3-d location parameters using dual number quaternions.  CVGIP: Image Understanding, 54(3):358–367, 1991.
  * [95] Y. Wang, Y. Zheng, Y. Peng, W. Zhang, and F. Li.  Feature detection and attenuation in embeddings.  arXiv, 2019.
  * [96] K. Webster, M. Recasens, V. Axelrod, and J. Baldridge.  Mind the gap: A balanced corpus of gendered ambiguous pronouns.  TACL, 6:605–617, 2018.
  * [97] K. Weinberger, A. Dasgupta, J. Langford, A. Smola, and J. Attenberg.  Feature hashing for large scale multitask learning.  page 1113–1120. ACM, 2009.
  * [98] A. Williams, N. Nangia, and S. Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In Proceedings of NAACL: HLT, Volume 1 (Long Papers), pages 1112–1122. ACL, 2018.
  * [99] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al.  Transformers: State-of-the-art natural language processing.  arXiv, 2019.
  * [100] M. Xie, H. Yin, H. Wang, F. Xu, W. Chen, and S. Wang.  Learning graph-based poi embedding for location-based recommendation.  ACM, 2016.
  * [101] C.-C. M. Yeh, D. Gelda, Z. Zhuang, Y. Zheng, L. Gou, and W. Zhang.  Towards a flexible embedding learning framework.  arXiv, 2020.
  * [102] B. H. Zhang, B. Lemoine, and M. Mitchell.  Mitigating unwanted biases with adversarial learning.  In Proceedings of AAAI/ACM FAT*, pages 335–340, 2018.
  * [103] J. Zhao, S. Mukherjee, S. Hosseini, K.-W. Chang, and A. H. Awadallah.  Gender bias in multilingual embeddings and cross-lingual transfer.  arXiv, 2020.
  * [104] J. Zhao, T. Wang, M. Yatskar, R. Cotterell, V. Ordonez, and K.-W. Chang.  Gender bias in contextualized word embeddings.  In Proceedings of NAACL: HLT, Volume 1 (Long and Short Papers), pages 629–634. ACL, 2019.
  * [105] J. Zhao, T. Wang, M. Yatskar, V. Ordonez, and K.-W. Chang.  Men also like shopping: Reducing gender bias amplification using corpus-level constraints.  In Proceedings of EMNLP, pages 2941–2951. ACL, 2017.
  * [106] J. Zhao, Y. Zhou, Z. Li, W. Wang, and K.-W. Chang.  Learning gender-neutral word embeddings.  In Proceedings of EMNLP, pages 4847–4853. ACL, 2018.