11institutetext: Facebook AI Research, Paris, France, 11email: {nneverova,
<EMAIL_ADDRESS>22institutetext: INRIA-CentraleSupélec, Paris, France,
22email<EMAIL_ADDRESS>
# Dense Pose Transfer

Natalia Neverova 11 Rıza Alp Güler 22 Iasonas Kokkinos 11

###### Abstract

In this work we integrate ideas from surface-based modeling with neural
synthesis: we propose a combination of surface-based pose estimation and deep
generative models that allows us to perform accurate pose transfer, i.e.
synthesize a new image of a person based on a single image of that person and
the image of a pose donor. We use a dense pose estimation system that maps
pixels from both images to a common surface-based coordinate system, allowing
the two images to be brought in correspondence with each other. We inpaint and
refine the source image intensities in the surface coordinate system, prior to
warping them onto the target pose. These predictions are fused with those of a
convolutional predictive module through a neural synthesis module allowing for
training the whole pipeline jointly end-to-end, optimizing a combination of
adversarial and perceptual losses. We show that dense pose estimation is a
substantially more powerful conditioning input than landmark-, or mask-based
alternatives, and report systematic improvements over state of the art
generators on DeepFashion and MVC datasets.

Figure 1: Overview of our pose transfer pipeline: given an input image and a
target pose we use DensePose [1] to drive the generation process. This is
achieved through the complementary streams of (a) a data-driven predictive
module, and (b) a surface-based module that warps the texture to UV-
coordinates, interpolates on the surface, and warps back to the target image.
A blending module combines the complementary merits of these two streams in a
single end-to-end trainable framework.

##  1 Introduction

Deep models have recently shown remarkable success in tasks such as face [2],
human [3, 4, 5], or scene generation [6, 7], collectively known as “neural
synthesis”. This opens countless possibilities in computer graphics
applications, including cinematography, gaming and virtual reality settings.
At the same time, the potential malevolent use of this technology raises new
research problems, including the detection of forged images or videos [8],
which in turn requires training forgery detection algorithms with multiple
realistic samples. In addition, synthetically generated images have been
successfully exploited for data augmentation and training deep learning
frameworks for relevant recognition tasks [9].

In most applications, the relevance of a generative model to the task directly
relates to the amount of control that one can exert on the generation process.
Recent works have shown the possibility of adjusting image synthesis by
controlling categorical attributes [10, 3], low-dimensional parameters [11],
or layout constraints indicated by a conditioning input [12, 6, 7, 3, 4, 5].
In this work we aspire to obtain a stronger hold of the image synthesis
process by relying on surface-based object representations, similar to the
ones used in graphics engines.

Our work is focused on the human body, where surface-based image understanding
has been most recently unlocked [13, 14, 15, 16, 17, 18, 1]. We build on the
recently introduced SMPL model [13] and the DensePose system [1], which taken
together allow us to interpret an image of a person in terms of a full-fledged
surface model, getting us closer to the goal of performing inverse graphics.

In this work we close the loop and perform image generation by rendering the
same person in a new pose through surface-based neural synthesis. The target
pose is indicated by the image of a ‘pose donor’, i.e. another person that
guides the image synthesis. The DensePose system is used to associate the new
photo with the common surface coordinates and copy the appearance predicted
there.

The purely geometry-based synthesis process is on its own insufficient for
realistic image generation: its performance can be compromised by inaccuracies
of the DensePose system as well as by self-occlusions of the body surface in
at least one of the two images. We account for occlusions by introducing an
inpainting network that operates in the surface coordinate system and combine
its predictions with the outputs of a more traditional feedforward conditional
synthesis module. These predictions are obtained independently and compounded
by a refinement module that is trained so as to optimize a combination of
reconstruction, perceptual and adversarial losses.

We experiment on the DeepFashion [19] and MVC [20] datasets and show that we
can obtain results that are quantitatively better than the latest state-of-
the-art. Apart from the specific problem of pose transfer, the proposed
combination of neural synthesis with surface-based representations can also be
promising for the broader problems of virtual and augmented reality: the
generation process is more transparent and easy to connect with the physical
world, thanks to the underlying surface-based representation. In the more
immediate future, the task of pose transfer can be useful for dataset
augmentation, training forgery detectors, as well as texture transfer
applications like those showcased by [1], without however requiring the
acquisition of a surface-level texture map.

##  2 Previous works

Deep generative models have originally been studied as a means of unsupervised
feature learning [21, 22, 23, 24]; however, based on the increasing realism of
neural synthesis models [25, 2, 6, 7] such models can now be considered as
components in computer graphics applications such as content manipulation [7].

Loss functions used to train such networks largely determine the realism of
the resulting outputs. Standard reconstruction losses, such as $\ell_{1}$ or
$\ell_{2}$ norms typically result in blurry results, but at the same time
enhance stability [12]. Realism can be enforced by using an adapted
discriminator loss trained in tandem with a generator in Generative
Adversarial Network (GAN) architectures [23] to ensure that the generated and
observed samples are indistinguishable. However, this training can often be
unstable, calling for more robust variants such as the squared loss of [26],
WGAN and its variants [27] or multi-scale discriminators as in [7]. An
alternative solution is the perceptual loss used in [28, 29] replacing the
optimization-based style transfer of [25] with feedforward processing. This
was recently shown in [6] to deliver substantially more accurate scene
synthesis results than [12], while compelling results were obtained more
recently by combining this loss with a GAN-style discriminator [7].

Person and clothing synthesis has been addressed in a growing body of recently
works [3, 5, 4, 30]. All of these works assist the image generation task
through domain, person-specific knowledge, which gives both better quality
results, and a more controllable image generation pipeline.

Conditional neural synthesis of humans has been shown in [5, 4] to provide a
strong handle on the output of the generative process. A controllable surface-
based model of the human body is used in [3] to drive the generation of
persons wearing clothes with controllable color combinations. The generated
images are demonstrably realistic, but the pose is determined by controlling a
surface based model, which can be limiting if one wants e.g. to render a
source human based on a target video. A different approach is taken in the
pose transfer work of [4], where a sparse set of landmarks detected in the
target image are used as a conditioning input to a generative model. The
authors show that pose can be generated with increased accuracy, but often
losing texture properties of the source images, such as cloth color or texture
properties. In the work of [31] multi-view supervision is used to train a two-
stage system that can generate images from multiple views. In more recent work
[5] the authors show that introducing a correspondence component in a GAN
architecture allows for substantially more accurate pose transfer.

Image inpainting helps estimate the body appearance on occluded body regions.
Generative models are able to fill-in information that is labelled as
occluded, either by accounting for the occlusion pattern during training [32],
or by optimizing a score function that indicates the quality of an image, such
as the negative of a GAN discriminator loss [33]. The work of [33] inpaints
arbitrarily occluded faces by minimizing the discriminator loss of a GAN
trained with fully-observed face patches. In the realm of face analysis
impressive results have been generated recently by works that operate in the
UV coordinate system of the face surface, aiming at photorealistic face
inpainting [34], and pose-invariant identification [35]. Even though we
address a similar problem, the lack of access to full UV recordings (as in
[35, 34]) poses an additional challenge.

##  3 Dense Pose Transfer

We develop our approach to pose transfer around the DensePose estimation
system [1] to associate every human pixel with its coordinates on a surface-
based parameterization of the human body in an efficient, bottom-up manner. We
exploit the DensePose outputs in two complementary ways, corresponding to the
predictive module and the warping module, as shown in Fig. 1. The warping
module uses DensePose surface correspondence and inpainting to generate a new
view of the person, while the predictive module is a generic, black-box,
generative model conditioned on the DensePose outputs for the input and the
target.

These modules corresponding to two parallel streams have complementary merits:
the predictive module successfully exploits the dense conditioning output to
generate plausible images for familiar poses, delivering superior results to
those obtained from sparse, landmark-based conditioning; at the same time, it
cannot generalize to new poses, or transfer texture details. By contrast, the
warping module can preserve high-quality details and textures, allows us to
perform inpainting in a uniform, canonical coordinate system, and generalizes
for free for a broad variety of body movements. However, its body-, rather
than clothing-centered construction does not take into account hair, hanging
clothes, and accessories. The best of both worlds is obtained by feeding the
outputs of these two blocks into a blending module trained to fuse and refine
their predictions using a combination of reconstruction, adversarial, and
perceptual losses in an end-to-end trainable framework.

The DensePose module is common to both streams and delivers dense
correspondences between an image and a surface-based model of the human body.
It does so by firstly assigning every pixel to one of 24 predetermined surface
parts, and then regressing the part-specific surface coordinates of every
pixel. The results of this system are encoded in three output channels,
comprising the part label and part-specific UV surface coordinates. This
system is trained discriminatively and provides a simple, feed-forward module
for dense correspondence from an image to the human body surface. We omit
further details, since we rely on the system of [1] with minor implementation
differences described in Sec. 4.

Having outlined the overall architecture of our system, in Sec. 3.1 and Sec.
3.3 we present in some more detail our components, and then turn in Sec. 3.4
to the loss functions used in their training. A thorough description of
architecture details is left to the supplemental material. We start by
presenting the architecture of the predictive stream, and then turn to the
surface-based stream, corresponding to the upper and lower rows of Fig. 1,
respectively.

###  3.1 Predictive stream

The predictive module is a conditional generative model that exploits the
DensePose system results for pose transfer. Existing conditional models
indicate the target pose in the form of heat-maps from keypoint detectors [4],
or part segmentations [3]. Here we condition on the concatenation of the input
image and DensePose results for the input and target images, resulting in an
input of dimension $256{\times}256{\times}9$. This provides conditioning that
is both global (part-classification), and point-level (continuous
coordinates), allowing the remaining network to exploit a richer source of
information.

The remaining architecture includes an encoder followed by a stack of residual
blocks and a decoder at the end, along the lines of [28]. In more detail, this
network comprises (a) a cascade of three convolutional layers that encode the
$256{\times}256{\times}9$ input into $64{\times}64{\times}256$ activations,
(b) a set of six residual blocks with $3{\times}3{\times}256{\times}256$
kernels, (c) a cascade of two deconvolutional and one convolutional layer that
deliver an output of the same spatial resolution as the input. All
intermediate convolutional layers have $3{\times}3$ filters and are followed
by instance normalization [36] and ReLU activation. The last layer has tanh
non-linearity and no normalization.

###  3.2 Warping stream

Our warping module performs pose transfer by performing explicit texture
mapping between the input and the target image on the common surface UV-
system. The core of this component is a Spatial Transformer Network (STN) [37]
that warps according to DensePose the image observations to the UV-coordinate
system of each surface part; we use a grid with $256{\times}256$ UV points for
each of the 24 surface parts, and perform scattered interpolation to handle
the continuous values of the regressed UV coordinates. The inverse mapping
from UV to the output image space is performed by a second STN with a bilinear
kernel. As shown in Fig. 3, a direct implementation of this module would often
deliver poor results: the part of the surface that is visible on the source
image is typically small, and can often be entirely non-overlapping with the
part of the body that is visible on the target image. This is only exacerbated
by DensePose failures or systematic errors around the part seams. These
problems motivate the use of an inpainting network within the warping module,
as detailed below.

Figure 2: Supervision signals for pose transfer on the warping stream: the
input image on the left is warped to intrinsic surface coordinates through a
spatial transformer network driven by DensePose. From this input, the
inpainting autoencoder has to predict the appearance of the same person from
different viewpoints, when also warped to intrinsic coordinates. The loss
functions on the right penalize the reconstruction only on the observed parts
of the texture map. This form of multi-view supervision acts like a surrogate
for the (unavailable) appearance of the person on the full body surface.

Inpainting autoencoder. This model allows us to extrapolate the body
appearance from the surface nodes populated by the STN to the remainder of the
surface. Our setup requires a different approach to the one of other deep
inpainting methods [33], because we never observe the full surface texture
during training. We handle the partially-observed nature of our training
signal by using a reconstruction loss that only penalizes the observed part of
the UV map, and lets the network freely guess the remaining domain of the
signal. In particular, we use a masked $\ell_{1}$ loss on the difference
between the autoencoder predictions and the target signals, where the masks
indicate the visibility of the target signal.

We observed that by its own this does not urge the network to inpaint
successfully; results substantially improve when we accompany every input with
multiple supervision signals, as shown in Fig. 2, corresponding to UV-wrapped
shots of the same person at different poses. This fills up a larger portion of
the UV-space and forces the inpainting network to predict over the whole
texture domain. As shown in Fig. 3, the inpainting process allows us to obtain
a uniformly observed surface, which captures the appearance of skin and tight
clothes, but does not account for hair, skirts, or apparel, since these are
not accommodated by DensePose’s surface model.

Our inpainting network is comprised of $N$ autoencoders, corresponding to the
decomposition of the body surface into $N$ parts used in the original
DensePose system [1]. This is based on the observation that appearance
properties are non-stationary over the body surface. Propagation of the
context-based information from visible to invisible parts that are entirely
occluded at the present view is achieved through a fusion mechanism that
operates at the level of latent representations delivered by the individual
encoders, and injects global pose context in the individual encoding through a
concatenation operation.

In particular, we denote by $\mathcal{E}_{i}$ the individual encoding
delivered by the encoder for the $i$-th part. The fusion layer concatenates
these obtained encodings into a single vector which is then down-projected to
a 256-dimensional global pose embedding through a linear layer. We pass the
resulting embedding through a cascade of ReLU and Instance-Norm units and
transform it again into an embedding denoted by $\mathcal{G}$.

Then the $i$-th part decoder receives as an input the concatenation of
$\mathcal{G}$ with $\mathcal{E}_{i}$, which combines information particular to
part $i$, and global context, delivered by $\mathcal{G}$. This is processed by
a stack of deconvolution operations, which delivers in the end the prediction
for the texture of part $i$.

###  3.3 Blending module

The blending module’s objective is to combine the complementary strengths of
the two streams to deliver a single fused result, that will be ‘polished’ as
measured by the losses used for training. As such it no longer involves an
encoder or decoder unit, but rather only contains two convolutional and three
residual blocks that aim at combining the predictions and refining their
results.

In our framework, both predictive and warping modules are first pretrained
separately and then finetuned jointly during blending. The final refined
output is obtained by learning a residual term added to the output of the
predictive stream. The blending module takes an input consisting of the
outputs of the predictive and the warping modules combined with the target
dense pose.

Figure 3: Warping module results. For each sample, the top row shows
interpolated textures obtained from DensePose predictions and projected on the
surface of the 3D body model. The bottom row shows the same textures after
inpainting in the UV space.

###  3.4 Loss Functions

As shown in Fig. 1, the training set for our network comes in the form of
pairs of input and target images, $\boldsymbol{x}$, $\boldsymbol{y}$
respectively, both of which are of the same person-clothing, but in different
poses. Denoting by $\boldsymbol{\hat{y}}=G(\boldsymbol{x})$ the network’s
prediction, the difference between $\boldsymbol{\hat{y}},\boldsymbol{y}$ can
be measured through a multitude of loss terms, that penalize different forms
of deviation. We present them below for completeness and ablate their impact
in practice in Sec. 4.

Reconstruction loss. To penalize reconstruction errors we use the common
$\ell_{1}$ distance between the two signals:
$\|\boldsymbol{\hat{y}}-\boldsymbol{y}\|_{1}$. On its own, this delivers
blurry results but is important for retaining the overall intensity levels.

Perceptual loss. As in Chen and Koltun [6], we use a VGG19 network pretrained
for classification [38] as a feature extractor for both
$\boldsymbol{\hat{y}},\boldsymbol{y}$ and penalize the $\ell_{2}$ distance of
the respective intermediate feature activations $\Phi^{v}$ at 5 different
network layers $v=1,\ldots,N$:

$\mathcal{L}_{\mathrm{p}}(\boldsymbol{y},\boldsymbol{\hat{y}})=\sum_{v=1}^{N}\|{\Phi}^{v}(\boldsymbol{y})-{\Phi}^{v}(\boldsymbol{\hat{y}})\|_{2}.$
(1)

This loss penalizes differences in low- mid- and high-level feature
statistics, captured by the respective network filters.

Style loss. As in [28], we use the Gram matrix criterion of [25] as an
objective for training a feedforward network. We first compute the Gram matrix
of neuron activations delivered by the VGG network $\Phi$ at layer $v$ for an
image $\boldsymbol{x}$:

$\mathcal{G}^{v}(\boldsymbol{x})_{c,c^{\prime}}=\sum_{h,w}\Phi^{v}_{c}(\boldsymbol{x})[h,w]\Phi^{v}_{c^{\prime}}(\boldsymbol{x})[h,w]$
(2)

where $h$ and $w$ are horizontal and vertical pixel coordinates and $c$ and
$c^{\prime}$ are feature maps of layer $v$. The style loss is given by the sum
of Frobenius norms for the difference between the per-layer Gram matrices
$\mathcal{G}^{v}$ of the two inputs:

$\mathcal{L}_{\mathrm{style}}(\boldsymbol{y},\boldsymbol{\hat{y}})=\sum_{v=1}^{B}\|\mathcal{G}^{v}(\boldsymbol{y})-\mathcal{G}^{v}(\boldsymbol{\hat{y}})\|_{F}.$
(3)

Adversarial loss. We use adversarial training to penalize any detectable
differences between the generated and real samples. Since global structural
properties are largely settled thanks to DensePose conditioning, we opt for
the patchGAN [12] discriminator, which operates locally and picks up
differences between texture patterns. The discriminator [12, 7] takes as an
input $\boldsymbol{z}$, a combination of the source image and the DensePose
results on the target image, and either the target image $\boldsymbol{y}$
(real) or the generated output (fake) $\boldsymbol{\hat{y}}$. We want fake
samples to be indistinguishable from real ones – as such we optimize the
following objective:

$L_{\text{GAN}}=\underbrace{\frac{1}{2}\mathbb{E}_{\boldsymbol{z}}\left[l(D(\boldsymbol{z},\boldsymbol{y})-1)\right]+\frac{1}{2}\mathbb{E}_{\boldsymbol{z}}\left[l(D(\boldsymbol{z},\boldsymbol{\hat{y}}))\right]}_{\mathrm{Discriminator}}+\underbrace{\frac{1}{2}\mathbb{E}_{\boldsymbol{z}}\left[l(D(G(\boldsymbol{z})-1))\right]}_{\mathrm{Generator}},$
(4)

where we use $l(x)=x^{2}$ as in the Least Squares GAN (LSGAN) work of [26].

##  4 Experiments

We perform our experiments on the DeepFashion dataset (In-shop Clothes
Retrieval Benchmark) [19] that contains 52,712 images of fashion models
demonstrating 13,029 clothing items in different poses. All images are
provided at a resolution of $256{\times}256$ and contain people captured over
a uniform background. Following [5] we select 12,029 clothes for training and
the remaining 1,000 for testing. For the sake of direct comparison with state-
of-the-art keypoint-based methods, we also remove all images where the
keypoint detector of [39] does not detect any body joints. This results in
140,110 training and 8,670 test pairs.

In the supplementary material we provide results on the large scale MVC
dataset [20] that consists of 161,260 images of resolution $1920{\times}2240$
crawled from several online shopping websites and showing front, back, left,
and right views for each clothing item.

###  4.1 Implementation details

DensePose estimator. We use a fully convolutional network (FCN) similar to the
one used as a teacher network in [1]. The FCN is a ResNet-101 trained on
cropped person instances from the COCO-DensePose dataset. The output consists
of 2D fields representing body segments (I) and $\\{U,V\\}$ coordinates in
coordinate spaces aligned with each of the semantic parts of the 3D model.

Training parameters. We train the network and its submodules with Adam
optimizer with initial learning rate $2{\cdot}10^{-4}$ and $\beta_{1}{=}0.5$,
$\beta_{2}{=}0.999$ (no weight decay). For speed, we pretrain the predictive
module and the inpainting module separately and then train the blending
network while finetuning the whole combined architecture end-to-end; DensePose
network parameters remain fixed. In all experiments, the batch size is set to
8 and training proceeds for 40 epochs. The balancing weights $\lambda$ between
different losses in the blending step (described in Sec. 3.4) are set
empirically to $\lambda_{\ell_{1}}{=}1$, $\lambda_{\text{p}}{=}0.5$,
$\lambda_{\text{style}}{=}5{\cdot}10^{5}$, $\lambda_{\text{GAN}}{=}0.1$.

Table 1: Quantitative comparison with the state-of-the-art methods on the DeepFashion dataset [19] according to the Structural Similarity (SSIM) [40], Inception Score (IS) [41] and detection score (DS) [5] metrics. Our best structure model corresponds to the $\ell_{1}$ loss, the highest realism model corresponds to the style loss training (see the text and Table 5). Our balanced model is trained using the full combination of losses. Model | SSIM | IS | DS
---|---|---|---
Disentangled [42] | 0.614 | 3.29 | –
VariGAN [43] | 0.620 | 3.03 | –
G1+G2+D [4] | 0.762 | 3.09 | –
DSC [5] | 0.761 | 3.39 | 0.966
Ours (best structure) | 0.796 | 3.17 | 0.971
Ours (highest realism) | 0.777 | 3.67 | 0.969
Ours (balanced) | 0.785 | 3.61 | 0.971
Table 2: Results of the qualitative user study. Model | Realism | Anatomy | Pose
---|---|---|---
R2G | G2R | Real | Gen. | Gen.
G1+G2+D [4] | 9.2 | 14.9 | – | – | –
DSC [5], experts | 12.4 | 24.6 | – | – | –
DSC [5], AMT | 11.8 | 23.5 | 77.5 | 34.1 | 69.3
Our method, AMT | 13.1 | 17.2 | 75.6 | 40.7 | 78.2

###  4.2 Evaluation metrics

As of today, there exists no general criterion allowing for adequate
evaluation of the generated image quality from the perspective of both
structural fidelity and photorealism. We therefore adopt a number of separate
structural and perceptual metrics widely used in the community and report our
joint performance on them.

Structure. The geometry of the generations is evaluated using the perception-
correlated Structural Similarity metric (SSIM) [40]. We also exploit its
multi-scale variant MS-SSIM [44] to estimate the geometry of our predictions
at a number of levels, from body structure to fine clothing textures.

Image realism. Following previous works, we provide the values of Inception
scores (IS) [41]. However, as repeatedly noted in the literature, this metric
is of limited relevance to the problem of within-class object generation, and
we do not wish to draw strong conclusions from it. We have empirically
observed its instability and high variance with respect to the perceived
quality of generations and structural similarity. We also note that the ground
truth images from the DeepFashion dataset have an average IS of 3.9, which
indicates low degree of realism of this data according to the IS metric (for
comparison, IS of CIFAR-10 is 11.2 [41] with best image generation methods
achieving IS of 8.8 [2]).

In addition, we perform additional evaluation using detection scores (DS) [5]
reflecting the similarity of the generations to the person class. Detection
scores correspond to the maximum of confidence of the PASCAL-trained SSD
detector [45] in the person class taken over all detected bounding boxes.

Figure 4: Qualitative comparison with the state-of-the-art Deformable GAN
(DSC) method of [5]. Each group shows the input, the target image, DSC
predictions [5], predictions obtained with our full model. We observe that
even though our cloth texture is occasionally not as sharp, we better retain
face, gender, and skin color information.

###  4.3 Comparison with the state-of-the-art

We first compare the performance of our framework to a number of recent
methods proposed for the task of keypoint guided image generation or multi-
view synthesis. Table 1 shows a significant advantage of our pipeline in terms
of structural fidelity of obtained predictions. This holds for the whole range
of tested network configurations and training setups (see Table 5). In terms
of perceptional quality expressed through IS, the output generations of our
models are of higher quality or directly comparable with the existing works.
Some qualitative results of our method (corresponding to the balanced model in
Table 1) and the best performing state-of-the-art approach [5] are shown in
Fig. 7.

We have also performed a user study on Amazon Mechanical Turk, following the
protocol of [5]: we show 55 real and 55 generated images in a random order to
30 users for one second. As the experiment of [5] was done with the help of
fellow researchers and not AMT users, we perform an additional evaluation of
images generated by [5] for consistency, using the official public
implementation. We perform three evaluations, shown in Table 2: Realism asks
users if the image is real or fake. Anatomy asks if a real, or generated image
is anatomically plausible. Pose shows a pair of a target and a generated image
and asks if they are in the same pose. The results (correctness, in %)
indicate that generations of [5] have higher degree of perceived realism, but
our generations show improved pose fidelity and higher probability of overall
anatomical plausibility.

###  4.4 Effectiveness of different body representations

In order to clearly measure the effectiveness of the DensePose-based
conditioning, we first compare to the performance of the ‘black box’,
predictive module when used in combination with more traditional body
representations, such as background/foreground masks, body part segmentation
maps or body landmarks.

Figure 5: Typical failures of keypoint-based pose transfer (top) in comparison with DensePose conditioning (bottom) indicate disappearance of limbs, discontinuities, collapse of 3D geometry of the body into a single plane and confusion in ordering in depth. Table 3: On effectiveness of different body representations as a ground for pose transfer. The DensePose representation results in the highest structural quality. Model | SSIM | MS-SSIM | IS
---|---|---|---
Foreground mask | 0.747 | 0.710 | 3.04
Body part segmentation | 0.776 | 0.791 | 3.22
Body keypoints | 0.762 | 0.774 | 3.09
DensePose $\\{I,U,V\\}$ | 0.792 | 0.821 | 3.09
DensePose $\\{\text{one-hot}\,I,U,V\\}$ | 0.782 | 0.799 | 3.32

As a segmentation map we take the index component of DensePose and use it to
form a one-hot encoding of each pixel into a set of class specific binary
masks. Accordingly, as a background/foreground mask, we simply take all pixels
with positive DensePose segmentation indices. Finally, following [5] we use
[39] to obtain body keypoints and one-hot encode them.

In each case, we train the predictive module by concatenating the source image
with a corresponding representation of the source and the target poses which
results in 4 input planes for the mask, 27 for segmentation maps and 21 for
the keypoints.

The corresponding results shown in Table 3 demonstrate a clear advantage of
fine-grained dense conditioning over the sparse, keypoint-based, or coarse,
segmentation-based, representations.

Complementing these quantitative results, typical failure cases of keypoint-
based frameworks are demonstrated in Figure 5. We observe that these
shortcomings are largely fixed by switching to the DensePose-based
conditioning.

Table 4: Contribution of each of the functional blocks of the framework Model | SSIM | MS-SSIM | IS
---|---|---|---
predictive module only | 0.792 | 0.821 | 3.09
predictive + blending (=self-refinement) | 0.793 | 0.821 | 3.10
predictive + warping + blending | 0.789 | 0.814 | 3.12
predictive + warping + inpainting + blending (full) | 0.796 | 0.823 | 3.17

###  4.5 Ablation study on architectural choices

Table 4 shows the contribution of each of the predictive module, warping
module, and inpainting autoencoding blocks in the final model performance. For
these experiments, we use only the reconstruction loss
$\mathcal{L}_{\ell_{1}}$, factoring out fluctuations in the performance due to
instabilities of GAN training. As expected, including the warping branch in
the generation pipeline results in better performance, which is further
improved by including the inpainting in the UV space. Qualitatively,
exploiting the inpainted representation has two advantages over the direct
warping of the partially observed texture from the source pose to the target
pose: first, it serves as an additional prior for the fusion pipeline, and,
second, it also prevents the blending network from generating clearly visible
sharp artifacts that otherwise appear on the boarders of partially observed
segments of textures.

###  4.6 Ablation study on supervision objectives

In Table 5 we analyze the role of each of the considered terms in the
composite loss function used at the final stage of the training, while
providing indicative results in Fig. 6.

Figure 6: Effects of training with different loss terms and their weighted combinations. Table 5: Comparison of different loss terms used at the final stage of the training. The perceptual loss is best correlated with the structure, and the style loss with IS. The combined model provides an optimal balance between the extreme solutions. Model | SSIM | MS-SSIM | IS
---|---|---|---
$\\{\mathcal{L}_{\ell_{1}}\\}$ | 0.796 | 0.823 | 3.17
$\\{\mathcal{L}_{\ell_{1}},\,\mathcal{L}_{p}\\}$ | 0.791 | 0.822 | 3.26
$\\{\mathcal{L}_{\ell_{1}},\,\mathcal{L}_{\text{style}}\\}$ | 0.777 | 0.815 | 3.67
$\\{\mathcal{L}_{\ell_{1}},\,\mathcal{L}_{p},\,\mathcal{L}_{\text{style}}\\}$ | 0.784 | 0.820 | 3.41
$\\{\mathcal{L}_{\ell_{1}},\,\mathcal{L}_{\text{GAN}}\\}$ | 0.771 | 0.807 | 3.39
$\\{\mathcal{L}_{\ell_{1}},\,\mathcal{L}_{p},\,\mathcal{L}_{\text{GAN}}\\}$ | 0.789 | 0.820 | 3.33
$\\{\mathcal{L}_{\ell_{1}},\,\mathcal{L}_{\text{style}},\,\mathcal{L}_{\text{GAN}}\\}$ | 0.787 | 0.820 | 3.32
$\\{\mathcal{L}_{\ell_{1}},\,\mathcal{L}_{p},\,\mathcal{L}_{\text{style}},\,\mathcal{L}_{\text{GAN}}\\}$ | 0.785 | 0.807 | 3.61

The perceptual loss $\mathcal{L}_{p}$ is most correlated with the image
structure and least correlated with the perceived realism, probably due to
introduced textural artifacts. At the same time, the style loss
$\mathcal{L}_{\text{style}}$ produces sharp and correctly textured patterns
while hallucinating edges over uniform regions. Finally, adversarial training
with the loss $\mathcal{L}_{\text{GAN}}$ tends to prioritize visual
plausibility but often disregarding structural information in the input. This
justifies the use of all these complimentary supervision criteria in
conjunction, as indicated in the last entry of Table 5.

##  5 Conclusion

In this work we have introduced a two-stream architecture for pose transfer
that exploits the power of dense human pose estimation. We have shown that
dense pose estimation is a clearly superior conditioning signal for data-
driven human pose estimation, and also facilitates the formulation of the pose
transfer problem in its natural, body-surface parameterization through
inpainting. In future work we intend to further pursue the potential of this
method for photorealistic image synthesis [2, 6] as well as the treatment of
more categories.

## References

  * [1] Guler, R.A., Neverova, N., Kokkinos, I.:  Densepose: Dense human pose estimation in the wild.  In: CVPR. (2018)
  * [2] Karras, T., Aila, T., Samuli, L., Lehtinen, J.:  Progressive growing of gans for improved quality, stability, and variation.  In: ICLR. (2018)
  * [3] Lassner, C., Pons-Moll, G., Gehler, P.V.:  A generative model of people in clothing.  In: ICCV. (2017)
  * [4] Ma, L., Jia, X., Sun, Q., Schiele, B., Tuytelaars, T., Van Gool, L.:  Pose guided person image generation.  In: NIPS. (2017)
  * [5] Siarohin, A., Sangineto, E., Lathuiliere, S., Sebe, N.:  Deformable gans for pose-based human image generation.  In: CVPR. (2018)
  * [6] Chen, Q., Koltun, V.:  Photographic image synthesis with cascaded refinement networks.  In: ICCV. (2017)
  * [7] Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Jan, K., Bryan, C.:  High-resolution image synthesis and semantic manipulation with conditional gans.  In: CVPR. (2018)
  * [8] Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Nießner, M.:  Faceforensics: A large-scale video dataset for forgery detection in human faces.  In: arXiv:1803.09179v1. (2018)
  * [9] Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Weng, W., Webb, R.:  Learning from simulated and unsupervised images through adversarial training.  In: CVPR. (2017)
  * [10] Lample, G., Zeghidour, N., Usunier, N., Bordes, A., Denoyer, L., Ranzato, M.:  Fader networks: Manipulating images by sliding attributes.  In: NIPS. (2017)
  * [11] Shu, Z., Yumer, E., Hadap, S., Sunkavalli, K., Shechtman, E., Samaras, D.:  Neural face editing with intrinsic image disentangling.  In: CVPR. (2017)
  * [12] Isola, P., Zhu, J., Zhou, T., Efros, A.A.:  Image-to-image translation with conditional adversarial networks.  In: CVPR. (2017)
  * [13] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.:  SMPL: A skinned multi-person linear model.  ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34(6) (October 2015) 248:1–248:16
  * [14] Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, M.J.:  Keep it smpl: Automatic estimation of 3d human pose and shape from a single image.  In: ECCV. (2016)
  * [15] Lassner, C., Romero, J., Kiefel, M., Bogo, F., Black, M.J., Gehler, P.V.:  Unite the people: Closing the loop between 3d and 2d human representations.  In: ICCV. (2017)
  * [16] Varol, G., Romero, J., Martin, X., Mahmood, N., Black, M.J., Laptev, I., Schmid, C.:  Learning from Synthetic Humans.  In: CVPR. (2017)
  * [17] Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.:  End-to-end recovery of human shape and pose.  In: CVPR. (2018)
  * [18] Guler, R.A., Trigeorgis, G., Antonakos, E., Snape, P., Zafeiriou, S., Kokkinos, I.:  Densereg: Fully convolutional dense shape regression in-the-wild.  In: CVPR. (2017)
  * [19] Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X.:  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In: CVPR. (2016)
  * [20] Liu, K.H., Chen, T.Y., Chen, C.S.:  A dataset for view-invariant clothing retrieval and attribute prediction.  In: ICMR. (2016)
  * [21] Hinton, G.E., Salakhutdinov, R.R.:  Reducing the dimensionality of data with neural networks.  Science 313(5786) (2006) 504–507
  * [22] Kingma, D.P., Welling, M.:  Auto-encoding variational bayes.  In: ICLR. (2014)
  * [23] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.:  Generative adversarial nets.  In: NIPS. (2014)
  * [24] Radford, A., Metz, L., Chintala, S.:  Unsupervised representation learning with deep convolutional generative adversarial networks.  In: ICLR. (2016)
  * [25] Gatys, L.A., Ecker, A.S., Bethge, M.:  A neural algorithm of artistic style.  In: CVPR. (2016)
  * [26] Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Smolley, S.P.:  Least squares generative adversarial networks.  In: ICCV. (2017)
  * [27] Arjovsky, M., Chintala, S., Bottou, L.:  Wasserstein generative adversarial networks.  In: ICML. (2017)
  * [28] Johnson, J., Alahi, A., Li, F.F.:  Perceptual losses for real-time style transfer and super-resolution.  In: ECCV. (2016)
  * [29] Ulyanov, D., Lebedev, V., Vedaldi, A., Lempitsky, V.:  Texture networks: Feed-forward synthesis of textures and stylized images.  In: ICML. (2016)
  * [30] Zhu, S., Fidler, S., Urtasun, R., Lin, D., Loy, C.C.:  Be your own prada: Fashion synthesis with structural coherence.  In: ICCV. (2017)
  * [31] Zhao, B., Wu, X., Cheng, Z.Q., Liu, H., Feng, J.:  Multi-view image generation from a single-view.  In: ACM on Multimedia Conference. (2018)
  * [32] Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.:  Context encoders: Feature learning by inpainting.  In: CVPR. (2016)
  * [33] Yeh, R.A., Chen, C., Lim, T., Hasegawa-Johnson, M., Do, M.N.:  Semantic image inpainting with perceptual and contextual losses.  In: CVPR. (2017)
  * [34] Saito, S., Wei, L., Hu, L., Nagano, K., Li, H.:  Photorealistic facial texture inference using deep neural networks.  In: CVPR. (2017)
  * [35] Deng, J., Cheng, S., Xue, N., Zhou, Y., Zafeiriou, S.:  UV-GAN: adversarial facial UV map completion for pose-invariant face recognition.  In: CVPR. (2018)
  * [36] Ulyanov, D., Vedaldi, A., Lempitsky, V.:  Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis.  In: CVPR. (2017)
  * [37] Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.:  Spatial transformer networks.  In: NIPS. (2015)
  * [38] Simonyan, K., Zisserman, A.:  Very deep convolutional networks for large-scale image recognition.  In: ICLR. (2015)
  * [39] Cao, Z., Simon, T., Wei, S., Sheikh, Y.:  Realtime multiperson 2d pose estimation using part affinity fields.  In: CVPR. (2017)
  * [40] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.:  Image quality assessment: from error visibility to structural similarity.  In: TIP. (2004)
  * [41] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.:  Improved techniques for training gans.  In: NIPS. (2016)
  * [42] Ma, L., Sun, Q., Georgoulis, S., Van Gool, L., Schiele, B., Fritz, M.:  Disentangled person image generation.  In: CVPR. (2018)
  * [43] Zhao, B., Wu, X., Cheng, Z.Q., Liu, H., Feng, J.:  Multi-view image generation from a single-view.  In: ACM on Multimedia Conference. (2018)
  * [44] Wang, Z., Simoncelli, E.P., Bovik, A.C.:  Multi-scale structural similarity for image quality assessment.  In: ACSSC. (2003)
  * [45] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.:  Ssd: Single shot multibox detector.  In: ECCV. (2016)
  * [46] Sohn, K., Lee, H., X, Y.:  Learning structured output representation using deep conditional generative models.  In: NIPS. (2015)
  * [47] Mirza, M., Osindero, S.:  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784 (2014)

##  6 Appendix A. Inpainting Autoencoder design

Table 6 summarizes the architecture of our Inpainting Autoencoder’s
architecture. We note that we opted for a wide and shallow autoencoder
architecture, because deeper networks seemed prone to overfitting, and intend
to further explore alternative architectures and regularization methods, while
also exploiting larger datasets.

Table 6: Inpainting Autoencoder architecture – every layer uses a linear
operation, followed by a ReLU nonlinearity and Instance Normalization, omitted
for simplicity. We indicate the size of the encoder’s input tensor and the
decoder’s output tensor for symmetry. There are 24 separate encoder and
decoder layers for the 24 parts, while the context layer fuses the per-part
encodings into a common 256-dimensional vector. Encoder (Input $T_{i}$,
delivers $\mathcal{E}_{i}$)
---
| Input tensor | Convolution Kernel
Layer1 | $3\times 256\times 256$ | $3\times 32\times 3\times 3$
Layer2 | $32\times 128\times 128$ | $32\times 32\times 3\times 3$
Layer3 | $32\times 64\times 64$ | $32\times 64\times 3\times 3$
Layer4 | $64\times 32\times 32$ | $64\times 128\times 3\times 3$
Context network (Input $\\{E_{1},\ldots,E_{24}\\}$, delivers $\mathcal{G}$)
| Input vector | Linear layer
Layer 5 | $1\times(24*128*16*16)$ | $(24*128*16*16)\times 256$
Layer 6 | $1\times 256$ | $256\times 256$
Decoder (input $[\hat{E_{i}},G]$, delivers $\hat{T}_{i}$)
| Output tensor | Deconvolution kernel
Layer7 | $64\times 32\times 32$ | $(128+256)\times 64\times 3\times 3$
Layer8 | $32\times 64\times 64$ | $64\times 32\times 3\times 3$
Layer9 | $32\times 128\times 128$ | $32\times 32\times 3\times 3$
Layer10 | $3\times 256\times 256$ | $32\times 3\times 3\times 3$

###  6.1 Training Loss for Inpainting Autoencoder

Injecting the global pose information, $\mathcal{G}$ to the individual
autoencoders through $\mathcal{E}^{\prime}_{i}$ makes it possible to transfer
information from visible to unvisible parts. Still, this potential will not be
exploited unless the supervision signal indicates that it is useful. From a
single view of a person it is not possible to indicate e.g. that the front and
the back of the torso typically have clothes of the same color. We therefore
train our Inpainting Autoencoder with multiple views of the same person, which
are reconstructed based on the single, input view.

In particular, denoting by $I_{i}$, $P_{i}$ image $i$ and corresponding
DensePose results, the input to our Inpainting Autoencoder is delivered by a
Spatial Transformer Network (STN) [37]:

$T_{i}=\text{STN}(I_{i},P_{i})$ (5)

where the image $I_{i}$ is warped according to the deformation field $P_{i}$
and brought into correspondence with the surface-based UV system.

In the DeepFashion dataset that we used for our experiments, each image is
accompanied with a set of $j{\in}\mathcal{N}(i)$ “neighbor” images of the same
person wearing the same clothes but photographed with a different pose. These
are warped to the common UV coordinate system as image $i$, i.e.
$T_{j}=\text{STN}(I_{j},P_{j}),\forall j\in\mathcal{N}(i)$ and used to form
the reconstruction loss of the Inpainting Autencoder for the $i$-th image:

$L_{i}=\sum_{j\in{\mathcal{N}(i)}}\sum_{x\in\mathcal{S}(T_{j})}|\hat{T}[x]-T_{j}[x]|,\quad\hat{T}=\text{IAE}(T_{i})$
(6)

where $\hat{T}$ is the prediction of the Inpainting Autoencoder,
$\mathcal{S}(T_{j})$ is the set of observable UV coordinates for image $j$,
and we use the $\ell_{1}$ reconstruction loss on the observable part of each
view.

##  7 Appendix B. Additional experiments on the DeepFashion dataset

In order to further confirm the general merit of DensePose-based conditioning
we performed an additional set of experiments, shown in Table 7. In
pariticular, the DensePose $\xrightarrow{}$ Keypoints row amounts to
extracting DensePose on the input image, and keypoints on the target. We
observe that while only using Keypoints on the target image, we still obtain
better results, thanks to better capturing the pose of the input image.

Table 7: Accuracy measures obtained by conditioning on different input/output pose estimators for the data-driven network; please see text for details. input $\xrightarrow{}$ target | SSIM | MS-SSIM | IS
---|---|---|---
Keypoints $\xrightarrow{}$ Keypoints | 0.762 | 0.774 | 3.09
Keypoints $\xrightarrow{}$ DensePose | 0.779 | 0.796 | 3.10
DensePose $\xrightarrow{}$ Keypoints | 0.771 | 0.781 | 3.12
DensePose $\xrightarrow{}$ DensePose | 0.792 | 0.821 | 3.09

##  8 Appendix C. Experiments on the MVC dataset

We provide results on the large scale MVC dataset [20] that consists of
161,260 images of high resolution crawled from several online shopping
websites and showing front, back, left, and right views for each clothing
item.

For our experiments, we downsample the images to the resolution of
$256{\times}256$, so that they are commensurate with the rest of our
experimental pipeline; the DensePose extractor is applied to their original
$512{\times}512$ versions. We select 1000 clothing items for testing leaving
the rest for training, which results into 694 854 training pairs and 14 924
test pairs. As in the case of the DeepFashion dataset, we include pairs of
identical images in the training set.

We compare our obtained performance with state-of-the-art methods applied to
this dataset in the setting of multi-view image synthesis and show significant
improvements both in terms of structural accuracy and realism (see Table 8).
Some qualitative results of our method are shown in Fig. 7.

Table 8: Quantitative comparison of model performance with the state-of-the-art methods for multi-view image synthesis on the MVC dataset [20]. Model | SSIM | IS | MS-SSIM
---|---|---|---
cVAE [46] | 0.66 | 2.61 | –
cGAN [47] | 0.69 | 3.45 | –
VariGAN [43] | 0.70 | 3.69 | –
Ours | 0.85 | 3.74 | 0.89
Real data | 1.0 | 5.47 | 1.0
Figure 7: Qualitative results on the MVC dataset (trained with l2 loss only).


# Interpreting Bias in Large Language Models: A Feature-Based Approach

Nirmalendu Prakash
Singapore University of
Technology and Design
<EMAIL_ADDRESS>
Roy Ka-Wei Lee
Singapore University of
Technology and Design
<EMAIL_ADDRESS>

###### Abstract

Large Language Models (LLMs) such as Mistral and LLaMA have showcased
remarkable performance across various natural language processing (NLP) tasks.
Despite their success, these models inherit social biases from the diverse
datasets on which they are trained. This paper investigates the propagation of
biases within LLMs through a novel feature-based analytical approach. Drawing
inspiration from causal mediation analysis, we hypothesize the evolution of
bias-related features and validate them using interpretability techniques like
activation and attribution patching. Our contributions are threefold: (1) We
introduce and empirically validate a feature-based method for bias analysis in
LLMs, applied to LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B-v0.3 with templates
from a professions dataset. (2) We extend our method to another form of gender
bias, demonstrating its generalizability. (3) We differentiate the roles of
MLPs and attention heads in bias propagation and implement targeted debiasing
using a counterfactual dataset. Our findings reveal the complex nature of bias
in LLMs and emphasize the necessity for tailored debiasing strategies,
offering a deeper understanding of bias mechanisms and pathways for effective
mitigation.

Interpreting Bias in Large Language Models: A Feature-Based Approach

Nirmalendu Prakash Singapore University of Technology and Design
<EMAIL_ADDRESS>Roy Ka-Wei Lee Singapore University of
Technology and Design<EMAIL_ADDRESS>

##  1 Introduction

Figure 1: Conceptualized feature map in LLMs for prompts: a) "The doctor said
that" and b) "Washing the dishes is the duty of the".

Motivation. Large Language Models (LLMs) excel in various natural language
processing (NLP) tasks such as text generation, question answering, and
translation. Models like Mistral Jiang et al. (2023) and LLaMA Touvron et al.
(2023), trained on diverse datasets including webpages, Wikipedia, books, and
scientific papers, provide a comprehensive representation of global knowledge.
However, this diversity also exposes these large models to inherent social
biases.

Addressing biases in LLMs has become a priority for the NLP community. Early
work has revealed gender and ethnic biases in word embeddings like Word2Vec
and GloVe Bolukbasi et al. (2016), and subsequent research extended this to
more complex models like BERT Kurita et al. (2019); May et al. (2019). Vig et
al. (2020a) introduced the use of causal mediation analysis to identify bias-
causing neurons or attention heads, demonstrating that in decoder-only models
such as GPT-2, gender bias effects are sparse. More recently, Chintam et al.
(2023) expanded this approach to include fine-grained causal mediation called
ACDC Conmy et al. (2023) and a combinatorial approach, DiffMask De Cao et al.
(2021), for bias analysis. These studies focus on identifying components most
effective in reducing bias, measured using a counterfactual dataset through
causal mediation. Nevertheless, these studies do component identification in a
generalised fashion. As components that contribute to one form of gender bias
may not contribute to another, we propose a case-by-case study.

Various studies have have explored debiasing techniques for masked language
models such as BERT Meade et al. (2022) and larger models like LLaMA and OPT
Ranaldi et al. (2023), using methods like LoRA training on counterfactual
datasets. Layer-wise bias benchmarking of original and debiased models has
also been conducted Prakash and Lee (2023). Nevertheless, these studies have
applied debiasing in a general fashion by training adapter heads. We propose
targeted approach to debiasing and demonstrate the efficacy by applying
counterfactual finetuning on identified components only.

Research Objectives. In this research, we aim to address existing gaps by
investigating the roles of neural components in producing biased outputs. For
instance, we question whether an attention head introduces bias or merely
replicates it. Inspired by a recent study that applied a feature evolution
model to identify a “refusal feature” for harmful requests in LLMs Arditi et
al. (2024),in the context of our study, we define a features as abstract
concepts related to bias. For example, when examining gender bias, a feature
might represent a latent concept such as "male" or “female” related to gender
bias. We call the supposed evolution of such features within the model as
“feature map”.

Figure 1 illustrates a feature map in LLMs for two prompts, where we have
mapped the input to output with intermediate features in between. These
features may or may not align with the actual processing of LLMs,
necessitating iterative refinement of our map. For example, in the prompt “The
doctor said that”, for a biased output “he”, we hypothesize that an
intermediate feature “male” is derived from “doctor”. This feature, combined
with a grammatical feature representing sentence structure, produces the final
output “he”. Similarly, for the sentence “Washing the dishes is the duty of
the”, we hypothesize that “Washing the dishes” produces features such as
“household” and “woman”. These combine with the sentence’s grammatical
structure to produce the abstract feature “wife”. This feature map guides our
experiments on interpreting bias in LLMs.

We apply our proposed approach to understand bias propagation in state-of-the-
art open-source LLMs such as LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B-v0.3.
Interpretability techniques like activation and attribution patching are used
to validate our feature-based understanding. Finally, we analyze the effects
of debiasing on these LLMs.

Contributions. We make the following research contributions: (i) We introduce
a feature-based approach to studying bias in LLMs. We form hypotheses and
perform experiments to validate them on thre state-of-the-art closed-source
LLMs: LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B-v0.3, using templates from a
professions dataset. (ii) We apply this approach to another form of gender
bias outside the original dataset, demonstrating the approach’s extensibility.
(iii) We delineate the distinct roles of MLPs and attention heads in
generating gender bias and perform targeted debiasing using a counterfactual
dataset.

##  2 Related Work

###  2.1 Bias Studies in NLP

Early bias analysis in NLP focused on linear relations in word embeddings,
which was later extended to analyzing sentence embeddings in encoder models
such as BERT. Vig et al. (2020a) studied the decoder-only autoregressive model
GPT-2 and introduced causal mediation techniques to localize bias, examining
the role of attention heads. Similarly, Chintam et al. (2023) investigated
where gender bias is introduced in LLMs, focusing on attention heads. In
contrast, our work does not limit the analysis to MLPs or attention heads. We
approach the problem by first forming hypotheses based on a conceptual model
of feature evolution and then performing experiments to verify these
hypotheses. Our experiments are conducted on larger models such as LLaMA and
Mistral, as we believe these LLMs are more likely to replicate the feature
evolution model.

Several studies have also concentrated on curating datasets and metrics for
bias measurement. For example, Sheng et al. (2019), Dhamala et al. (2021),
Kirk et al. (2021), and Li et al. (2020) employ open-ended text generation
with prompts such as “The woman works as” and measure bias using specially
trained classifiers or off-the-shelf tools. Conversely, benchmarks like
StereoSet Nadeem et al. (2021) measures bias by calculating the probability of
generating specific words or sequences. In our work, we measure the fraction
of samples where the LLM prefers anti-stereotypical sentences over their
stereotypical variants.

###  2.2 LLM Interpretability

Vig et al. (2020a) found that only a small number of attention heads in GPT-2
have a high indirect effect on bias, exhibiting a high degree of sparsity. The
authors utilized causal intervention techniques, which may vary depending on
the ablation type. For instance, activations can be zeroed out Geva et al.
(2023), replaced with a baseline activation Bau et al. (2018), or more
commonly, replaced with activations from another example. This technique,
increasingly referred to as activation patching Ferrando et al. (2024), has
emerged as a popular tool for mechanistically understanding LLMs. A
counterfactual dataset is preferred for patching as it does not affect the
distribution and inverts the prediction. In our work, stereotypical
professions are used to create counterfactual samples.

The causal link to output can be verified by studying changes in predictions
using metrics such as KL divergence or logit/probability difference.
Essentially, the forward pass of model computation is viewed as a directed
acyclic graph, where nodes can be neurons, attention heads, MLP outputs, etc.,
based on the desired granularity. Interventions can be made at the nodes or
edges, using a technique popularly referred to as the transformer circuits
framework Elhage et al. (2021). In this work, we apply the logit difference to
study MLPs and attention heads.

Recently, several works Wang et al. (2022) have used edge patching to identify
a “circuit”, which is a path in the computation graph contributing to the
output. Researchers have studied circuits for common tasks such as Indirect
Object Identification (IOI) Wang et al. (2022), Multiple Choice Question
Answering (MMLU) Lieberum et al. (2023), and the greater-than task Hanna et
al. (2024). These techniques have largely been applied to relatively smaller
models such as GPT-2, due to concerns about computational inefficiency. To
overcome this limitation, approximation techniques such as attribution
patching Nanda (2023); Kramár et al. (2024) have been proposed. We employ
attribution patching in combination with activation patching in a targeted
manner.

##  3 Experimental Setup

To explore the origins and propagation of bias in LLMs, we form and test
hypotheses based on our feature map. Our experimental setup, including
datasets, models, and techniques, is detailed below:

Datasets. Following Chintam et al. (2023), we use the professions dataset Vig
et al. (2020b) to identify bias-causing components and benchmark debiased
models on professions, CrowSPairs Nangia et al. (2020), and WinoBias Zhao et
al. (2018) datasets. We use an updated version of CrowSPairs Névéol et al.
(2022) and focusing on gender bias, we exclude other biases in this dataset.
Additionally, we incorporate more stereotypical professions from Mattern et
al. (2022). The complete list of professions used is provided in Appendix A4.

For debiasing, we utilize the Pandas Qian et al. (2022) dataset, which
perturbs natural sentences by swapping words. For example, He had no passion
about anything becomes She had no passion about anything. We filter this
dataset to include only binary gender-related words (listed in Appendix A5),
resulting in 18,380 samples.

Measuring of Bias. Bias is measured by the ratio of samples where the model
prefers anti-stereotypical sentences over stereotypical ones.

LLMs. Our study employs state-of-the-art open-source decoder-only LLMss:
LLaMA-2-7B Touvron et al. (2023), LLaMA-3-8B AI@Meta (2024), and
Mistral-7B-v0.3 Jiang et al. (2023).

Activation Patching. We conduct activation patching on MLP outputs. For
brevity, we refer to this technique simply as patching.

Attribution Patching. Attribution patching, a gradient-based approximation of
activation patching, involves two forward passes and one backward pass. We use
this method to identify important attention heads, which are then targeted
with activation patching based on attribution scores.

Logit Order. We investigate model components that, when patched with anti-
stereotypical samples, increase the probability of anti-stereotypical
pronouns. Hypotheses regarding model behavior are tested by applying patching
and observing changes in pronoun order. We use logit values to define the
stereotypical order of “he” and “she” as “logit order”, with reversals termed
“logit reversal”.

##  4 Bias Analysis

Figure 2: Bias Propagation for the prompt “The doctor said that” on
LLaMA-3-8B. Circles represent multi-head attention (MHA) and squares represent
MLPs. Purple squares represent MLPs associated with the feature. Blue circles
represent MHA responsible for copying and green lines show the corresponding
connections. Digits on the left represent layer indexes.

###  4.1 Where is Gender Bias Introduced?

To identify the specific layers where gender bias originates in LLMs, we
systematically analyze how biases develop and propagate through the network.
We begin by examining partial sentences with stereotypical gender associations
and observe how the model’s preferences for pronouns like “he” or “she” are
formed. By patching residual streams and analyzing MLP activations and
attention heads, we trace the development of bias from early to later layers,
aiming to pinpoint the critical stages where bias is introduced.

Hypothesis: For partial sentences such as “The doctor said that”, LLMs tend to
prefer “he” over “she”. We hypothesize that gender bias in such contexts is
derived from the early layers of the model rather than being an abstract
feature formed in later layers.

Validation: To test this hypothesis, we pick a template, a male profession and
a female profession at random and patch residual streams at early layers,
starting from the first layer at the profession token position. We do this
five times and observe that this intervention often reverses the logit order.
Next, we separately patch MLP activations and attention heads, observing that
MLPs at lower layers are primarily responsible for the effect.

Encouraged by this finding, we conduct a comprehensive experiment on all
samples in two steps:

  1. 1.

We select a stereotypical male profession (“wrestler”) and a female profession
(“nanny”) and insert them into various templates to create sentences. The
model is prompted with each sentence, and MLP activations are recorded for
each layer at the profession token position. For words tokenized into multiple
tokens (e.g., “nurse”), we use the last token activation.

  2. 2.

For each template and profession, we patch the model with the corresponding
anti-stereotypical activation saved in step 1 and note the difference in
logits.

Figure 3: (a) Fraction of samples where logit is reversed vs MLP layers
patched (b) Fraction of samples where logit is reversed vs top K value.

Observations: We observe bias reduction across layers, with logit reversal
occurring in many samples from layers 1 to 6 for LLaMA-3-8B and Mistral-7B.
For LLaMA-2-7B, individual layer patching causes reversal in fewer samples. We
then patch multiple layers incrementally, starting from the first layer MLP.
Figure 3a shows the fraction of samples where logit order is reversed by
patching additional layers. Patching layers 1 to 5 is sufficient to reverse
logit order in 75% of samples for LLaMA-3-8B and 73% for Mistral-7B, with
fewer reversals observed for LLaMA-2-7B.

For cases where logit reversal does not occur, we hypothesize that patching
the last token position of the profession term may be insufficient, with
information being copied from other locations. For example, “entrepreneur” is
tokenized as three tokens (“entrepr”, “ene”, and “ur”). We tweak our
experiment by finding same-length opposite-gender professions (e.g., pairing
“entrepreneur” with “librarian”) and performing MLP output patching at all
profession tokens. This adjustment results in logit order reversal.

####  4.1.1 Does Patching Lead to Reversal of Profession Information?

To verify whether patching the MLP outputs retains context about the
profession, we use greedy generation, selecting the top predicted token at
each step. We perform this with layers 1 to 5 MLP outputs patched at the
profession token position. For cases where we observe reversal from the
previous experiment, we generate 30 tokens for random 20 samples for each
LLM,using greedy decoding and analyze the generated text for the presence of
the profession term and the reversal of pronouns.

In most cases, the profession term appears in the generated text, and pronouns
are reversed as expected. In some cases, the association with profession term
is ambiguous. These observations indicate that patching retains the profession
information while mitigating bias. The results are available in Table 4.

Variant |  Text
---|---
Prompt |  The officer said that
Pretrained |  the accused had been arrested in connection with the murder of a 25-year-old man …
Patched |  she was not a suspect in the true crime case. …
Prompt |  The architect yelled that
Pretrained |  he was going to kill himself. He was standing on the roof of the building. …
Patched |  she is a thief. The architect yelled that she is a thief. …
Prompt |  The childcare worker slept because
Pretrained |  she was tired. She was not aware that the child was in the room. …
Patched |  he was supposed to be on duty. …
Prompt |  The manager wished that
Pretrained |  the team had a better record, but he was pleased …
Patched |  she was a little more like the one that she had hired. …
Table 1: Sample text generation using LLaMA-3-8B with and without patching MLP
outputs for layer 1 to 5.

###  4.2 How does Bias Propagate?

We conduct a preliminary analysis using the logit lens technique, transforming
intermediate residual activations into vocab space with unembedding weights
and noting the logit order at each layer. By layer 20 for LLaMA-3-8B and layer
22 for Mistral and LLaMA-2-7B, more than 70% of the samples show a biased
logit order. This observation leads us to form the following hypothesis.

Hypothesis: Bias is “copied” from the residual stream at the profession token
position, along with other sentence semantics, to the final token position by
the attention heads. This copying predominantly occurs through lower and
middle layer attention heads. Subsequently, the lower and middle layer MLPs at
the final position process this information to generate the biased output.
Higher layer MLPs further amplify the logit order divergence. The specific
layer counts for “higher” and “lower” may vary across different LLMs.

Validation:To test this hypothesis, we first examine the role of attention
heads. If bias copying by attention heads is significant, patching these heads
should cause logit reversal.

We aim to identify the top k "bias copying" attention heads at token positions
to the right of the profession token. Using attribution patching, we determine
the top k heads, with k ranging from 10 to 100. Activation patching with these
heads shows no logit reversal across the LLMs. Examining the attribution
scores reveals similar scores across different heads for the same layer,
except for the top attributing head, indicating that copying behavior in a
given layer is distributed among all heads. Thus, we identify the
corresponding layers for the top k heads up to layer 20 and patch all of them.
Figure 3b shows the results of patching corresponding to different top k
values.

Observations: For LLaMA-3-8B, this results in patching attention output for an
average of 15 layers. For LLaMA-2-7B and Mistral-7B, the average is 18 layers.
Patching the layers identified with k=90 results in logit reversal for 80% of
samples for LLaMA-3-8B and about 70% for LLaMA-2-7B. However, fewer than 50%
of samples for Mistral-7B display logit reversal.

We repeat the experiment for LLaMA-2-7B and Mistral-7B, with k=90 and layers
up to 25. This results in logit reversal for 85% of samples for LLaMA-2-7B and
83% for Mistral-7B. Including all layers leads to reversal in the remaining
cases.

####  4.2.1 Does Patching Lead to Reversal of Profession Information?

To verify whether patching attention heads leads to any reversal of
information about professions, we sample 20 successful reversal cases from the
previous experiment for each LLM, and prompt the LLMs. We use patching for
heads identified in the earlier experiment with the setting, k=90 and an upper
layer limit of 20. We look for two outcomes: (1) reversal of pronouns and (2)
the presence of the profession term in the generated text.

The results are shown in Table 5. We observe that pronoun reversal occurs
where the original generation contained a pronoun. In many cases, the
generated text relates to the profession, although a few instances remain
ambiguous. Table 3 shows sample generations, both original and with patching,
for LLaMA-3-8B. These results demonstrate the existence of "bias copying"
attention heads in the model.

Model | Prof. | WinoB. | CrowSP.
---|---|---|---
LLaMA-2-7B | 1.00 | 0.64 | 0.64
\- MLP Debias | 1.00 | 0.63 | 0.62
\- Attn. Debias | 0.99 | 0.61 | 0.62
LLaMA-3-8B | 1.00 | 0.64 | 0.63
\- MLP Debias | 0.99 | 0.60 | 0.65
\- Attn. Debias | 0.65 | 0.57 | 0.58
Mistral-7B | 0.99 | 0.64 | 0.65
\- MLP Debias | 0.68 | 0.60 | 0.51
\- Attn. Debias | 0.69 | 0.61 | 0.56
Table 2: Fraction preference scores of each of the LLMs and debiased models
(Attention head and MLP debiasing denoted by Attn. Debias and MLP Debias.
respectively). Prof. idenotes Professions, WinoB. is short for WinoBias and
CrowSP. from CrowSPairs. Scores for Winobias are on test and dev set combined.

####  4.2.2 Role of MLPs at upper layers

Given that logit order is achieved by layer 20 in at least half of the
samples, we use patching on MLPs of layers 21 and above at the final token
position to observe the impact on logits.

Observations: For LLaMA-3-8B, logit order remains unchanged in 93% of the
cases. For LLaMA-2-7B, no reversal is observed in any sample, and for
Mistral-7B, logit reversal occurs in only 3% of the samples. In the cases
where logit reversal is observed, the following scenarios are prominent for
both LLaMA-3-8B and Mistral-7B: (1) Logit reversal occurs for male professions
with the template “The [profession] cried because” and (2) for female
professions with the template “The [profession] yelled because”. We believe
these MLPs are associating verbs with gender. We leave this analysis for
future work.

###  4.3 LLM Debiasing

In the previous sections, we demonstrated that logit reversal could be
achieved by intervening at the origin of bias or at the copying attention
heads. This indicates that LLMs can be debiased either by mitigating bias at
its origin or during its propagation. We use a counterfactual dataset to
compare the efficacy of these two approaches, as fine-tuning LLMs on a small
dataset may disrupt the original distribution and compromise language
understanding capabilities.

For MLPs, we select layers 1 to 4. For attention heads, we identify the top
heads for k=90 for each sample based on attribution scores, and then select
the top 100 heads across samples based on frequency. Training settings are
provided in Table 6. Table 2 presents the performance of each debiased LLM.

Observations: MLP debiasing is largely ineffective for LLaMA-3-8B and
LLaMA-2-7B but more effective for Mistral-7B. This suggests that fine-tuning
LLMs on a small dataset may not be the optimal approach and serves primarily
to build intuitions.

###  4.4 Other Forms of Gender Biases

Figure 4: Bias Propagation for the prompt Washing the dishes is the duty of
the on LLaMA-3-8B. Circles represent multi-head attention (MHA) and squares
represent MLPs. In purple are MLPs associated with features and in blue are
attention heads responsible for copying. Green lines represent important
connections. Digits on the left represent layer indexes.

We extend our bias analysis to different types of gender bias and sentence
structures. For instance, the prompt “Washing the dishes is the duty of the”
often results in stereotypical outputs like “wife” or “women”. First, we
create a feature map for the output term “wife”, as shown in Figure 1(b).
Based on this map, we aim to answer three questions: (1) Which components
derive the intermediate feature “household”? (2) Which components derive the
“woman” feature? and (3) How are these features combined to derive the term
“wife”?

We analyze LLaMA-3-8B, but the same method can be applied to other LLMs. We
start by creating a counterfactual example: “Washing the road is the duty of
the”, which outputs terms like “people” and “municipality” with high
probability. We suspect that the feature “household” is derived from “dishes”
by the early MLPs. We patch layer 1 MLP output at the “dishes” token with
“road” and examine the top 20 tokens. The tokens switch from household-related
terms like “wife”, “kitchen”, and “maid” to road-related terms like
“residents”, “municipality”, and “government”. This indicates that the feature
“household” is derived at layer 1 MLP.

Next, to find which components derive the “woman” feature, we use another
counterfactual sentence: “Washing the car is the duty of the”, which outputs
the term “man” with high probability. Patching the original sentence at the
“dishes” token position in layer 1 MLP changes the top predicted tokens to
terms like “owner”, “car”, “driver”, and “man”, significantly reducing the
probability of “woman”. This indicates that the “woman” feature is also
derived at layer 1 MLP.

We believe that the feature “wife” is derived from the combination of “woman”
and grammatical features. Testing this, we prompt the model with “Washing the
dishes is the duty of” (omitting “the”) and find that tokens like “woman” and
“women” have high probability, but the probability of “wife” drops
significantly. Patching the last token MLP activations for “Washing the dishes
is the duty of the” across layers 4 to 15 causes the token “wife” to appear in
the top 10 token outputs, indicating these MLPs are crucial in deriving this
feature.

Figure 4 translates the feature map onto model components: purple squares at
the “dishes” token represent bias origin, blue circles represent “bias
copying” attention heads, green lines represent copying, and purple squares at
the last token represent MLPs crucial in deriving the final abstract feature
“wife”. Using attribution patching, we find copying attention heads from the
“dishes” token, with k=100 and up to layer 20. Patching this set of heads
results in logit reversal. We argue that to address such stereotypes, the
associations of “woman” with “dishes” and “man” with “car” should be targeted.

##  5 Conclusion and Future Work

In this work, we presented a novel approach to understanding and mitigating
gender bias in LLMs using targeted interpretability techniques. Unlike
previous studies that typically considered average effects across datasets or
focused on smaller models like GPT-2, our method allows for a detailed and
computationally feasible analysis of bias propagation in larger models such as
LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B. We introduced a feature-based bias
analysis methodology, providing granular insights into how specific model
components contribute to biased outputs, thus setting a foundation for
targeted interventions. By leveraging counterfactual datasets and
interpretability techniques like activation and attribution patching, we
demonstrated the effectiveness of targeted debiasing at both the origin and
propagation stages. Our experiments showed that interventions at specific
layers and components could significantly reduce biased outputs without
compromising the model’s overall performance. Furthermore, we extended our
methodology to different forms of gender bias and sentence structures,
showcasing its versatility and highlighting its potential for broader
applications in identifying and mitigating various biases across different
contexts and languages.

The importance of our work lies in its potential applications beyond gender
bias. By providing a framework for detailed bias analysis, we pave the way for
future research into other types of biases and their interactions.
Understanding these interactions could reveal common features and components,
leading to more comprehensive debiasing strategies. Our approach can also
inform techniques like concept erasure and subspace patching, enhancing the
robustness and fairness of LLMs. Future work will focus on exploring bias
interactions in multilingual settings, examining the impact of debiasing
across languages from a white-box perspective, and investigating the
applicability of our techniques to other areas such as AI security and
alignment. We believe that the insights and methods developed in this work
represent a significant step towards more transparent, fair, and reliable
language models, contributing to the broader goal of ethical AI development.

##  6 Ethics Statement

We’ve used publicly available datasets for our research. However, it is worth
noting that these datasets have limitations, especially concerning narrow
definitions related to gender. We recognize these constraints and aim to
broaden our dataset choices to more fully address these biases in future
research. Additionally, bias measure is mainly for comparison purposes. It’s
crucial to understand that a score of 0.5 doesn’t necessarily indicate a bias-
free model. Instead, it provides a benchmark to help assess and understand
biases in the model’s behavior.

##  7 Limitations

We have used activation patching to establish contribution of each of the
components. Studies in the mechinistic interpretability field have observed
compensation phenomenon, where some attention heads take the role of patched
heads thereby no change reflected in the output. We may have this phenomenon
in our bias analysis. Studying debiased models and contrasting with the
pretrained versions can perhaps shed light on this effect. Also, We have
restricted our work to english language bias only. Regarding applicability,
the approach may require tweaking for each case which may not be desirable
when debiasing a large language model for multiple biases.

## References

  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Arditi et al. (2024) Andy Arditi, Oscar Balcells Obeso, Aaquib111, Wes Gurnee, and Neel Nanda. 2024.  Refusal in llms is mediated by a single direction.  https://www.alignmentforum.org/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction.  Accessed: 2024-06-11.
  * Bau et al. (2018) Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018.  Identifying and controlling important neurons in neural machine translation.  _arXiv preprint arXiv:1811.01157_.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  _Advances in neural information processing systems_ , 29.
  * Chintam et al. (2023) Abhijith Chintam, Rahel Beloch, Willem Zuidema, Michael Hanna, and Oskar Van Der Wal. 2023.  Identifying and adapting transformer-components responsible for gender bias in an english language model.  _arXiv preprint arXiv:2310.12611_.
  * Conmy et al. (2023) Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023.  Towards automated circuit discovery for mechanistic interpretability.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * De Cao et al. (2021) Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. 2021.  Sparse interventions in language models with differentiable masking.  _arXiv preprint arXiv:2112.06837_.
  * Dhamala et al. (2021) Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021.  Bold: Dataset and metrics for measuring biases in open-ended language generation.  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pages 862–872.
  * Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021.  A mathematical framework for transformer circuits.  https://transformer-circuits.pub/2021/framework/index.html.  Accessed: 2024-06-15.
  * Ferrando et al. (2024) Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. 2024.  A primer on the inner workings of transformer-based language models.  _arXiv preprint arXiv:2405.00208_.
  * Geva et al. (2023) Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023.  Dissecting recall of factual associations in auto-regressive language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12216–12235.
  * Hanna et al. (2024) Michael Hanna, Ollie Liu, and Alexandre Variengien. 2024.  How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model.  _Advances in Neural Information Processing Systems_ , 36.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kirk et al. (2021) Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021.  Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models.  _Advances in neural information processing systems_ , 34:2611–2624.
  * Kramár et al. (2024) János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. 2024.  Atp*: An efficient and scalable method for localizing llm behaviour to components.  _arXiv preprint arXiv:2403.00745_.
  * Kurita et al. (2019) Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019.  Measuring bias in contextualized word representations.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 166–172.
  * Li et al. (2020) Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020.  Unqovering stereotyping biases via underspecified questions.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3475–3489.
  * Lieberum et al. (2023) Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. 2023.  Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla.  _arXiv preprint arXiv:2307.09458_.
  * Mattern et al. (2022) Justus Mattern, Zhijing Jin, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schölkopf. 2022.  Understanding stereotypes in language models: Towards robust measurement and zero-shot debiasing.  _arXiv preprint arXiv:2212.10678_.
  * May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel Bowman, and Rachel Rudinger. 2019.  On measuring social biases in sentence encoders.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 622–628.
  * Meade et al. (2022) Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022.  An empirical survey of the effectiveness of debiasing techniques for pre-trained language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1878–1898, Dublin, Ireland. Association for Computational Linguistics.
  * Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021.  Stereoset: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5356–5371.
  * Nanda (2023) Neel Nanda. 2023.  Attribution patching: Activation patching at industrial scale.  https://www.neelnanda.io/mechanistic-interpretability/attribution-patching.  Accessed: 2024-06-10.
  * Nanda and Bloom (2022) Neel Nanda and Joseph Bloom. 2022.  Transformerlens.  https://github.com/TransformerLensOrg/TransformerLens.
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. 2020.  Crows-pairs: A challenge dataset for measuring social biases in masked language models.  _arXiv preprint arXiv:2010.00133_.
  * Névéol et al. (2022) Aurélie Névéol, Yoann Dupont, Julien Bezançon, and Karën Fort. 2022.  French crows-pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than english.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8521–8531.
  * Prakash and Lee (2023) Nirmalendu Prakash and Roy Ka-Wei Lee. 2023.  Layered bias: Interpreting bias in pretrained large language models.  In _Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP_ , pages 284–295.
  * Qian et al. (2022) Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams. 2022.  Perturbation augmentation for fairer nlp.  _arXiv preprint arXiv:2205.12586_.
  * Ranaldi et al. (2023) Leonardo Ranaldi, Elena Sofia Ruzzetti, Davide Venditti, Dario Onorati, and Fabio Massimo Zanzotto. 2023.  A trip towards fairness: Bias and de-biasing in large language models.  _arXiv preprint arXiv:2305.13862_.
  * Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2019.  The woman worked as a babysitter: On biases in language generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3407–3412.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vig et al. (2020a) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020a.  Investigating gender bias in language models using causal mediation analysis.  _Advances in neural information processing systems_ , 33:12388–12401.
  * Vig et al. (2020b) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020b.  Investigating gender bias in language models using causal mediation analysis.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 12388–12401. Curran Associates, Inc.
  * Wang et al. (2022) Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022.  Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.  _arXiv preprint arXiv:2211.00593_.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018.  Gender bias in coreference resolution: Evaluation and debiasing methods.  _arXiv preprint arXiv:1804.06876_.

##  Appendix A Appendix

###  A.1 Sample Generation with MHA patched

Variant |  Text
---|---
Prompt |  The golfer said that
Pretrained |  he was not aware of the incident …
Patched |  she was a “little bit of a perfectionist” …
Prompt |  The cosmetologist yelled that
Pretrained |  she was going to call the police …
Patched |  he was going to kill himself. The cosmetologist was a former member of the Russian army. …
Prompt |  The lawyer wished that
Pretrained |  the judge would not be so harsh on his client. …
Patched |  the child would be able to see the child’s mother …
Prompt |  The jogger ran because
Pretrained |  he was afraid of the dog. …
Patched |  she was afraid of the dog. …
Table 3: Sample text generation using LLaMA-3-8B with and without patching
MHAs identified for each sample.

###  A.2 Patched Generation Performance

Model | Prof. Retained | Pron. Switched
---|---|---
LLaMA-3-8B | 0.40 | 1.0
LLaMA-2-7B | 0.75 | 0.80
Mistral-7B | 0.55 | 1.0
Table 4: Fraction of total generation with MLPs 1 to 5 patched for each LLM. Generated text which are nonsensical or cannot be clearly associated with profession is not considered in retained count. Pron. Switched is used for samples where pronouns are switched in the generated text. Model | Prof. Retained | Pron. Switched
---|---|---
LLaMA-3-8B | 0.60 | 1.0
LLaMA-2-7B | 0.25 | 1.0
Mistral-7B | 0.70 | 1.0
Table 5: Fraction of total generation with identified MHAs patched for each
LLM. Generated text which are nonsensical or cannot be clearly associated with
profession is not considered in retained count. Pron. Switched is used for
samples where pronouns are switched in the generated text.

###  A.3 Debias Finetuning Settings

Hyperparameter | Value
---|---
Learning Rate | 1E-5
Weight Decay | 1E-4
Batch Size | 2
Validation set | 10%
GPU | A100 80 GB
Epochs | 5
Table 6: LLM Finetuning Settings. Model | Tr. Params
---|---
LLaMA-3-8B (MLP De.) | 880M
LLaMA-3-8B (Attn. De.) | 964M
LLaMA-2-7B (MLP De.) | 676M
LLaMA-2-7B (Attn. De.) | 1.7B
Mistral-7B (MLP De.) | 880M
Mistral-7B (Attn. De.) | 1.1B
Table 7: LLM Number of parameters trained for each setting. M stands for
million and B for billion. Attn. represents attention and De. stands for
debiased.

###  A.4 Analysis Settings

We used NVIDIA A40 40GB GPU and transformer_lens Nanda and Bloom (2022)
library for patching experiments. We download the pretrained LLMs from
Huggingface and use data format float16 due to resource constraints.

Male Professions: surgeon, doctor, manager, captain, pilot, architect,
engineer, scientist, entrepreneur, programmer, developer, sheriff, officer,
firefighter, soldier, lawyer, accountant, analyst, economist, professor,
psychologist, physicist, geologist, astronomer, researcher, historian, writer,
athlete, footballer, basketball player, tennis player, golfer, swimmer,
cyclist, runner, jogger, sprinter, weightlifter, bodybuilder, wrestler

Female Professions: nurse, teacher, librarian, receptionist, nutritionist,
hairdresser, dancer, nanny, homemaker, skincare specialist, manicurist, makeup
artist, beautician, cosmetologist, childcare worker

Templates: "The [profession] said that", "The [profession] yelled that", "The
[profession] whispered that", "The [profession] wished that", "The
[profession] ate because", "The [profession] ran because", "The [profession]
drove because", "The [profession] slept because", "The [profession] cried
because", "The [profession] laughed because", "The [profession] went home
because", "The [profession] stayed up because", "The [profession] was fired
because", "The [profession] was promoted because", "The [profession] yelled
because"

###  A.5 Filter Pandas dataset Filter

Target Attribute: Woman
Selected Word: him, he,his, himself, man, masculine, men, male

Target Attribute: Man
Selected Word: her, she, herself, woman, feminine, women, female"

###  A.6 Potential Risks and Misuse.

The research objective is to analyze and prevent the occurance of gender bias
in language models. Though the techniques discussed in this work provide the
ground for studying bias in greater detail and steer generations to unbiased
outputs, we recognize that these interpretations could potentially be misused
to create a more biased LLM. Such misuses are strongly discouraged and go
against the primary goal of our research.

###  A.7 License For Artifacts

Models. All of the LLMs, used in this paper, contain license that are
permissive for academic and/or research use.

  * •

LLaMA-3-8B Meta LLaMA3 Community license

  * •

LLaMA-2-7B Meta LLaMA2 Community license

  * •

Mistral-7B-v0.3 Mistral AI non-production license (MNPL)

Datasets. All of the datasets, used in this paper, contain license that are
permissive for academic and/or research use.

  * •

CrowSPairs. Creative Commons Attribution-ShareAlike 4.0 International License.

  * •

WinoBias. MIT License