# Solutions of a slowly rotating Kerr flat-horizon black hole in dynamical
Chern-Simons modified gravity

G.G.L. Nashed<EMAIL_ADDRESS>Centre for Theoretical Physics, The British
University, P.O. Box 43, El Sherouk City, Cairo 11837, Egypt  Kazuharu Bamba
<EMAIL_ADDRESS>Faculty of Symbiotic Systems Science, Fukushima
University, Fukushima 960-1296, Japan

(August 28, 2024)

###### Abstract

Solutions pertaining to a Kerr black hole with a flat horizon undergoing
gradual rotation are explored in the context of gravitational theories
modified by dynamical Chern-Simons terms with cylindrical metrics, which
approach asymptotically the anti de Sitter spacetime. It is shown that the
cross-term of a metric component is unaffected by the perturbations of the
Chern-Simons scalar independently of whether the dynamical Chern-Simons field
equation is uncharged or charged with an electric field. From this result, it
is ensured that the Chern-Simons scalar field can affect the spaces of the
metric that approach asymptotically the flat spacetime only.

##  I Introduction

Modified gravity theories (as well as dark energy models) have been studied to
account for the late-time cosmic acceleration (including inflation in the
early universe) and solve cosmological and astrophysical issues which cannot
be explained in the context of General Relativity (GR) padmanabhan2007 ;
Durrer2008 ; Sotiriou:2008rp ; DeFelice2010 ; Nojiri:2010wj ; Capozziello2011
; Clifton:2011jh ; Joyce:2014kja ; Cai:2015emx ; Nojiri:2017ncd .

When we consider the physics of black holes, however, it is not a trivial task
to choose one theory among different modified gravity theories. It is well-
known that the solutions of the Schwarzschild and Kerr black holes are popular
in different alternative constructions, and that the dynamical property can be
analyzed in the regime of strong gravitational field for the observational
limits. Hence, it is more useful to discuss differences from GR with an
appropriate class of parameters, by measuring the shift starting from
established metric solutions and ensuring that the Schwarzschild and Kerr
geometries can be smoothly derived under certain suitable conditions. Such an
idea has been explored in, e.g., Ref. Johannsen:2011dh , where alterations to
the Kerr solution occur organically, accommodating arbitrary spin values, in
the context of modified gravity, specifically in scenarios characterized by
rotating solutions, it pertains to a stationary, axisymmetric, and
asymptotically flat metric, irrespective of the specific theoretical framework
in use.

This approach offers the advantage of addressing the majority of issues that
typically surface in perturbation theory within GR. These challenges primarily
stem from the broken of the no-hair theorems Gair:2007kr ; Johannsen:2010xs ;
Sotiriou:2015pka ; Herdeiro:2014goa ; Herdeiro:2015waa ; Cardoso:2016ryw ,
which significantly limit the forecasts of certain scenarios, such as maximum
mass-disparity inspirals Barack:2006pq  alternatively, the orbital dynamics of
stars and pulsars in proximity to black holes Wex:1998wt ; Will:2007pp ;
Broderick:2013rlq . Nonetheless, the absence of a clearly established model
elucidating the genesis of the metric presented in Ref. Johannsen:2011dh
constrains its utility in scenarios extending beyond black holes. This raises
questions about how such a solution can be systematically derived..

Due to the above mentioned point, the Chern-Simons (CS) modified gravity is a
beneficial theory. In the following, we first explain the significance of the
CS theories as well as the previous related studies.

The CS theory could address the issues of black holes and quantum aspects of
gravitation in a single theoretical framework. The CS gravitational theory as
influenced by the alteration of electrodynamics, as presented by Jackiw and Pi
in their work Jackiw:2003pm ; Carroll:1989vb . The Lagrangian for the Maxwell
field in the theory of $U(1)$ gauge undergoes modification through the
inclusion of a (pseudo)-scalar field $\vartheta(x)$ that is linked to the
topology of the Pontryagin density of $U(1)$
${}^{*}FF\equiv^{*}F_{\mu\nu}F^{\mu\nu}$. Despite keeping the gauge
invariance, such modification admits for the violation of the Lorentz/CPT
symmetry Carroll:1989vb . This is understood from the expression of the
Carroll-Field-Jackiw form ${\mathit{v}}_{\mu}\,^{*}F^{\mu\nu}A_{\nu}$, with
${\mathit{v}}_{\mu}\equiv\partial_{\mu}\vartheta$ being the axial vector which
is the source of the violation of the Lorentz/CPT symmetry. If we extend the
theoretical construction, ${\mathit{v}}_{\mu}$ connects to one of the
coefficients associated with Lorentz CPT violation within the framework of the
extension to the standard model Colladay:1996iz ; Colladay:1998fq ;
Kostelecky:2003fs . In the context of CS modified gravity, as seen in
modifications like the CS alteration to Maxwell’s electrodynamics, where an
introduction of a non-minimal coupling is made between $\vartheta(x)$ and the
gravitational Pontryagin density as follows
${}^{*}RR\equiv{}^{*}R^{\mu\nu\alpha\beta}R_{\nu\mu\alpha\beta}$ (denoted in
the Lagrangian of GR).

The relation between the (anti) de-Sitter (A)dS and the conformal field theory
(CFT) Maldacena:1997re ; Witten:1998qj ; Witten:1998zw ; Gubser:1998bc ;
Aharony:1999ti  has gained a lot of interest in the asymptotic AdS spacetimes
and their corresponding thermodynamics. A number of phenomena in gauge field
theories such as phase transitions, the encoding of confinement, and the
conformal anomalies in the physics of semi-classical asymptotically AdS black
holes has been investigated Witten:1998qj ; Witten:1998zw ; Henningson:1998gx
. Aside from its critical role in the preceding duality, asymptotic (A)dS has
different attractive characteristics. In contrast to the asymptotic flat
situation where $\Lambda=0$, the case with a negative cosmological constant
($\Lambda<0$) can be permitted for different kinds of topologies for the black
holes horizons. Such horizons could have different forms i.e., hyperbolic,
spherical or planar, global identifications can result in the formation of
cylinders (in the planar case) or the tori and Riemann surfaces (in the
hyperbolic case) in four dimensions Mann:1997iz . In the present paper, we
apply a rotating flat horizon spacetime to the CS dynamical theory and try to
derive a black hole solution.

The structure of the current investigation is outlined as follows. In Sec. II,
we present the cornerstone of the CS gravitational theory. In Sec. III, we
explain the necessary tools that we use in the techniques of perturbations. In
Sec. IV, we give a brief review of the flat horizon black hole (BH) solution.
In Sec. V, we apply the metric or rotating flat horizon to the dynamical CS
field equations and show that the scalar field does not affect the metric
component, $g_{t,\theta}$. In Sec. VI, we take the procedure described in Sec.
V with a charge rotating flat horizon, and we demonstrate that the scalar
field has no effect the metric component $g_{t\,\theta}$.In conclusion, we
conclude our investigation by presenting the main findings in Sec. VII.

The predominant choice for spacetime signature is mostly positive, and the
gravitational coupling is defined as $\kappa^{2}=8\pi$, describing in
geometric units where $G=c=1$. Levi-Civita tensor which is denoted as
$\varepsilon_{\mu\nu\rho\sigma}$, can be expressed in terms of fully
antisymmetric symbol $\varepsilon_{\mu\nu\rho\sigma}$, where
$\varepsilon_{0123}=1$.

##  II Chern-Simons amended gravitational theory

In this section we are going to present the main tools of the CS gravitational
theory that support the present study Nashed:2023cyr .

In CS gravitational theory, the Lagrangian density takes the form:

$L=L_{1}+L_{2}+L_{3}+L_{4}\,.$ (1)

Here, $L_{1}$ is the Einstein-Hilbert Lagrangian involving the cosmological
constant, defined as:

$L_{1}=\kappa\int_{V}d^{4}x\sqrt{-g}(R-2\Lambda),$ (2)

$L_{2}$ is the Lagrangian of the scalar field, given by:

$L_{2}=-\frac{\alpha_{1}}{2}\int_{V}d^{4}x\sqrt{-g}\left[g^{\alpha\beta}\left(\nabla_{\alpha}\vartheta\right)\left(\nabla_{\beta}\vartheta\right)+2V(\vartheta)\right],\quad$
(3)

${L}_{3}$ is the Chern-Simons Lagrangian, expressed as:

$L_{3}=\frac{\alpha}{4}\int_{V}d^{4}x\sqrt{-g}\;\vartheta\;{\,{}^{\ast}\\!}R\,R\,,$
(4)

$L_{4}$ is the Lagrangian of the matter field, represented as:

$L_{M}=\int_{V}d^{4}x\sqrt{-g}{L}_{mat.}.$ (5)

The following symbols are used through this study: $\alpha$ and $\alpha_{1}$
are dimensional constants; $\nabla_{\beta}$ is the covariant derivative; $g$
is the metric determinant; $R$ is the Ricci scalar tensor, $\Lambda$ is the
cosmological constant that can rewritten as $\Lambda=\frac{-3}{\iota^{2}}$,
and $\kappa=\frac{1}{16\pi G}$. The symbol ${\,{}^{\ast}\\!}R\,R$ refers to
the Pontryagin density and is defined as

${\,{}^{\ast}\\!}R\,R={\,{}^{\ast}\\!}R^{\alpha}{}_{\beta}{}^{\gamma\delta}R^{\beta}{}_{\alpha\gamma\delta}\,,$
(6)

with ${\,{}^{\ast}\\!}R^{\alpha}{}_{\beta}{}^{\gamma\delta}$ the dual Riemann-
tensor, defined as:

${\,{}^{\ast}\\!}R^{\alpha}{}_{\beta}{}^{\gamma\delta}=\frac{1}{2}\varepsilon^{\gamma\delta\gamma_{1}\delta_{1}}R^{\alpha}{}_{\beta\gamma_{1}\delta_{1}}\,,$
(7)

where $\varepsilon^{\gamma\delta\gamma_{1}\delta_{1}}$ is completely anti-
symmetric tensor and $\varepsilon^{0123}=-1$.

The expression for the CS scalar field $\vartheta$ is the one that
parameterizes the deformation in GR. If $\vartheta=const.$, GR is recovered
since the Pontryagin density becomes a total derivative of the CS topological
field which is defined as:

$\nabla_{\alpha}\Psi^{a}=\frac{1}{2}{\,{}^{\ast}\\!}R\,R\,,$ (8)

where

$\Psi^{\alpha}=\varepsilon^{\alpha\beta\gamma\delta}\Gamma^{\rho}_{{\beta}\epsilon}\left(\partial_{\gamma}\Gamma^{\epsilon}_{\delta\rho}+\frac{2}{3}\Gamma^{\epsilon}_{\gamma\gamma_{1}}\Gamma^{\gamma_{1}}_{\delta\rho}\right)\,,$
(9)

with $\Gamma$ the second kind of Christoffel symbol. By using Eq. (9), we
describe $L_{3}$ in the form Yunes:2007ss

$L_{{CS}}=\alpha\left(\vartheta\;{\Psi}^{\alpha}\right)|_{\partial{{V}}}-\frac{\alpha}{2}\int_{{{V}}}d^{4}x\sqrt{-g}\;\left(\nabla_{\alpha}\vartheta\right){\Psi}^{\alpha}\,.$
(10)

In general, we do not take into account the first term in Eq. (10) because it
is described at the boundary of the spacetime Grumiller:2008ie . On the other
hand, the second term is regarded as the CS correction term.

The variation of the Lagrangian (1) with respect to (w.r.t) the metric, and
the CS coupling leads to the following field equations:

$\displaystyle
R_{ab}-2g_{\alpha\beta}\Lambda+\frac{\alpha_{1}}{\kappa}C_{\alpha\beta}=\frac{1}{2\kappa}\left(T_{\alpha\beta}-\frac{1}{2}g_{\alpha\beta}[T-4\Lambda]\right),$
(11)
$\displaystyle\alpha_{1}\;\square\vartheta=\alpha_{1}\;\frac{dV}{d\vartheta}-\frac{\alpha}{4}{\,{}^{\ast}\\!}R\,R\,.$
(12)

Here the Ricci second order tensor is $R_{ab}$ while
$\square=\nabla_{\mu}\nabla^{\mu}$ is the D’Alembertian operator. The term
$C_{ab}$ is the C-tensor, defined as

$C^{\alpha\beta}=v_{\gamma}\varepsilon^{\gamma\gamma_{1}\gamma_{2}(\alpha}\nabla_{\gamma_{2}}R^{\beta){}_{\gamma_{1}}}+v_{\alpha_{1}\alpha_{2}}{\,{}^{\ast}\\!}R^{\alpha_{2}(\alpha\beta)\gamma}\,,$
(13)

where

$v_{\alpha}=\nabla_{\alpha}\vartheta\,,\qquad
v_{\alpha\beta}=\nabla_{\alpha}\nabla_{\beta}\vartheta\,.$ (14)

The term $T_{\alpha\beta}$ is the total stress-energy tensor which can be
expressed as:

$T_{\alpha\beta}=T^{{mat}}_{\alpha\beta}+T_{\alpha\beta}^{\varrho}\,.$ (15)

Here, $T^{mat}_{\alpha\beta}$ is the matter source, which we do not consider
in this study, and $T_{\alpha\beta}^{\vartheta}$ is defined as

$\displaystyle
T_{\alpha\beta}^{\vartheta}=\alpha_{1}\left[\left(\nabla_{\alpha}\vartheta\right)\left(\nabla_{\beta}\vartheta\right)-\frac{1}{2}g_{\alpha\beta}\left(\nabla^{\alpha_{1}}\vartheta\right)\left(\nabla^{\alpha_{1}}\vartheta\right)-g_{\alpha\beta}V(\vartheta)\right]\,.$
(16)

The principle of strong equivalence which is denoted as
($\nabla_{\beta}T^{\alpha\beta}_{{mat}}=0$), holds true in the case of CS
theory when the field equations (12) for the scalar field $\vartheta$ are
fulfilled. This is due to the fact that the derivative of Eq. (11), the first
term, becomes nullified which is the characteristics of the Bianchi
identities. However, the next term which is linked to the Pontryagin density
in a proportionate manner, as per the following expression:

$\nabla_{\alpha}C^{\alpha\beta}=-\frac{1}{8}v^{\beta}{\,{}^{\ast}\\!}R\,R.$
(17)

The confirmation of Eq. (17) leads to Eq. (12).

The dimensional characteristics of the constants employed in this
investigation and the scalar field $\vartheta$ are considered as follows. The
fixation of one of $(\alpha,\alpha_{1},\vartheta)$ fixes the units of the
others. For example, if the unit of the CS scalar field is
$[\vartheta]=l^{b}$, $[\alpha]=l^{2-b}$ and $[\alpha_{1}]=l^{-2b}$, with $l$
being the standard unit of length. To make the CS scalar $\vartheta$ a
dimensionless quantity, $[\alpha]=l^{2}$ and $\alpha_{1}$ must be
dimensionless in the scalar-tensor theories as usual 111We are going to put
$G=c=1$, and thus, the Lagrangian takes the measurement of $l^{2}$. Therefore,
when employing natural units then $G=c=1$, therefore, the Lagrangian will have
no dimensions, and in the event that $[\vartheta]=l^{b}$ then
$[\alpha]=l^{-b}$ and $[\alpha_{1}]=l^{-2b-2}$.. Another option is
$\alpha=\alpha_{1}$, which results in $[\vartheta]=l^{-2}$ if $L_{2}$ and the
Lagrangian of CS are on equal footing. Since no formulation forces us to
choice special measurement units for $\vartheta$, we keep it unspecified, as a
result of previous studies that have opted for a different selection.

##  III Rotating flat horizon BH solution

In this section, we use two different schemes and resolve the modified
dynamical CS equation of motions till the second order perturbation term to
examine the properties of the rotating flat horizon BH solution in the
dynamical CS theory of gravity.

###  III.1 The approximate method

The procedures of two approximations developed in this work are as follows.
$a$ is the slow rotation parameter and we consider the approximations of
small-coupling. In this process, one considers the amended CS like a minor
alteration of the GR, enabling us to make an assumption about the metric
decomposition (up to the second order) in the following manner:

$\displaystyle g_{ab}=g_{ab}^{(0)}+\xi
g^{(1)}_{ab}(\vartheta)+\xi^{2}g^{(2)}_{ab}(\vartheta)\,,$

where $g_{ab}^{(0)}$ represents the underlying metric which verifies the
equation of motions of GR, like (A)dS-Kerr metric, while
$g_{ab}^{(1)}(\vartheta)$ and $g_{ab}^{(2)}(\vartheta)$ constitute the
perturbations at both first and second orders of the CS theory. The parameter
$\xi$ represents the degree of approximation within the small-coupling scheme.

Moreover, We can re-express the base configuration and the $\xi$-perturbations
expressed in powers of (A)dS-Kerr through the utilization of the slow rotation
approximation. Thus, the metric background and the metric perturbations give:

$\displaystyle g_{ab}^{(0)}$ $\displaystyle=$
$\displaystyle\eta_{ab}^{(0,0)}+\epsilon\;h_{ab}^{(1,0)}+\epsilon^{2}h_{ab}^{(2,0)},$
$\displaystyle\xi g_{ab}^{(1)}$ $\displaystyle=$ $\displaystyle\xi
h_{ab}^{(0,1)}+\xi\epsilon\;h_{ab}^{(1,1)}+\xi\epsilon^{2}h_{ab}^{(2,1)},$
$\displaystyle\xi^{2}g_{ab}^{(2)}$ $\displaystyle=$
$\displaystyle\xi^{2}h_{ab}^{(0,2)}+\xi^{2}\epsilon\;h_{ab}^{(1,2)}+\xi^{2}\epsilon^{2}h_{ab}^{(2,2)}\,,$
(19)

where $\epsilon$ represents the degree of expansion within the slow rotation
approximation. It is noted that the notation $h^{o\;p}_{a\;b}$ is the
expressions of ${\mathcal{O}}(o\;p)$, which points to an expression of
${\mathcal{O}}(\epsilon^{o})$ and ${\mathcal{O}}(\xi^{p})$. In Eq. (III.1),
$\eta_{ab}^{(0,0)}$ represents the base metric in the absence of the rotation
parameter, i.e., $a=0$, whereas $h_{ab}^{(1,0)}$, $h_{ab}^{(2,0)}$,
$\xi_{a}^{(1,0)}$ and $\xi_{a}^{(2,0)}$ constitute the perturbations of the
base metric at both the first and second orders. We obtain the bivariate
expansion by using the two independent parameter $\xi$ and $\epsilon$,which
take on the following metric form with perturbations up to the second order:

$\displaystyle g_{ab}=\eta_{ab}^{(0,0)}+\epsilon h_{ab}^{(1,0)}+\xi
h_{ab}^{(0,1)}+\epsilon\xi
h_{ab}^{(1,1)}+\epsilon^{2}h_{ab}^{(2,0)}+\xi^{2}h_{ab}^{(0,2)}\,.$

The terms ${\mathcal{O}}(1,0)$ or ${\mathcal{O}}(0,1)$ corresponding to the
first-order while ${\mathcal{O}}(2,0)$, or ${\mathcal{O}}(0,2)$ or
${\mathcal{O}}(1,1)$ corresponding to the second-order.

Slow-rotation involves expanding the parameter $a$, and such expansion must be
dimensionless, represented as $a/M$. Hence, any equation multiplied by
$\epsilon^{n}$ is ${\mathcal{O}}\left((a/M)^{n}\right)$. Before applying the
above procedure to (A)dS-Kerr solution, we present a brief review of this BH
in the next section.

##  IV Flat horizon black hole solution

The action for GR with the Maxwell field of asymptotic AdS spacetime is given
by

$\displaystyle L=-{1\over 16\pi
G_{d}}\int_{\mathcal{M}}d^{4}x\sqrt{-g}\left(R+{6\over\iota^{2}}\right)\,.$
(21)

The variation of the action w.r.t. the metric gives the following field
equations:

$G_{\mu\nu}-\Lambda g_{\mu\nu}=2\,T_{\mu\nu}\,.$ (22)

Lemos obtained a four-dimensional solution having rotation and flat horizon,
is presented in Lemos:1994xp ; Lemos:1995cm , and it is expressed in the
following manner:

$ds^{2}=-Q(r)\,(\Xi\,dt-a\,d\theta)^{2}+{dr^{2}\over
Q(r)}+{r^{2}\over\iota^{4}}(a\,dt-\Xi\iota\,d\theta)^{2}+{r^{2}\over\iota^{2}}\,d{\rho}^{2}\,,$
(23)

where $Q(r)$ is given by

$Q(r)={r^{2}\over\iota^{2}}-{M\over r}.\\\ $ (24)

The entities $\theta$ and $\rho$ are supposed to be $0\leq\theta<2\pi$ and
$-\infty<\rho<\infty$, respectively. In this context, $a$ represents the
rotation, $M$ is the mass, and $\Xi$ is figures as
$\Xi=\sqrt{1+a^{2}/\iota^{2}}$. The solution given in (23) is investigated in
previous works like the one presented in Lemos:1994xp ; Lemos:1995cm ;
Dehghani:2002rr ; Cardoso:2001vs ; Awad:2002cz , and preserves supersymmetry
Lemos:2000wp . It’s evident that a transformation exists, which linking the
solution with $a$ to the one without $a$, and this transformation is defined
as:

$t^{\prime}=\Xi\,t-a\,\theta\hskip 72.26999pt\theta^{\prime}={a\over
l^{2}}\,t-\Xi\,\theta\,.$ (25)

Due to its combination of the compactified $\theta$ coordinate and the time
coordinate $t$. The above transformation is locally permissible however, it is
not valid globally Lemos:1994xp . References Lemos:1994xp ; Stachel:1981fg
have demonstrated that in cases where the first Betti number of the manifold
is non-zero, there exist no global diffeomorphisms capable of mapping the two
metrics Lemos:1994xp ; Stachel:1981fg . References Lemos:1994xp ;
Stachel:1981fg  have demonstrated that solution (23) possesses a first Betti
number of one when cylindrical or toroidal horizons are present.

##  V Slowly rotating (A)dS BH solution

In the regime of slow rotation, the expansion of the background metric can be
characterized by the following line element:

$\displaystyle
ds^{2}=-Q\left[1+Q_{1}(r)\right]dt^{2}+\frac{1}{Q}\left[1+Q_{2}(r)\right]dr^{2}+\frac{r^{2}}{\iota^{2}}\left[1+Q_{3}(r)\right]\left[d\phi-\omega(r)dt\right]^{2}+\frac{r^{2}}{\iota^{2}}\left[1+Q_{4}(r)\right]d\rho^{2}\,,$

where $Q$ takes the form $Q=\frac{r^{2}}{\iota^{2}}-\frac{M}{r}$. This is the
(A)dS-Schwarzschild solution which we take as the background metric, and $M$
is the mass of the BH in the absence of the CS expression. In Eq. (V), we use
the cylindrical coordinates with the perturbations of the metric are
$Q_{1}(r)$, $Q_{2}(r)$, $Q_{3}(r)$, $Q_{4}(r)$ and $\omega(r)$.

The metric (V) can be rewritten similarly to the one presented in Refs.
Lemos:1994xp ; Lemos:1995cm , and The expansions of the metric perturbations
need to be carried out in asymptote of $\xi$ as well as $\epsilon$. The
expression for the second-order expansion is given by:

$\displaystyle Q_{1}(r)$ $\displaystyle=$
$\displaystyle\epsilon\;Q_{1}{{}_{(1,0)}}+\epsilon\;\xi\;Q_{1}{{}_{(1,1)}}+\epsilon^{2}\;Q_{1}{{}_{(2,0)}},$
$\displaystyle Q_{2}(r)$ $\displaystyle=$
$\displaystyle\epsilon\;Q_{2}{{}_{(1,0)}}+\epsilon\;\xi\;Q_{2}{{}_{(1,1)}}+\epsilon^{2}\;Q_{2}{{}_{(2,0)}},$
$\displaystyle Q_{3}(r)$ $\displaystyle=$
$\displaystyle\epsilon\;Q_{3}{{}_{(1,0)}}+\epsilon\;\xi\;Q_{3}{{}_{(1,1)}}+\epsilon^{2}\;Q_{3}{{}_{(2,0)}},$
$\displaystyle Q_{4}(r)$ $\displaystyle=$
$\displaystyle\epsilon\;Q_{4}{{}_{(1,0)}}+\epsilon\;\xi\;Q_{4}{{}_{(1,1)}}+\epsilon^{2}\;Q_{4}{{}_{(2,0)}},$
$\displaystyle\omega(r)$ $\displaystyle=$
$\displaystyle\epsilon\;\omega_{(1,0)}+\epsilon\;\xi\;\omega_{(1,1)}+\epsilon^{2}\;\omega_{(2,0)}.$
(27)

Equations (V) have no terms of ${\mathcal{O}}(0,0)$ because it exists in the
(A)dS-Schwarzschild structure of Eq. (V), in which we have bivariated the
diagonal components of (A)dS-Kerr in terms of the rotation parameter $a$.
Moreover, it is supposed that if the rotation parameter of (A)dS-Kerr is
vanishing, we can show that the Schwarzschild-(A)dS is a solution. This
ensures that all the terms of ${\mathcal{O}}(0,s)$ are vanishing. Therefore,
the CS expression must be linear in terms of the (A)dS-Kerr rotation. When the
rotation parameter $a$ is perturbed, the metric perturbation proportional to
$\xi^{0}$ up to the first order takes the following form

$\displaystyle Q_{1}{{}_{(1,0)}}$ $\displaystyle=$ $\displaystyle
Q_{2}{{}_{(1,0)}}=Q_{3}{{}_{(1,0)}}=Q_{4}{{}_{(1,0)}}=0\,,\qquad\omega_{(1,0)}=-2\frac{M}{r}\frac{a}{\iota}\,,$
(28)

and up to second order as

$\displaystyle Q_{1}{{}_{(2,0)}}$ $\displaystyle=$
$\displaystyle\frac{M}{r}\frac{a^{2}}{\iota^{2}}\,,\qquad\qquad
Q_{2}{{}_{(2,0)}}=Q_{4}{{}_{(2,0)}}=\omega_{(2,0)}=0\,,\qquad\qquad
Q_{3}{{}_{(2,0)}}=\frac{M}{r}\frac{a^{2}}{\iota^{2}}\,,$ (29)

which corresponds to the (A)dS Schwarzschild solution when the rotation
parameter vanishes, namely, $a=0$ Nashed:2021xtt ; Nashed:2021sey ;
Nashed:2020kdb . All the fields are bivariate in terms of the rotation
parameter $a$, in addition within the framework of the small-coupling
approximation, incorporating the CS field. To derive the leading-order of the
scalar field $\vartheta$, we analyze the evolution equation in Eq. (12). This
equation yields
$\partial^{2}\vartheta\sim(\alpha_{1}/\alpha){\,{}^{\ast}\\!}R\,R$, from which
the Pontryagin density is vanishing till order $a/M$. As a result, the CS
first order behavior should behaves as
$\vartheta\sim(\alpha_{1}/\alpha)((a/M))$, i.e., $\propto\epsilon$. We should
have $\vartheta^{(0,s)}=0$ for all $s$ should the Schwarzschild-(A)dS metric
which is the unique solution when $a$ is zero. In the present paper, we
describe slowly rotating solution. By utilizing Eq. (12) in Eq. (V) and with
Eq. (V), we get

$\vartheta=\epsilon\;\vartheta^{(1,0)}(r)+\epsilon\;\xi\;\vartheta^{(1,1)}(r)+\epsilon^{2}\;\vartheta^{(2,0)}(r)\,.$
(30)

By applying the method outlined above for solving the modified field equations
and highlighting the evolutionary equation for the dynamical CS scalar field,
up to ${\mathcal{O}}(1,0)$ we acquire the form

$\displaystyle
Q\vartheta^{(1,0)}_{,rr}+\frac{1}{r}\vartheta^{(1,0)}_{,r}\left(\frac{4r^{2}}{\iota^{2}}-\frac{M}{r}\right)=0\,.$
(31)

The solution of the above differential equation (42) is given by

$\vartheta^{(1,0)}(r)=\frac{c_{0}}{3\,M\,\iota^{2}}\ln\left(\frac{1}{1-\frac{M}{r}\,\frac{\iota^{2}}{r^{2}}}\right).$
(32)

where $c_{0}$ is a dimensional constant of integration.

The asymptotic form of Eq. (43) is expressed as

$\displaystyle\vartheta^{(1,0)}(r)\approx\frac{c_{0}}{r^{3}}\left(1+\frac{M}{2r}\frac{\iota^{2}}{r^{2}}\right)+{\mathcal{O}}\left(\frac{1}{r^{9}}\right)\,.$
(33)

Since we present the dynamical CS field, we derive the dynamical CS
corrections of the metric perturbations. We omit the sharing of the energy-
momentum tensor in Eq. (16) to the metric perturbation because of its sharing
to the modified equation of motions in Eq. (11) up to ${\mathcal{O}}(2,1)$.
The modified Einstein equations are divided into two sets. The initial set
constructs a system of differential equations that have ${h_{1}}^{(1,1)}$,
${h_{2}}^{(1,1)}$, ${h_{3}}^{(1,1)}$ and ${h_{4}}^{(1,1)}$, that are the
components $(t,t)$, $(r,r)$, $(r,\theta)$, $(\theta,\theta)$ and
$(\rho,\rho)$. The second set gives one equation of $\omega^{(1,1)}$, that is
the $(t,\theta)$.

The initial group is not dependent on the dynamics of the CS, $\vartheta$, and
therefore its contribution vanishes identically. Hence, we investigate the
second set and consider $(t,\theta)$-component, which leads to

$\displaystyle
Q\omega^{(1,1)}_{rr}-\frac{4Q}{r}\omega^{(1,1)}_{r}-\frac{6\alpha_{1}}{\iota^{2}}\omega^{(1,1)}=6\alpha_{1}\frac{M}{r}\frac{a}{\iota}\frac{\alpha_{1}}{r\iota}\,.$
(34)

The equation above is a non-homogeneous differential equation, possessing both
homogeneous and particular solutions. The homogeneous solution is structured
as follows:

$\omega^{(1,1)}=r^{\beta}\left(1-\frac{M}{r}\frac{\iota^{2}}{r^{2}}\right)\left\\{c_{1}H([b],c,d)+c_{2}H([-b],-c,d)\right\\}\,,$
(35)

where $H$ denotes a hypergeometric function222The hypergeometric function is
defined as  $H(n,d,z)=Sum(z^{k}/k!*product(pochhammer(n[i],k),i=1\cdots
p)/product(pochhammer(d[j],k),j=1..p),k=0\cdots\infty)\,.$ and
$\beta=\frac{-6+\sqrt{96\beta+36}}{4}$,
$b=[\frac{6-\sqrt{96\beta+36}}{12},\frac{18-\sqrt{96\beta+36}}{12}]$,
$\frac{6-\sqrt{96\beta+36}}{6}$, $d=\frac{M}{r}\frac{\iota^{2}}{r^{2}}$. The
homogenous solution (35) is not a physical one because it has no definite
value as $r\to\infty$, and therefore we do not examine it further. The
particular solution of Eq. (34) has the form

$\omega^{(1,1)}_{Part.}=\frac{c_{3}}{r^{3}}\,,$ (36)

where $c_{3}$ is a dimensional constant.

We obtain the non-vanishing metric components of the above solution

$\displaystyle g_{tt}=$
$\displaystyle\,-\left(\frac{r^{2}}{l^{2}}-\frac{M}{r}\right)-\frac{M}{r}\frac{a^{2}}{l^{2}}\,,\qquad\qquad
g_{t\theta}=-\frac{Ma}{r}-\frac{c_{1}\xi}{r}\equiv-\frac{\mathrm{const}}{r}\,,$
$\displaystyle\qquad g_{rr}=$
$\displaystyle\,\frac{1}{\left(\frac{r^{2}}{l^{2}}-\frac{M}{l}\right)}\,,\qquad\qquad
g_{\theta\theta}=\frac{r^{2}}{l^{2}}+\frac{M}{r}\frac{a^{2}}{l^{2}}\,,\qquad
g_{\rho\,\rho}=\frac{r^{2}}{l^{2}}\,.$ (37)

Equation (V) shows that the CS scalar field has no effect on the metric. This
means that any spacetime asymptote as (A)dS of the CS scalar field will have
no effect. In the next section, we examine this conclusion and show that it is
true if we include the charge field.

##  VI Slowly charged rotating (A)dS BH solution

In the charged BH solution, the Lagrangian of the gravitational theory is
represented as

$L=L_{1}+L_{2}+L_{3}+L_{4}+L_{5}\,,$ (38)

where $L_{1}$, $L_{2}$, $L_{3}$, $L_{4}$ are defined in Eqs. (2), (3), (4),
and (5), while $L_{E.M.}$ is the electromagnetic field, defined as

$S_{4}=\frac{\alpha_{1}}{2}\int_{V}d^{4}x\sqrt{-g}\left(\frac{F^{2}}{2}\right)\,,\quad\textrm{where}\quad
F^{2}=F_{\mu\nu}F^{\mu\nu}\,,\quad\textrm{and}\quad
F_{\mu\nu}=2A_{[\mu,\nu]}\,,$ (39)

where $A$ represents the gauge potential for electromagnetism.

The action’s variation with respect to the metric and the electromagnetic
gauge potential, as described in (38), yields the following field equations:

$\displaystyle
R_{ab}+\frac{\alpha_{1}}{\kappa}C_{ab}=\frac{1}{2\kappa}\left({\mathcal{T}}_{ab}-\frac{1}{2}g_{ab}{\mathcal{T}}\right),$
(40) $\displaystyle 0=\partial_{b}\left(\sqrt{-g}{\textrm{F}}^{ab}\right)\,.$
(41)

Conversely, when the action (38) is varied with respect to the CS term, it
results in Eq. (12).

We employ the procedure outlined earlier to the modified equations of motions
and the equation of evolution for the dynamical CS. Till ${\mathcal{O}(1,0)}$
we obtain the evolution equation in the following format:

$\displaystyle{\mathit{Q}}\vartheta^{(1,0)}_{,rr}+\frac{1}{r}\vartheta^{(1,0)}_{,r}\left(\frac{4r^{2}}{l^{2}}-\frac{M}{r}\right)=0\,,$
(42)

where
${\textit{Q}}=\left(\frac{r^{2}}{\iota^{2}}-\frac{M}{r}+\frac{q^{2}}{r^{2}}\right)$.
Solution of the differential equation (42) with homogeneous terms is
represented as follows:

$\vartheta^{(1,0)}(r)=-c_{4}\int\frac{r^{2}\,\iota^{2}}{\mathit{Q}}\,dr\approx\frac{c_{4}}{r^{3}}\left(1+\frac{M}{2r}\frac{\iota^{2}}{r^{2}}-\frac{3q^{2}}{7r^{2}}\frac{\iota^{2}}{r^{2}}\right)+{\mathcal{O}}\left(\frac{1}{r^{9}}\right)\,.$
(43)

where $c_{4}$ is a dimensional constant of integration.

We deal with the second set $(t,\theta)$-component, which yields

$\displaystyle
Q\omega^{(1,1)}_{rr}-\frac{4Q}{r}\omega^{(1,1)}_{r}-\frac{6\alpha_{1}}{\iota^{2}}\omega^{(1,1)}=6\alpha_{1}\left(\frac{M}{r}-\frac{q}{r^{2}}\right)\frac{a}{\iota}\frac{\alpha_{1}}{r\iota}\,.$
(44)

The equation above is a non-homogeneous differential equation with both
homogeneous and particular solutions. Homogeneous solution is represented as
follows:

$\displaystyle\omega^{(1,1)}\approx
c_{5}\,{r}^{-\frac{3+\,\sqrt{3}\sqrt{{{3\,+8\,\beta}}}}{2}}{e^{{\frac{\,\sqrt{6}{\iota}^{2}\alpha_{1}\,\left(Mr-{q}^{2}\right)}{2\sqrt{-{q}^{2}{\iota}^{2}\alpha_{1}}{r}^{2}}}}}HeunB\Bigg{[}\sqrt{3}\sqrt{{{3\,+8\,\alpha_{1}}}},{\frac{i\alpha_{1}\,{}^{2}M{\iota}^{2}\sqrt[4]{6}}{\left(-{q}^{2}{\iota}^{2}\alpha_{1}\right)^{3/4}}},-{\frac{{\alpha_{1}}^{2}{M}^{2}{\iota}^{4}\sqrt{6}}{4\left(-{q}^{2}{\iota}^{2}\alpha_{1}\right)^{3/2}}},0,{\frac{i\sqrt[4]{6}\sqrt[4]{-{}^{3}{q}^{2}{\iota}^{2}\alpha_{1}}}{\,r}}\Bigg{]}$
$\displaystyle+c_{6}\,{r}^{-\frac{3-\,\sqrt{3}\sqrt{{{3\,+8\,\beta}}}}{2}}HeunB\bigg{[}-\sqrt{3}\sqrt{{{3\,+8\,\alpha_{1}}}},{\frac{i\alpha_{1}\,{}^{2}M{\iota}^{2}\sqrt[4]{6}}{\left(-{q}^{2}{\iota}^{2}\alpha_{1}\right)^{3/4}}},-{\frac{{\alpha_{1}}^{2}{M}^{2}{\iota}^{4}\sqrt{6}}{4\left(-{q}^{2}{\iota}^{2}\alpha_{1}\right)^{3/2}}},0,{\frac{i\sqrt[4]{6}\sqrt[4]{-{}^{3}{q}^{2}{\iota}^{2}\alpha_{1}}}{\,r}}\Bigg{]}$
$\displaystyle\times\,e^{-\frac{\left\\{i{r}^{2}\left(\sqrt{3}\sqrt{{\frac{3\,+8\,\alpha_{1}}{}}}-1\right)\pi\,\sqrt{-{q}^{2}{\iota}^{2}\alpha_{1}}-2\,\,\sqrt{6}{\iota}^{2}\alpha_{1}\,\left(Mr-{q}^{2}\right)\right\\}}{{4{r}^{2}\sqrt{-{q}^{2}{\iota}^{2}\alpha_{1}}}}}\,,$
(45)

where $HeunB$ function is the solution of the Heun Biconfluent equation333The
Heun function, $HeunB(\alpha,\beta,\gamma,\delta,z)$, is the solution of the
Heun Biconfluent equation which has the form  $Y^{\prime\prime}(z)-\frac{\beta
z-\alpha+2z^{2}-1}{z}Y^{\prime}(z)-\frac{2(\alpha-\gamma+2)z+\delta+\beta+\beta\alpha}{2z}Y(z)\,.$
(46) The above differential equation should satisfy the following boundary
conditions: $Y(0)=1$,
$Y^{\prime}(0)=\frac{\delta+\beta+\beta\alpha}{2(1+\alpha)}$.. The homogenous
solution of Eq. (44) in Eq.(VI) is not a physical one because the term
multiplied by the constant $c_{6}$ has no definite value as $r\to\infty$.
Therefore, we set the constant $c_{6}=0$. Similarly, the term multiplied by
the constant $c_{5}$ is a not real one, and hence we take the constant
$c_{5}=0$. The particular solution of Eq. (44) gives the form

$\omega^{(1,1)}_{Part.}=\frac{c_{7}}{r^{3}}\,,$ (47)

where $c_{3}$ is a dimensional constant.

We obtain the non-vanishing metric components of the above solution

$\displaystyle g_{tt}=$
$\displaystyle\,-\left(\frac{r^{2}}{l^{2}}-\frac{M}{r}+\frac{q^{2}}{r^{2}}\right)-\left(\frac{M}{r}-\frac{q^{2}}{r^{2}}\right)\frac{a^{2}}{l^{2}}\,,\qquad\qquad
g_{t\theta}=-\left(\frac{M}{r}-\frac{q^{2}}{r^{2}}\right)\frac{a}{\iota}-\frac{c_{7}\xi}{r}\frac{a}{\iota}\equiv-\left(\frac{\mathrm{const}}{r}-\frac{q^{2}}{r^{2}}\right)\frac{a}{\iota}\,,$
$\displaystyle\qquad g_{rr}=$
$\displaystyle\,\frac{1}{\left(\frac{r^{2}}{l^{2}}-\frac{M}{l}+\frac{q^{2}}{r^{2}}\right)}\,,\qquad\qquad
g_{\theta\theta}=\frac{r^{2}}{l^{2}}+\left(\frac{M}{r}-\frac{q^{2}}{r^{2}}\right)\frac{a^{2}}{l^{2}}\,,\qquad
g_{\rho\,\rho}=\frac{r^{2}}{l^{2}}\,.$ (48)

Equation (VI) shows that the CS scalar field does not affect the metric in the
charged case. This means that any spacetime asymptote as (A)dS in the CS
scalar field will have no effect.

##  VII Conclusion and discussions

In this article, we have investigated the (A)dS-Kerr spacetime by using the
dynamical CS field equations. We have studied two cases in detail: i) The non-
charged case and ii) the charged one.
In the non-charged case, it has been shown that the scalar field of the SC
theory, Eq. (12), is affected compared with the solution presented in
Yunes:2007ss . Consequently, the components of the CS, i.e.,
$\vartheta^{(1,0)}$ differ from those presented in Ref. Yunes:2007ss . Despite
this fact, if we use the scalar field $\vartheta^{(1,0)}$ in the equation of
motions (11), we do not obtain any extra terms in the metric, up to order
$\epsilon$. Although the correction of $\omega^{(1,1)}$ of slow Kerr gives an
asymptotic form of order ${\mathcal{O}}\left(\frac{1}{r^{6}}\right)$, we have
shown that the asymptotic form of $\omega$ is not different from the form of
$\omega^{(1,1)}$ up to order $\xi$ in the case of (A)dS-Kerr spacetime.

In the charging case, we have followed the same procedure carried out for the
case of the non-charged case. We have shown that the CS scalar field (12) is
affected compared to the slowly rotating Kerr solution discussed in Ref.
Yunes:2007ss . Moreover, we have shown that using this scalar field in the
field equations (11), the asymptotic of $\omega^{(1,0)}$ has no effect on the
asymptote of $\omega^{(1,1)}$ up to order $\xi^{1}$. It is ensured from the
above two cases, charged and non-charged, that any rotating spacetime that
asymptotes as (A)dS the cross-term, which is responsible for the rotation,
cannot be affected by the CS scalar field, unlike the slowly rotating Kerr
solution that asymptotes as flat spacetime.

Finally, we end this study by mentioning the following point. In this study,
we have obtained a slow Kerr-(A)dS BH solution in the dynamical theory by
employing a linear form of angular momentum, i.e., $a$, without considering
the potential. The issue that the addition of the potential may lead to any
new physics different from (A)dS-Kerr without the potential will be studied in
the near future.

## Acknowledgments

The work of KB was partially supported by the JSPS KAKENHI Grant Number
21K03547.

## Data Availability Statement

No new data were created or analyzed in this study.

## References

  * (1) T. Padmanabhan, Gen. Rel. Grav. 40, 529-564 (2008).
  * (2) R. Durrer and R. Maartens, Gen. Rel. Grav. 40, 301-328 (2008).
  * (3) T. P. Sotiriou and V. Faraoni, Rev. Mod. Phys. 82, 451-497 (2010).
  * (4) A. De Felice and S. Tsujikawa, Living Rev. Rel. 13, 3 (2010).
  * (5) S. Nojiri and S. D. Odintsov, Phys. Rept. 505, 59-144 (2011).
  * (6) S. Capozziello and M. De Laurentis, Phys. Rept. 509, (2011) 167-321.
  * (7) T. Clifton, P. G. Ferreira, A. Padilla and C. Skordis, Phys. Rept. 513, 1-189 (2012).
  * (8) A. Joyce, B. Jain, J. Khoury and M. Trodden, Phys. Rept. 568, 1-98 (2015).
  * (9) Y. F. Cai, S. Capozziello, M. De Laurentis and E. N. Saridakis, Rept. Prog. Phys. 79, no.10, 106901 (2016).
  * (10) S. Nojiri, S. D. Odintsov and V. K. Oikonomou, Phys. Rept. 692, 1-104 (2017).
  * (11) T. Johannsen and D. Psaltis, Phys. Rev. D 83, 124015 (2011).
  * (12) J. R. Gair, C. Li and I. Mandel, Phys. Rev. D 77, 024035 (2008).
  * (13) T. Johannsen and D. Psaltis, Astrophys. J. 716, 187-197 (2010).
  * (14) T. P. Sotiriou, Class. Quant. Grav. 32, no.21, 214002 (2015).
  * (15) C. A. R. Herdeiro and E. Radu, Phys. Rev. Lett. 112, 221101 (2014).
  * (16) C. A. R. Herdeiro and E. Radu, Int. J. Mod. Phys. D 24, no.09, 1542014 (2015).
  * (17) V. Cardoso and L. Gualtieri, Class. Quant. Grav. 33, no.17, 174001 (2016).
  * (18) L. Barack and C. Cutler, Phys. Rev. D 75, 042003 (2007).
  * (19) N. Wex and S. Kopeikin, Astrophys. J. 514, 388 (1999).
  * (20) C. M. Will, Astrophys. J. Lett. 674, L25-L28 (2008).
  * (21) A. E. Broderick, T. Johannsen, A. Loeb and D. Psaltis, Astrophys. J. 784, 7 (2014).
  * (22) R. Jackiw and S. Y. Pi, Phys. Rev. D 68, 104012 (2003).
  * (23) S. M. Carroll, G. B. Field and R. Jackiw, Phys. Rev. D 41, 1231 (1990).
  * (24) D. Colladay and V. A. Kostelecky, Phys. Rev. D 55, 6760-6774 (1997).
  * (25) D. Colladay and V. A. Kostelecky, Phys. Rev. D 58, 116002 (1998).
  * (26) V. A. Kostelecky, Phys. Rev. D 69, 105009 (2004).
  * (27) J. M. Maldacena, Adv. Theor. Math. Phys. 2, 231-252 (1998).
  * (28) E. Witten, Adv. Theor. Math. Phys. 2, 253-291 (1998).
  * (29) E. Witten, Adv. Theor. Math. Phys. 2, 505-532 (1998).
  * (30) S. S. Gubser, I. R. Klebanov and A. M. Polyakov, Phys. Lett. B 428, 105-114 (1998).
  * (31) O. Aharony, S. S. Gubser, J. M. Maldacena, H. Ooguri and Y. Oz, Phys. Rept. 323, 183-386 (2000).
  * (32) M. Henningson and K. Skenderis, JHEP 07, 023 (1998).
  * (33) R. B. Mann, Annals Israel Phys. Soc. 13, 311 (1997).
  * (34) G. G. L. Nashed and K. Bamba, Nucl. Phys. B 994, 116325 (2023).
  * (35) N. Yunes and C. F. Sopuerta, Phys. Rev. D 77, 064007 (2008).
  * (36) D. Grumiller, R. B. Mann and R. McNees, Phys. Rev. D 78 (2008), 081502 doi:10.1103/PhysRevD.78.081502 [arXiv:0803.1485 [gr-qc]].
  * (37) J. P. S. Lemos, Phys. Lett. B 353, 46-51 (1995).
  * (38) J. P. S. Lemos and V. T. Zanchin, Phys. Rev. D 54, 3840-3853 (1996).
  * (39) M. H. Dehghani, Phys. Rev. D 66, 044006 (2002).
  * (40) V. Cardoso and J. P. S. Lemos, Class. Quant. Grav. 18, 5257-5267 (2001). [arXiv:gr-qc/0107098 [gr-qc]].
  * (41) A. M. Awad, Class. Quant. Grav. 20, 2827-2834 (2003).
  * (42) J. P. S. Lemos, Nucl. Phys. B 600, 272-284 (2001).
  * (43) J. Stachel, Phys. Rev. D 26, 1281-1290 (1982).
  * (44) G. G. L. Nashed and K. Bamba, Phys. Dark Univ. 33, 100859 (2021).
  * (45) G. G. L. Nashed, Phys. Lett. B 815, 136133 (2021).
  * (46) G. G. L. Nashed and E. N. Saridakis, Phys. Rev. D 102, no.12, 124072 (2020).


# MultiContrievers: Analysis of Dense Retrieval Representations

Seraphina Goldfarb-Tarrant♡♠ , Pedro Rodriguez♢, Jane Dwivedi-Yu♢, Patrick
Lewis♡
♡ Cohere, ♠ University of Edinburgh, ♢ FAIR, Meta
{seraphina<EMAIL_ADDRESS>
{par<EMAIL_ADDRESS>Work done while interning at FAIR, Meta.

###### Abstract

Dense retrievers compress source documents into (possibly lossy) vector
representations, yet there is little analysis of what information is lost
versus preserved, and how it affects downstream tasks. We conduct the first
analysis of the information captured by dense retrievers compared to the
language models they are based on (e.g., BERT versus Contriever). We use 25
MultiBert checkpoints as randomized initialisations to train MultiContrievers,
a set of 25 contriever models. We test whether specific pieces of
information—such as gender and occupation—can be extracted from contriever
vectors of wikipedia-like documents. We measure this extractability via
information theoretic probing. We then examine the relationship of
extractability to performance and gender bias, as well as the sensitivity of
these results to many random initialisations and data shuffles. We find that
(1) contriever models have significantly increased extractability, but
extractability usually correlates poorly with benchmark performance 2) gender
bias is present, but is not caused by the contriever representations 3) there
is high sensitivity to both random initialisation and to data shuffle,
suggesting that future retrieval research should test across a wider spread of
both.111We release our 25 MultiContrievers (including intermediate
checkpoints), all code, and all results, to facilitate further analysis.
https://github.com/facebookresearch/multicontrievers-analysis

##  1 Introduction

Figure 1: Part 1: We train 25 Contrievers from the 25 MultiBerts, and compare
the information theoretic extractability of gender and occupation from each of
their representations of documents. Part 2: We then compare these to metrics
of performance and of gender bias to better understand the properties of dense
retrievers.

Dense retrievers (Karpukhin et al., 2020; Izacard et al., 2022; Hofstätter et
al., 2021) are a standard component of retrieval augmented Question Answering
(QA) (Lewis et al., 2020a), and other retrieval systems such as fact-checking
(Thorne et al., 2018), argumentation (Wachsmuth et al., 2018), and others.
Despite their ubiquity, we lack an understanding of the information
recoverable from dense retriever representations, and how it affects retrieval
system behaviour. This lack of analytical work is surprising. Retrievers are
widespread, and are used in contexts that require trust: increasing factuality
and decreasing hallucination (Shuster et al., 2021), and providing trust and
transparency (Lewis et al., 2020b) via a source document that has provenance
and can be examined. The information a representation retains from a source
document constrains these abilities. Dense retrievers lossily encode input
documents into N-dimensional representations, and by doing so necessarily
emphasise some pieces of information over others. A biography of Mary
Somerville will contain many details about her: her profession (astronomy and
mathematics), her gender (female), her political influence (women’s suffrage),
her country of origin (Scotland) and others. Each of these features are
relevant to different kinds of queries. Which ones will a given retriever
represent most recoverably?

Some analysis of this type exists for Masked Language Models (MLMs) (§2.2),
but there is no such analysis for retrievers, which optimise a contrastive
loss. Contrastive training is a very different objective than MLM, based on
(dis)similarity of paired samples. The choice of pair affects feature
suppression – what is recoverable and what is not (Robinson et al., 2021). So
we extend this previous analytical work into the retrieval domain, by training
25 MultiContrievers initialised from MultiBert checkpoints (Sellam et al.,
2022). This is the first study that includes variability over a large number
of retriever initialisations, with some surprising results from this alone. We
use information theoretic probing, also known as minimum description length
(MDL) probing (Voita and Titov, 2020), to measure the information in
MultiContriever representations. We evaluate the models on 14 retrieval
datasets from the BEIR benchmark Thakur et al. (2021). We test how well
retrievers preserve information in a document, like gender and occupation,
which we refer to as features. We adapt the existing datasets to better test
for knowledge of these, by creating a new manually annotated gender subset of
Natural Questions, NQ-gender. We ultimately test if gender information is
predictive of gender bias, as it was in previous MLM work (§2.2).We address
the following four research questions:

#### Q1

To what extent do retrievers preserve information like gender and occupation
in an encoded document? (§4.1)

For both MultiBerts and MultiContrievers, gender is more extractable than
occupation, which can cause a model to rely on gender heuristics (a source of
gender bias). But there are noticeable differences in the models. Both
features are more extractable in MultiContrievers than MultiBerts, but there
is a lower ratio (less difference) between gender and occupation. This
indicates MultiContrievers are less likely to rely less on gender heuristics
(Lovering et al., 2021), but still might.

#### Q2

How sensitive is this to random initialisation and data shuffle? (§4.2)

In MultiBerts, extractability is very sensitive to random initialisation and
shuffle, in MultiContrievers it is not. MultiContrievers have a much smaller
variance between the 25 seeds, suggesting a regularising effect. However,
MultiContriever performance is surprisingly sensitive to both random
initialisation and to data shuffle. MultiContrievers have a very wide range of
performance on BEIR benchmarks, despite identical loss curves. But it is not
easy to select a ‘best’ model, since the best and worst model is not
consistent across datasets - the ranking of each model can change, sometimes
drastically.

#### Q3

Do differences in this information correlate with performance on retrieval
benchmarks? (§4.3)

On partitions of examples that ostensibly require gender information (NQ-
gender), we show that gender extractability is highly correlated with
retrieval performance. However, overall retrieval performance on benchmarks
like BEIR is poorly correlated with extractability. This suggests that while
some benchmark examples do reward models for preserving gender information,
most examples do not require that, so the benchmark as a whole does not
require that capability.

#### Q4

Is gender information in retrievers predictive of their gender bias? (§4.4)

Despite the evidence that extractability of gender information is helpful to a
model, it is not the cause of gender bias in the NQ-gender dataset. When we do
a causal analysis by removing gender from MultiContriever representations,
gender bias persists, suggesting that the source of bias is in the queries or
corpus.

Our contributions are: 1) the first information theoretic analysis of dense
retrievers, 2) an analysis of variability in performance and social bias
across random retriever seeds, 3) the first causal analysis of sources of
social bias in dense retrievers, 4) NQ-gender, an annotated subset of Natural
Questions for queries that constrain gender, and 5) a suite of 25
MultiContrievers for use in future work, with all training and evaluation
code.

##  2 Background and Related Work

The below covers dense retrievers, information theoretic probing for
extractability, and what extractability can tell us about model behaviour.

###  2.1 What is a retriever?

Retrievers take an input query and return relevance scores for documents from
a corpus. We encode documents $D$ and queries $Q$ separately by the same model
$f_{\theta}$. Given a query $q_{i}$ and document $d_{i}$, relevance is the dot
product between the document and query representations.

$\displaystyle s(d_{i},q_{i})=f_{\theta}(q_{i})\cdot f_{\theta}(d_{i})$ (1)

Training $f_{\theta}$ is a challenge. Language models like BERT (Devlin et
al., 2019), are not good retrievers out-of-the-box, but retrieval training
resources are limited and labour intensive to create, since they involve
matching candidate documents to a query from a corpus of potentially millions.
So retrievers are either trained on one of the few corpora available, such as
Natural Questions (NQ) (Kwiatkowski et al., 2019) or MS MARCO (Campos et al.,
2016) as supervision (Hofstätter et al., 2021; Karpukhin et al., 2020), or on
a self-supervised proxy for the retrieval task (Izacard et al., 2022). Both
approaches result in a domain shift between training and later inference,
making retrieval a generalisation task. This motivates our analysis, as
Lovering et al. (2021)’s work shows that information theoretic probing is
predictive of where a model would generalise and where it relies on simple
heuristics and dataset artifacts.

In this work, we focus on the self-supervised Contriever (Izacard et al.,
2022), initialised from a BERT model and then fine-tuned with a contrastive
objective.222We choose Contriever for societal relevance of our results, as it
has two orders of magnitude more monthly downloads than other popular models:
https://huggingface.co/facebook/contriever. For this objective, all documents
in a large corpus are broken into chunks, where chunks from the same document
are positive pairs and chunks from different documents are negative pairs.
This is a loose proxy for ‘relevance’ in the retrieval sense, so we are
interested in what information this objective encourages contriever to
emphasise, what to retain, what to lose, and what this means for the eventual
retrieval task.

###  2.2 What is Information Theoretic (MDL) probing?

Diagnostic classifiers, or probes, are a powerful tool for determining what
information is in a model representation Belinkov and Glass (2019). Let
$DS=\\{(d_{i},y_{i}),...,d_{n},y_{n})\\}$ be a dataset, where $d$ is a
document (e.g. a chunk of a Wikipedia biography about Mary Somerville) and $y$
is a label from a set of $k$ discrete labels $y_{i}\in Y$, $Y=\\{1,...k\\}$
for some information in that document (e.g. mathematics, astronomy if probing
for occupation).

In a probing task, we want to measure how well $f_{\theta}(d_{i})$ encodes
$y_{i}$, for all $d_{1:n}$, $y_{1:n}$. We use Minimum Description Length (MDL)
probing (Voita and Titov, 2020), or information theoretic probing, in our
experiments. This measures extractability of $Y$ via compression of
information $y_{1:n}$ from $f_{\theta}(d_{i:n})$ via the ratio of uniform
codelength to online codelength.

$\displaystyle Compression=\frac{L_{uniform}}{L_{online}}$ (2)

where $L_{uniform}(y_{1:n}|f_{\theta}(d_{i:n})=n\log_{2}k$ and $L_{online}$ is
calculated by training the probe on increasing subsets of the dataset, and
thus measures quality of the probe relative to the number of training
examples. Better performance with less examples will result in a shorter
online codelength, and a higher compression, showing that $Y$ is more
extractable from $f_{\theta}(d_{i:n})$.

In this work, we probe for binary gender, where $Y$ = $\\{m,f\\}$ and
occupation, where $Y$ = $\\{lawyer,doctor,...\\}$

Extractability, as measured by MDL probing, is predictive of shortcutting;
when a model relies on a heuristic feature to solve a task, which has
sufficient correlation with the actual task to have high accuracy on the
training set, but is not the true task (Geirhos et al., 2020). Shortcutting
causes failure to generalise; a heuristic that worked on the training set due
to a spurious correlation will not work after a distributional shift: e.g.
relying on the word ‘not’ to predict negation may work for one dataset but not
all (Gururangan et al., 2018). Lovering et al. (2021) look at linguistic
information in MLM representations (such as subject verb agreement) which is
necessary for the task of grammaticality judgments, and find that spurious
features are relied on if they are very extractable. This is of particular
interest to retrievers, which depend on generalisation, but which are also
contrastively trained, which can encourage shortcutting (Robinson et al.,
2021).

Shortcutting is also often the cause of social biases. Orgad et al. (2022)
find that extractability of gender in language models is predictive of gender
bias in coreference resolution and biography classification. So when some
information, such as gender, is more extractable than other information, such
as anaphora resolution, the model is risk of using gender as a heuristic, if
the data supports this usage. And thus of both failing to generalise and of
propagating biases. For instance, for the case of Mary Somerville, if gender
is easier for a model to extract than profession, then a model might have
actually learnt to identify mathematicians via male, instead of via maths (the
true relationship), since it is both easier to learn and the error penalty on
that is small, as there are not many female mathematicians.

##  3 Methodology

We analyse the relationship between information in different model
representations, and their performance & fairness. This requires at minimum a
model, a probing dataset (with labels for information we want to probe for),
and a performance dataset. We need some of the performance dataset to have
gender metadata to calculate performance difference across gender demographics
(Fig 1) also called gender bias, or more precisely, allocational fairness.

###  3.1 Models

For the majority of our experiments, we compare our 25 MultiContriever models
to the 25 MultiBerts models (Sellam et al., 2022). We access the MultiBerts
via huggingface333e.g. https://huggingface.co/google/MultiBerts-seed_[SEED]
and train the contrievers via modifying the repository released in Izacard et
al. (2022). We use the same contrastive training data as Izacard et al.
(2022), to maximise comparability. This comprises a 50/50 mix of Wikipedia and
CCNet from 2019. As a result, five of the fourteen performance datasets
involve temporal generalisation, since they postdate both the MultiContriever
and the MultiBert training data. This most obviously affects the TREC-COVID
dataset (QA), though also four additional datasets: Touché-2020
(argumentation), SCIDOCS (citation prediction), and Climate-FEVER and Scifact
(fact-checking). Further details on contriever training and infrastructure are
in Appendix A.

We train 25 random seeds as both generalisation and bias vary greatly by
random seed initialisation (McCoy et al., 2020). MultiContrievers have no new
parameters, so the random seed affects only their data shuffle. The MultiBerts
each have a different random seed for both weight initialisation and data
shuffle.

###  3.2 Probing Datasets

We verify that results are not dataset specific, or the result of dataset
artifacts, by using two probing datasets. First the BiasinBios dataset (De-
Arteaga et al., 2019), which contains biographies from the web annotated with
labels of the subject’s binary gender (male, female) and biography topic
(lawyer, journalist, etc). We also use the Wikipedia dataset from md_gender
(Dinan et al., 2020), which contains Wikipedia pages about people, annotated
with binary gender labels.444This dataset does contain non-binary labels, but
there are few (0.003%, or 1̃80 examples out of 6 million). Uniform codelength
($dataset\\_size*log2(num\\_classes)$) affects information theoretic probing;
additional class with very few examples can significantly affect results. This
dataset was also noisier, making small data subsets less trustworthy. For
gender labels, BiasinBios is close to balanced, with 55% male and 45% female
labels, but Wikipedia is very imbalanced, with 85% male and 15% female. For
topic labels, BiasinBios has a long-tail zipfian distribution over 28
professions, with professor and physician together as a third of examples and
rapper and personal trainer as 0.7%. Examples from both datasets can be found
in Appendix B.

To verify the quality of each dataset’s labels, we manually annotated 20
random samples and compared to gold labels. BiasinBios agreement with our
labels was 100%, and Wikipedia’s was 88%.555We investigated other md_gender
datasets in the hope of replicating these results on a different domain such
as dialogue (e.g. Wizard of Wikipedia), but found the labels to be of
insufficiently high agreement to use. We focus on the higher quality
BiasinBios dataset for most of our graphs and analysis, though we replicate
all experiments on Wikipedia.

###  3.3 Evaluation Datasets and Metrics

We evaluate on the BEIR benchmark, which covers retrieval for seven different
tasks (fact-checking, citation prediction, duplicate question retrieval,
argument retrieval, question answering, bio-medical information retrieval, and
entity retrieval).666The BEIR benchmark itself contains two additional tasks,
tweet retrieval, and news retrieval, but these datasets are not publicly
available.. We initially analysed all standard metrics used in BEIR and TREC
(e.g. NDCG, Recall, MAP, MRR, @10 and @100). We observed similar trends across
all metrics, somewhat to our surprise, since many retrieval papers focus on
the superiority of a particular metric (Wang et al., 2013). We thus
predominantly report NDCG@10, but more metrics (NDCG@100, and Recall@100) are
included in Appendix G.

For allocational fairness evaluation, we create NQ-gender, a subset of Natural
Questions (NQ) about entities, annotated with male, female, and neutral (no
gender). Further details on annotation in Appendix C. We measure allocational
fairness as the difference between the female and male query performance. We
use the neutral/no gender entity queries as a control to make sure the system
performs normally on this type of query.

(a) Compression (gender)

(b) Compression (occupation)

(c) Compression (gender:occupation ratio)

Figure 2: Bert and Contriever compression for gender and occupation over all
seeds. Y-axes have different scales (gender is much larger); higher numbers
mean more extractability and more regular representations. Contriever has more
uniform compression across seeds, and a lower ratio of gender:occupation,
which means less shortcutting.

##  4 Results

We address our four research questions: how does extractability change (Q1),
how sensitive are retrievers to random initialisation (Q2), do changes in
extractability correlate with performance (Q3), and is it predictive of
allocational bias (Q4).

###  4.1 Q1: Information Extractability

(a) NQ

(b) MSMARCO

(c) NQ entities (gendered)

(d) NQ entities (control)

Figure 3: Scatterplots of the correlation between x-axis compression (ratio of
uniform to online codelength) and y-axis performance (NDCG@10), for different
datasets (NQ, MSMARCO) at left and entity subsets of NQ at right. Colours are
different seeds, and are held constant across graphs.

Both gender (Fig 2(a)) and occupation (Fig 2(b)) are more extractable in
MultiContrievers than MultiBerts. Gender compression ranges for
MultiContrievers are 4-12 points higher, or a 9-47% increase (depending on
seed initialisation), than the corresponding MultiBerts. Occupation
compression ranges are 1.7-2.12 points higher for MultiContrievers; as the
overall compression is much lower this is a 19-38% increase over MultiBerts.
Both graphs also show a regularisation effect; MultiBerts have a large range
of compression across random seeds, whereas MultiContrievers have similar
values.

Figure 2(c) shows that though MultiContrievers have higher extractability for
gender and occupation, the ratio between them decreases. So while
MultiContrievers do represent gender far more strongly than occupation, this
effect is lessened vs. MultiBerts, which means they should be slightly less
likely to shortcut based on gender.

###  4.2 Q2: Sensitivity to Random Initialisation

Figure 4: Distribution of performance (NDCG@10) for the 24 MultiContrievers,
per BEIR dataset, performance on x-axis, number of models with that value on
y-axis. Dashed line indicates reference performance from previous work. While
for some datasets the reference performance sits at or near the mean of the
MultiContriever distribution, for some the reference performance is an
outlier. There is sufficiently high variance that performance improvements
from random seed can exceed those from continuing to train on supervised fine-
tuning data.

We analysed the distribution of performance by dataset for 24 seeds, as both
generalisation and fairness are sensitive to initialisation in MLMs (Sellam et
al., 2022).777Seed 13 (ominously) is excluded from our analysis because of
extreme outlier behaviour, which was not reported in (Sellam et al., 2022). We
investigated this behaviour, and it is fascinating, but orthogonal to this
work, so we have excluded the seed from all analysis. Our investigation can be
found in Appendix E and should be of interest to researchers investigating
properties of good representations (e.g. anisotropy) and of random
initialisations. Figure 4 shows this data, broken out by dataset, with a
dashed line at previous reference performance (Izacard et al., 2022).

A few things are notable: first, there is a large range of benchmark
performance across seeds with for identical contrastive losses. During
training, MultiContrievers converge to the same accuracy (Appendix A) and
(usually) have the same aggregate BEIR performance reported in Izacard et al.
(2022). However, the range of scores per dataset is often quite large, and for
some datasets the original reference Contriever is in the tail of the
distribution: e.g in Climate-Fever (row 1 column 2) it performs much worse
than all 24 models. It is also worse than almost all models for Fiqa and
Arguana.888For Fiqa 19 models are up to 2.5 points better, for Arguana 20
models are up to 6.3 points better. Nothing changed between the different
MultiContrievers except the random seed for MultiBert initialisation, and the
random seed for the data shuffle for contrastive fine-tuning.999There are a
few small differences between the original released BERT, which Contriever was
trained on, and the MultiBerts, which we trained on, detailed in Sellam et al.
(2022). But not between our 25 MultiBerts.

Second, the potential increase in performance across random seeds can exceed
the increase in performance from training on supervised data (e.g. MSMARCO).
We see this effect for half the datasets in BEIR. The higher performing seeds
surpass the performance on all supervised models from Thakur et al.
(2021)101010The BEIR benchmark reports performance on all datasets for four
dense retrieval systems—DPR(Karpukhin et al., 2020), ANCE (Xiong et al.,
2021), TAS-B (Hofstätter et al., 2021), and GENQ (their own system)—which all
use supervision of some kind. DPR uses NQ and Trivia QA, as well as two
others, ANCE, GENQ, and TAS-B all use MSMARCO. Note that the original
Contriever underperformed these other models until supervision was added. on
three datasets (Fever, Scifact, and Scidocs) and surpass all but one model
(TAS-B) on Climate-fever. These datasets are the fact-checking and citation
prediction datasets in the benchmark, suggesting that even under mild task
shifts from supervision data (which is always QA), random initialisation can
have a greater effect than supervision. This effect exists across diverse non-
QA tasks; for four additional datasets the best random seeds are better than
all but one supervised model: this is true for Arguana and Touché
(argumentation), HotpotQA (multihop QA), and Quora (duplicate question
retrieval).

Figure 5: Ranking of best performing seed per dataset (one colour per seed).
For legibility, NDCG@10 values are scaled, and all seeds with middle
performance are not pictured (10 included). One seed is arbitrarily given a
star marker to aid visual interpretation.

Third, the best and worst model across the BEIR benchmark datasets is not
consistent (Figure 5); not only is the range large across seeds but the
ranking of each seed is very variable. The best model on average, seed 24, is
top-ranked on only one dataset, and the second-best average model, seed 2, is
best on no individual datasets. The best or worst model on any given dataset
is almost always the best or worst on only that dataset and none of the other
14. Sometimes, the best model on one dataset is worst on another, e.g. seed 4
is best on NQ and worst on FiQA, seed 5 is best on Scifact and worst on
Scidocs.111111This best-worst flip exists for seeds 8, 18, and 23 also.. Even
seed 10, which is the only model that is worst on more than 2 datasets (it is
worst on 6) is still best on TREC-Covid.121212This is to be taken with a grain
of salt - that dataset is interesting for generalisation (as these models are
trained on only pre-Covid data), but it is only 50 datapoints. We note also
that analysis on seed 13 revealed that seed 10 was also unusual, that analysis
can also be found in E.

Our results show that there is no single best retriever, which both supports
the motivation of the BEIR benchmark (to give a more well rounded view on
retriever performance via a combination of diverse datasets) and shows the
need for more analysis into random initialisation and shuffle.

As an addendum, we note that Sellam et al. (2022) did extensive experiments
with both random initialisation and data shuffle, and found initialisation to
matter more. We did our own experiments to this effect where we trained five
MultiContrievers from the same MultiBert initialisation with different data
shuffles, from the best, worst, and middle performing seeds. This additional
analysis is in Appendix D.

###  4.3 Q3: Correlation between Extractability & Performance

We tested for correlations across all datasets and common metrics, and present
a selection here (Fig 3). Neither NQ (Fig 3(a)) nor MSMARCO (Fig 3(b))
correlate with compression metrics. NQ and MSMARCO are the most widely used of
the BEIR benchmark datasets, and we hypothesised them to be most likely to
correlate. Both are search engine queries (from Google and Bing, respectively)
and contain queries that require occupation-type information (what is cabaret
music?, MSMARCO) and that require gender information (who is the first foreign
born first lady?, NQ). However, as the dispersed points on the scatterplots
show (Figures 3(a), 3(b)), neither piece of information correlates to
performance on either dataset. NQ and MSMARCO are representative; we include
plots for all datasets in Appendix F.

This result was somewhat surprising; since the contriever training both
regularises and increases extractability of gender and occupation, we might
expect this to be important for the task. But perhaps it is relevant for only
the contrastive objective, and not for the retrieval benchmark. Alternatively,
it is possible that this information is important, but only up to some
threshold that MultiContriever models exceed. Finally, it’s possible that this
information doesn’t matter for most queries in these datasets, and so there is
some correlation but it is lost, as these datasets are extremely large. This
is somewhat supported by the exception cases with correlations being smaller,
more curated datasets (F), and so we investigated this as the most tractable
to implement.

(a) NDCG@10 on the neutral vs. gendered NQ entity subsets. Representations are
raw (blue) vs. INLP (orange) with gender removed. INLP performance degrades on
only gender constrained queries: gender is used in those queries, but is not
in the control.

(b) Difference in performance between male (blue) and female (orange) entity
queries, for raw (left) and INLP (right). The performance gap is constant even
when gender is removed via INLP, remains; so the bias is not due to gender in
the representations.

Figure 6: INLP experiments

Our NQ-gender subset of gendered queries (§3.3) does show a strong correlation
between gender extractability and performance (Fig 3(c)). And the NQ-gender
subset of neutral non-gendered queries shows no correlation (Fig 3(d)). So we
find that if we isolate to a topical dataset, as e here, extractability is
predictive of performance, it just isn’t over a large diverse dataset.

We strengthen this analysis, testing whether gender information is necessary,
rather than simply correlated. We use Iterative Nullspace Projection (INLP)
(Ravfogel et al., 2020) to remove gender information from MultiContriever
representations. INLP learns a projection matrix $W$ onto the nullspace of a
gender classifier, which we apply before computing relevance scores between
corpus and query. So with INLP, the previous Equation 1 becomes:

$\displaystyle
s(d_{i},q_{i})=\textbf{W}f_{\theta}(q_{i})\cdot\textbf{W}f_{\theta}(d_{i})$
(3)

Then we calculate performance of retrieval with these genderless
representations. No drop in performance on the gendered query set with INLP
would mean extractable gender information was not necessary. A drop in
performance on both gender and control queries would support the ‘minimum
threshold’ explanation, or mean that the representation was sufficiently
degraded by the removal of gender that other functions were harmed.

Gender information post-INLP drops to 1.4 (nearly none, as 1 is no compression
over uniform, Eq 2). Performance on non-gendered entity queries is unaffected,
but performance on gendered entity questions drops significantly (5 points)
(Fig 6(a)). From these two experiments we conclude that the increased
information extractability was useful for answering specific questions that
require that information. But most queries in the available benchmarks simply
don’t require that information to answer them.

###  4.4 Q4: Gender Extractability and Allocational Gender Bias

Orgad et al. (2022) found gender extractability in representations to be
predictive of allocational gender bias for classification tasks; when gender
information was reduced or removed, bias also reduced.131313Orgad et al.
(2022) use a lexical method to remove gender, but we chose INLP as a more
elegant, extensible solution. We replicated their paper with INLP, showing
equivalence. We found that gender information is used (§4.3) so now we ask: is
it predictive of gender bias? At least for our dataset, it is not (Fig 6(b).
This graph shows that there is allocational bias between the female and male
queries, and also that the bias remains after we remove gender via INLP. All
performance drops, as we saw for the gendered entities in §4.3. But
performance drops by equivalent amounts for female and male entities. These
results diverge from what we expected based on the findings of Orgad et al.
(2022) for MLMs, who found gender in representations did matter. Our findings
suggest that in this case the gender bias comes from the retrieval corpus or
the queries, or from a combination. The corpus could have lower quality or
less informative articles about female entities (as was found for Wikipedia by
Sun and Peng (2021)), or queries about women could be structurally harder in
some way.

##  5 Discussion, Future Work, Conclusion

We trained a suite of 25 MultiContrievers, analysed their performance on the
BEIR benchmark, probed them for gender and occupation information, and removed
gender information from representations to analyse gender bias.

We showed performance to be extremely variable by random seed initialisation,
as was the performance ranking of different random initialisations across
datasets, despite equal losses during training. Best seed performances often
exceed the performance of more complex dense retrievers that use explicit
supervision. Future analysis of retriever loss basins to look for differing
generalisation strategies could be valuable (Juneja et al., 2023). Our results
show that a better understanding of initialisations may be more valuable than
developing new models. Our work also highlights the usefulness of metadata
enriched datasets for analysis, and we were limited by what was available.
Future work could create these datasets and then probe for additional targeted
information to learn more about retrievers. This would also enable analysis of
demographic biases beyond binary gender.

Gender and occupation extractability was not predictive of performance except
in subsets of queries that require gender information. Though both gender and
occupation increase in MultiContrievers, the ratio between them decreases, so
MultiContrievers should be less likely to shortcut based on gender compared to
MultiBerts. We established that the gender bias that we found was not caused
by the representations, as it persists when gender is removed. Future work
should test in a pipeline is best to correct bias, and how various parts
interact. This work also shows the utility of information removal (INLP,
others) for causality and interpretability, rather than just debiasing. More
availability of test sets for shortcutting could increase the scope of these
preliminary results.

Finally, we have analysed only the retriever component of a retrieval system.
In an eventual retrieval augmented generation task, the retrieval
representation will have to compete with language model priors. The generation
will be a composition between unconditionally probable text, and text attested
by the retrieved data. Future work could investigate the role of information
extractability in the full system, and how this bears on vital questions like
hallucination in retrieval augmented generation. We have done the first
information theoretic analysis of retrieval systems, and the first causal
analysis of the reasons for allocational gender bias in retrievers. We release
our code and resources for the community to expand and continue this line of
enquiry. This is particularly important in the current generative NLP
landscape, which is increasingly reliant on retrievers and where understanding
of models lags so far behind development.

##  6 Limitations

This work is limited by analysing only one architecture of dense retriever; we
chose to experiment instead with random initialisations and shuffles rather
than different architectures, so we focused on only the most popular one. So
these results may not generalise to all retriever architectures. Our analysis
covered only English, and there is work that shows that gender is encoded in a
more complex way in other languages (Gonen et al., 2022). INLP, the method we
used for causal analysis, is linear, so it might not even work beyond English,
though there are recent non-linear extensions of it (Iskander et al., 2023)
that could be used in future work.

## References

  * Barocas et al. (2019) Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019.  _Fairness and Machine Learning: Limitations and Opportunities_.  fairmlbook.org.  http://www.fairmlbook.org.
  * Belinkov and Glass (2019) Yonatan Belinkov and James Glass. 2019.  Analysis methods in neural language processing: A survey.  _Transactions of the Association for Computational Linguistics_ , 7:49–72.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In _NIPS_.
  * Campos et al. (2016) Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016.  Ms marco: A human generated machine reading comprehension dataset.  _NIPS_.
  * De-Arteaga et al. (2019) Maria De-Arteaga, Alexey Romanov, H. Wallach, J. Chayes, C. Borgs, A. Chouldechova, S. C. Geyik, K. Kenthapadi, and A. Kalai. 2019.  Bias in bios: A case study of semantic representation bias in a high-stakes setting.  _Proceedings of the Conference on Fairness, Accountability, and Transparency_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dinan et al. (2020) Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020.  Multi-dimensional gender bias classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 314–331, Online. Association for Computational Linguistics.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 55–65, Hong Kong, China. Association for Computational Linguistics.
  * Geirhos et al. (2020) Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11):665–673.
  * Gonen et al. (2022) Hila Gonen, Shauli Ravfogel, and Yoav Goldberg. 2022.  Analyzing gender representation in multilingual models.  In _Proceedings of the 7th Workshop on Representation Learning for NLP_ , pages 67–77, Dublin, Ireland. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hofstätter et al. (2021) Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy J. Lin, and Allan Hanbury. 2021.  Efficiently teaching an effective dense retriever with balanced topic aware sampling.  _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_.
  * Iskander et al. (2023) Shadi Iskander, Kira Radinsky, and Yonatan Belinkov. 2023.  Shielded representations: Protecting sensitive attributes through iterative gradient-based projection.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 5961–5977, Toronto, Canada. Association for Computational Linguistics.
  * Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022.  Unsupervised dense information retrieval with contrastive learning.  _Transactions on Machine Learning Research_.
  * Juneja et al. (2023) Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. 2023.  Linear connectivity reveals generalization strategies.  In _The Eleventh International Conference on Learning Representations_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Klasnja et al. (2022) Anja Klasnja, Negar Arabzadeh, Mahbod Mehrvarz, and Ebrahim Bagheri. 2022.  On the characteristics of ranking-based gender bias measures.  In _Proceedings of the 14th ACM Web Science Conference 2022_ , WebSci ’22, page 245–249, New York, NY, USA. Association for Computing Machinery.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lewis et al. (2020a) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020a.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Lewis et al. (2020b) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Lovering et al. (2021) Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. 2021.  Predicting inductive biases of pre-trained models.  In _International Conference on Learning Representations_.
  * McCoy et al. (2020) R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2020.  BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 217–227, Online. Association for Computational Linguistics.
  * Orgad et al. (2022) Hadas Orgad, Seraphina Goldfarb-Tarrant, and Yonatan Belinkov. 2022.  How gender debiasing affects internal model representations, and why it matters.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2602–2628, Seattle, United States. Association for Computational Linguistics.
  * Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020.  Null it out: Guarding protected attributes by iterative nullspace projection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7237–7256, Online. Association for Computational Linguistics.
  * Rekabsaz and Schedl (2020) Navid Rekabsaz and Markus Schedl. 2020.  Do neural ranking models intensify gender bias?  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2065–2068.
  * Robinson et al. (2021) Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. 2021.  Can contrastive learning avoid shortcut solutions?  _Advances in neural information processing systems_ , 34:4974–4986.
  * Sellam et al. (2022) Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander Nicholas D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick, editors. 2022.  _The MultiBERTs: BERT Reproductions for Robustness Analysis_.
  * Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.  Retrieval augmentation reduces hallucination in conversation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3784–3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sun and Peng (2021) Jiao Sun and Nanyun Peng. 2021.  Men are elected, women are married: Events gender bias on Wikipedia.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 350–360, Online. Association for Computational Linguistics.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Voita and Titov (2020) Elena Voita and Ivan Titov. 2020.  Information-theoretic probing with minimum description length.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 183–196, Online. Association for Computational Linguistics.
  * Wachsmuth et al. (2018) Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018.  Retrieval of the best counterargument without prior topic knowledge.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 241–251, Melbourne, Australia. Association for Computational Linguistics.
  * Wang et al. (2013) Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie-Yan Liu. 2013.  A theoretical analysis of ndcg type ranking measures.  In _Proceedings of the 26th Annual Conference on Learning Theory_ , volume 30 of _Proceedings of Machine Learning Research_ , pages 25–54, Princeton, NJ, USA. PMLR.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  In _International Conference on Learning Representations_.

##  Appendix A Contriever Training

Each MultiContriever model was initialised from a MultiBert checkpoint for
each of the 25 seeds from 0 - 24, accessed at
https://huggingface.co/google/multiberts-seed_X where X is an integer from 0 -
24. NB: MultiBerts released many checkpoints to enable study of training
dynamics, we use only the final complete checkpoint.

Hyperparameters and training regime is exactly matched to the original
Contriever work of (Izacard et al., 2022). Hyperparams can be found in Table
1. Data used was identical to in (Izacard et al., 2022) (from 2019) and was a
50/50 CCNet Wikipedia split.

Each MultiContriever was trained across 4 nodes with 8 GPUs per node (32 GPUs
total) for on average 2.5 days. Each MultiContriever was trained for the full
500,000 steps, and checkpointed often; but in all but one seed the best
performing checkpoint was the final one (so for that one we use the model at
450,000 steps). This is excepting seed 13, which was anomalous in many other
ways (see E).

All MultiContrievers have similar loss and accuracy curves, with seeds 12 and
13 excerpted in Figure 7. All models steeply increase accuracy/decrease loss
within 10,000 steps, and then asymptotically approach 69% accuracy by 50,000
steps.

sampling coefficient | 0
---|---
pooling | average
augmentation | delete
probability_augmentation | 0.1
momentum | 0.9995
temperature | 0.05
queue_size | 131072
chunk_length | 256
warmup_steps | 20000
total_steps | 500000
learning_rate | 0.00005
scheduler | linear
optimizer | adamw
batch_size (per gpu) | 64
Table 1: Hyperparameters used for training MultiContrievers.

(a) Loss (12)

(b) Train Accuracy (12)

(c) Loss (13)

(d) Train Accuracy (13)

Figure 7: Loss and accuracy for seeds 12 and 13, steps on x-axis and loss or
accuracy on y-axis.

##  Appendix B Probing Datasets

We rely on two datasets. The first is BiasinBios (De-Arteaga et al., 2019),
which is a dataset of web biographies labelled with binary gender, and
biography profession. We use De-Arteaga et al. (2019)’s train/dev/test splits
of 65:10:25, yielding 255,710 train 39,369 dev, and 98,344 test datapoints.
Second is the Wikipedia slice of the md_gender dataset (Dinan et al., 2020).
This has only labels for gender, which we restrict to be binary since non-
binary gender is so small and would adversely affect this analysis. We filter
out texts below 10 words (words, not tokens) leaving a dataset of size
10,681,700, split 65:10:25 into 6,943,105 train, 934,649 dev, 2,803,946 test.
For practical reasons, we shard it to 9 shards (650,000 train examples each)
and then check the results on each shard. All shards behaved consistently. As
noted in the text, BiasinBios is nearly balanced with regard to gender labels,
but Wikipedia is severely imbalanced.

For both datasets, we use the train set for probing, and the test set for
measuring accuracy on the final probe. We investigated using other datasets,
but none were of sufficient quality that they were usable. We tested usability
very simply: each of the authors labelled a different random sample of 20
examples by hand, and we measured accuracy of dataset labels against our
labels, and only took datasets with over 80% accuracy, since our probing task
is sensitive to errors in labelling. No other subsets of md_gender nor
external datasets that we surveyed passed this bar. We didn’t multiply
annotate as we found no examples to be at ambiguous.

##  Appendix C Annotation of NQ gender subset

To do our experiments we create a subset of Natural Questions, NQ-gender.

We subsample Natural Questions to entity queries by filtering automatically
for queries containing any of who, whose, whom, person, name. We similarly
filter this set into gendered entity queries by using a modified subset of
gender terms from Bolukbasi et al. (2016). From this we get a set of queries
that is just about entities Who was the first prime minister of Finland?, and
gendered entities (a female query is Who was the first female prime minister
of Finland? and a male query is Who was the first male prime minister of
Finland?).

This automatic process is low precision/high recall. It captures queries with
gendered terms in prepositional phrases, (Who starred in O Brother Where Art
Thou?) which are common false positives in QA datasets, as they are not about
brothers. So we manually filter these results by annotating with two criteria:
gender of the subject (male, female, or neutral/none (in cases where the
gender term was actually in a title or other prepositional phrase as in the
example), and a binary tag with whether the query actually constrains the
gender of the answer. This second annotation is somewhat subtle, but very
important. For example, in our dataset there is the query Who was the actress
that played Bee, which contains a gendered word (actress) but it is not
necessary to answer the question; all actors that played Bee are female, and
the question could be as easily answered in the form Who played Bee?. Whereas
in another example query, Who plays the sister in Home Alone 3? the query does
constrain the gender of the answer. We annotated 816 queries with both of
these attributes, of which 51% have a gender constraint, with a gender
breakdown of 59% female and 41% male.

We do this annotation ourselves (two of the authors), and we throw out
examples that we don’t agree on. We are not a representative sample of people
(we are all NLP researchers after all) but we consider this lack of diversity
to be acceptable since we are not making subjective judgments but are just
providing metadata labels.

It is also worth mentioning that two very different types of gender bias in
retriever works do create artifacts also, but they are unsuitable for our type
of analysis for the following reasons. Rekabsaz and Schedl (2020) and Klasnja
et al. (2022) release subsets of MSMARCO, which we did examine and use in
initial tests early in this work. Those works define bias very differently, as
the genderedness of retrieved documents based on lexical terms, making the
implicit normative statement that lack of bias means equal representation of
male and female documents in non-gendered queries. This is essentially an
independence assertion from fairness literature (Barocas et al., 2019). This
is quite different to our approach, which looks at performance disparity
between queries that require male and female gender information to answer. Our
approach has more immediate practical utility for a real world retriever, and
also ties in to the work on information theory by restricting to queries that
require gender information. So the lexical document based approach cannot be
adapted for our purpose.

##  Appendix D Data Shuffle Experiments

We wanted to answer the question of If you begin from a worse random
initialisation, can you fix it via data shuffle?. This is of significant
practical utility to researchers, who often cannot retrain an existing model
from scratch before adapting it to their purpose. Figure 8 shows the best,
worst, and a middle performing seed with five additional different data
shuffles, and the variance in performance over the datasets. We can see that
the worst performing seed is characterised by high variability overall, and
the best seed by low variability. So the overall picture is that, on average,
the different initialisations determine the quality of the retriever more than
the data shuffle. This is in agreement with the findings of Sellam et al.
(2022) for MLMs. However, variability is sufficiently high enough that you
could get lucky and get the best performance from varying the shuffle, if that
is the option available. It would be valuable to extend these to explicit
generalisation tasks and interpretabilty challenge sets to see if the high
performing shuffles of very variable seeds can be trusted in all settings.

Figure 8: Performance for 5 random datashuffles for a fixed MultiBERT seed -
the worst, the best, and a middling seed based on previous experiments. This
answers the question of how much variance comes from the random initialisation
of parameters, and how much from the data shuffle. It also answers the
practical question of ‘if you are fine-tuning one model, are you doomed based
on the state of the initial model?’ The answer is, sort of, but not entirely.

##  Appendix E Seed 13

MultiContrievers were trained with seeds 0-24 based on respective MultiBerts
0-24. Seed 13 was excluded from all analysis as it displayed repeatedly
anomalous behaviour. During the course of contriever training it appeared
indistinguishable from other seeds, loss curves looked normal, there were no
signs of overfitting. Performance converged to the same level as other
MultiContrievers. However, when applied to the datasets of the BEIR benchmark
it did not perform at all, with NDCG of between 2 and 20 on each dataset. We
retrained once to replicate the behaviour, and then twice more with different
seeds for data shuffle, with identical results. We thus exclude it from all
analysis. To aid in future investigations we include our initial analysis of
seed 13 irregularities here. We follow the method of analysis of
representation spaces from Ethayarajh (2019). We measure the L2 norm of all
representations in the BiasinBios dataset (272k) as well as average self-
similarity of 1000 randomly sampled representations of those bigraphies, as
measured by cosine similarity and by dot product. The former answers the
question of how much volume the representations occupy, the latter describes
the vector space via how conical (anisotropic) or spherical it is.

In Figure 9, we observe that the vector space of MultiContriever 13 is both
larger volume and more obtusely anisotropic (i.e. it occupies a wider cone)
than other MultiContrievers. The more obtuse anisotropy originates from
MultiBert 13, as can be seen in the high variances for both seeds in cosine
similarity. But the larger relative volume happens during the training of the
MultiContriever and is unique to it. For MultiBert 13, L2 norm is within
normal range, and the anomalous seeds are seeds 10 and 23, which both have
larger norms and 5x the variance of other seeds. MultiContriever 13, however,
has 1.5x the average norms of all other seeds (which have regularised and
become closer in values) and 6x the variance of others. Both MultiBert 13 and
MultiContriever 13 have very high variance to average cosine similarity, where
the effective range of MultiContriever 13 is -0.03 to 0.53, and MultiBert 13
is 0.02 to 0.58, as compared to other models have a range of 0.28-0.32, for
both types of models.

We hypothesise that this reveals a limitation of reliance on the dot product
for retrieval, any operation reliant on the dot product loses information when
there is a chance of a cosine similarity of zero. We leave other investigation
– such as why this would persist from a difference of only random seed
initialisation, or why this issue would appear in retrieval, but not in any
tasks in the MultiBerts paper, or in the contrastive training process – to
future work.

We also note that seed 10 was anomalous in performance compared to the other
seeds on the BEIR benchmark; not so anomalous as to be excluded, but it was
reliably performing poorly. We can see the higher variance in L2 norms for 10
and 23 in MultiBerts, and then for 10 still in MultiContriever (though nothing
noticeable in cosine similarity). Seeds 10 and 13 were not found to be
anomalous by Sellam et al. (2022), but they did find seed 23 to display
strange behaviour and be extremely unbiased (or even anti-biased) on the
Winogender benchmark.

We hope that future work will use our models and continue this line of
analysis.

(a) MultiBerts (mean)

(b) MultiBerts (var)

(c) MultiContriever (mean)

(d) MultiContriever (var)

(e) MultiBerts (mean)

(f) MultiBerts (var)

(g) MultiContriever (mean)

(h) MultiContriever (var)

Figure 9: Top row: mean and var of L2 norms of the full BiasinBios dataset for
all MultiBert and MultiContriever seeds. Bottom row: mean and var cosine
similarity between 1000 random biographies.

##  Appendix F Full set of results for correlation between extractability and
performance

Full set of correlations between gender compression and performance in Figure
10 and between profession compression and performance in Figure 11. The latter
(profession correlation) have misleading regression lines as only three of 24
models had large differences in compression, such that the line is based off
insufficient datapoints. It is included for completeness but left out of
analysis for that reason. Gender compression numbers (Figure 10) are
distributed more evenly. There are four statistically significant correlations
(referred to as by row 1-4, and column a-d, such that the upper left cell is
1a and the lower right cell is 4d). Arguana (1a), Scifact (2b), Webis-Touche
(3a), and NQ (4b). All have middling correlation coefficients: Arguana -0.41,
Scifact 0.41, Webis-Touche 0.31, NQ 0.42. There is also little in common
between these datasets, Arguana and Webis-Touche are argumentation, Scifact is
fact-checking, and NQ is google-search style questions. As this leaves most
datasets with no correlations, we consider the correlation overall to be weak.
We do note that the temporal generalisation datasets are overrepresented in
this set (Webis-Touche and Scifact), but leave an investigation of that for
future work.

Arguana in particular is unique in having a significant negative correlation.
We have no answers as to why this might be. It may be a fluke due to
peculiarities of this dataset: the dataset is small (less thank 2k
datapoints), and is not structured in the same way with query (input) and
passage (retrieved) but instead uses a full document passage as the query. It
is unclear why this might cause a deterioration in performance from better
gender or profession encoding (as we observe the same in profession
compression). The Arguana task should match the unsupervised training much
more closely since they both are matching the relevance of to document chunks.
We leave an investigation into the peculiarities of that dataset also to
future work.

Figure 10: Full set of scatterplots of the correlation between x-axis gender
compression (ratio of uniform to online codelength) and y-axis performance
(NDCG@10), for all datasets individually, and for the average of all BEIR
datasets (lower-right). Shaded region is 95% confidence interval. Figure 11:
Full set of scatterplots of the correlation between x-axis profession
compression (ratio of uniform to online codelength) and y-axis performance
(NDCG@10), for all datasets individually, and for the average of all BEIR
datasets (lower-right). Shaded region is 95% confidence interval.

##  Appendix G Additional metrics