# From spherical to periodic symmetry:
the analog of orbital angular momentum for semiconductor crystals

Monique Combescot1, and Shiue-Yuan Shiau2 (1) Institut des NanoSciences de
Paris, Sorbonne Université, CNRS, 4 place Jussieu, 75005 Paris  (2) Physics
Division, National Center for Theoretical Sciences, Taipei, 10617, Taiwan

###### Abstract

The angular momentum formalism provides a powerful way to classify atomic
states. Yet, requiring a spherical symmetry from the very first line, this
formalism cannot be used for periodic systems, even though cubic semiconductor
states are commonly classified according to atomic notations. Although never
noted, it is possible to define the analog of the orbital angular momentum, by
only using the potential felt by the electrons. The spin-orbit interaction for
crystals then takes the $\mathbfcal{\hat{L}}\cdot\hat{{\bf S}}$ form, with
$\mathbfcal{\hat{L}}$ reducing to $\hat{{\bf L}}={\bf r}\times\hat{{\bf p}}$
for spherical symmetry. This provides the long-missed support for using the
eigenvalues of $\mathbfcal{\hat{L}}$ and
$\mathbfcal{\hat{J}}=\mathbfcal{\hat{L}}+\hat{{\bf S}}$, as quantum indices to
label cubic semiconductor states. Importantly, these quantum indices also
control the phase factor that relates valence electron to hole operators, in
the same way as particle to antiparticle, in spite of the fact that the hole
is definitely not the valence-electron antiparticle. Being associated with a
broader definition, the ($\mathbfcal{\hat{L}},\mathbfcal{\hat{J}}$) analogs of
the $(\hat{{\bf L}},\hat{{\bf J}})$ angular momenta, must be distinguished by
names: we suggest “spatial momentum” for $\mathbfcal{\hat{L}}$ that acts in
the real space, and “hybrid momentum” for $\mathbfcal{\hat{J}}$ that also acts
on spin, the potential symmetry being specified as “cubic spatial momentum”.
This would cast $\hat{{\bf J}}$ as a “spherical hybrid momentum”, a bit
awkward for the concept is novel.

It is well establishedLandau01 ; Sakurai ; Cohenphotonatom ; Pethick  that the
state classification for electrons in a spherical potential, i.e., a potential
that only depends on the modulus of the electron coordinate ${\bf r}$, is
driven by the orbital angular momentum operator

$\hat{{\bf L}}={\bf r}\times\hat{{\bf p}}$ (1)

where $\hat{{\bf p}}=-i\hbar\nabla$ is the electron momentum operator. This
operator controls the interplay between the spin and the spatial degree of
freedom of atomic electrons, via the spin-orbit interactionLS  that for a
spherical potential, takes the simple $\hat{{\bf L}}\cdot\hat{{\bf S}}$ form.
This scalar product is handled by introducing the angular momentum operator
$\hat{{\bf J}}=\hat{{\bf L}}+\hat{{\bf S}}$. Since $\hat{{\bf
L}}\cdot\hat{{\bf S}}$ is equal to $(\hat{{\bf J}}^{2}-\hat{{\bf
L}}^{2}-\hat{{\bf S}}^{2})/2$, while $\hat{{\bf L}}^{2}$ and $\hat{{\bf
S}}^{2}$ commute with the atomic Hamiltonian, the derivation of the spin-orbit
eigenstates just amounts to deriving the $\hat{{\bf J}}^{2}$ eigenstates,
which is easy to do with the help of the angular momentum formalism. Although
mathematically smart, the concept of angular momentum is physically odd
because $\hat{{\bf J}}$ is the sum of two operators that act in different
subspaces. Yet, the angular momentum formalism provides a very elegant way to
derive the degeneracy of the various atomic states.

While $\hat{{\bf L}}$ commutes with the Hamiltonian for atomic electrons in a
spherical potential, this is not so for semiconductor electrons in a periodic
potentialVAKu ; so, the orbital angular momentum ${\bf L}$ would not be a
conserved quantity. Moreover, this quantity would be infinite in the large
sample limit: indeed, semiconductor electrons have a wave function that
extends over the whole sampleKittelbook ; Merminbook . The mean value of the
${\bf r}$ coordinate for such “itinerant” electron goes to infinity, whereas
it stays finite for a bound electron orbiting around a nucleus, whatever the
sample size.

Still, valence electron states in a Zinc-Blende-like cubic crystal are
commonly labeled along the atomic notations, which is fully questionable from
the very first line because these notations are related to problems having a
spherical symmetry. The valence electrons of such crystals have a threefold
spatial degeneracyCardona , labeled along the crystal axes $(x,y,z)$. Like
atomic electrons, they suffer a spin-orbit interaction that split this spatial
degeneracyDresselhaus1955 ; KL ; Luttinger ; Warren  in a way that should be
similar with how the threefold $\ell=1$ atomic level splits, because the
$(x,y,z)$ cyclic permutations fundamentally corresponds to a spherical
symmetry “restricted to” a $\pi/2$ rotation. Consequently, even if
semiconductor electrons do not have an orbital angular momentum due to the
lack of spherical symmetry, they should have a vector operator that plays a
similar role.

The goal of the present work is to identify the vector operator for periodic
potentials that is conceptually similar to the orbital angular momentum
operator for a spherical potential, and to possibly classify the energy levels
of semiconductor electrons without borrowing atomic notations that rely on a
concept invalid for a periodic system. To do it, we go back to the foundation
of the spin-orbit interaction, in order to follow the spirit of the procedure
that leads to the classification of atomic states.

The general form of the spin-orbit interaction for whatever the system
symmetry, readsThomas

$\hat{H}_{so}=\lambda_{so}\Big{(}\nabla\mathcal{V}({\bf r})\times\hat{{\bf
p}}\Big{)}\cdot\hat{{\bf S}}$ (2)

where $\lambda_{so}$ is a constant and $\mathcal{V}({\bf r})$ is the internal
electrostatic potential felt by the electrons. This interaction splits into an
operator $(\nabla\mathcal{V}({\bf r})\times\hat{{\bf p}})$ that only acts in
the real space, and $\hat{{\bf S}}$ that only acts in the spin space.

For a spherical potential, as in the case of atomic electrons, the operator
that acts on space reduces to

$\displaystyle\hat{\bf\Lambda}^{(at)}$ $\displaystyle=$
$\displaystyle\nabla\mathcal{V}^{(at)}({\bf r})\times\hat{{\bf
p}}=\left(\frac{d}{dr}\mathcal{V}^{(at)}(r)\,\,\,\frac{{\bf
r}}{r}\right)\times\hat{{\bf p}}$ $\displaystyle=$
$\displaystyle\lambda_{at}(r)\,\hat{{\bf L}}$

from which the usual $\hat{{\bf L}}\cdot\hat{{\bf S}}$ form for the spin-orbit
interaction follows readily.

When the potential is periodic, as in the case of crystals, we can, by
analogy, introduce the vector operator $\hat{\mathbfcal{L}}$ defined
throughMonicPRB2019

$\hat{\bf\Lambda}=\nabla\mathcal{V}({\bf r})\times\hat{{\bf
p}}\equiv\lambda(\textbf{r})\,\hat{\mathbfcal{L}}$ (4)

for the spin-orbit interaction to take a similar
$\hat{\mathbfcal{L}}\cdot\hat{{\bf S}}$ form. The $\lambda(\textbf{r})$ scalar
follows from enforcing $\hat{\mathbfcal{L}}$ to have the same dimensionality
as $\hat{{\bf L}}$, that is, eigenvalues that scale as $\hbar$.

The vector operator $\hat{\mathbfcal{L}}$ reduces to $\hat{{\bf L}}={\bf
r}\times\hat{{\bf p}}$ when the $\mathcal{V}({\bf r})$ potential has a
spherical symmetry; but it is fundamentally different in the case of an
itinerant electron in a periodic crystal; such electron does not orbit around
a particular nucleus; so, its position can not be described by a distance $r$
plus two angles $(\theta,\varphi)$. Still, we will show that, in the case of a
cubic crystal, the matrix representation of the corresponding
$\hat{\mathbfcal{L}}^{(cb)}$ operator in the threefold basis $(x,y,z)$ taken
along the crystal axes, is identical to the one of the orbital angular
momentum $\hat{{\bf L}}$ in the threefold basis of the $\ell=1$ atomic level,
provided that the $(X,Y,Z)$ atomic axes, that are arbitrary for spherical
symmetry, are chosen along the cubic crystal axes. As a direct consequence,
the spin-orbit interaction $\hat{\mathbfcal{L}}^{(cb)}\cdot\hat{{\bf S}}$ must
split the threefold spatial level of electrons in a cubic crystal, in exactly
the same way as $\hat{{\bf L}}\cdot\hat{{\bf S}}$ does for the $\ell=1$ atomic
level: we just have to introduce the vector operator
$\hat{\mathbfcal{J}}^{(cb)}=\hat{\mathbfcal{L}}^{(cb)}+\hat{{\bf S}}$; its
eigenstates will give the spin-orbit eigenstates for cubic semiconductors,
just as the angular momentum $\hat{{\bf J}}$ resolves the spin-orbit
eigenstates of atomic electrons.

Yet, the difference that exists between $\hat{\mathbfcal{L}}$ defined in Eq.
(4) for whatever potential symmetry, and $\hat{{\bf L}}$ defined in Eq. (1),
calls for different names. The vector $\hat{{\bf L}}$ has been named “orbital
angular momentum” because the atomic electron is best described by two angles
and a distance with respect to the nucleus around which it orbits. It is clear
that neither orbital nor angular can be used to qualify $\hat{\mathbfcal{L}}$
because these words are too closely related to a spherical symmetry. We are
left with what the operator $\hat{\mathbfcal{L}}$ is: it acts in the real
space. This is why we suggest to call it “spatial momentum”. By adding the
spin to get $\hat{\mathbfcal{J}}=\hat{\mathbfcal{L}}+\hat{{\bf S}}$, we
construct a hybrid operator that acts in two subspaces. We suggest to call it
“hybrid momentum”. Of course, these two names encompass $\hat{{\bf L}}$ and
$\hat{{\bf J}}$, just as when a broader concept is introduced in physics.
Still, the precise matrix forms of these operators depend on the symmetry of
the problem. This is why it is necessary to add this symmetry to their name:
$\hat{\mathbfcal{L}}^{(cb)}$ for cubic crystals should be named “cubic spatial
momentum”, while the “orbital angular momentum” $\hat{{\bf J}}$ is a
“spherical spatial momentum” within this broader understanding. Note that
since $\hat{\textbf{S}}$ has a spherical symmetry, the operators
$\hat{\mathbfcal{L}}$ and $\hat{\mathbfcal{J}}$ have the same symmetry.

The paper is organized as follows.

In Section I, we reconsider the spin-orbit interaction in the case of atoms,
to settle the procedure.

In Section II, we consider the $\hat{\mathbfcal{L}}$ operator defined in Eq.
(4), and we calculate its matrix representation in the threefold valence-
electron subspace when the periodic potential has a cubic symmetry. We show,
by using the spatial basis $|v,\ell_{z}\rangle$ with $\ell_{z}=(\pm 1,0)$
instead of $|v,\mu\rangle$ with $\mu=(x,y,z)$, that the
$\hat{\mathbfcal{L}}^{(cb)}$ operator has exactly the same matrix elements as
the orbital angular momentum $\hat{{\bf L}}$ in the $\ell=1$ subspace. As a
direct consequence, the matrix elements of the “cubic hybrid momentum”
$\hat{\mathbfcal{J}}^{(cb)}=\hat{\mathbfcal{L}}^{(cb)}+\hat{{\bf S}}$ in the
sixfold spin-orbit eigenstate subspace are just the same as the ones of the
angular momentum $\hat{{\bf J}}$ in the six $j=(3/2,1/2)$ atomic states. As a
result, the operators $\hat{\mathbfcal{J}}^{(cb)}$ and $\hat{{\bf J}}$ have
the same eigenvalues and their eigenstates have the same forms. This brings
the long-missed support for using atomic notations for cubic semiconductor.

In Section III, we show that although the valence hole is for sure not a naïve
antiparticle, the relation between the destruction operator of a particle and
the creation operator of its antiparticleLandau01 , established in
relativistic quantum theoryBerestetskii , stays valid for valence electron and
hole in a cubic semiconductor, on condition that we label the valence
electrons along quantum indices $\mathcal{L}$ or $\mathcal{J}$ that correspond
to the $\hat{\mathbfcal{L}}^{(cb)}$ or $\hat{\mathbfcal{J}}^{(cb)}$
eigenvalues. This provides a secure way for turning from valence electron
operator to hole operator when writing the semiconductor physics in terms of
electrons and holes, as it should for problems dealing with semiconductor
excitations.

In Section IV, we discuss the possibility to extend the $\hat{\mathbfcal{L}}$
concept defined in Eq. (4), that we here use for electrons in a cubic
semiconductor, to crystals having a different symmetry. This should provide a
smart way to derive the spin-orbit eigenstates for semiconductor crystals,
without relying on the physically obscure group theory formalism.

We then conclude.

##  I Spin-orbit interaction for atoms

###  I.1 Angular momentum formalism

We consider an electron in the $|\ell,\ell_{Z}\rangle$ eigenstate of the
operators $\hat{{\bf L}}^{2}$ and $\hat{L}_{Z}$, with $\ell=1$ and
$\ell_{Z}=(\pm 1,0)$ for a threefold orbital level, the quantization axis $\bf
Z$ being chosen at will due to spherical symmetry. The eigenstates of the
operators $\hat{{\bf J}}^{2}$ and $\hat{J}_{Z}$, with $\hat{{\bf J}}=\hat{{\bf
L}}+\hat{{\bf S}}$, are $|j,j_{Z}\rangle$ with $j=\ell\pm 1/2$ and
$j_{Z}=(j,j-1,\cdots,-j)$. From $\hat{{\bf
J}}^{2}|j,j_{Z}\rangle=\hbar^{2}j(j+1)|j,j_{Z}\rangle$ and similar relations
for $\hat{{\bf L}}^{2}$ and $\hat{{\bf S}}^{2}$, we readily find that
$\hat{{\bf L}}\cdot\hat{{\bf S}}=(\hat{{\bf J}}^{2}-\hat{{\bf
L}}^{2}-\hat{{\bf S}}^{2})/2$ acting on the state
$\left|j=3/2,j_{Z}\right\rangle$ made of $\ell=1$ orbital states and $s=1/2$
spin states, reads as

$\displaystyle\hat{{\bf L}}\cdot\hat{{\bf
S}}\left|j=\frac{3}{2},j_{Z}\right\rangle$ $\displaystyle=$
$\displaystyle\frac{\hbar^{2}}{2}\left(\frac{3}{2}\cdot\frac{5}{2}-1\cdot
2-\frac{1}{2}\cdot\frac{3}{2}\right)\left|\frac{3}{2},j_{Z}\right\rangle$ (5)
$\displaystyle=$
$\displaystyle\frac{\hbar^{2}}{2}\left|\frac{3}{2},j_{Z}\right\rangle$

In the same way, $\hat{{\bf L}}\cdot\hat{{\bf
S}}\left|j=1/2,j_{Z}\right\rangle=-\hbar^{2}\left|1/2,j_{Z}\right\rangle$.
Since the spin-orbit interaction for a spherical potential is proportional to
$\hat{{\bf L}}\cdot\hat{{\bf S}}$, the above results show that this
interaction splits the $(3\times 2)$-fold atomic level $(\ell_{Z},s_{Z})$ into
a fourfold level $(j=3/2,j_{Z})$ and a twofold level $(j=1/2,j_{Z})$.

###  I.2 Matrix representations

An easy way to relate atomic operator to cubic semiconductor operator is
through their matrix representations. To possibly do it, let us recall the
matrix forms of $\hat{{\bf L}}$ and $\hat{{\bf J}}$.

(a) Orbital angular momentum $\hat{{\bf L}}$

The basis for the threefold atomic level $\ell=1$ corresponds to
$|\ell_{Z}\rangle$ with $\ell_{Z}=(\pm 1,0)$. The angular momentum formalism
tells that the operators $\hat{L}_{Z}$ and $\hat{L}_{\pm}=\hat{L}_{X}\pm
i\hat{L}_{Y}$ are such that
$\hat{L}_{Z}|\ell_{Z}\rangle=\hbar\ell_{Z}|\ell_{Z}\rangle$ and
$\hat{L}_{\pm}|\ell_{Z}\rangle=\hbar\sqrt{1\cdot 2-\ell_{Z}(\ell_{Z}\pm
1)}\,\,|\ell_{Z}\pm 1\rangle$. This gives the matrices representing these
operators in the $\ell_{Z}=(1_{Z},0_{Z},-1_{Z})$ basis, as

$\hat{L}_{Z}=\hbar\left(\begin{matrix}1&0&0\\\ 0&0&0\\\
0&0&-1\end{matrix}\right)_{1;Z}\quad\hat{L}_{+}=\hat{L}^{\dagger}_{-}=\hbar\sqrt{2}\left(\begin{matrix}0&1&0\\\
0&0&1\\\ 0&0&0\end{matrix}\right)_{1;Z}$ (6)

from which we get

$\hat{L}_{X}=\frac{\hbar}{\sqrt{2}}\left(\begin{matrix}0&1&0\\\ 1&0&1\\\
0&1&0\end{matrix}\right)_{1;Z}\quad\hat{L}_{Y}=\frac{\hbar}{\sqrt{2}}\left(\begin{matrix}0&-i&0\\\
i&0&-i\\\ 0&i&0\end{matrix}\right)_{1;Z}$ (7)

We can check that $\left[\hat{L}_{X},\hat{L}_{Y}\right]_{-}=i\hbar\hat{L}_{Z}$
and $\hat{{\bf L}}^{2}=\hat{L}_{X}^{2}+\hat{L}_{Y}^{2}+\hat{L}_{Z}^{2}=(1\cdot
2)\hbar^{2}\,\hat{\textrm{I}}^{(3)}$, where $\hat{\textrm{I}}^{(3)}$ is the
$3\times 3$ identity matrix.

(b) Angular momentum $\hat{{\bf J}}$

The angular momentum formalism gives the $\hat{{\bf J}}$ matrices in the
$j=3/2$ eigenstate basis $|j_{Z}\rangle$ with $j_{Z}=(\pm 3/2,\pm 1/2)$,
through $\hat{J}_{Z}|j_{Z}\rangle=\hbar j_{Z}|j_{Z}\rangle$ and
$\hat{J}_{\pm}|j_{Z}\rangle=\hbar\sqrt{3/2\cdot 5/2-j_{Z}(j_{Z}\pm
1)}\,\,|j_{Z}\pm 1\rangle$. We then find in the
$j_{Z}=(3/2_{Z},1/2_{Z},-1/2_{Z},-3/2_{Z})$ basis

$\hat{J}_{Z}{=}\hbar\left(\begin{matrix}\frac{3}{2}&0&0&0\\\
0&\frac{1}{2}&0&0\\\ 0&0&{-}\frac{1}{2}&0\\\
0&0&0&{-}\frac{3}{2}\end{matrix}\right)_{\frac{3}{2};Z}\,\,\hat{J}_{+}{=}\hat{J}_{-}^{\dagger}{=}\hbar\left(\begin{matrix}0&\sqrt{3}&0&0\\\
0&0&2&0\\\ 0&0&0&\sqrt{3}\\\ 0&0&0&0\end{matrix}\right)_{\frac{3}{2};Z}$ (8)

from which we get, for $\hat{J}_{\pm}=\hat{J}_{X}\pm i\hat{J}_{Y}$,

$\displaystyle\hat{J}_{X}=\hbar\left(\begin{matrix}0&\frac{\sqrt{3}}{2}&0&0\\\
\frac{\sqrt{3}}{2}&0&1&0\\\ 0&1&0&\frac{\sqrt{3}}{2}\\\
0&0&\frac{\sqrt{3}}{2}&0\end{matrix}\right)_{\frac{3}{2};Z}$
$\displaystyle\hat{J}_{Y}=\hbar\left(\begin{matrix}0&-i\frac{\sqrt{3}}{2}&0&0\\\
i\frac{\sqrt{3}}{2}&0&-i&0\\\ 0&i&0&-i\frac{\sqrt{3}}{2}\\\
0&0&i\frac{\sqrt{3}}{2}&0\end{matrix}\right)_{\frac{3}{2};Z}$ (9)

As for $\hat{{\bf L}}$, the $\hat{{\bf J}}$ operator fulfills
$\left[\hat{J}_{X},\hat{J}_{Y}\right]_{-}=i\hbar\hat{J}_{Z}$ and $\hat{{\bf
J}}^{2}=\hat{J}_{X}^{2}+\hat{J}_{Y}^{2}+\hat{J}_{Z}^{2}=\left(\frac{3}{2}\cdot\frac{5}{2}\right)\hbar^{2}\,\hat{\textrm{I}}^{(4)}$
where $\hat{\textrm{I}}^{(4)}$ is the $4\times 4$ identity matrix.

##  II Spin-orbit interaction for semiconductor crystals

We now turn to semiconductor crystals and first derive the spatial operator
$\hat{\bf\Lambda}$ defined in Eq. (4).

(a) Periodic potential

The first problem is to handle the periodicity of the $\mathcal{V}({\bf r})$
potential felt by electrons in a crystal, $\mathcal{V}({\bf
r})=\mathcal{V}({\bf r}+{\bf R}_{\ell})$ for any lattice vector ${\bf
R}_{\ell}$. The way to do it is to expand $\mathcal{V}({\bf r})$ on reciprocal
lattice vectors ${\bf Q}$ that fulfill $e^{i{\bf Q}\cdot{\bf R}_{\ell}}=1$,
namely

$\mathcal{V}({\bf r})=\sum_{\bf Q}\mathcal{V}_{\bf Q}e^{i{\bf Q}\cdot{\bf r}}$
(10)

This gives $\nabla\mathcal{V}({\bf r})=i\sum_{\bf Q}{\bf
Q}\,\,\mathcal{V}_{\bf Q}\,e^{i{\bf Q}\cdot{\bf r}}$, which is obviously not a
vector along ${\bf r}$. So, the orbital angular momentum $\hat{{\bf L}}={\bf
r}\times\hat{{\bf p}}$ is not going to appear in the spin-orbit interaction of
semiconductor electrons.

(b) Spatial basis for cubic crystals

The next problem is to perform calculations using a spatial basis relevant to
semiconductor electrons. From the Bloch theorem, we know that electrons in a
periodic crystal are characterized by a band index $n$ and a wave vector ${\bf
k}$ that is quantized as $2\pi/L$ for a sample volume $L^{3}$, in order to
fulfill the Born-von Karman boundary condition,
$f(\textbf{r})=f(\textbf{r}+\textbf{L})$, which allows extending the crystal
periodicity to a finite volume.

Valence electrons in the threefold level of a cubic crystal have an additional
spatial index that can be taken as $\mu=(x,y,z)$ along the crystal axes, in
contrast to the $(X,Y,Z)$ axes for atoms that can be chosen at will due to
spherical symmetry. The valence-electron wave function then readsMerminbook

$\langle{\bf r}|v,\mu,{\bf k}\rangle=\frac{e^{i{\bf k}\cdot{\bf
r}}}{L^{3/2}}\,u_{v,\mu,{\bf k}}({\bf r})$ (11)

the Bloch function having the lattice periodicity $u_{v,\mu,{\bf k}}({\bf
r})=u_{v,\mu,{\bf k}}({\bf r}+{\bf R}_{\ell})$.

We want to find how the spin-orbit interaction splits the degeneracy of the
threefold spatial level $(v,\mu)$, for ${\bf k}=\bf 0$, that is, at the
valence band maximum9 . To handle the periodicity of the Bloch function, we
also expand it on reciprocal lattice vectors:

$\langle{\bf r}|v,\mu,{\bf k}={\bf 0}\rangle=\frac{1}{L^{3/2}}\,u_{v,\mu,{\bf
0}}({\bf r})=\frac{1}{L^{3/2}}\sum_{\bf Q}u_{v,\mu;{\bf Q}}e^{i{\bf
Q}\cdot{\bf r}}$ (12)

(c) $\hat{\bf\Lambda}^{(cb)}$ matrix in the $\mu$ basis

$\bullet$ To calculate the $\hat{\bf\Lambda}^{(cb)}$ matrix elements in the
$\mu$ basis, we first note, from Eq. (12), that

$\langle{\bf r}|\hat{{\bf p}}|v,\mu,{\bf
0}\rangle=\frac{\hbar}{i}\nabla\langle{\bf r}|v,\mu,{\bf
0}\rangle=\frac{\hbar}{L^{3/2}}\sum_{\bf Q}{\bf Q}\,\,u_{v,\mu;{\bf
Q}}\,e^{i{\bf Q}\cdot{\bf r}}$ (13)

which leads, for the $\mathcal{V}({\bf r})$ potential given in Eq. (10) and
$\hat{\bf\Lambda}^{(cb)}$ defined in Eq. (4), to

$\displaystyle\langle{\bf r}|\hat{\bf\Lambda}^{(cb)}|v,\mu,{\bf 0}\rangle=$
(14) $\displaystyle\,\,\,\,\,\,\,\,\,\frac{i\hbar}{L^{3/2}}\sum_{{\bf
Q}_{1}}\sum_{\bf Q}\mathcal{V}_{{\bf Q}_{1}}\,u_{v,\mu;{\bf Q}}\,e^{i({\bf
Q}+{\bf Q}_{1})\cdot{\bf r}}\,\Big{(}{\bf Q}_{1}\times{\bf Q}\Big{)}$

The matrix elements of the $\hat{\bf\Lambda}^{(cb)}$ operator in the
degenerate subspace $|v,\mu,{\bf 0}\rangle$ then read

$\displaystyle\langle v,\mu^{\prime},{\bf
0}|\hat{\bf\Lambda}^{(cb)}|v,\mu,{\bf 0}\rangle\\!\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\\!i\hbar\sum_{{\bf Q}_{1}}\sum_{{\bf Q}^{\prime}{\bf
Q}}\mathcal{V}_{{\bf Q}_{1}}\,u^{*}_{v,\mu^{\prime};{\bf
Q}^{\prime}}u_{v,\mu;{\bf Q}}$ $\displaystyle\Big{(}{\bf Q}_{1}\times{\bf
Q}\Big{)}\int\frac{d^{3}r}{L^{3}}e^{i({\bf Q}+{\bf Q}_{1}-{\bf
Q}^{\prime})\cdot{\bf r}}$

The integral over ${\bf r}$ imposes ${\bf Q}^{\prime}={\bf Q}+{\bf Q}_{1}$;
so, the above matrix element reduces to

$i\hbar\sum_{{\bf Q}^{\prime}{\bf Q}}\mathcal{V}_{{\bf Q}^{\prime}-{\bf
Q}}\,u^{*}_{v,\mu^{\prime};{\bf Q}^{\prime}}u_{v,\mu;{\bf Q}}\,\Big{(}{\bf
Q}^{\prime}\times{\bf Q}\Big{)}$ (16)

$\bullet$ To go further, we note that for crystals with inversion symmetry,
all valence-electron wave functions are even, and all conduction-electron wave
functions are oddCardona . So, the upper threefold valence level has an even
parity, in spite of the fact that it has been misleadingly called P according
to the atomic notation for a threefold level. This parity then imposes in Eq.
(12)

$\langle{\bf r}|v,\mu,{\bf 0}\rangle=\langle-{\bf r}|v,\mu,{\bf
0}\rangle\Longleftrightarrow u_{v,\mu;{\bf Q}}=u_{v,\mu;-{\bf Q}}$ (17)

which for a threefold level with cubic symmetry, leads us to take, due to
cyclic permutations,

$u_{v,\mu;{\bf Q}}=\frac{Q_{x}Q_{y}Q_{z}}{Q_{\mu}}G_{v,Q}$ (18)

where $G_{v,Q}$ depends on $Q=|{\bf Q}|$ only.

$\bullet$ The last step is to show that all $\hat{\bf\Lambda}^{(cb)}$ matrix
elements in the $|v,\mu,{\bf 0}\rangle$ basis are equal to zero, except

$\langle v,y,{\bf 0}|\hat{\Lambda}_{z}^{(cb)}|v,x,{\bf 0}\rangle\equiv
i\hbar\lambda_{cb}$ (19)

and its cyclic permutations, the constant $\lambda_{cb}$ being given by

$\displaystyle\lambda_{cb}$ $\displaystyle=$ $\displaystyle\sum_{{\bf
Q}^{\prime}{\bf Q}}\mathcal{V}_{{\bf Q}^{\prime}-{\bf
Q}}\,G^{*}_{v,Q^{\prime}}G_{v,Q}$ (20)
$\displaystyle\times\big{(}Q^{\prime}_{x}Q^{\prime}_{y}Q^{\prime}_{z}\big{)}\big{(}Q_{x}Q_{y}Q_{z}\big{)}\frac{Q^{\prime}_{x}Q_{y}-Q^{\prime}_{y}Q_{x}}{Q^{\prime}_{y}Q_{x}}$

which does not depend on $(x,y)$ due to cyclic permutations. The derivation of
this key result, which requires

$\mathcal{V}(x,y,z)=\mathcal{V}(y,x,z)=\mathcal{V}(-x,y,z)$ (21)

as fulfilled by the potential $\mathcal{V}({\bf r})$ of a cubic crystal, is
given in Appendix A.

(d) Cubic spatial momentum $\hat{\mathbfcal{L}}^{(cb)}$

By using Eq. (19), we can derive the components of the “spatial momentum”

$\hat{\mathbfcal{L}}^{(cb)}=\frac{\hat{\bf\Lambda}^{(cb)}}{\lambda_{cb}}$ (22)

in the $|v,\mu,{\bf 0}\rangle$ basis. From them, we find that the matrix
representations of the $\hat{\mathbfcal{L}}^{(cb)}$ components read as

$\displaystyle\hat{\mathcal{L}}_{x}^{(cb)}=\hbar\begin{pmatrix}0&0&0\\\
0&0&-i\\\
0&i&0\end{pmatrix}_{\mu}\qquad\hat{\mathcal{L}}_{y}^{(cb)}=\hbar\begin{pmatrix}0&0&i\\\
0&0&0\\\ -i&0&0\end{pmatrix}_{\mu}$
$\displaystyle\hat{\mathcal{L}}_{z}^{(cb)}=\hbar\begin{pmatrix}0&-i&0\\\
i&0&0\\\ 0&0&0\end{pmatrix}_{\mu}\hskip 56.9055pt$ (23)

We can check that like for the orbital angular momentum $\hat{{\bf L}}$, these
components fulfill
$\left[\hat{\mathcal{L}}_{x}^{(cb)},\hat{\mathcal{L}}_{y}^{(cb)}\right]_{-}=i\hbar\hat{\mathcal{L}}_{z}^{(cb)}$,
and $(\hat{\mathbfcal{L}}^{(cb)})^{2}=(1\cdot
2)\hbar^{2}\,\hat{\textrm{I}}^{(3)}$.

$\bullet$ We now anticipate that, when the spin is included, the appropriate
spatial basis will not be $|v,\mu,{\bf 0}\rangle$, but
$|v,\mathcal{L}_{z},{\bf 0}\rangle$ with $\mathcal{L}_{z}=(\pm 1,0)$, that we
define according to the Landau-Lifshitz phase factor for spherical
harmonicsLandau01  as

$\displaystyle|v,\pm 1_{z},{\bf 0}\rangle$ $\displaystyle=$
$\displaystyle\frac{\mp i|v,x,{\bf 0}\rangle+|v,y,{\bf 0}\rangle}{\sqrt{2}}$
(24) $\displaystyle|v,0_{z},{\bf 0}\rangle$ $\displaystyle=$ $\displaystyle
i|v,z,{\bf 0}\rangle$ (25)

By using Eq. (23), we find that the matrix elements of the
$\hat{\mathbfcal{L}}^{(cb)}$ operator in this basis are given by

$\displaystyle\langle v,\eta^{\prime}_{z},{\bf
0}|\hat{\mathcal{L}}_{z}^{(cb)}|v,\eta_{z},{\bf 0}\rangle$ $\displaystyle=$
$\displaystyle\hbar\frac{\eta^{\prime}_{z}+\eta_{z}}{2}$ (26)
$\displaystyle\langle v,\eta_{z},{\bf
0}|\hat{\mathcal{L}}_{\pm}^{(cb)}|v,0_{z},{\bf 0}\rangle$ $\displaystyle=$
$\displaystyle\hbar\frac{1\pm\eta_{z}}{\sqrt{2}}$ (27)

for $\eta=\pm 1$. Using them, we get the matrix representations of these
operators in the $|v,\mathcal{L}_{z},{\bf 0}\rangle$ basis as

$\displaystyle\hat{\mathcal{L}}_{z}^{(cb)}=\hbar\begin{pmatrix}1&0&0\\\
0&0&0\\\ 0&0&-1\end{pmatrix}_{1;z}$ (28)
$\displaystyle\quad\hat{\mathcal{L}}_{+}^{(cb)}=(\hat{\mathcal{L}}_{-}^{(cb)})^{\dagger}=\hbar\sqrt{2}\,\begin{pmatrix}0&1&0\\\
0&0&1\\\ 0&0&0\end{pmatrix}_{1;z}$

These matrices have the same form as the ones, in Eq. (6), for the orbital
angular momentum $\hat{{\bf L}}$, provided that the atomic axes $(X,Y,Z)$ are
chosen along the cubic axes $(x,y,z)$.

(e) Cubic hybrid momentum $\hat{\mathbfcal{J}}^{(cb)}$

$\bullet$ The last step is to introduce the spin. The spin-orbit interaction
given in Eq. (2) reads

$\hat{H}_{so}=\lambda_{so}\hat{\bf\Lambda}^{(cb)}\cdot\hat{{\bf
S}}=\lambda_{so}\,\lambda_{cb}\,\,\hat{\mathbfcal{L}}^{(cb)}\cdot\hat{{\bf
S}}$ (29)

that we rewrite as

$\hat{H}_{so}=\lambda_{so}\lambda_{cb}\left(\frac{\hat{\mathcal{L}}_{+}^{(cb)}\hat{S}_{-}+\hat{\mathcal{L}}_{-}^{(cb)}\hat{S}_{+}}{2}+\hat{\mathcal{L}}_{z}^{(cb)}\hat{S}_{z}\right)$
(30)

with $\hat{S}_{\pm}=\hat{S}_{x}\pm i\hat{S}_{y}$.

Using Eqs. (26,27), we can deduce the matrix elements of the
$\hat{\mathbfcal{L}}^{(cb)}\cdot\hat{{\bf S}}$ operator in the six-state basis
$|(1_{z},0_{z},{-}1_{z})\rangle\otimes|1/2_{z}\rangle$ and
$|(1_{z},0_{z},{-}1_{z})\rangle\otimes|{-}1/2_{z}\rangle$. The corresponding
matrix appears as

$\hat{\mathbfcal{L}}^{(cb)}\cdot\hat{{\bf
S}}=\frac{\hbar^{2}}{2}\begin{pmatrix}1&0&0&0&0&0\\\ 0&0&0&\sqrt{2}&0&0\\\
0&0&-1&0&\sqrt{2}&0\\\ 0&\sqrt{2}&0&-1&0&0\\\ 0&0&\sqrt{2}&0&0&0\\\
0&0&0&0&0&1\\\ \end{pmatrix}$ (31)

It can be made block-diagonal by interchanging the states
$|{-}1_{z}\rangle\otimes|1/2_{z}\rangle$ and
$|1_{z}\rangle\otimes|{-}1/2_{z}\rangle$, to get

$\hat{\mathbfcal{L}}^{(cb)}\cdot\hat{{\bf
S}}=\frac{\hbar^{2}}{2}\begin{pmatrix}1&0&0&0&0&0\\\ 0&0&\sqrt{2}&0&0&0\\\
0&\sqrt{2}&-1&0&0&0\\\ 0&0&0&-1&\sqrt{2}&0\\\ 0&0&0&\sqrt{2}&0&0\\\
0&0&0&0&0&1\\\ \end{pmatrix}$ (32)

The eigenvalues of the upper-left $(3\times 3)$ submatrix follow from the
determinant

$0=\left|\begin{matrix}1{-}u&0&0\\\ 0&{-}u&\sqrt{2}\\\
0&\sqrt{2}&{-}1{-}u\end{matrix}\right|=(1-u)(u^{2}+u-2)$ (33)

And similarly for the lower-right $(3\times 3)$ submatrix. Their eigenvalues
are $u=(1,1,-2)$.

The basis is made of the states $|\eta_{z}\rangle\otimes|\eta_{z}/2\rangle$,
$|0_{z}\rangle\otimes|\eta_{z}/2\rangle$ and
$|\eta_{z}\rangle\otimes|-\eta_{z}/2\rangle$, with $\eta=+1$ for the upper-
left $(3\times 3)$ submatrix, and $\eta=-1$ for the lower-right $(3\times 3)$
submatrix. The four eigenstates that correspond to the $u=1$ eigenvalue, that
is, a spin-orbit shift equal to $(\hbar^{2}/2)\lambda_{so}\lambda_{cb}$, read

$\displaystyle|\pm 1_{z}\rangle\otimes|(\pm\frac{1}{2})_{z}\rangle$
$\displaystyle\equiv$ $\displaystyle|\pm A\rangle$ (34)
$\displaystyle\frac{|\pm
1_{z}\rangle\otimes\left|(\mp\frac{1}{2})_{z}\right\rangle+\sqrt{2}|0_{z}\rangle\otimes\left|(\pm\frac{1}{2})_{z}\right\rangle}{\sqrt{3}}$
$\displaystyle\equiv$ $\displaystyle|\pm A^{\prime}\rangle$ (35)

while the two eigenstates that correspond to the $u=-2$ eigenvalue, that is, a
spin-orbit shift equal to $-\hbar^{2}\lambda_{so}\lambda_{cb}$, read

$\pm\frac{\sqrt{2}|\pm
1_{z}\rangle\otimes\left|(\mp\frac{1}{2})_{z}\right\rangle-|0_{z}\rangle\otimes\left|(\pm\frac{1}{2})_{z}\right\rangle}{\sqrt{3}}\equiv|\pm
B\rangle$ (36)

We see that the $|\pm A\rangle$ and $|\pm A^{\prime}\rangle$ eigenstates are
just the atomic states $|j=3/2,j_{Z}=\pm 3/2\rangle$ and $|j=3/2,j_{Z}=\pm
1/2\rangle$, while $|\pm B\rangle$ correspond to $|j=1/2,j_{Z}=\pm
1/2\rangle$, for $Z$ chosen along $z$.

$\bullet$ This remark leads us to introduce the vector operator
$\hat{\mathbfcal{J}}^{(cb)}$, formally defined as

$\hat{\mathbfcal{J}}^{(cb)}=\hat{\mathbfcal{L}}^{(cb)}+\hat{{\bf S}}$ (37)

where the operators $\hat{\mathbfcal{L}}^{(cb)}$ and $\hat{{\bf S}}$
respectively act in the real space and the spin space. From
$\left[\hat{\mathcal{L}}_{x}^{(cb)},\hat{\mathcal{L}}_{y}^{(cb)}\right]_{-}=i\hbar\hat{\mathcal{L}}_{z}^{(cb)}$
and $\left[\hat{S}_{x},\hat{S}_{y}\right]_{-}=i\hbar\hat{S}_{z}$, we readily
get

$\left[\hat{\mathcal{J}}_{x}^{(cb)},\hat{\mathcal{J}}_{y}^{(cb)}\right]_{-}=i\hbar\hat{\mathcal{J}}_{z}^{(cb)}$
(38)

with cyclic permutations.

As
$(\hat{\mathbfcal{J}}^{(cb)})^{2}=(\hat{\mathbfcal{L}}^{(cb)})^{2}+\hat{{\bf
S}}^{2}+2\hat{\mathbfcal{L}}^{(cb)}\cdot\hat{{\bf S}}$, while
$(\hat{\mathbfcal{L}}^{(cb)})^{2}$ gives $(1\cdot
2)\hbar^{2}\,\hat{\textrm{I}}^{(3)}$ on any spatial states
$|\mathcal{L}_{z}\rangle$ with $\mathcal{L}_{z}=(\pm 1,0)$, we find that the
operator $(\hat{\mathbfcal{J}}^{(cb)})^{2}$ is diagonal in the $(|\pm
A\rangle,|\pm A^{\prime}\rangle)$ subspace that corresponds to the
$(\hbar^{2}/2)\lambda_{so}\lambda_{cb}$ spin-orbit shift; the associated
eigenvalue, equal to

$1\cdot
2\hbar^{2}+\frac{1}{2}\frac{3}{2}\hbar^{2}+2\,\frac{\hbar^{2}}{2}=\frac{3}{2}\frac{5}{2}\hbar^{2}$
(39)

just corresponds to $\mathcal{J}(\mathcal{J}+1)\hbar^{2}$ for
$\mathcal{J}=3/2$. In the same way, we find that the
$(\hat{\mathbfcal{J}}^{(cb)})^{2}$ eigenvalue in the $|\pm B\rangle$ subspace
that corresponds to the $(-\hbar^{2})\lambda_{so}\lambda_{cb}$ spin-orbit
shift, is equal to

$1\cdot
2\hbar^{2}+\frac{1}{2}\frac{3}{2}\hbar^{2}-2\,\hbar^{2}=\frac{1}{2}\frac{3}{2}\hbar^{2}$
(40)

which just corresponds to $\mathcal{J}(\mathcal{J}+1)\hbar^{2}$ for
$\mathcal{J}=1/2$.

Moreover, we note that the $|\pm A\rangle$, $|\pm A^{\prime}\rangle$, and
$|\pm B\rangle$ states also are eigenstates of
$\hat{\mathcal{J}}_{z}^{(cb)}=\hat{\mathcal{L}}_{z}^{(cb)}+\hat{S}_{z}$,

$\hat{\mathcal{J}}_{z}^{(cb)}|\pm A\rangle&=\pm\frac{3}{2}\hbar|\pm
A\rangle\\\ \hat{\mathcal{J}}_{z}^{(cb)}|\pm
A^{\prime}\rangle=\pm\frac{1}{2}\hbar|\pm A^{\prime}\rangle\\\
\hat{\mathcal{J}}_{z}^{(cb)}|\pm B\rangle=\pm\frac{1}{2}\hbar|\pm B\rangle$
(41)

The fact that these states are eigenstates of the
$((\hat{\mathbfcal{J}}^{(cb)})^{2},\hat{\mathcal{J}}_{z}^{(cb)})$ operators
leads to the following identification

$|\pm
A\rangle&\equiv\left|\mathcal{J}=\frac{3}{2},\mathcal{J}_{z}=\pm\frac{3}{2}\right\rangle\\\
|\pm
A^{\prime}\rangle\equiv\left|\mathcal{J}=\frac{3}{2},\mathcal{J}_{z}=\pm\frac{1}{2}\right\rangle\\\
|\pm
B\rangle\equiv\left|\mathcal{J}=\frac{1}{2},\mathcal{J}_{z}=\pm\frac{1}{2}\right\rangle$
(42)

From them, it is easy to obtain the matrix representation of the vector
operator $\hat{\mathbfcal{J}}^{(cb)}$ in the $(|\pm A\rangle,|\pm
A^{\prime}\rangle)$ subspace and in the $|\pm B\rangle$ subspace. In
particular, the $\hat{\mathcal{J}}_{z}^{(cb)}$ components read in these
subspaces as

$\hat{\mathcal{J}}_{z}^{(cb)}=\hbar\begin{pmatrix}\frac{3}{2}&0&0&0\\\
0&\frac{1}{2}&0&0\\\ 0&0&-\frac{1}{2}&0\\\
0&0&0&-\frac{3}{2}\end{pmatrix}_{\frac{3}{2};z}\quad\hat{\mathcal{J}}_{z}^{(cb)}=\hbar\begin{pmatrix}\frac{1}{2}&0\\\
0&-\frac{1}{2}\end{pmatrix}_{\frac{1}{2};z}$ (43)

##  III Phase factor between valence-electron and hole operators

Relativistic quantum theoryBerestetskii  gives the link between the
destruction operator of a particle with quantum indices $(j,j_{Z})$ and the
creation operator of its antiparticleLandau01 , as

$\hat{a}_{j,j_{Z}}=(-1)^{j-j_{Z}}\hat{b}^{\dagger}_{j,-j_{Z}}\,$ (44)

To provide a strong physical support to these cubic operators
$\hat{\mathbfcal{L}}^{(cb)}$ and $\hat{\mathbfcal{J}}^{(cb)}$, let us show
that, although the hole is definitely not a naïve antiparticle of the valence
electron due to its interactions with the electrons that remain in the valence
bandMonicbook , so that the hole is a many-body object in itself, the above
relation is still valid for cubic semiconductorsSS2021 , provided that we
label the valence electron states along the eigenvalues of the cubic spatial
momentum $\hat{\mathbfcal{L}}^{(cb)}$ or the cubic hybrid momentum
$\hat{\mathbfcal{J}}^{(cb)}$ defined in the preceding section.

$\bullet$ The derivation of this result relies on

$\hat{a}_{(\pm\frac{1}{2})_{z}}=\pm\,\hat{b}^{\dagger}_{(\mp\frac{1}{2})_{z}}$
(45)

that follows from Eq. (44) taken for the spin, that is, $j=1/2$, and

$\hat{a}_{\mu}=\hat{b}^{\dagger}_{\mu}\,\,\,\,\,\,\,\,\,\,\,\textrm{for}\,\,\,\,\,\,\,\,\,\mu=(x,y,z)$
(46)

as imposed by cyclic permutations in a cubic crystal.

$\bullet$ Equations (24,25) then give

$\displaystyle\hat{a}_{\pm 1_{z}}$ $\displaystyle=$ $\displaystyle\frac{\pm
i\hat{a}_{x}+\hat{a}_{y}}{\sqrt{2}}=\frac{\pm
i\hat{b}^{\dagger}_{x}+\hat{b}^{\dagger}_{y}}{\sqrt{2}}=\hat{b}^{\dagger}_{\mp
1_{z}}\,$ (47) $\displaystyle\hat{a}_{0_{z}}$ $\displaystyle=$
$\displaystyle-i\hat{a}_{z}=-i\hat{b}^{\dagger}_{z}=-\hat{b}^{\dagger}_{0_{z}}\,$
(48)

which agree with Eq. (44) for $j=1$ and $j_{z}=(\pm 1,0)$ with $Z$ taken along
$z$.

$\bullet$ When turning to the four eigenstates defined in Eqs. (34,35), the
above equations give for $|\pm A\rangle$ seen as
$(\mathcal{J}=3/2,\mathcal{J}_{z}=\pm 3/2)$,

$\hat{a}^{\dagger}_{\frac{3}{2},(\pm\frac{3}{2})_{z}}=\hat{a}^{\dagger}_{\pm
1_{z},(\pm\frac{1}{2})_{z}}=\pm\hat{b}_{\mp
1_{z},(\mp\frac{1}{2})_{z}}=\pm\hat{b}_{\frac{3}{2},(\mp\frac{3}{2})_{z}}$
(49)

while for $|\pm A^{\prime}\rangle$ seen as
$(\mathcal{J}=3/2,\mathcal{J}_{z}=\pm 1/2)$, we get

$\displaystyle\hat{a}^{\dagger}_{\frac{3}{2},(\pm\frac{1}{2})_{z}}$
$\displaystyle=$ $\displaystyle\frac{\hat{a}^{\dagger}_{\pm
1_{z},(\mp\frac{1}{2})_{z}}+\sqrt{2}\,\hat{a}^{\dagger}_{0_{z},(\pm\frac{1}{2})_{z}}}{\sqrt{3}}$
(50) $\displaystyle=$ $\displaystyle\frac{(\mp)\hat{b}^{\dagger}_{\mp
1_{z},(\pm\frac{1}{2})_{z}}-(\pm)\sqrt{2}\,\hat{b}^{\dagger}_{0_{z},(\mp\frac{1}{2})_{z}}}{\sqrt{3}}$
$\displaystyle=$
$\displaystyle\mp\hat{b}^{\dagger}_{\frac{3}{2},(\mp\frac{1}{2})_{z}}$

We again see that the change from valence-electron destruction operator to
hole creation operator agrees with Eq. (44).

$\bullet$ We can check that this is also true for the $|\pm B\rangle$ states
of Eq. (36), seen as $(\mathcal{J}=1/2,\mathcal{J}_{z}=\pm 1/2)$. Indeed,

$\displaystyle\hat{a}^{\dagger}_{\frac{1}{2},(\pm\frac{1}{2})_{z}}\\!\\!\\!$
$\displaystyle=$
$\displaystyle\\!\\!\\!\pm\frac{\sqrt{2}~{}\hat{a}^{\dagger}_{\pm
1_{z},(\mp\frac{1}{2})_{z}}-\hat{a}^{\dagger}_{0_{z},(\pm\frac{1}{2})_{z}}}{\sqrt{3}}$
(51) $\displaystyle=$
$\displaystyle\pm\frac{(\mp)\sqrt{2}~{}\hat{b}^{\dagger}_{\mp
1_{z},(\pm\frac{1}{2})_{z}}+(\pm)\hat{b}^{\dagger}_{0_{z},(\mp\frac{1}{2})_{z}}}{\sqrt{3}}$
$\displaystyle=$
$\displaystyle\pm\hat{b}^{\dagger}_{\frac{1}{2},(\mp\frac{1}{2})_{z}}$

##  IV Possible extension

We wish to stress that the definition of the vector operator
$\hat{\mathbfcal{L}}$ given in Eq. (4) is completely general: it only reads in
terms of the electron momentum operator $\hat{{\bf p}}$ and the internal
electrostatic potential $\mathcal{V}({\bf r})$ felt by semiconductor
electrons, without any restriction on the potential symmetry. When this
potential has the cubic symmetry, it does not come as a surprise to find that
$\hat{\mathbfcal{L}}^{(cb)}$ has exactly the same matrix representation as the
orbital angular momentum $\hat{{\bf L}}$ for the $\ell=1$ atomic level,
because the $(x,y,z)$ orthogonal axes play the same role for a sphere and a
cube.

We can go further and ask whether the $\hat{\mathbfcal{L}}$ concept can be
extended to a crystal having less symmetry than the cubic symmetry, for
example when the crystal axes $(x,y,z)$ are not equivalent, or even not
orthogonal. The identity between the $\hat{\mathbfcal{L}}$ and $\hat{{\bf L}}$
matrices will not exist anymore, but the $\hat{\mathbfcal{J}}$ eigenstates,
for $\hat{\mathbfcal{J}}$ constructed in the same way out of
$\hat{\mathbfcal{L}}$, may still provide a convenient way to derive the
semiconductor eigenstates in the presence of spin-orbit interaction, without
having to resort to the group theory which is physically obscure when turning
to the double group, as required to handle the spin-orbit interaction. The
possible extension of the present work to periodic systems with other
potential symmetries deserves further investigation; it just constitutes the
starting point to move forward.

##  V Conclusion

Through a microscopic procedure, we here show that the spin-orbit interaction
in semiconductors has the same $\hat{\mathbfcal{L}}\cdot\hat{{\bf S}}$ form as
for atomic electrons, with $\hat{\mathbfcal{L}}$ being the analog of the
orbital angular momentum $\hat{{\bf L}}$, as supported by their identical
matrix representations in the atomic basis and the Bloch-state basis for cubic
semiconductors. Our work thus provides the long-missed support for using
atomic notations to label valence electron states in GaAs-like semiconductors,
in spite of the fact that the electrostatic potential felt by semiconductor
electrons is not spherical but periodic. Up to now, the only clean
classification for these electrons relied on the group theory, which is
definitively correct but overly heavy when dealing with a crystal symmetry as
simple as cubic.

As a strong support to this labeling, we also show that the transformation
from valence electron to hole operators in a cubic semiconductor, has the same
phase factor as the one from particle to antiparticle in relativistic quantum
theory. This provides a secure way to describe semiconductor physics in terms
of electrons and holes through effective electron-hole HamiltonianKL ;
Monicbook  and effective coupling to the electromagnetic fieldMC1990 ;
Hartmutbook .

##  VI Acknowledgment

It is our pleasure to acknowledge constructive discussions with the Referee
and with Benoit Eble, about the appropriate way to name $\hat{\mathbfcal{L}}$
and $\hat{\mathbfcal{J}}$.

##  Appendix A Derivation of Eq. (19)

We here derive Eq. (19) starting from Eq. (16), namely

$\displaystyle\langle v,\mu^{\prime},{\bf
0}|\hat{\bf\Lambda}^{(cb)}|v,\mu,{\bf 0}\rangle$ (52)
$\displaystyle\,\,\,\,\,\,\,\,\,\,\,\,=i\hbar\sum_{{\bf Q}^{\prime}{\bf
Q}}\mathcal{V}_{{\bf Q}^{\prime}-{\bf Q}}\,u^{*}_{v,\mu^{\prime};{\bf
Q}^{\prime}}u_{v,\mu;{\bf Q}}\,\Big{(}{\bf Q}^{\prime}\times{\bf Q}\Big{)}$

(i) To do it, we first consider $\mu^{\prime}=\mu$. By using Eq. (18), the
above equation gives the $\hat{\Lambda}_{x}^{(cb)}$ component as

$\displaystyle\langle v,\mu,{\bf 0}|\hat{\Lambda}_{x}^{(cb)}|v,\mu,{\bf
0}\rangle=i\hbar\sum_{{\bf Q}^{\prime}{\bf Q}}\mathcal{V}_{{\bf
Q}^{\prime}-{\bf Q}}\,G^{*}_{v,Q^{\prime}}G_{v,Q}\hskip 28.45274pt$ (53)
$\displaystyle\times\left(\frac{Q^{\prime}_{x}Q^{\prime}_{y}Q^{\prime}_{z}}{Q^{\prime}_{\mu}}\frac{Q_{x}Q_{y}Q_{z}}{Q_{\mu}}\right)\,\big{(}Q_{y}^{\prime}Q_{z}-Q_{y}Q_{z}^{\prime}\big{)}$

When reversing the direction of the z axis, $(Q_{z},Q^{\prime}_{z})$ change
into $(-Q_{z},-Q^{\prime}_{z})$; so, the first factor
$\left(\frac{Q^{\prime}_{x}Q^{\prime}_{y}Q^{\prime}_{z}}{Q^{\prime}_{\mu}}\frac{Q_{x}Q_{y}Q_{z}}{Q_{\mu}}\right)$
keeps its sign whatever $\mu$, while the second factor
$(Q_{y}^{\prime}Q_{z}-Q_{y}Q_{z}^{\prime})$ changes sign. Since a potential
with cubic symmetry is not affected by reversing the direction of the z axis,
namely, $\mathcal{V}(x,y,z)=\mathcal{V}(x,y,-z)$, we end with $\langle
v,\mu,{\bf 0}|\hat{\Lambda}_{x}^{(cb)}|v,\mu,{\bf 0}\rangle=\,-\,\langle
v,\mu,{\bf 0}|\hat{\Lambda}_{x}^{(cb)}|v,\mu,{\bf 0}\rangle$, which proves
that this matrix element is equal to zero.

The same is true for $\hat{\Lambda}_{y}^{(cb)}$ by reversing the direction of
the x axis, and for $\hat{\Lambda}_{z}^{(cb)}$ by reversing the direction of
the y axis. So, we end with

$\displaystyle\langle v,\mu,{\bf
0}|\hat{\Lambda}_{\mu^{\prime\prime}}^{(cb)}|v,\mu,{\bf 0}\rangle=0$ (54)

whatever $\mu$ and $\mu^{\prime\prime}$.

(ii) We now consider $\mu^{\prime}\neq\mu$. By reversing the directions of the
$\mu^{\prime}$ axis and the $\mu$ axis, the cubic potential still stays
unchanged and the factor
$\left(\frac{Q^{\prime}_{x}Q^{\prime}_{y}Q^{\prime}_{z}}{Q^{\prime}_{\mu^{\prime}}}\frac{Q_{x}Q_{y}Q_{z}}{Q_{\mu}}\right)$
still keeps its sign. In order for the matrix elements of
$\hat{\Lambda}_{\mu^{\prime\prime}}^{(cb)}$ to also keep its sign,
$\mu^{\prime\prime}$ must be different from $(\mu^{\prime},\mu)$, like $x$ for
$(y,z)$. The component of $\big{(}{\bf Q}^{\prime}\times{\bf Q}\big{)}$ along
$\mu^{\prime\prime}$ in Eq. (A) then reads in terms of
$Q^{\prime}_{\mu^{\prime}}Q_{\mu}$. So, to get nonzero matrix elements, that
is,

$\displaystyle\langle v,\mu^{\prime},{\bf
0}|\hat{\Lambda}_{\mu^{\prime\prime}}^{(cb)}|v,\mu,{\bf 0}\rangle\neq 0$ (55)

we must have the $(\mu,\mu^{\prime},\mu^{\prime\prime})$ indices all
different, the values of the above matrix elements being independent of these
indices due to cubic symmetry.

## References

  * (1) L. D. Landau, and E. M. Lifshitz, Quantum Mechanics: Non-Relativistic Theory, Vol. 3 (Pergamon Press, 1977).
  * (2) J. J. Sakurai, Modern Quantum Mechanics (Benjamin/Cummings, Menlo Park, CA, 1985).
  * (3) C. Cohen-Tannoudji, J. Dupont-Roc, and G. Grynberg, Photons and Atoms: Introduction to Quantum Electrodynamics, Wiley (USA, 1992).
  * (4) C. J. Pethick and H. Smith, Bose-Einstein Condensation in Dilute Gases, 2nd ed. (Cambridge University Press, New York, 2008).
  * (5) C. Cohen-Tannoudji, B. Diu, and F. Laloë, Quantum Mechanics (Wiley, New York, 1977).
  * (6) V. A. Kuzkin, ZAMM Z. Angew. Math. Mech. 95(11), 1290 (2015).
  * (7) C. Kittel, Introduction to Solid State Physics, 7th ed. (Wiley, New York, 1996).
  * (8) N. W. Ashcroft, and N. D. Mermin, Solid State Physics (Holt, Rinehart and Winston, New York, 1976).
  * (9) P. Y. Yu, and M. Cardona, Fundamentals of Semiconductors, 3rd ed. (Springer, Berlin, 2005).
  * (10) G. Dresselhaus, A. F. Kip, and C. Kittel, Phys. Rev. 98, 368 (1955).
  * (11) J. M. Luttinger, and W. Kohn, Phys. Rev. 97, 869 (1955).
  * (12) J. M. Luttinger, Phys. Rev. 102, 1030 (1956).
  * (13) W. J. Elder, R. M. Ward, J. Zhang, Phys. Rev. B 83, 165210 (2011).
  * (14) L. H. Thomas, Nature (London) 117, 514 (1926).
  * (15) M. Combescot, S.-Y. Shiau, and V. Voliotis, Phys. Rev. B 99, 245202 (2019).
  * (16) V. B. Berestetskii, E. M. Lifshitz, and L. P. Pitaevskii, Relativistic Quantum Theory, Vol. 4, 1st ed. (Pergamon Press, 1971).
  * (17) M. Combescot and S.-Y. Shiau, Excitons and Cooper Pairs: two composite bosons in many-body physics, Oxford University Press (Oxford, 2015).
  * (18) For finite k, treated through a $\textbf{k}\cdot\textbf{p}$ perturbative approach, the $\mathcal{J}=3/2$ fourfold spin-orbit level, obtained for $\textbf{k}=\textbf{0}$, splits into two branches that lead to heavy and light valence electrons when the warping is neglected. See, for example, M. Combescot, and S.-Y. Shiau, Phys. Rev. B 101, 195203 (2020).
  * (19) S.-Y. Shiau, and M. Combescot, Semiconductors (2022); http://doi.org/10.1134/S1063782621090190.
  * (20) M. Combescot, Phys. Rev. B 41, 3517 (1990).
  * (21) H. Haug, and S. W. Koch, Quantum Theory of the Optical and Electronic Properties of Semiconductors (World Scientific, 5th ed., 2009).


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

CLiC-it 2023: Ninth Italian Conference on Computational Linguistics, November
30–December 2, 2023, Venice, IT

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

# How To Build Competitive Multi-gender Speech Translation Models For
Controlling Speaker Gender Translation

Marco Gaido  Fondazione Bruno Kessler  Dennis Fucci  University of Trento
Matteo Negri  Luisa Bentivogli

(2021)

###### Abstract

When translating from notional gender languages (e.g., English) into
grammatical gender languages (e.g., Italian), the generated translation
requires explicit gender assignments for various words, including those
referring to the speaker. When the source sentence does not convey the
speaker’s gender, speech translation (ST) models either rely on the possibly-
misleading vocal traits of the speaker or default to the masculine gender, the
most frequent in existing training corpora. To avoid such biased and not
inclusive behaviors, the gender assignment of speaker-related expressions
should be guided by externally-provided metadata about the speaker’s
gender.111Throughout the paper, we use the word gender to indicate the
preferred linguistic expression of gender and not the gender identity. While
previous work has shown that the most effective solution is represented by
separate, dedicated gender-specific models, the goal of this paper is to
achieve the same results by integrating the speaker’s gender metadata into a
single “multi-gender” neural ST model, easier to maintain. Our experiments
demonstrate that a single multi-gender model outperforms gender-specialized
ones when trained from scratch (with gender accuracy gains up to 12.9 for
feminine forms), while fine-tuning from existing ST models does not lead to
competitive results.

###### keywords:

gender bias gradient reversal speech translation

##  1 Introduction

11footnotetext: Throughout the paper, we use the word gender to indicate the
preferred linguistic expression of gender and not the gender identity.

Spurred by growing concerns about fairness in language technologies, research
on understanding and mitigating gender bias in automatic translation is
gaining traction [1]. The bias of automatic systems is extremely evident when
it comes to ambiguous sentences or expressions, where there are no explicit
cues in the source content about the correct gender1 assignment of a referent
(e.g., en: The doctor arrived – it: Il/La dottore/essa è arrivato/a). In this
setting, the state-of-the-art neural models often choose the masculine forms
or perpetuate stereotypical assignments, as they reflect the condition
statistically more likely based on their (biased) training data [2, 3].

This situation frequently occurs when the source language is genderless or
employs notional gender, expressing gender in a limited set of parts of
speech, and the target language follows a grammatical gender system, embedding
gender distinctions throughout a broad inventory of parts of speech. Focusing
on the case in which the source language is English, a notional gender
language, and the target language is Italian, a grammatical gender language, a
frequent instance of this condition is represented by first-person references,
i.e. by the words and expressions referred to the speaker (e.g., en: I am a
young researcher – it: Sono una/un giovane ricercatrice/tore). In this case,
text-to-text machine translation (MT) models mostly output masculine forms,
while direct (or end-to-end) speech-to-text translation (ST) systems partly
rely on the biological cue of the speaker’s vocal traits to assign gender [4,
5]. However, direct ST models are still largely biased toward producing
masculine forms, and, most importantly, biological aspects are related to the
sex rather than to the gender of an individual. Hence, their exploitation is
not inclusive of all people, harming several groups such as transgenders [6].

As a solution, [7] proposed to leverage external metadata about the speaker’s
gender to control the gender assignment of words referred to the speaker.
Specifically, they investigated two approaches: i) the development of two
separate gender-specialized models, fine-tuned on gender-specific data as also
proposed later in MT [8], and ii) a single multi-gender model, where the
speaker gender is a tag fed to a single model as in multilingual systems [9].
While the second solution would be preferable (as the specialized solution
involves the higher cost of maintaining two separate models), the experiments
in [7] demonstrate that specialized models outperform the multi-gender
approach by a large margin in terms of gender accuracy.

In light of the above, in this paper we address the following research
questions: i) why do specialized models outperform multi-gender ones? ii) Can
we build competitive multi-gender systems? Through experiments on English-
Italian translation of TED talks, we show that the low accuracy of multi-
gender models comes from the initialization with the weights of a gender-
unaware ST system and the inability to override the behavior of the base ST
model (i.e., the reliance on vocal cues) during the fine-tuning stage. We also
try to address this problem with two solutions: i) a contrastive loss that
penalizes the extraction of gender cues from speech input, and ii) altering
vocal properties of training data to misalign gender cues with gender tags and
gender translations.222Our code is released open source under Apache 2.0
Licence at: https://github.com/hlt-mt/FBK-fairseq/ Despite the slight
improvements brought by these solutions in gender accuracy and overall
translation quality, none of them effectively close the performance gap with
the specialized solution. However, training multi-gender models from scratch
yields competitive results, outperforming the specialized approach with gender
accuracy gains of up to 12.9 points for feminine translations. Therefore, we
recommend building multi-gender models from scratch, while building them on
top of existing systems remains an open research question.

##  2 Background

In this section, we introduce the basic concepts useful for understanding the
rest of the paper. First, we provide an overview of the methods proposed in
the literature to integrate language tags into neural multilingual translation
models (§2.1), from which multi-gender models draw inspiration. Then, we
present how gender information has been removed from neural representations
through adversarial training in previous works (§2.2), from which we derive
our solution presented in §3.1.

###  2.1 Tags Integration in Multilingual Models

State-of-the-art models in MT and ST are sequence-to-sequence models made of
an encoder and an autoregressive Transformer decoder [10]. The autoregressive
decoder predicts the next-token probability over a predefined vocabulary at
every iteration by looking at the encoder output and at the previously
generated tokens, which are pre-pended a special token named beginning of
sentence ($<$bos$>$). Formally, the probability $p_{V}(y_{t})$ over the
vocabulary $V$ at time step $t$ is:

$\text{softmax}(D(E(X);\text{{$<$bos$>$}},y_{0},...,y_{t-1}))$ (1)

where E is the encoder, D is the decoder, X is the input sequence, and $y_{i}$
is the token generated at the $i$-th time step.

While early attempts to build multilingual MT models were based on training
dedicated encoders and decoders for each language [11, 12], nowadays the
preferred solution is a model made of a single universal encoder and decoder
where the language is represented as a tag pre-pended to the text [13, 14, 9].
In the case of one-to-many multilingual models, this means that the $<$bos$>$
token is replaced with a token that indicates the language, so that Eq. 1
becomes:

$\text{softmax}(D(E(X);\text{LID},y_{0},...,y_{t-1}))$ (2)

where LID is the identifier of the desired target language.

In direct ST, [15] demonstrated the effectiveness of this solution, also known
as “target forcing”, while [16] proposed other methods to integrate the
language information into the architecture. Thanks to its simplicity and
effectiveness, target forcing is currently the most widespread method to build
multilingual ST systems [17, 18], also when using large pre-trained textual
models such as mBART [19] to initialize the ST decoder [20, 21]. In line with
this trend, [7] obtained their best multi-gender models with target forcing.
As such, we build multi-gender models using target forcing with the F and M
tags representing the two grammatical genders instead of the language
identifiers.

###  2.2 Gender Information Removal

With the goal of fairer technology that does not rely on spurious cues
reflecting stereotypical biases in the available data, researchers have tried
to build systems that achieve “equalized odds” among different demographic
groups [22]. Formally, this means that, given an attribute $z$ representing
the belonging to one of the $Z$ demographic groups, the predicted probability
$p(\hat{Y})$ of a fair system should be independent of the variable $z$, i.e.
$p(\hat{Y})=p(\hat{Y}|z),\forall z\in Z$. The variable $z$ is named the
protected attribute and in the context of gender bias literature represents
the gender of the involved person. So in this work we consider
$Z=\\{F,M\\}$.333Although this paper does not aim at perpetuating a binary
vision of gender, in this work we limit to the feminine and masculine
categories for the sake of simplicity, as the available benchmarks currently
do not cover the non-binary case.

The first attempts to achieve equalized odds across genders in neural systems
have focused on deep neural network (DNN) classifiers [23, 24, 25, 26]. In
this line of work, the last hidden representation of the DNN is passed both to
a linear layer that predicts the classification scores $\hat{Y}$ and to a
linear layer (the discriminator) devoted to predicting the protected attribute
$z$. The DNN is then trained in an adversarial manner [27], i.e. it is
alternatively trained to i) predict $z$ (while keeping the shared DNN freezed)
and ii) predict $\hat{Y}$ while minimizing the ability to predict $z$ (keeping
the protected attribute classification layer freezer). As this training
procedure is often unstable, similar practices based on minmax optimizations
have been proposed [28], even with discriminators made of functions different
from linear projections [29], or using more than one discriminator [30]. [31]
also proposed methods to automatically extract the protected attributes in
case they have not been provided.

Such adversarial training procedures can be seen as an extension of the
gradient reversal [32], where the training alternatively freezes the base
model (to refine the discriminator) and the discriminator (inverting its loss
to train the base model to be unable to discriminate). In fact, the gradient
reversal layer, applied to the hidden representations before feeding them to
the discriminator, is an identity function in the forward pass, while inverts
the gradient in the backward pass, scaling it by a positive factor $\lambda$.
By naming $x$ the hidden representation, $D$ the discriminator, and $GRL$ the
gradient reversal layer, this means that:

$GRL(x)=x,\nabla D(GRL(x))=-\lambda\nabla D(x)$ (3)

where $\lambda$ is a hyperparameter that can either be fixed or updated
according to the following equation:

$\lambda=\frac{2}{1+e^{-\gamma p}}-1$ (4)

where $p$ is the ratio between the number of parameter updates performed and
the total number of updates needed to complete the training.

##  3 Solutions for Multi-gender Models

To create multi-gender ST models that solely rely on the gender tag, ignoring
spurious cues related to speakers’ vocal traits, we test two approaches.
First, we try to create gender-invariant encoder representations by adding a
gradient-reverted discriminator on the speaker’s gender (§3.1). Second, we
manipulate the input audio by altering the speaker’s pitch, so that the
correlation between the gender tag (and output text) and the speaker’s vocal
traits is lost (§3.2).

###  3.1 Gradient Reversal

As seen in §2.1, the decoder of a multi-gender model has three inputs: the
encoder output, a tag representing the speaker’s gender, and the previously
generated tokens. As we want the decoder to have the tag as the only source of
information about the speaker’s gender, we propose to create encoder outputs
that do not convey any information regarding the speaker’s gender by adding a
gradient-reverted discriminator on top of the encoder, motivated by the
success of this approach in MT with sentences where there is a single referent
whose gender has to be determined [33]. The discriminator is made of two
fully-connected layers with ReLU activation function [34], whose output is
averaged over the temporal dimension to obtain a single vector representing
the logit444The logit is the vector of raw predictions before a function
(commonly, the softmax) that maps it into probabilities. of the discriminator.

Furthermore, we experiment with assigning dedicated class weights to the loss
of the discriminator, as a countermeasure to the class imbalance between
female and male speakers in the training data. Specifically, we assigned the
weights ($w_{f},w_{m}$) proportionally to the inverse of the frequency of each
class ($f_{f},f_{m}$) in the training data:

$\begin{cases}w_{f}\propto\frac{1}{f_{f}},w_{m}\propto\frac{1}{f_{m}}\\\
w_{f}*\frac{f_{f}}{f_{m}+f_{f}}+w_{m}*\frac{f_{m}}{f_{m}+f_{f}}=1\end{cases}$

resulting in $w_{f}=1.4$, $w_{m}=0.8$ in our case.

###  3.2 Audio Manipulation

Our second approach aims to break the correlation between the vocal
characteristics of the speaker on one side and the gender tag and target
translation on the other. To this aim we manipulate part of the training data
using the Opposite pitch manipulation strategy by [35]. The amount of data
that is manipulated at each iteration (epoch) is controlled with a
hyperparameter, $p$, which determines the probability of altering an
utterance, regardless of whether it is produced by a male or female speaker.
The manipulation is performed by altering two crucial acoustic parameters
distinguishing between male and female voices [36, 37]: $f0$ and formants. In
particular, we first estimate the $\tilde{f\mathit{0}}$ median of the $f0$
contour of the considered speech segment. Then, we sample a new
$\tilde{f\mathit{0}}^{\prime}$ median of the desired output audio from a
normal distribution whose mean and standard deviation depend on the target
gender: for feminine voices, we use 250 Hz as the mean and 17 as the standard
deviation so that the sampled value is between 199 Hz and 301 Hz with 99.7%
probability; for masculine voices, the mean is 140 Hz and the standard
deviation is 20 to obtain a 99.7% probability range within 80 Hz and 200 Hz.
Once $\tilde{f\mathit{0}}$ and $\tilde{f\mathit{0}}^{\prime}$ are defined, we
compute a scaling factor $\alpha$ as the ratio
$\tilde{f\mathit{0}^{\prime}}/\tilde{f\mathit{0}}$. Lastly, the original $f0$
contour is scaled by the $\alpha$ factor, while the formants are scaled by 1.2
when converting from male to female voices, or by 0.8 otherwise. This
perturbation is applied independently to each sample during each training
epoch, so as to maximize the variability of the training data.

Model | BLEU ($\uparrow$) | Gender Accuracy ($\uparrow$)
---|---|---
1F | 1M | 1F-Tag M | 1M-Tag F
Fine-tuning
Specialized | 27.4 |  73.3 |  92.5 | 80.9 | 56.1
Multi-gender | 26.0 |  66.8 |  78.0 | 64.6 | 47.1
\+ gradient reversal | 26.7 |  60.7 |  85.9 | 77.0 | 43.3
\+ gradient reversal weighted | 26.3 |  62.4 |  83.5 | 77.7 | 45.9
\+ audio manipulation (50%) | 26.4 |  56.0 |  82.6 | 60.7 | 33.9
\+ audio manipulation (80%) | 26.3 |  69.3 |  81.1 | 69.0 | 47.7
Training from scratch
Multi-gender | 27.2 |  84.0 |  92.7 | 93.4 | 69.0
\+ gradient reversal | 24.9 |  70.9 |  93.2 | 94.1 | 58.7
\+ gradient reversal weighted | 24.2 |  75.8 |  92.6 | 92.8 | 63.5
\+ audio manipulation (50%) | 26.2 |  79.6 |  92.4 | 91.5 | 67.9
\+ audio manipulation (80%) | 25.7 |  81.7 |  92.6 | 91.0 | 65.3
Table 1: BLEU and gender accuracy scores for the specialized models
(Specialized) and the multi-gender models (Multi-gender) both trained from
scratch and fine-tuned, also with gradient reversal and audio manipulation.

##  4 Experimental Settings

Our ST models are composed of a Conformer [38] encoder with 12 layers and a
Transformer [10] decoder with 6 layers. We used the Conformer implementation
by [39], which does not contain bugs related to the presence of padding. The
embedding size was 512, and the dropout was set to 0.1. We optimized label-
smoothed cross entropy using Adam. The learning rate followed the Noam
scheduler with 25,000 warmup updates and a maximum value of $2e^{-3}$. We
train for 50,000 updates and average the last 7 checkpoints.

We train our models on MuST-C [40], an ST corpus built from TED data, for
which is also available the annotation of the gender of the speaker [7]. We
extract 80 features with log mel-filterbank from the input audio and normalize
them with cepstral mean and variance [41]. The target text is encoded into
subwords with 8,000 BPE merge rules [42] learned on the training set. We
evaluate on the MuST-SHE benchmark [4], which contains a section (“Category
1”) dedicated to assessing the gender assignment of words referring to the
speaker. We compute
SacreBLEU555case:mixed$|$eff:no$|$tok:13a$|$smooth:exp$|$version:2.0.0 [43] on
the whole MuST-SHE test set to evaluate the translation quality of our models
and gender accuracy [7] on the feminine and masculine sections of “Category 1”
to evaluate the ability of each model to correctly assign gender to words
referring to the speaker.

#### Gradient Reversal.

The loss of the auxiliary speaker-classification task is summed to the loss on
the decoder output scaling it by a $0.5$ factor. For the gradient reversal
layer, we tested both fixed values of $\lambda$ and controlling its value with
$\gamma$. For fine-tunings, we set $\lambda=10$, so as to give similar weight
to the gender classification loss and the cross-entropy loss for the
translation. When training from scratch, instead, despite different attempts
the training is unstable and diverges unless lambda is set to a fixed, small
value, where its contribution is negligible. We report results for
$\lambda=0.5$, which is the highest $\lambda$ value for which the loss on the
validation set does not explode during training.

#### Audio Manipulation.

In our experiments, we tested two values (0.5 and 0.8) for the hyperparameter
$p$, which controls the probability of manipulating a speech segment. In the
first case, 50% of the data is manipulated, leading to a complete loss of
correlation between gender tags and vocal traits (50% of the samples with the
F tag would exhibit frequency characteristics typical of masculine voices, and
50% of the samples with the M tag would have frequency characteristics typical
of feminine voices). In the second case, instead, the correlation between the
gender tag and the vocal traits is negative, to counteract the patterns
learned by a gender-unaware ST model. In any case, as the training data is
imbalanced (70% of the samples are uttered by male speakers, and 30% by female
speakers), and the manipulation probability is the same for segments uttered
by male and female speakers, the gender imbalance in the training data is not
mitigated.

##  5 Results

We investigate the performance of multi-gender models trained in two different
ways: i) fine-tuning a base, gender-unaware ST model, and ii) training from
scratch. In both cases, we study the effect of the introduction of the
discriminator with gradient reversal and of the audio manipulation techniques.
Table 1 presents BLEU and gender accuracy scores (separately for segments
spoken by female (1F) and male (1M) speakers) for all the models, comparing
them with the specialized models. To assess the inclusivity of our solution in
cases where speakers exhibit vocal traits that do not conform with traditional
gender perceptions, we also report gender accuracy for tests where the gender
tag is inverted compared to the original audio segment (1F-Tag M and 1M-Tag
F). In these instances, the gender translation is expected to align with the
gender tag, and we use the “wrong” reference of MuST-SHE, which swaps the
speaker’s references to the opposite gender.

#### Fine-tuning.

The fine-tuned models from a base ST system consistently yield lower scores
compared to the specialized systems. The simple multi-gender model performs
1.4 BLEU points worse than the specialized models in terms of overall
translation quality. However, when audio manipulation and especially gradient
reversal techniques are employed during fine-tuning, the performance gap is
reduced by up to half. Regarding gender accuracy, the multi-gender model
achieves considerably lower scores than the specialized models, confirming
previous findings from [7]. This indicates that a fine-tuned multi-gender
model struggles to accurately follow the gender tag for gender translation.
The accuracy gap is particularly high when the tag conflicts with vocal traits
(-16.3 in 1F-Tag M, -9.0 in 1M-Tag F), where multi-gender models show below-
chance accuracy for feminine forms, being below 50%. Both gradient reversal
and audio manipulation techniques seem to further bias the model towards
masculine forms. This likely indicates that the reduced ability to rely on the
speakers’ vocal traits is not compensated by the model looking at the gender
tag, rather it strengthens its tendency to default to the most frequent
masculine forms. The only technique that consistently improves both masculine
and feminine translations compared to the simple fine-tuned multi-gender model
is the introduction of audio manipulation with high probability (80%).
However, the gains (2.5 for 1F and 3.1 for 1M, and 4.4 for 1F-Tag M and 0.7
for 1M-Tag F) are limited and the gap with specialized models remains large.

#### Training from scratch.

Unlike the fine-tuned models, the multi-gender models trained from scratch
yield comparable or even higher results than the specialized models. This
suggests that when an ST model is trained from scratch with gender tags, it
learns to effectively follow them. Specifically, the simple multi-gender model
trained from scratch achieves comparable translation quality to the
specialized system (-0.2 BLEU) and significantly outperforms it in gender
accuracy, with gains ranging from 0.2 (1M) to 12.5 (1F-Tag M). As in the fine-
tuning case, neither gradient reversal nor audio manipulations increase the
reliance of the model on the tag and the resulting models are more biased
toward masculine forms. In fact, the multi-gender with gradient reversal
reaches the highest accuracies in producing masculine forms (93.2 in 1M and
94.1 in 1F-Tag M), while suffering substantial drops in feminine accuracy.
This effect is reduced when the weight of the F class is increased in the
discriminator. In addition, in both cases the translation quality suffers a
considerable drop. In the case of audio manipulation, the translation quality
drop is lower (although still present), as well as the differences in terms of
gender accuracy. They do not provide, though, any benefit compared to the
simple multi-gender training.

In summary, the results demonstrate that the previous finding about the low
performance of multi-gender models is due to the adoption of a fine-tuning
strategy. In this setting, the model cannot effectively override the reliance
on speakers’ vocal traits of the gender-unaware base ST model. In addition,
techniques aimed at avoiding the exploitation of speakers’ vocal traits seem
ineffective. However, training the multi-gender model effectively solves the
problem and the model is capable of following the indication given by the
gender tag, outperforming even the specialized strategy by up to 12.9 gender
accuracy (1M-Tag F).

##  6 Conclusions

In this paper, we studied the effect of different training strategies to build
multi-gender ST models, i.e. models that are informed of the gender of the
speaker by an explicit gender tag. Focusing on English-Italian translations,
we demonstrated that the low accuracy of multi-gender models shown by previous
work stems from the their initialization with gender-unaware ST system weights
and the inability of effectively overriding the reliance on vocal cues during
fine-tuning. On the other hand, training multi-gender models from scratch
proved to be an effective solution, outperforming the approach based on the
creation of two gender-specialized models. As training from scratch is not
always feasible, we also experimented with two methods to enhance the reliance
on the gender tag in fine-tuned multi-gender models: penalizing the extraction
of gender cues from speech input, and altering the vocal properties of the
speakers in the training data to avoid the alignment between biological cues
and gender tags and translations. While these solutions partially improved
gender accuracy and overall translation quality in fine-tuned multi-gender
models, they did not close the gap with specialized models. Therefore, further
research is needed in this direction.

###### Acknowledgements.

This work is part of the project “Bias Mitigation and Gender Neutralization
Techniques for Automatic Translation”, which is financially supported by an
Amazon Research Award AWS AI grant. We acknowledge the support of the PNRR
project FAIR - Future AI Research (PE00000013), under the NRRP MUR program
funded by the NextGenerationEU.

## References

  * Savoldi et al. [2021] B. Savoldi, M. Gaido, L. Bentivogli, M. Negri, M. Turchi,  Gender bias in machine translation,  Transactions of the Association for Computational Linguistics 9 (2021) 845–874. URL: https://aclanthology.org/2021.tacl-1.51. doi:10.1162/tacl_a_00401.
  * Prates et al. [2020] M. O. R. Prates, P. H. C. Avelar, L. C. Lamb,  Assessing gender bias in machine translation: a case study with google translate,  Neural Computing and Applications 32 (2020) 6363–6381. doi:10.1007/s00521-019-04144-6.
  * Cho et al. [2019] W. I. Cho, J. W. Kim, S. M. Kim, N. S. Kim,  On Measuring Gender bias in Translation of Gender-neutral Pronouns,  in: Proceedings of the First Workshop on Gender Bias in Natural Language Processing, Association for Computational Linguistics, Florence, Italy, 2019, pp. 173–181. URL: https://www.aclweb.org/anthology/W19-3824. doi:10.18653/v1/W19-3824.
  * Bentivogli et al. [2020] L. Bentivogli, B. Savoldi, M. Negri, M. A. Di Gangi, R. Cattoni, M. Turchi,  Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus,  in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 6923–6933. URL: https://www.aclweb.org/anthology/2020.acl-main.619.
  * Gaido et al. [2021] M. Gaido, M. A. Di Gangi, M. Negri, M. Turchi,  On Knowledge Distillation for Direct Speech Translation,  in: Proceedings of CLiC-IT 2020, Online, 2021. URL: http://ceur-ws.org/Vol-2769/paper_28.pdf.
  * Zimman [2020] L. Zimman,  Transgender language, transgender moment: Toward a trans linguistics,  in: K. Hall, R. Barrett (Eds.), The Oxford Handbook of Language and Sexuality, 2020. doi:10.1093/oxfordhb/9780190212926.013.45.
  * Gaido et al. [2020] M. Gaido, B. Savoldi, L. Bentivogli, M. Negri, M. Turchi,  Breeding Gender-aware Direct Speech Translation Systems,  in: Proceedings of the 28th International Conference on Computational Linguistics, International Committee on Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 3951–3964. URL: https://www.aclweb.org/anthology/2020.coling-main.350. doi:10.18653/v1/2020.coling-main.350.
  * Choubey et al. [2021] P. K. Choubey, A. Currey, P. Mathur, G. Dinu,  GFST: Gender-filtered self-training for more accurate gender in translation,  in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021, pp. 1640–1654. URL: https://aclanthology.org/2021.emnlp-main.123. doi:10.18653/v1/2021.emnlp-main.123.
  * Johnson et al. [2017] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, M. Hughes, J. Dean,  Google’s multilingual neural machine translation system: Enabling zero-shot translation,  Transactions of the Association for Computational Linguistics 5 (2017) 339–351. URL: https://aclanthology.org/Q17-1024. doi:10.1162/tacl_a_00065.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, I. Polosukhin,  Attention is all you need,  in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Proceedings of the 31st International Conference on Neural Information Processing Systems, Curran Associates Inc., Long Beach, USA, 2017. URL: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Firat et al. [2016a] O. Firat, K. Cho, Y. Bengio,  Multi-way, multilingual neural machine translation with a shared attention mechanism,  in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, San Diego, California, 2016a, pp. 866–875. URL: https://aclanthology.org/N16-1101. doi:10.18653/v1/N16-1101.
  * Firat et al. [2016b] O. Firat, B. Sankaran, Y. Al-onaizan, F. T. Yarman Vural, K. Cho,  Zero-resource translation with multi-lingual neural machine translation,  in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, Texas, 2016b, pp. 268–277. URL: https://aclanthology.org/D16-1026. doi:10.18653/v1/D16-1026.
  * Sennrich et al. [2016] R. Sennrich, B. Haddow, A. Birch,  Improving Neural Machine Translation Models with Monolingual Data,  in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Berlin, Germany, 2016, pp. 86–96. URL: https://aclanthology.org/P16-1009. doi:10.18653/v1/P16-1009.
  * Ha et al. [2016] T.-L. Ha, J. Niehues, A. Waibel,  Toward multilingual neural machine translation with universal encoder and decoder,  in: Proceedings of the 13th International Conference on Spoken Language Translation, International Workshop on Spoken Language Translation, Seattle, Washington D.C, 2016. URL: https://aclanthology.org/2016.iwslt-1.6.
  * Inaguma et al. [2019] H. Inaguma, K. Duh, T. Kawahara, S. Watanabe,  Multilingual end-to-end speech translation,  in: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019, pp. 570–577. doi:10.1109/ASRU46091.2019.9003832.
  * Di Gangi et al. [2019] M. A. Di Gangi, M. Negri, M. Turchi,  One-to-many multilingual end-to-end speech translation,  in: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019, pp. 585–592. doi:10.1109/ASRU46091.2019.9004003.
  * Wang et al. [2020] C. Wang, Y. Tang, X. Ma, A. Wu, D. Okhonko, J. Pino,  Fairseq S2T: Fast speech-to-text modeling with fairseq,  in: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Suzhou, China, 2020, pp. 33–39. URL: https://aclanthology.org/2020.aacl-demo.6.
  * Salesky et al. [2021] E. Salesky, M. Wiesner, J. Bremerman, R. Cattoni, M. Negri, M. Turchi, D. W. Oard, M. Post,  The Multilingual TEDx Corpus for Speech Recognition and Translation,  in: Proc. Interspeech 2021, 2021, pp. 3655–3659. doi:10.21437/Interspeech.2021-11.
  * Liu et al. [2020] Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, L. Zettlemoyer,  Multilingual denoising pre-training for neural machine translation,  Transactions of the Association for Computational Linguistics 8 (2020) 726–742. URL: https://aclanthology.org/2020.tacl-1.47. doi:10.1162/tacl_a_00343.
  * Liu et al. [2023] D. Liu, T. Binh Nguyen, S. Koneru, E. Yavuz Ugan, N.-Q. Pham, T. Nam Nguyen, T. Anh Dinh, C. Mullov, A. Waibel, J. Niehues,  KIT’s multilingual speech translation system for IWSLT 2023,  in: Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), Association for Computational Linguistics, Toronto, Canada (in-person and online), 2023, pp. 113–122. URL: https://aclanthology.org/2023.iwslt-1.6.
  * Gow-Smith et al. [2023] E. Gow-Smith, A. Berard, M. Zanon Boito, I. Calapodescu,  NAVER LABS Europe’s multilingual speech translation systems for the IWSLT 2023 low-resource track,  in: Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), Association for Computational Linguistics, Toronto, Canada (in-person and online), 2023, pp. 144–158. URL: https://aclanthology.org/2023.iwslt-1.10.
  * Hardt et al. [2016] M. Hardt, E. Price, N. Srebro,  Equality of opportunity in supervised learning,  in: Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, Curran Associates Inc., Red Hook, NY, USA, 2016, p. 3323–3331.
  * Beutel et al. [2017] A. Beutel, E. H. Chi, J. Chen, Z. Zhao,  Data decisions and theoretical implications when adversarially learning fair representations,  2017\. URL: https://arxiv.org/pdf/1707.00075.pdf.
  * Zhang et al. [2018] B. H. Zhang, B. Lemoine, M. Mitchell,  Mitigating unwanted biases with adversarial learning,  in: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 335–340. URL: https://doi.org/10.1145/3278721.3278779. doi:10.1145/3278721.3278779.
  * Elazar and Goldberg [2018] Y. Elazar, Y. Goldberg,  Adversarial removal of demographic attributes from text data,  in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium, 2018, pp. 11–21. URL: https://aclanthology.org/D18-1002. doi:10.18653/v1/D18-1002.
  * Gao et al. [2022] L. Gao, H. Zhan, A. Chen, V. Sheng, Mitigate gender bias using negative multi-task learning, 2022. URL: https://doi.org/10.21203/rs.3.rs-2024101/v1. doi:10.21203/rs.3.rs-2024101/v1.
  * Goodfellow et al. [2020] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio,  Generative adversarial networks,  Commun. ACM 63 (2020) 139–144. URL: https://doi.org/10.1145/3422622. doi:10.1145/3422622.
  * Ravfogel et al. [2022a] S. Ravfogel, M. Twiton, Y. Goldberg, R. D. Cotterell,  Linear adversarial concept erasure,  in: K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, S. Sabato (Eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, PMLR, 2022a, pp. 18400–18421. URL: https://proceedings.mlr.press/v162/ravfogel22a.html.
  * Ravfogel et al. [2022b] S. Ravfogel, F. Vargas, Y. Goldberg, R. Cotterell,  Adversarial concept erasure in kernel space,  in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022b, pp. 6034–6055. URL: https://aclanthology.org/2022.emnlp-main.405.
  * Han et al. [2021] X. Han, T. Baldwin, T. Cohn,  Diverse adversaries for mitigating bias in training,  in: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Association for Computational Linguistics, Online, 2021, pp. 2760–2765. URL: https://aclanthology.org/2021.eacl-main.239. doi:10.18653/v1/2021.eacl-main.239.
  * Shao et al. [2023] S. Shao, Y. Ziser, S. B. Cohen,  Erasure of unaligned attributes from neural representations,  Transactions of the Association for Computational Linguistics 11 (2023) 488–510. URL: https://aclanthology.org/2023.tacl-1.29. doi:10.1162/tacl_a_00558.
  * Ganin and Lempitsky [2015] Y. Ganin, V. Lempitsky,  Unsupervised Domain Adaptation by Backpropagation,  in: F. Bach, D. Blei (Eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, PMLR, Lille, France, 2015, pp. 1180–1189. URL: https://proceedings.mlr.press/v37/ganin15.html.
  * Fleisig and Fellbaum [2022] E. Fleisig, C. Fellbaum, Mitigating Gender Bias in Machine Translation through Adversarial Learning, 2022. arXiv:2203.10675.
  * Nair and Hinton [2010] V. Nair, G. E. Hinton,  Rectified linear units improve restricted boltzmann machines,  in: Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, Omnipress, Madison, WI, USA, 2010, p. 807–814.
  * Fucci et al. [2023] D. Fucci, M. Gaido, M. Negri, M. Cettolo, L. Bentivogli,  No Pitch Left Behind: Addressing Gender Unbalance in Automatic Speech Recognition through Pitch Manipulation,  in: IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Taipei, Taiwan, 2023.
  * Coleman [1976] R. O. Coleman,  A comparison of the contributions of two voice quality characteristics to the perception of maleness and femaleness in the voice,  Journal of Speech & Hearing Research 19(1) (1976) 168–180. doi:https://doi.org/10.1044/jshr.1901.168.
  * Hillenbrand and Clark [2009] J. M. Hillenbrand, M. J. Clark,  The role of f0 and formant frequencies in distinguishing the voices of men and women,  Attention Perception & Psychophysics 71(5) (2009) 1150–1166. doi:10.3758/APP.71.5.1150.
  * Gulati et al. [2020] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, R. Pang,  Conformer: Convolution-augmented Transformer for Speech Recognition,  in: Proceedings of the 21st Annual Conference of the International Speech Communication Association, International Speech Communication Association, Shanghai, China (Online), 2020, pp. 5036–5040. doi:10.21437/Interspeech.2020-3015.
  * Papi et al. [2023] S. Papi, M. Gaido, A. Pilzer, M. Negri, When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP, 2023. arXiv:2303.16166.
  * Cattoni et al. [2021] R. Cattoni, M. A. Di Gangi, L. Bentivogli, M. Negri, M. Turchi,  Must-c: A multilingual corpus for end-to-end speech translation,  Computer Speech & Language 66 (2021) 101–155. URL: https://www.sciencedirect.com/science/article/pii/S0885230820300887. doi:https://doi.org/10.1016/j.csl.2020.101155.
  * Viikki and Laurila [1998] O. Viikki, K. Laurila,  Cepstral domain segmental feature vector normalization for noise robust speech recognition,  Speech Communication 25 (1998) 133–147. URL: https://www.sciencedirect.com/science/article/pii/S0167639398000338. doi:https://doi.org/10.1016/S0167-6393(98)00033-8.
  * Di Gangi et al. [2020] M. A. Di Gangi, M. Gaido, M. Negri, M. Turchi,  On target segmentation for direct speech translation,  in: Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), Association for Machine Translation in the Americas, Virtual, 2020, pp. 137–150. URL: https://aclanthology.org/2020.amta-research.13.
  * Post [2018] M. Post,  A call for clarity in reporting BLEU scores,  in: Proceedings of the Third Conference on Machine Translation: Research Papers, Association for Computational Linguistics, Belgium, Brussels, 2018, pp. 186–191. URL: https://www.aclweb.org/anthology/W18-6319.