# The self-dual Lorentz violating model: quantization, scattering and dual
equivalence

M. A. Anacleto<EMAIL_ADDRESS>Departamento de Física, Universidade
Federal de Campina Grande Caixa Postal 10071, 58429-900 Campina Grande,
Paraíba, Brazil  F. A. Brito<EMAIL_ADDRESS>Departamento de Física,
Universidade Federal de Campina Grande Caixa Postal 10071, 58429-900 Campina
Grande, Paraíba, Brazil  Departamento de Física, Universidade Federal da
Paraíba, Caixa Postal 5008, 58051-970 João Pessoa, Paraíba, Brazil  E. Passos
<EMAIL_ADDRESS>Departamento de Física, Universidade Federal de Campina
Grande Caixa Postal 10071, 58429-900 Campina Grande, Paraíba, Brazil

###### Abstract

In this paper, we analysis the dynamics, at the quantum level, of the self-
dual field minimally coupled to bosons with Lorentz symmetry breaking. We
quantize the model by applying the Dirac bracket canonical quantization
procedure. In addition, we test the relativistic invariance of the model by
computing the boson-boson elastic scattering amplitude. Therefore, we show
that the Lorentz symmetry breaking has been restored at the quantum level. We
finalize our analysis by computing the dual equivalence between the self-dual
model with Lorentz symmetry breaking coupled with bosonic matter and the
Maxwell-Chern-Simons with Lorentz invariance violation coupled with bosonic
field.

##  I Introduction

In physics, symmetries play important roles. In particular, the violation of
Lorentz symmetry initiated by Kostelecky and collaborators [1] has been widely
investigated in various fields. The motivation comes from the theory of
superstrings indicating that the Lorentz symmetry would be broken at high
energies. It is well known that Lorentz symmetry is of fundamental importance
in the construction of physical theories. Therefore, this symmetry was tested
with great precision [2]. From a theoretical point of view, it is conjectured
that violations of Lorentz symmetry can occur on the Planck scale. In
addition, these effects are observed even in the low-energy region. In [3],
the breaking of Lorentz symmetry in field theory was studied. Lorentz symmetry
breaking in QED was investigated in [4, 5, 6]. Here, our focus of attention is
on the self-dual field model coupled with bosons, in the background violating
Lorentz symmetry.

The self-dual field model proposed by Townsend, Pilch and Van Nieuwenhuizen
[7] has received a lot of attention. The equivalence between the free self-
dual model and the Maxwell Chern-Simons model was established in [8, 9]. Also,
in [10], equivalence at the level of the Green function has been established.
In [11], Gomes and collaborators explore the equivalence between the self-dual
model minimally coupled with a fermion field and the Maxwell Chern-Simons
model coupled with fermions. In addition, the quantization of the self-dual
model coupled with fermions and the relativistic invariance test via
calculation of the fermion-fermion scattering amplitude was performed [12].
Furthermore, the quantization of the self-dual model coupled with bosons, as
well as the relativistic invariance test computing the boson-boson scattering
was also studied in [13]. Later, the dual equivalence of the self-dual model
coupled with bosons and the Maxwell Chern-Simons model coupled with bosons was
established [14]. An analysis on the coupling of the self-dual field to
dynamic U(1) matter and its dual theory has been carried out in [15] Duality
in the context of Lorentz symmetry breaking has been investigated in four-
dimensional [16, 17, 18] and three-dimensional theories [19]. Besides,
different aspects of duality were constructed [20, 21, 22, 23, 24]. Also see
[25, 26, 27, 28, 29, 30, 31, 32] for other applications. Recently, in [33] the
duality between the Maxwell-Chern-Simons and self-dual models in very special
relativity has been investigated.

In this paper, we are interested in studying the canonical quantization of the
self-dual field model coupled with bosons violating Lorentz symmetry. In this
study, we will quantize the model by applying the canonical quantization
procedure of the Dirac brackets. Besides, we will compute the boson-boson
scattering amplitude to test the Lorentz invariance of the model. Therefore,
as a result of this analysis, we show that the combined action of the non-
covariant parts of the interaction Hamiltonian is replaced by the minimum
covariant field-current interaction. In addition, we verify that Lorentz
invariance is preserved at the quantum level. Moreover, we investigate the
duality between the self-dual and Maxwell-Chern-Simons models with Lorentz
symmetry breaking and coupled with bosonic matter.

The paper is organized as follows. In Sec. II we introduce the Lorentz
symmetry breaking effect in the self-dual field model coupled with bosons and
quantize the model via Dirac brackets. In Sec. III we compute the boson-boson
elastic scattering amplitude to test the Lorentz invariance of the model. In
Sec. IV we explore the dual equivalence between the self-dual and Maxwell-
Chern-Simons models with Lorentz symmetry breaking and coupled with bosonic
matter. Finally in Sec. V we present our final considerations.

##  II The Lorentz-breaking self-dual model

In this section, we consider the self-dual field model coupled to bosons with
Lorentz symmetry breaking, given by the following Lagrangian:

$\displaystyle\mathcal{L}=\mathcal{L}_{sd}+\mathcal{L}_{\phi
f}+\mathcal{L}_{\varphi}+\mathcal{L}_{int},$ (1)

being

$\displaystyle\mathcal{L}_{sd}$ $\displaystyle=$
$\displaystyle\frac{m}{2}\epsilon^{\mu\nu\rho}(f_{\mu}\partial_{\nu}f_{\rho})-\frac{m^{2}}{2}f_{\mu}f^{\mu},$
(2) $\displaystyle\mathcal{L}_{\phi f}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\partial_{\mu}\phi\partial^{\mu}\phi+2m\phi
v^{\mu}f_{\mu},$ (3) $\displaystyle\mathcal{L}_{\varphi}$ $\displaystyle=$
$\displaystyle\partial_{\mu}\varphi^{\ast}\partial^{\mu}\varphi-M^{2}\varphi^{\ast}\varphi,$
(4) $\displaystyle\mathcal{L}_{int}$ $\displaystyle=$ $\displaystyle-
ef_{\mu}J^{\mu}+e^{2}f_{\mu}f^{\mu}\varphi^{\ast}\varphi-e^{2}\phi^{2}\varphi^{\ast}\varphi,$
(5)

where
$J_{\mu}=i(\varphi^{\ast}\partial_{\mu}\varphi-\partial_{\mu}\varphi^{\ast}\varphi)$
is the current, $m$ is a parameter with dimensions of mass, $\varphi$ is the
charged scalar field, $M$ is the boson mass, $f_{\mu}$ is the self-dual field
and $\phi$ is the real scalar field interacting with the self-dual field
$f_{\mu}$. Here, $2m\phi v^{\mu}f^{\mu}$ is the term carrying the Lorentz
symmetry breaking with the constant 3-vector introducing a preferred frame of
reference in spacetime. The model consisting of $\mathcal{L}_{sd}$ and
$\mathcal{L}_{\phi f}$ was considered in [19] to examine the self-
dual/Maxwell-Chern-Simons duality with Lorentz symmetry-breaking. In the
Maxwell-Chern-Simons model with Lorentz symmetry breaking, the mixed term,
$\phi\epsilon^{\mu\nu\rho}F_{\nu\rho}$, was obtained through the dimensional
reduction of the Jackiw term [34, 35, 36]. For other applications of this
term, see also [37, 38, 39].

Now we start the development of the canonical quantization approach of the
self-dual model coupled to bosons and with Lorentz symmetry breaking. Then, by
computing the canonically conjugate moments of the model, we find

$\displaystyle\pi_{\alpha}$ $\displaystyle=$ $\displaystyle\frac{\partial{\cal
L}}{\partial(\partial_{0}f_{\alpha})}=-\frac{m}{2}\epsilon_{0\alpha\mu}f^{\mu},$
(6) $\displaystyle\Pi_{\phi}$ $\displaystyle=$
$\displaystyle\frac{\partial{\cal
L}}{\partial(\partial_{0}\phi)}=\partial_{0}\phi,$ (7) $\displaystyle{\Pi}$
$\displaystyle=$ $\displaystyle\frac{\partial{\cal
L}}{\partial(\partial_{0}\varphi)}=\partial_{0}\varphi^{\ast}-ief^{0}\varphi^{\ast},$
(8) $\displaystyle{\Pi}^{\ast}$ $\displaystyle=$
$\displaystyle\frac{\partial{\cal
L}}{\partial(\partial_{0}\varphi^{\ast})}=\partial_{0}\varphi+ief^{0}\varphi.$
(9)

Hence, we find that the primary constraints are given by

$\displaystyle\Phi^{(1)}_{0}$ $\displaystyle=$ $\displaystyle\pi_{0}\approx
0,$ (10) $\displaystyle\Phi^{(1)}_{i}$ $\displaystyle=$
$\displaystyle\pi_{i}+\frac{m}{2}\epsilon_{ij}f^{j}\approx 0.$ (11)

According to the Dirac quantization procedure, the symbol ($\approx$)
introduced above means weak equality and with equality occurring on the
constraint surface [40]. However, the canonical Hamiltonian density is given
by

$\displaystyle{\cal H}$ $\displaystyle=$
$\displaystyle\Pi_{\phi}\partial_{0}\phi+\Pi\partial_{0}\varphi+\Pi^{\ast}\partial_{0}\varphi^{\ast}+\pi_{i}\partial_{0}f^{i}-{\cal
L},$ (12) $\displaystyle=$
$\displaystyle\Pi^{\ast}\Pi+\frac{\Pi_{\phi}^{2}}{2}-\frac{1}{2}\partial_{i}\phi\partial^{i}\phi-\partial_{i}\varphi^{\ast}\partial^{i}\varphi+M^{2}\varphi^{\ast}\varphi+e^{2}\phi^{2}\varphi^{\ast}\varphi+ef^{i}J_{i}-e^{2}f_{i}f^{i}\varphi^{\ast}\varphi$
$\displaystyle+$
$\displaystyle\frac{m^{2}}{2}f_{\mu}f^{\mu}-2m\phi{v_{i}f^{i}}-f^{0}[m\epsilon_{ij}\partial^{i}f^{j}+ie(\Pi\varphi-\Pi^{\ast}\varphi^{\ast})+2m\phi{v_{0}}].$

Thus, the primary Hamiltonian can be presented as

$\displaystyle H_{p}=\int d^{2}x({\cal
H}+U^{0}\Phi^{(1)}_{0}+U^{i}\Phi^{(1)}_{i}),$ (13)

being $U^{0}$ and $U^{i}$ the Lagrange multipliers.

Next, we apply the Dirac algorithm by imposing the condition of conservation
in time, such that:

$\displaystyle\dot{\Phi}^{(1)}_{0}=\\{\Phi^{(1)}_{0},H_{p}\\}_{p}=\\{\pi_{0}(\vec{x}),H_{p}(\vec{y})\\}_{p}\approx
0.$ (14)

This condition gives rise to the secondary constraint

$\displaystyle\Phi^{(2)}$ $\displaystyle=$ $\displaystyle
m^{2}\left(f^{0}-\frac{1}{m}\epsilon_{ij}\partial^{i}f^{j}-\frac{ie}{m^{2}}(\Pi\varphi-\Pi^{\ast}\varphi^{\ast})-\frac{2\phi{v^{0}}}{m}\right)\approx
0.$ (15)

Then, to complete the Dirac algorithm we must again check whether the
conservation of the constraints $\Phi^{(2)}$ implies new secondary
constraints. It is easy to see that no more constraints arise and the Lagrange
multipliers are determined. We can verify that the system of constraints is of
the second class.

At this point, we begin the process of quantizing the theory with second-class
constraints by introducing the Dirac brackets defined for arbitrary functions
of fields and momenta as follows:

$\displaystyle\\{\Lambda,\Omega\\}_{D}=\\{\Lambda,\Omega\\}-\sum_{q=0}^{3}\sum_{p=0}^{3}\int\int{d^{2}ud^{2}v\\{\Lambda,\xi_{p}(\vec{u})\\}}R^{-1}_{pq}\\{\xi_{q}(\vec{v}),\Omega\\},$
(16)

where $\xi_{0}\equiv\Phi^{(1)}_{0}$, $\xi_{l}\equiv\Phi^{(1)}_{l},l=1,2$ ,
$\xi_{3}\equiv\Phi^{(2)}$ and
$R_{pq}(\vec{u},\vec{v})=\\{\xi_{p}(\vec{u}),\xi_{q}(\vec{v})\\}$ is a matrix
of the Poisson brackets of the constraints (it is non-degenerate as it should
be for the system of the second-class constraints).

In the sequence to quantize the theory, we define the fields and momenta as
being operators and thus we have the following relation involving the
commutator and Dirac bracket:

$\displaystyle[\Lambda,\Omega]=i\\{\Lambda,\Omega\\}_{D}.$ (17)

Thus, after computing the Dirac brackets, we find the following commutation
relations at equal times for the dynamic variables

$\displaystyle[f^{0}(\vec{x}),f^{j}(\vec{y})]=i\partial^{j}_{x}\delta(\vec{x}-\vec{y}),$
$\displaystyle[f^{k}(\vec{x}),f^{j}(\vec{y})]=-im\epsilon^{kj}\delta(\vec{x}-\vec{y}),$
$\displaystyle[f^{0}(\vec{x}),\pi_{k}(\vec{y})]=-\frac{i}{2m}\epsilon_{kj}\partial^{j}_{x}\delta(\vec{x}-\vec{y}),$
$\displaystyle[f^{j}(\vec{x}),\pi_{k}(\vec{y})]=\frac{i}{2}g_{k}^{j}\delta(\vec{x}-\vec{y}),$
$\displaystyle[\pi_{j}(\vec{x}),\pi_{k}(\vec{y})]=-\frac{i}{4m}\epsilon_{jk}\delta(\vec{x}-\vec{y}),$
$\displaystyle[f^{0}(\vec{x}),\varphi(\vec{y})]=e\varphi\delta(\vec{x}-\vec{y}),$
$\displaystyle[f^{0}(\vec{x}),\varphi^{\dagger}(\vec{y})]=-e\varphi^{\dagger}\delta(\vec{x}-\vec{y}),$
$\displaystyle[\phi(\vec{x}),\Pi_{\phi}(\vec{y})]=i\delta(\vec{x}-\vec{y}),$
$\displaystyle[\varphi(\vec{x}),\Pi(\vec{y})]=i\delta(\vec{x}-\vec{y}),$
$\displaystyle[\phi^{\dagger}(\vec{x}),\Pi^{\dagger}(\vec{y})]=i\delta(\vec{x}-\vec{y}).$
(18)

The Hamiltonian operator (12), with use of the constraint (15), can be
rewritten as

$\displaystyle H$ $\displaystyle=$
$\displaystyle\int{d^{2}x}\left(\Pi^{\dagger}\Pi+\frac{\Pi_{\phi}^{2}}{2}-\frac{1}{2}\partial_{i}\phi\partial^{i}\phi-\partial_{i}\varphi^{\dagger}\partial^{i}\varphi+M^{2}\varphi^{\dagger}\varphi+e^{2}\phi^{2}\varphi^{\dagger}\varphi\right.$
(19) $\displaystyle+$
$\displaystyle\left.ef^{i}J_{i}-e^{2}f_{i}f^{i}\varphi^{\dagger}\varphi+\frac{m^{2}}{2}f_{i}f^{i}-\frac{m^{2}}{2}f_{0}f^{0}-2m\phi{v_{i}f^{i}}\right),$

and applying condition (15) to eliminate the $f^{0}$ operators, the
Hamiltonian takes the form

$H^{I}=H_{0}^{I}+H_{int\,1}^{I}++H_{int\,2}^{I},$ (20)

being

$\displaystyle H_{0}^{I}$ $\displaystyle=$
$\displaystyle\int{d^{2}x}\left(\Pi^{I\dagger}\Pi^{I}+\frac{(\Pi_{\phi}^{I})^{2}}{2}-\frac{1}{2}\partial_{i}\phi^{I}\partial^{i}\phi^{I}-2\phi^{I2}v_{0}^{2}-\partial_{i}\varphi^{I\dagger}\partial^{i}\varphi^{I}+M^{2}\varphi^{I\dagger}\varphi^{I}\right.$
(21) $\displaystyle+$
$\displaystyle\left.\frac{m^{2}}{2}f_{i}^{I}f^{Ii}-\frac{1}{2}\epsilon^{ij}\epsilon^{kl}(\partial_{i}f_{j}^{I})(\partial_{k}f_{l}^{I})\right),$
$\displaystyle H_{int\,1}^{I}$ $\displaystyle=$
$\displaystyle-2\int{d^{2}x}\left(mv_{i}\phi^{I}{f^{Ii}}+v_{0}\phi^{I}\epsilon^{ij}\partial_{i}f_{j}^{I}\right),$
(22)

and

$\displaystyle H_{int\,2}^{I}$ $\displaystyle=$
$\displaystyle\int{d^{2}x}\left(ef^{Ik}J_{k}^{I}-\frac{ie}{m}\epsilon^{kj}(\partial_{k}f_{j})(\Pi^{I}\varphi^{I}-\Pi^{I\dagger}\varphi^{I\dagger})-\frac{2ie}{m}v_{0}\phi^{I}(\Pi^{I}\varphi^{I}-\Pi^{I\dagger}\varphi^{I\dagger})\right.$
(23) $\displaystyle+$
$\displaystyle\left.\frac{e^{2}}{2m^{2}}(\Pi^{I}\varphi^{I}-\Pi^{I\dagger}\varphi^{I\dagger})^{2}-e^{2}f_{k}^{I}f^{Ik}\varphi^{I\dagger}\varphi^{I}+e^{2}\phi^{I2}\varphi^{I\dagger}\varphi^{I}\right).$

In the Hamiltonian operator above, we write the superscript $I$ to indicate
that the field operators belong to the interaction picture. Note that in the
Hamiltonian $H^{I}_{0}$ the Lorentz symmetry breaking term generated a mass
term for the scalar field $\phi$. In the Hamiltonian $H^{I}_{int\,1}$ we have
the interaction of the scalar field $\phi$ with the gauge field $f$, whereas
in the Hamiltonian $H^{I}_{int\,2}$ we have the interaction of the field
$\varphi$ with the gauge field $f$ and with the scalar field $\phi$. Then, the
equations of motion that $\varphi^{I}$ and $\varphi^{I\dagger}$ obey are
respectively given by:

$\displaystyle\dot{\varphi}(\vec{x})$ $\displaystyle=$ $\displaystyle
i[H_{0}^{I},\varphi^{I}(\vec{x})]=\Pi^{I\dagger},$ (24)
$\displaystyle\dot{\varphi}^{I\dagger}(\vec{x})$ $\displaystyle=$
$\displaystyle i[H_{0}^{I},\varphi^{I\dagger}(\vec{x})]=\Pi^{I}.$ (25)

Thus, the propagator of the boson field in the momentum space is

$\displaystyle\Delta(p)=\frac{i}{p^{2}-M^{2}+i\epsilon}.$ (26)

For the self-dual field ($f_{i}^{I},i=1,2$), the propagator is given by [19]

$\displaystyle\frac{1}{m^{2}}D^{lj}(k)=-\frac{i}{k^{2}-m^{2}+i\epsilon}\left(g^{lj}-\frac{k^{l}k^{j}}{m^{2}}+\frac{i}{m}\epsilon^{lj}k_{0}-\frac{4v^{l}v^{j}}{k^{2}}-\frac{v^{l}k^{j}}{k^{2}}\right),$
(27)

and for the mixed propagators, we have

$\displaystyle\langle
f^{i}\phi\rangle=\frac{i}{k^{2}-m^{2}+i\epsilon}\left(\frac{2mv^{i}}{k^{2}}\right),$
(28)

and

$\displaystyle\langle\phi\phi\rangle=-\frac{i}{p^{2}-m^{2}+i\epsilon}\left(\frac{4m^{2}}{k^{2}}\right).$
(29)

Therefore, we can now write the Hamiltonian of interactions in terms of the
fundamental fields as follows:

$\displaystyle H_{int\,2}^{I}$ $\displaystyle=$
$\displaystyle\int{d^{2}x}\left(ief^{Ik}(\varphi^{I\dagger}\partial_{k}\varphi^{I}-\partial_{k}\varphi^{I\dagger}\varphi^{I})-\frac{ie}{m}\epsilon^{lj}(\partial_{l}f_{j})({\dot{\varphi}}^{I\dagger}\varphi^{I}-\varphi^{I\dagger}\dot{\varphi}^{I})\right.$
(30) $\displaystyle+$
$\displaystyle\left.\frac{e^{2}}{2m^{2}}(\dot{\varphi}^{I\dagger}\varphi^{I}-\varphi^{I\dagger}\dot{\varphi}^{I})^{2}-\frac{2ie}{m}v_{0}\phi^{I}(\dot{\varphi}^{I\dagger}\varphi^{I}-\varphi^{I\dagger}\dot{\varphi}^{I})\right.$
$\displaystyle-$
$\displaystyle\left.e^{2}f_{k}^{I}f^{Ik}\varphi^{I\dagger}\varphi^{I}+e^{2}\phi^{I2}\varphi^{I\dagger}\varphi^{I}\right).$

Note that the Hamiltonian of interactions above has six terms. The first term
corresponds to the spatial part of the dual field-current interaction. The
second term, third term, and fourth term arise when we use the constraint
condition to eliminate $f^{0}$ from the Hamiltonian. Being the second term the
interaction of the magnetic field with the time component of the current,
while the third is the interaction of the temporal part of currents. Moreover,
the extra terms are local in spacetime and not renormalizable by power
counting. The fourth term is the scalar field interacting with the temporal
component of the current. The fifth term is the spatial part of the gauge-
boson field interaction, and the sixth term is the scalar-boson field
interaction.

Here, we mention that the introduced Feynman rules are not manifestly
covariant. Therefore, we must clarify whether or not this can lead to a
relativistically invariant $S$ matrix. In the following, we will focus on
testing the relativistic invariance of the model in connection with the
specific process of elastic boson-boson scattering. Since we are dealing with
a non-renormalizable theory, we restrict our calculations to the tree
approximation.

##  III boson-boson scattering

At this point, we will investigate the Lorentz invariance of the model. For
this purpose, we will compute the $e^{2}$ order contribution to the lowest
order elastic boson-boson scattering amplitude. Here, due to the non-
renormalizability of the model, we will restrict the calculation to the tree-
level approximation. Hence, for the scattering amplitude, we have seven
different types of terms grouped as follows:

$\displaystyle S^{(2)}=\sum_{\mu=1}^{7}{S^{(2)}_{\mu}}$ (31)

where

$\displaystyle S^{(2)}_{1}$ $\displaystyle=$
$\displaystyle\frac{e^{2}}{2}\int\int{d^{3}xd^{3}y}\langle\Phi_{f}\mid
T\\{:[\varphi^{I\dagger}(x)\partial_{k}\varphi^{I}(x)-\partial_{k}\varphi^{I\dagger}(x)\varphi^{I}(x)]f^{k}(x):$
(32) $\displaystyle\times$
$\displaystyle:[\varphi^{I\dagger}(y)\partial_{j}\varphi^{I}(y)-\partial_{j}\varphi^{I\dagger}(y)\varphi^{I}(y)]f^{j}(y):\\}\mid\Phi_{i}\rangle,$
$\displaystyle S^{(2)}_{2}$ $\displaystyle=$
$\displaystyle-\frac{e^{2}}{m}\int\int{d^{3}xd^{3}y}\langle\Phi_{f}\mid
T\\{:[\varphi^{I\dagger}(x)\partial_{k}\varphi^{I}(x)-\partial_{k}\varphi^{I\dagger}(x)\varphi^{I}(x)]f^{k}(x):$
(33) $\displaystyle\times$
$\displaystyle:[\epsilon^{lj}\partial_{l}f_{j}(y)({\dot{\varphi}}^{I\dagger}(y)\varphi^{I}(y)-\varphi^{I\dagger}(y)\dot{\varphi}^{I}(y))]:\\}\mid\Phi_{i}\rangle,$
$\displaystyle S^{(2)}_{3}$ $\displaystyle=$
$\displaystyle\frac{e^{2}}{2m^{2}}\int\int{d^{3}xd^{3}y}\langle\Phi_{f}\mid
T\\{:[\epsilon^{kj}\partial_{k}f_{j}(x)({\dot{\varphi}}^{I\dagger}(x)\varphi^{I}(x)-\varphi^{I\dagger}(x)\dot{\varphi}^{I}(x))]:$
(34) $\displaystyle\times$
$\displaystyle:[\epsilon^{il}\partial_{i}f_{l}(y)({\dot{\varphi}}^{I\dagger}(y)\varphi^{I}(y)-\varphi^{I\dagger}(y)\dot{\varphi}^{I}(y))]:\\}\mid\Phi_{i}\rangle,$
$\displaystyle S^{(2)}_{4}$ $\displaystyle=$
$\displaystyle-\frac{ie^{2}}{2m^{2}}\int\int{d^{3}xd^{3}y}\delta(x-y)\langle\Phi_{f}\mid
T\\{:[{\dot{\varphi}}^{I\dagger}(x)\varphi^{I}(x)-\varphi^{I\dagger}(x)\dot{\varphi}^{I}(x)]:$
(35) $\displaystyle\times$
$\displaystyle:[{\dot{\varphi}}^{I\dagger}(y)\varphi^{I}(y)-\varphi^{I\dagger}(y)\dot{\varphi}^{I}(y)]:\\}\mid\Phi_{i}\rangle,$
$\displaystyle S^{(2)}_{5}$ $\displaystyle=$
$\displaystyle-\frac{2e^{2}v_{0}}{m}\int\int{d^{3}xd^{3}y}\langle\Phi_{f}\mid
T\\{:[\varphi^{I\dagger}(x)\partial_{k}\varphi^{I}(x)-\partial_{k}\varphi^{I\dagger}(x)\varphi^{I}(x)]f^{k}(x):$
(36) $\displaystyle\times$
$\displaystyle:[{\dot{\varphi}}^{I\dagger}(y)\varphi^{I}(y)-\varphi^{I\dagger}(y)\dot{\varphi}^{I}(y)]\phi(y):\\}\mid\Phi_{i}\rangle,$
$\displaystyle S^{(2)}_{6}$ $\displaystyle=$
$\displaystyle-\frac{2e^{2}v_{0}}{m^{2}}\int\int{d^{3}xd^{3}y}\langle\Phi_{f}\mid
T\\{:[{\dot{\varphi}}^{I\dagger}(x)\varphi^{I}(x)-\varphi^{I\dagger}(x)\dot{\varphi}^{I}(x)]\phi(x):$
(37) $\displaystyle\times$
$\displaystyle:[{\dot{\varphi}}^{I\dagger}(y)\varphi^{I}(y)-\varphi^{I\dagger}(y)\dot{\varphi}^{I}(y)]\epsilon^{kj}\partial_{k}f_{j}(y):\\}\mid\Phi_{i}\rangle,$
$\displaystyle S^{(2)}_{7}$ $\displaystyle=$
$\displaystyle\frac{e^{2}v_{0}^{2}}{m^{2}}\int\int{d^{3}xd^{3}y}\langle\Phi_{f}\mid
T\\{:[{\dot{\varphi}}^{I\dagger}(x)\varphi^{I}(x)-\varphi^{I\dagger}(x)\dot{\varphi}^{I}(x)]\phi(x):$
(38) $\displaystyle\times$
$\displaystyle:[{\dot{\varphi}}^{I\dagger}(x)\varphi^{I}(y)-\varphi^{I\dagger}(y)\dot{\varphi}^{I}(y)]\phi(y):\\}\mid\Phi_{i}\rangle.$

Being $\mid\Phi_{i}\rangle$ and $\langle\Phi_{f}\mid$ the initial and final
state of the process and $T$ the chronological ordering operator. Here both
$\mid\Phi_{i}\rangle$ and $\langle\Phi_{f}\mid$ are two-boson states.

In the next step, we compute partial amplitudes (32)-(38) which in terms of
initial $(p_{1},p_{2})$ and final $(p_{1}^{\prime},p_{2}^{\prime})$ moments
become

$\displaystyle S^{(2)}_{1}$ $\displaystyle=$
$\displaystyle-e^{2}N_{p}(2\pi)^{3}\delta(p_{1}^{\prime}+p_{2}^{\prime}-p_{1}-p_{2})\left[(p_{1}^{\prime}+p_{1})_{j}(p_{2}^{\prime}+p_{2})_{l}\frac{1}{m^{2}}D^{jl}(k)+p_{1}\leftrightarrow
p_{2}\right],$ (39) $\displaystyle S^{(2)}_{2}$ $\displaystyle=$
$\displaystyle-e^{2}N_{p}(2\pi)^{3}\delta(p_{1}^{\prime}+p_{2}^{\prime}-p_{1}-p_{2})\left[(p_{1}^{\prime}+p_{1})_{j}(p_{2}^{\prime}+p_{2})_{0}\frac{1}{m^{2}}\Gamma^{j}(k)\right.$
(40) $\displaystyle+$
$\displaystyle\left.(p_{1}^{\prime}+p_{1})_{0}(p_{2}^{\prime}+p_{2})_{j}\frac{1}{m^{2}}\Gamma^{j}(-k)+p_{1}\leftrightarrow
p_{2}\right],$ $\displaystyle S^{(2)}_{3}$ $\displaystyle=$
$\displaystyle-e^{2}N_{p}(2\pi)^{3}\delta(p_{1}^{\prime}+p_{2}^{\prime}-p_{1}-p_{2})\left[(p_{1}^{\prime}+p_{1})_{0}(p_{2}^{\prime}+p_{2})_{0}\frac{1}{m^{2}}\Lambda(k)+p_{1}\leftrightarrow
p_{2}\right],$ (41) $\displaystyle S^{(2)}_{4}$ $\displaystyle=$
$\displaystyle-e^{2}N_{p}(2\pi)^{3}\delta(p_{1}^{\prime}+p_{2}^{\prime}-p_{1}-p_{2})\left[(p_{1}^{\prime}+p_{1})_{0}(p_{2}^{\prime}+p_{2})_{0}\frac{i}{m^{2}}+p_{1}\leftrightarrow
p_{2}\right],$ (42) $\displaystyle S^{(2)}_{5}$ $\displaystyle=$
$\displaystyle-e^{2}N_{p}(2\pi)^{3}\delta(p_{1}^{\prime}+p_{2}^{\prime}-p_{1}-p_{2})\left[(p_{1}^{\prime}+p_{1})_{j}(p_{2}^{\prime}+p_{2})_{0}\left(\frac{v_{0}v^{j}}{m^{4}}\right)\right.$
(43) $\displaystyle+$
$\displaystyle\left.(p_{1}^{\prime}+p_{1})_{0}(p_{2}^{\prime}+p_{2})_{j}\left(\frac{v_{0}v^{j}}{m^{4}}\right)+p_{1}\leftrightarrow
p_{2}\right],$ $\displaystyle S^{(2)}_{6}$ $\displaystyle=$
$\displaystyle-e^{2}N_{p}(2\pi)^{3}\delta(p_{1}^{\prime}+p_{2}^{\prime}-p_{1}-p_{2})\left[(p_{1}^{\prime}+p_{1})_{0}(p_{2}^{\prime}+p_{2})_{0}\left(\frac{k^{l}v_{l}}{m^{4}}\right)+p_{1}\leftrightarrow
p_{2}\right],$ (44) $\displaystyle S^{(2)}_{7}$ $\displaystyle=$
$\displaystyle-e^{2}N_{p}(2\pi)^{3}\delta(p_{1}^{\prime}+p_{2}^{\prime}-p_{1}-p_{2})\left[(p_{1}^{\prime}+p_{1})_{0}(p_{2}^{\prime}+p_{2})_{0}\left(\frac{v_{0}^{2}}{m^{2}k^{2}}\right)+p_{1}\leftrightarrow
p_{2}\right],$ (45)

where we have

$\displaystyle\frac{1}{m^{2}}D^{jl}(k)=\frac{i}{k^{2}-m^{2}+i\epsilon}\left(-g^{jl}+\frac{k^{j}k^{l}}{m^{2}}-\frac{i}{m}\epsilon^{jl}k_{0}-\frac{4v^{j}v^{l}}{k^{2}}-\frac{v^{j}k^{l}}{k^{2}}\right),$
(46)
$\displaystyle\frac{1}{m^{2}}\Gamma^{j}(k)=\frac{i}{k^{2}-m^{2}+i\epsilon}\left(\frac{k^{j}k^{0}}{m^{2}}+\frac{i}{m}\epsilon^{jl}k_{l}-\frac{4v^{j}v^{0}}{k^{2}}-\frac{v^{0}k^{j}}{k^{2}}\right),$
(47)
$\displaystyle\frac{1}{m^{2}}\Lambda(k)=\frac{i}{k^{2}-m^{2}+i\epsilon}\left(-\frac{k^{l}k_{l}}{m^{2}}-\frac{4v^{0}v^{0}}{k^{2}}-\frac{v^{0}k^{0}}{k^{2}}\right),$
(48)
$\displaystyle\frac{1}{m^{2}}\Sigma^{j}(k)=\frac{i}{k^{2}-m^{2}+i\epsilon}\left(-\frac{4v^{0}v^{j}}{k^{2}}\right),$
(49)
$\displaystyle\frac{1}{m^{2}}Q(k)=\frac{i}{k^{2}-m^{2}+i\epsilon}\left(-\frac{4v_{l}k^{l}}{k^{2}}\right),$
(50)
$\displaystyle\frac{1}{m^{2}}\Phi(k)=\frac{i}{k^{2}-m^{2}+i\epsilon}\left(-\frac{4v^{0}v^{0}}{k^{2}}\right),$
(51)

with

$\displaystyle k\equiv(p_{1}^{\prime}-p_{1})=-(p_{2}^{\prime}-p_{2}),$ (52)

the momentum transfer. Now, by replacing equations (39)-(45) into (31) we get

$\displaystyle
S^{(2)}=-e^{2}N_{p}(2\pi)^{3}\delta(p_{1}^{\prime}+p_{2}^{\prime}-p_{1}-p_{2})\left[(p_{1}^{\prime}+p_{1})_{\mu}(p_{2}^{\prime}+p_{2})_{\nu}\frac{1}{m^{2}}D^{\mu\nu}(k)+p_{1}\leftrightarrow
p_{2}\right],$ (53)

being

$\displaystyle\frac{1}{m^{2}}D^{\mu\nu}(k)=-\frac{i}{k^{2}-m^{2}+i\epsilon}\left(g^{\mu\nu}-\frac{k^{\mu}k^{\nu}}{m^{2}}+\frac{i}{m}\epsilon^{\mu\nu\alpha}k_{\alpha}-\frac{4v^{\mu}v^{\nu}}{k^{2}}-\frac{v^{\mu}k^{\nu}}{k^{2}}\right),$
(54)

the propagator of the self-dual field $f^{\mu}$. Note that the scattering
amplitude above is a Lorentz scalar and thus the model under consideration has
passed the test of relativistic invariance at the quantum level. Furthermore,
in the tree-level approximation we can replace all non-covariant terms in the
Hamiltonian of interactions by a minimum covariant interaction given by:

$\displaystyle H^{I}_{int\,2}=eJ^{I}_{\mu}f^{\mu},$ (55)

where

$\displaystyle
J^{I}_{\mu}=i(\varphi^{I\ast}\partial_{\mu}\varphi^{I}-\partial_{\mu}\varphi^{I\ast}\varphi^{I})-ef^{I}_{\mu}\varphi^{I\ast}\varphi^{I}-\frac{2m}{e}\phi
v_{\mu},$ (56)

is the effective bosonic matter current. Moreover, by taking $v_{\mu}=0$
(without the effect of breaking the Lorentz symmetry), we have

$\displaystyle\frac{1}{m^{2}}D^{\mu\nu}(k)=-\frac{i}{k^{2}-m^{2}+i\epsilon}\left(g^{\mu\nu}-\frac{k^{\mu}k^{\nu}}{m^{2}}+\frac{i}{m}\epsilon^{\mu\nu\alpha}k_{\alpha}\right),$
(57)

with

$\displaystyle H^{I}_{int}=eJ^{I}_{\mu}f^{\mu},$ (58)

where

$\displaystyle
J^{I}_{\mu}=i(\varphi^{I\ast}\partial_{\mu}\varphi^{I}-\partial_{\mu}\varphi^{I\ast}\varphi^{I})-ef^{I}_{\mu}\varphi^{I\ast}\varphi^{I}.$
(59)

Notice that in this way we recover the results found in [13].

##  IV Dual Equivalence

In this section, we study the duality between the self-dual and Maxwell-Chern-
Simons models with Lorentz symmetry breaking coupled with bosonic matter. For
this we write the Lagrangian as follows

$\displaystyle\tilde{\cal L}$ $\displaystyle=$
$\displaystyle\frac{m}{2}\epsilon^{\mu\nu\rho}(f_{\mu}\partial_{\nu}f_{\rho})-\frac{\tilde{m}^{2}}{2}f_{\mu}f^{\mu}+\frac{1}{2}\partial_{\mu}\phi\partial^{\mu}\phi+2m\phi
v^{\mu}f_{\mu}$ (60) $\displaystyle+$
$\displaystyle\partial_{\mu}\varphi^{\ast}\partial^{\mu}\varphi-\mathcal{M}^{2}\varphi^{\ast}\varphi-
ef_{\mu}J^{\mu},$

where $\tilde{m}^{2}=m^{2}-2e^{2}\varphi^{\ast}\varphi$ and
$\mathcal{M}^{2}=M^{2}+e^{2}\phi^{2}$. Note that the mass parameters are field
dependent, i.e., $\tilde{m}$ depends on the charged scalar field and
$\mathcal{M}$ depends on the real scalar field.

Equation of motion for the field $f_{\mu}$

$\displaystyle
m\epsilon^{\mu\nu\rho}(\partial_{\nu}f_{\rho})-\tilde{m}^{2}f^{\mu}-e\tilde{J}^{\mu}=0,$
(61)

where we have defined $\tilde{J}^{\mu}=J^{\mu}-2e^{-1}m\phi v^{\mu}$.

Now, to determine the Euler vector we apply the following gauge transformation
to the vector field $f_{\mu}$, such that $\delta
f_{\mu}=\partial_{\mu}\epsilon$, being $\epsilon$ a parameter of gauge
transformations. Thus, by varying the Lagrangian with respect to $f_{\mu}$, we
find $\delta\mathcal{\tilde{L}}=K^{\mu}\partial_{\mu}$, where

$\displaystyle
K^{\mu}=m\epsilon^{\mu\nu\rho}(\partial_{\nu}f_{\rho})-\tilde{m}^{2}f^{\mu}-e\tilde{J}^{\mu}=0,$
(62)

is the Euler vector.

From the Euler-Lagrange equation, we obtain the equations for the fields
$\phi$ and $\varphi$, given respectively by

$\displaystyle\partial_{\mu}\partial^{\mu}\phi=2mv^{\mu}f_{\mu},$ (63)

and

$\displaystyle(\partial_{\mu}\partial^{\mu}+\mathcal{M}^{2})\varphi=2ief^{\mu}\partial_{\mu}\varphi.$
(64)

The next step is to establish duality between the self-dual and Maxwell-Chern-
Simons models with Lorentz symmetry breaking coupled with bosonic matter.
Thus, we proceed by adopting the gauge embedding approach as done in [14, 19,
20]. In this way, we add to the original Lagrangian a term of the form
$F(K^{\mu})$, as follows:

$\displaystyle\mathcal{\tilde{L}}\rightarrow\mathcal{\tilde{L}}+F(K^{\mu}),$
(65)

being $F(0)=0$. Now to obtain the form of the function $F(K^{\mu})$, we define

$\displaystyle\mathcal{L}^{(1)}=\tilde{\mathcal{L}}+B_{\mu}K^{\mu},$ (66)

where $B_{\mu}$ acts as a Lagrange multiplier. In addition, we assume that
$\delta B_{\mu}=\partial_{\mu}\epsilon$ to cancel the variation of
$\tilde{\mathcal{L}}$. Therefore, proceeding as in Refs. [14, 19, 20] and
after eliminating $B_{\mu}$, we find the following gauge invariant effective
Lagrangian

$\displaystyle\mathcal{L}_{eff}=\tilde{\mathcal{L}}+\frac{1}{2\tilde{m}^{2}}K_{\mu}K^{\mu}.$
(67)

Hence, renaming $f_{\mu}\rightarrow A_{\mu}$ to show the invariant character
of the theory, we have

$\displaystyle\mathcal{L}_{eff}$ $\displaystyle=$
$\displaystyle\frac{1}{2}F_{\mu}F^{\mu}-\frac{m}{2}\epsilon^{\mu\nu\rho}A_{\mu}\partial_{\nu}A_{\rho}+\frac{1}{2}\partial_{\mu}\phi\partial^{\mu}\phi+\partial_{\mu}\varphi^{\ast}\partial^{\mu}\varphi-\mathcal{M}^{2}\varphi^{\ast}\varphi-\frac{em}{\tilde{m}^{2}}\tilde{J}_{\mu}F^{\mu}+\frac{e^{2}}{2\tilde{m}^{2}}\tilde{J}_{\mu}\tilde{J}^{\mu},$
(68)

being,

$\displaystyle F^{\mu}\equiv\frac{1}{2}\epsilon^{\mu\nu\rho}F_{\nu\rho},$ (69)

the dual of the tensor $F_{\nu\rho}$. By replacing $\tilde{J}_{\mu}$ we obtain

$\displaystyle\mathcal{L}_{eff}$ $\displaystyle=$
$\displaystyle\frac{1}{2}F_{\mu}F^{\mu}-\frac{m}{2}\epsilon^{\mu\nu\rho}A_{\mu}\partial_{\nu}A_{\rho}+\frac{1}{2}\partial_{\mu}\phi\partial^{\mu}\phi+\frac{2m^{2}}{\tilde{m}^{2}}\phi^{2}v_{\mu}v^{\mu}+\partial_{\mu}\varphi^{\ast}\partial^{\mu}\varphi-\mathcal{M}^{2}\varphi^{\ast}\varphi$
(70) $\displaystyle-$
$\displaystyle\frac{em}{\tilde{m}^{2}}{J}_{\mu}F^{\mu}+\frac{2m^{2}\phi}{\tilde{m}^{2}}{v}_{\mu}F^{\mu}+\frac{e^{2}}{2\tilde{m}^{2}}\left[{J}_{\mu}-\frac{4m}{e}\phi
v_{\mu}\right]{J}^{\mu}.$

Note that a mass term for the field $\phi$ has been generated, which has also
been previously obtained by the quantization procedure.

For this model, the equations for the $\phi$ field read

$\displaystyle\partial_{\mu}\partial^{\mu}\phi=2mv_{\mu}\left[\frac{m}{\tilde{m}^{2}}F^{\mu}+\frac{2m}{\tilde{m}^{2}}\phi
v^{\mu}-\frac{e}{\tilde{m}^{2}}J^{\mu}\right],$ (71)

and for the $\varphi$ field, we have

$\displaystyle\left(\partial_{\mu}\partial^{\mu}+\mathcal{M}^{2}\right)\varphi=-2ie\left[\frac{m}{\tilde{m}^{2}}F^{\mu}+\frac{2m}{\tilde{m}^{2}}\phi{v}^{\mu}-\frac{e}{\tilde{m}^{2}}{J}^{\mu}\right]\partial_{\mu}\varphi.$
(72)

Finally, by comparing equations (63), (64), (71) and (72), we obtain that the
correct map between the self-dual model and the Maxwell-Chern-Simons is given
by

$\displaystyle
f^{\mu}\rightarrow\frac{m^{2}}{\tilde{m}^{2}}\left(\frac{F^{\mu}}{{m}}+\frac{2\phi}{{m}}{v}^{\mu}-\frac{e}{{m}^{2}}{J}^{\mu}\right).$
(73)

In this way we have successfully completed the dual mapping between the two
models. The mapping obtained here is a generalization of the results obtained
in [14] and [19]. By setting $v^{\mu}=0$ (in the absence of Lorentz symmetry
breaking) we will recover the results obtained in [14].

$\displaystyle
f^{\mu}\rightarrow\frac{m^{2}}{\tilde{m}^{2}}\left(\frac{F^{\mu}}{{m}}-\frac{e}{{m}^{2}}{J}^{\mu}\right).$
(74)

On the other hand, without the presence of the field $\varphi$, but
maintaining the effect of breaking the Lorentz symmetry, we have the result
found in [19].

$\displaystyle
f^{\mu}\rightarrow\left(\frac{F^{\mu}}{{m}}+\frac{2\phi}{{m}}{v}^{\mu}\right).$
(75)

##  V Summary

In this work, we have considered the dynamics, at the quantum level, of the
self-dual model coupled to Lorentz symmetry-breaking bosons. Then, after
eliminating the degree of freedom $f^{0}$ the equal-time commutators with the
bosonic fields that are not zero, we have formulated the dynamics of the
interaction framework for the self-dual model minimally coupled to the bosons
by breaking the Lorentz symmetry. As a consequence, the Hamiltonian
formulation did not retain any relics of relativistic covariance. In addition,
we have shown that a mass term is generated for the real scalar field $\phi$.
However, we prove that the non-covariant parts in $H^{I}_{int\,2}$ are
equivalent to the minimum covariant field-current interaction. The high-energy
behavior of the $f$ propagator signals that the coupled theory is not
renormalizable. In this way, we have shown that relativistic invariance of the
model is preserved at the quantum level. We finish the paper by establishing
dual equivalence between the self-dual and Maxwell-Chern-Simons models with
Lorentz symmetry breaking and coupled with bosonic matter field.

###### Acknowledgements.

We would like to thank CNPq, CAPES and CNPq/PRONEX/FAPESQ-PB (Grant nos.
165/2018 and 015/2019), for partial financial support. M.A.A., F.A.B. and E.P.
acknowledge support from CNPq (Grant nos. 306398/2021-4, 309092/2022-1,
304290/2020-3).

## References

  * [1] V. A. Kostelecky and S. Samuel, Phys. Rev. D 39, 683 (1989) doi:10.1103/PhysRevD.39.683
  * [2] V. A. Kostelecky and N. Russell, Rev. Mod. Phys. 83, 11-31 (2011) doi:10.1103/RevModPhys.83.11 [arXiv:0801.0287 [hep-ph]].
  * [3] S. M. Carroll, G. B. Field and R. Jackiw, Phys. Rev. D 41, 1231 (1990) doi:10.1103/PhysRevD.41.1231
  * [4] V. A. Kostelecky, C. D. Lane and A. G. M. Pickering, Phys. Rev. D 65, 056006 (2002) doi:10.1103/PhysRevD.65.056006 [arXiv:hep-th/0111123 [hep-th]].
  * [5] M. Gomes, J. R. Nascimento, E. Passos, A. Y. Petrov and A. J. da Silva, Phys. Rev. D 76, 047701 (2007) doi:10.1103/PhysRevD.76.047701 [arXiv:0704.1104 [hep-th]].
  * [6] J. R. Nascimento, E. Passos, A. Y. Petrov and F. A. Brito, JHEP 06, 016 (2007) doi:10.1088/1126-6708/2007/06/016 [arXiv:0705.1338 [hep-th]].
  * [7] P. K. Townsend, K. Pilch and P. van Nieuwenhuizen, Phys. Lett. B 136, 38 (1984) doi:10.1016/0370-2693(84)92051-3
  * [8] S. Deser and R. Jackiw, Phys. Lett. B 139, 371-373 (1984) doi:10.1016/0370-2693(84)91833-1
  * [9] P. K. Townsend, K. Pilch and P. van Nieuwenhuizen, Phys. Lett. B 137, 443 (1984) doi.org/10.1016/0370-2693(84)91753-2
  * [10] R. Banerjee, H. J. Rothe and K. D. Rothe, Phys. Rev. D 52, 3750-3752 (1995) doi:10.1103/PhysRevD.52.3750 [arXiv:hep-th/9504067 [hep-th]].
  * [11] M. Gomes, L. C. Malacarne and A. J. da Silva, Phys. Lett. B 439, 137-141 (1998) doi:10.1016/S0370-2693(98)01008-9 [arXiv:hep-th/9711184 [hep-th]].
  * [12] H. O. Girotti, Int. J. Mod. Phys. A 14, 2495-2510 (1999) doi:10.1142/S0217751X99001238 [arXiv:hep-th/9803153 [hep-th]].
  * [13] M. A. Anacleto, J. R. S. Nascimento and R. F. Ribeiro, Phys. Rev. D 62, 067701 (2000) doi:10.1103/PhysRevD.62.067701 [arXiv:hep-th/0002055 [hep-th]].
  * [14] M. A. Anacleto, A. Ilha, J. R. S. Nascimento, R. F. Ribeiro and C. Wotzasek, Phys. Lett. B 504, 268-274 (2001) doi:10.1016/S0370-2693(01)00300-8 [arXiv:hep-th/0104152 [hep-th]].
  * [15] D. Dalmazi, J. Phys. A 37, 2487-2496 (2004) doi:10.1088/0305-4470/37/6/037 [arXiv:hep-th/0308150 [hep-th]].
  * [16] M. Botta Cantcheff, C. F. L. Godinho, A. P. Baeta Scarpelli and J. A. Helayel-Neto, Phys. Rev. D 68, 065025 (2003) doi:10.1103/PhysRevD.68.065025 [arXiv:hep-th/0304104 [hep-th]].
  * [17] M. S. Guimaraes, L. Grigorio and C. Wotzasek, [arXiv:hep-th/0609215 [hep-th]].
  * [18] M. S. Guimaraes, J. R. Nascimento, A. Y. Petrov and C. Wotzasek, EPL 95, no.5, 51002 (2011) doi:10.1209/0295-5075/95/51002 [arXiv:1010.3666 [hep-th]].
  * [19] C. Furtado, J. R. Nascimento, A. Y. Petrov and M. A. Anacleto, Phys. Rev. D 78, 065014 (2008) doi:10.1103/PhysRevD.78.065014 [arXiv:0805.3517 [hep-th]].
  * [20] A. F. Ferrari, M. Gomes, J. R. S. Nascimento, A. Y. Petrov and A. J. da Silva, Phys. Rev. D 73, 105010 (2006) doi:10.1103/PhysRevD.73.105010 [arXiv:hep-th/0604019 [hep-th]].
  * [21] A. Karlhede, U. Lindstrom, M. Rocek and P. van Nieuwenhuizen, Phys. Lett. B 186, 96-98 (1987) doi:10.1016/0370-2693(87)90519-3
  * [22] N. Bralic, E. H. Fradkin, V. Manias and F. A. Schaposnik, Nucl. Phys. B 446, 144-158 (1995) doi:10.1016/0550-3213(95)00225-H [arXiv:hep-th/9502066 [hep-th]].
  * [23] N. Banerjee, R. Banerjee and S. Ghosh, Nucl. Phys. B 481, 421-432 (1996) doi:10.1016/S0550-3213(96)90147-9 [arXiv:hep-th/9607065 [hep-th]].
  * [24] A. Ilha and C. Wotzasek, Nucl. Phys. B 604, 426-440 (2001) doi:10.1016/S0550-3213(01)00106-7 [arXiv:hep-th/0104115 [hep-th]].
  * [25] R. Banerjee, S. Kumar and S. Mandal, Phys. Rev. D 63, 125008 (2001) doi:10.1103/PhysRevD.63.125008 [arXiv:hep-th/0007148 [hep-th]].
  * [26] R. Menezes, J. R. S. Nascimento, R. F. Ribeiro and C. Wotzasek, Phys. Lett. B 564, 261-268 (2003) doi:10.1016/S0370-2693(03)00729-9 [arXiv:hep-th/0305112 [hep-th]].
  * [27] D. Bazeia, R. Menezes, J. R. Nascimento, R. F. Ribeiro and C. Wotzasek, J. Phys. A 36, 9943-9960 (2003) doi:10.1088/0305-4470/36/38/311 [arXiv:hep-th/0210311 [hep-th]].
  * [28] T. Mariz, R. Menezes, J. R. S. Nascimento, R. F. Ribeiro and C. Wotzasek, Phys. Rev. D 70, 085018 (2004) doi:10.1103/PhysRevD.70.085018 [arXiv:hep-th/0306265 [hep-th]].
  * [29] D. Dalmazi, JHEP 01, 132 (2006) doi:10.1088/1126-6708/2006/01/132 [arXiv:hep-th/0510153 [hep-th]].
  * [30] M. S. Guimaraes, T. Mariz, R. Menezes and C. Wotzasek, Phys. Lett. B 625, 351-356 (2005) doi:10.1016/j.physletb.2005.08.076 [arXiv:hep-th/0503107 [hep-th]].
  * [31] D. Dalmazi, JHEP 08, 040 (2006) doi:10.1088/1126-6708/2006/08/040 [arXiv:hep-th/0608129 [hep-th]].
  * [32] D. Dalmazi and E. L. Mendonca, J. Phys. A 39, 9355-9363 (2006) doi:10.1088/0305-4470/39/29/025 [arXiv:hep-th/0608223 [hep-th]].
  * [33] F. M. Belchior and R. V. Maluf, [arXiv:2312.00179 [hep-th]].
  * [34] H. Belich, Jr., M. M. Ferreira, Jr., J. A. Helayel-Neto and M. T. D. Orlando, Phys. Rev. D 67, 125011 (2003) [erratum: Phys. Rev. D 69, 109903 (2004)] doi:10.1103/PhysRevD.67.125011 [arXiv:hep-th/0212330 [hep-th]].
  * [35] H. Belich, M. M. Ferreira, Jr. and J. A. Helayel-Neto, Eur. Phys. J. C 38, 511-519 (2005) doi:10.1140/epjc/s2004-02033-x [arXiv:hep-th/0401235 [hep-th]].
  * [36] H. Belich, Jr., M. M. Ferreira, Jr., J. A. Helayel-Neto and M. T. D. Orlando, Phys. Rev. D 68, 025005 (2003) doi:10.1103/PhysRevD.68.025005 [arXiv:hep-th/0301224 [hep-th]].
  * [37] P. Gaete, [arXiv:hep-th/0501136 [hep-th]].
  * [38] P. Gaete and C. Wotzasek, Phys. Rev. D 75, 057902 (2007) doi:10.1103/PhysRevD.75.057902 [arXiv:hep-ph/0607321 [hep-ph]].
  * [39] M. A. Anacleto, Phys. Rev. D 92, no.8, 085035 (2015) doi:10.1103/PhysRevD.92.085035 [arXiv:1505.03238 [hep-th]].
  * [40] P. A. M. Dirac, Lectures on Quantum Mechanics, Belfer Graduate School on Science, Yeshiva University (New York, 1964).


# Adapting self-supervised models to multi-talker speech recognition
using speaker embeddings

###### Abstract

Self-supervised learning (SSL) methods which learn representations of data
without explicit supervision have gained popularity in speech-processing
tasks, particularly for single-talker applications. However, these models
often have degraded performance for multi-talker scenarios — possibly due to
the domain mismatch — which severely limits their use for such applications.
In this paper, we investigate the adaptation of upstream SSL models to the
multi-talker automatic speech recognition (ASR) task under two conditions.
First, when segmented utterances are given, we show that adding a target
speaker extraction (TSE) module based on enrollment embeddings is
complementary to mixture-aware pre-training. Second, for unsegmented mixtures,
we propose a novel joint speaker modeling (JSM) approach, which aggregates
information from all speakers in the mixture through their embeddings. With
controlled experiments on Libri2Mix, we show that using speaker embeddings
provides relative WER improvements of 9.1% and 42.1% over strong baselines for
the segmented and unsegmented cases, respectively. We also demonstrate the
effectiveness of our models for real conversational mixtures through
experiments on the AMI dataset.

Index Terms—  self-supervised learning, multi-talker ASR, target-speaker ASR

##  1 Introduction

Self-supervised learning (SSL), which obtains supervisory signals from the
input data itself, is an active sub-area of unsupervised learning [1, 2].
Generally speaking, the SSL pipeline consists of a pre-training stage, in
which an upstream model is trained on a large quantity of unlabeled data with
some pretext tasks, and a fine-tuning stage, in which the upstream model is
adapted for specific downstream tasks with annotated data.

SSL has recently gained popularity in the speech community due to its strong
performance and low adaptation cost. By simply replacing the acoustic features
with hidden representations from pre-trained upstream models [3, 4], or fine-
tuning the entire model with additional task-specific layers [5, 6], SSL
models have surpassed state-of-the-art downstream-only baselines for several
tasks including speech recognition [5, 6] and speaker recognition [7, 8].
Furthermore, these pre-trained upstream models often require very small
amounts of adaptation data; for instance, wav2vec 2.0 achieves a word error
rate (WER) of 8.2% on LibriSpeech [9] test-other after fine-tuning with 10
minutes of labeled speech [5].

Fig. 1: Our proposed Target Speaker Extraction (TSE) module (left) and Joint
Speaker Modeling (JSM) module (right) for adapting SSL-based upstream models
to multi-talker ASR.

Despite their strong performance for various downstream tasks, most of the
existing SSL models only focus on single-talker scenarios. The previous
studies on pre-training and fine-tuning upstream SSL models for multi-talker
scenarios are very limited [4, 10]. Such scenarios are usually more
challenging due to the involvement of multiple speakers, quick turn-taking,
and overlapping speech [11].

In this paper, we focus on the fine-tuning stage of the SSL pipeline and
investigate the adaptation of SSL-based upstream models to the multi-talker
ASR task, for both segmented utterances and unsegmented mixtures. For the
former, we build upon previous work on target speaker extraction and
recognition [12, 13, 14, 15]; we use target speaker embeddings (extracted from
enrollment audio) to direct the model to transcribe a specific speaker in the
segmented utterance containing mixed speech. We perform a thorough analysis
for several modeling aspects, including embedding type and dimensionality,
fusion methods for the adaptation layer, and the effect of upstream models.
For the case of unsegmented mixtures, we extend the TSE approach by
iteratively decoding each speaker in the mixtures, and we also propose a novel
joint speaker modeling (JSM) technique that aggregates information from all
speakers through their embeddings, and transcribes them simultaneously.

Our contributions are summarized as follows: (1) we propose methods to adapt
SSL models to the multi-talker ASR task based on the speaker embeddings; (2)
we conduct a detailed analysis of different modeling aspects of target speaker
extraction with SSL models; and (3) we evaluate our methods on both artificial
and real datasets, namely LibriMix and AMI. Our code will be released at the
time of publication.

##  2 The multi-talker ASR problem

In conventional ASR, the input is a single-talker, segmented utterance
$\mathbf{x}$ (in time or time-frequency domain), and the system is required to
predict hypothesis $\mathbf{y}=f(\mathbf{x};\Theta_{f})$, where $\mathbf{y}$
is a token sequence and $f$ is a function parametrized by $\Theta_{f}$ such as
a neural network. In multi-talker ASR, the input is an unsegmented mixture
$\mathbf{m}$ containing speech from multiple speakers (say, $K$ speakers),
i.e., $\mathbf{m}=\sum_{u=1}^{U}\mathbf{x}_{u}$, where $\mathbf{x}_{u}$ is the
$u$-th utterance and is shifted and zero-padded to the length of $\mathbf{m}$.
Here, we assume that the additive noise and convolutive reverberations are
included in the individual terms. Additionally, we may be provided enrollment
embeddings $\mathbf{e}_{k}\in\mathbb{R}^{d}$ for the $K$ speakers in the
mixture. The desired output is $\mathbf{y}=\bigcup_{u=1}^{U}\mathbf{y}_{u}$,
where $\mathbf{y}_{u}$ is the transcription corresponding to $\mathbf{x}_{u}$.
Depending on whether the mixture is segmented, two cases are possible.

###  2.1 Segmented utterances

In this easier case, we assume that the spans and speaker identities for all
$U$ utterances are known, such that $\mathbf{y}_{u}$ can be directly
estimated:
$\widehat{\mathbf{y}}_{u}=g_{\mathrm{S}}(\mathbf{m},\Delta_{u},\mathbf{e}_{k_{u}};\Theta_{g_{\mathrm{S}}})$,
where $\Delta_{u}$ denotes the span of $\mathbf{x}_{u}$ in the mixture,
$k_{u}$ is the speaker index corresponding to utterance $\mathbf{x}_{u}$. By
repeating this inference $U$ times, we get the desired multi-talker hypothesis
$\widehat{\mathbf{y}}$. In practice, $\Delta_{u}$ and $k_{u}$ may be obtained
using speaker diarization.

###  2.2 Unsegmented mixture

In this case, the individual segmentation of $\mathbf{x}_{u}$ is not known in
advance, therefore utterance-level transcription is not feasible. If we assume
that speakers are enrolled, we can instead perform speaker-level
transcription, i.e.,
$\widehat{\mathbf{y}}^{\prime}=\bigcup_{k=1}^{K}\widehat{\mathbf{y}}^{k}$,
where $\widehat{\mathbf{y}}^{k}$ denotes the hypothesis for speaker $k$. This
may be done by iteratively decoding each speaker using a same single-talker
parametrized model, i.e.,

$\widehat{\mathbf{y}}^{k}=g_{\mathrm{S}}(\mathbf{m},\Delta,\mathbf{e}_{k};\Theta_{g_{\mathrm{S}}}),$
(1)

where $\Delta$ is a dummy span indicating the entire duration of $\mathbf{m}$.
Alternatively, we can define a multi-talker parametrized model
$g_{\mathrm{M}}$ and perform joint decoding of all speakers, i.e.,

$\widehat{\mathbf{y}}^{\prime}=g_{\mathrm{M}}(\mathbf{m},\mathbf{E}_{1:K};\Theta_{g_{\mathrm{M}}}),$
(2)

where $\mathbf{E}\in\mathbb{R}^{d\times K}$ contains all $K$ speaker
embeddings.

In the next section, we will describe our proposed methods to use self-
supervised models for both $g_{\mathrm{S}}$ and $g_{\mathrm{M}}$, so that we
can handle segmented and unsegmented inputs.

##  3 Adapting self-supervised models with speaker embeddings

First, we will show how to adapt self-supervised models for the case of
single-talker transcription (i.e., $g_{\mathrm{S}}$) using speaker embeddings.
This model can be used for segmented utterances or iterative decoding of
unsegmented mixtures. Thereafter, we will develop a multi-talker extension
(i.e., $g_{\mathrm{M}}$) of the model to perform joint decoding of all
speakers in the mixture. Since our methods rely on the architecture of the
upstream SSL models, we will briefly describe them here.

We used three variants of the wav2vec 2.0 [5] class of SSL models: HuBERT [6],
WavLM Base, and WavLM Base+ [4]. These models consist of convolutional and
Transformer encoders. For all models, the convolutional encoder contains 7
layers that down-sample the audio input by a factor of 320, such that the
extracted features (say $\mathbf{h}_{\mathrm{conv}}$) have a stride of 20ms
and a receptive field of 25ms. The CNN encoder is followed by a 12-block
Transformer encoder, which produces contextual output representations
$\mathbf{h}_{\mathrm{trf}}$. Each Transformer block consists of a 768-dim
self-attention layer with 8 heads and a 3,072-dim feed-forward layer. WavLM
additionally uses relative gated positional bias in the self-attention layers
[16].

###  3.1 Target speaker extraction (TSE)

We adapt SSL models for single-talker transcription through a “target speaker
extraction” (TSE) component that models $g_{\mathrm{S}}$, as illustrated in
Fig. 1 (left). Specifically, we insert a speaker adaptation layer between the
CNN and Transformer blocks of the encoder. This layer adapts
$\mathbf{h}_{\mathrm{conv}}$ using the speaker embedding $\mathbf{e}_{k}$
before feeding them to the Transformer encoder. By fine-tuning the model with
multi-talker mixtures, speaker embeddings, and corresponding references, the
adaptation layer learns to extract only target-speaker features from the CNN
representations. The adapted CNN representations,
$\widehat{\mathbf{h}}_{\mathrm{conv}}$, are passed to the Transformer encoder
which produces speaker-specific contextual representations,
$\widehat{\mathbf{h}}_{\mathrm{trf}}$. A linear projection layer maps
$\widehat{\mathbf{h}}_{\mathrm{trf}}$ to the output dimensionality, and the
CTC loss [17] is used for end-to-end training.

In our experiments, we tried four different methods for the implementation of
the speaker adaptation layer.

  1. 1.

ADD: $\mathbf{e}_{k}$ is added to $\mathbf{h}_{\mathrm{conv}}$ through a
linear projection.

  2. 2.

CAT: $\mathbf{e}_{k}$ is appended to $\mathbf{h}_{\mathrm{conv}}$ and the
concatenated representation is projected back to the original dimension
through a linear projection.

  3. 3.

FiLM: Feature-wise linear modulation [18] estimates a feature-wise affine
transformation based on the conditioning information, i.e.,
$\widehat{\mathbf{h}}_{\mathrm{conv}}=\gamma(\mathbf{e}_{k})\cdot\mathbf{h}_{\mathrm{conv}}+\beta(\mathbf{e}_{k})$,
where $\gamma$ and $\beta$ can be arbitrary functions. In our experiment,
$\gamma$ and $\beta$ are two linear layers.

  4. 4.

CLN: In standard Transformer layers, layer normalization [19] has been applied
to normalize the distributions of intermediate layers, formulated as

$\widehat{\mathbf{h}}=\frac{\mathbf{h}-\mathbf{\mu}}{\mathbf{\sigma}}\cdot\gamma+\beta$
(3)

where $\mathbf{\mu}$ and $\mathbf{\sigma}$ are the mean and standard deviation
of input features $\mathbf{h}$, $\gamma$ and $\beta$ are learnable scaling and
shifting parameters for the affine transformation. Conditional layer
normalization (CLN) [20] replaces $\gamma$ with a speaker-specific scaling
$\widehat{\gamma}(\mathbf{e}_{k})$, i.e.,

$\widehat{\gamma}(\mathbf{e}_{k})=w(\mathbf{e}_{k})\cdot\gamma+b(\mathbf{e}_{k})$
(4)

Basically, it transforms $\gamma$ with a FiLM layer conditioned on the speaker
embeddings $\mathbf{e}_{k}$. Different from the three above methods which take
$\mathbf{h}_{\mathrm{conv}}$ as input, CLN is introduced to the bottom
Transformer block by replacing the original two LNs.

###  3.2 Joint speaker modeling (JSM)

In principle, the TSE module can be used for the transcription of unsegmented
mixtures by iterating over all speakers. However, this setup may be sub-
optimal for multi-talker ASR since the hypothesis produced for a speaker is
independent of other speakers. Some errors, like repetitive decoding of the
same utterance, could be potentially eliminated if the model has an overview
of all speakers presented in the speech. Furthermore, the TSE approach would
require $K$ decoding iterations, resulting in higher inference time. Inspired
by the architecture of target-speaker voice activity detection (TS-VAD) [14],
we propose “joint speaker modeling” (JSM) as illustrated in Fig. 1 (right). We
first use the TSE module to extract speaker-specific contextual
representations $\mathbf{h}_{\mathrm{trf}}^{k}$ for all speakers in the speech
mixture $\mathbf{m}$. Thereafter, the JSM module models $g_{\mathrm{M}}$ by
computing $\widehat{\mathbf{y}}^{\prime}$ directly using
$\\{\mathbf{h}_{\mathrm{trf}}^{k}|k=1,2,...,K\\}$. In our implementation, the
representations $\mathbf{h}_{\mathrm{trf}}^{k}$ are concatenated along the
feature axis and projected back to the original dimension. The combined input
then pass through a Transformer layer to get multi-talker contextual
representations $\widehat{\mathbf{h}}_{\mathrm{trf}}$. Finally, a linear layer
maps $\widehat{\mathbf{h}}_{\mathrm{trf}}$ to $K\times|V|$ output units (where
$|V|$ is vocabulary size) to get token-level posteriors for each speaker. The
model is trained with the sum of CTC losses over all $K$ speakers.

##  4 Experimental Setup

###  4.1 Data

We evaluated our method on LibriMix [21] and AMI [22] dataset. LibriMix is a
synthetic dataset consisting of simulated speech mixtures. In our experiments,
we used the “2-spk 16kHz max” condition (hereby denoted as Libri2Mix). The
training set consisted of 58 hours of speech mixtures created from LibriSpeech
[9] train-clean-100 while the dev and test set contained 11 hours each.

AMI [22] is a meeting-style corpus consisting of $\sim$100h of annotated
audio/video recordings. Each meeting has 3 to 5 participants, and was recorded
using headset microphones and two 8-ch microphone arrays. In our experiments,
we use the so-called “IHM-mix" evaluation condition, which is generated by
mixing all headset recordings from the same meeting. The resulting audios
contain a significant amount of overlap but no far-field artifacts, allowing
us to evaluate our models specifically for overlapping speech.

Our proposed methods require enrollment utterances for speaker embedding
extraction. For Libri2Mix, we collected 15 seconds of enrollment speech per
speaker from the Libri-Light [23] project (excluding the LibriSpeech portion).
For AMI IHM-mix, since the same speakers participate in multiple meetings, we
used clean segments from other meetings as enrollment utterances. For example,
if the speaker FEE041 participated in four meetings ES2011a-d, the clean
segments in meetings ES2011b-d would be considered for enrollment of FEE041 in
meeting ES2011a.

###  4.2 Evaluation metrics

Fig. 2: Utterance-based and utterance group-based evaluation schemes for
segmented utterances and unsegmented mixtures, respectively.

The two tasks with segmented utterances (§ 2.1) and unsegmented mixtures (§
2.2) require different evaluation schemes. Following [24], we refer to these
as utterance-based and utterance group-based evaluation, respectively, which
are illustrated in Fig. 2.

In utterance-based evaluation, the exact time boundary for each speaker
segment is known. We perform inference on each speaker segment, and our
hypothesis is compared with the reference to compute the word error rate
(WER). To evaluate unsegmented mixtures, we defined an “utterance group” as a
set of utterances connected by speaker overlaps. In practice, a voice activity
detector (VAD) may be used to estimate utterance group boundaries. If the
correspondence between the multi-talker hypotheses and references in an
utterance group is unknown (as is the case in [24]), concatenated minimum-
permutation WER (cpWER) [25] may be computed instead of WER. In our proposed
model, the output order of hypotheses is determined by the order of speaker
embeddings, therefore we can directly compute the WER between the hypotheses
and references.

###  4.3 Model configuration and baselines

The upstream models we used are all pre-trained with the same masked cluster
index prediction loss [6]. However, while HuBERT is trained only on single-
talker audios, WavLM additionally uses simulated mixtures for pre-training,
which makes it more suitable for multi-talker downstream tasks. HuBERT and
WavLM Base are pre-trained using LibriSpeech (960h), while WavLM Base+ uses
94k hours of mixtures comprising large-scale diverse data, such as LibriLight
[26], VoxPopuli [27], and GigaSpeech [28].

Besides the upstream model, the choice of pre-trained speaker embedding
extractor and the dimensionality of $\mathbf{e}_{k}$ is also important towards
the final performance. We compared the multi-talker ASR performance for three
different embedding methods — i-vectors [29], x-vectors [30], and ECAPA-TDNN
embeddings [31]. I-vectors are unsupervised embeddings that model the total
variability between an utterance and the universal background model. X-vectors
[30] and ECAPA-TDNN embeddings, on the other hand, are extracted from
discriminatively-trained neural networks after pooling layers. While the
former uses conventional time-delay neural networks (TDNNs) [32], the latter
applies several enhancements on the frame and segment pooling components
through squeeze-and-excitation [33] and channel-dependent attention,
respectively. For speaker recognition, DNN-based embeddings have been found to
outperform i-vectors; ECAPA-TDNN, in particular, has obtained state-of-the-art
performance on the VoxCeleb benchmark [31]. For our experiments, all these
three embedding extractors were pre-trained on the VoxCeleb [34, 35] dataset,
with a dimension of 400, 512, and 256 for i-vectors, x-vectors, and ECAPA-TDNN
embeddings, respectively.

For segmented utterances, we compared our models with a baseline that does not
use any speaker embeddings but is otherwise fine-tuned on the same data. For
unsegmented mixtures, we implemented a permutation invariant training (PIT)
[36] baseline as follows. We added a linear layer consisting of $K\times|V|$
output units on top of the pre-trained SSL model, where each of the $K$ sets
is expected to transcribe a distinct speaker. The model was then trained using
PIT, i.e., using the minimum sum of losses between all permutations of
reference and hypotheses speakers. Since this model does not have a
deterministic speaker ordering, we evaluated it using cpWER instead of regular
WER.

##  5 Results & Discussion

Table 1: WER (%) on Libri2Mix and AMI IHM-mix for utterance-based evaluations.
TSE used x-vectors fused by the CLN approach.

Model Libri2Mix AMI WavLM Base+ 8.34 18.97 \+ TSE 7.58 18.07

Fig. 3: WER decomposition by overlap ratio for AMI IHM-mix.

Table 2: WER (%) on Libri2Mix for different SSL-based upstream models with and
without the TSE module.

Model w/o TSE w/ TSE HuBERT Base 13.04 11.62 (↓10.9%) WavLM Base 9.81 9.03
(↓8.0%) WavLM Base+ 8.34 7.58 (↓9.1%)

Table 3: WER (%) on Libri2Mix for different types and dimensionality of
speaker embeddings.

Embed. type CLN ADD $d=64$ $d=128$ $d=64$ $d=128$ i-vector 7.82 8.04 8.25 8.54
ECAPA-TDNN 7.87 8.14 8.00 8.65 x-vector 7.58 7.65 7.86 7.91

Table 4: WER (%) on Libri2Mix for different fusion methods in the speaker
adaptation layer.

Method utterance utterance group dev test dev test ADD 7.52 7.63 12.13 14.28
CAT 7.45 7.65 11.96 13.33 FiLM 7.31 7.66 12.45 13.84 CLN 7.09 7.58 11.04 12.32

###  5.1 Utterance-based evaluation of segmented utterances

We evaluated the impact of our proposed TSE module on Libri2Mix and AMI IHM-
Mix using WavLM Base+, with x-vector embeddings fused by the CLN approach.
These results are shown in Table 1. For utterance-based evaluation, the
baseline WavLM system worked well even without any enrollment information.
Despite the high overlap ratio of Libri2Mix, it achieved 8.34% WER on the test
set. Similarly, it obtained a competitive WER of 18.97% on AMI. Adding the TSE
module improved the WERs by 9.1% and 4.7% relative on Libri2Mix and AMI,
respectively. To better understand where the improvements came from, we
analyzed the AMI WER across different overlap ratios, as shown in Fig. 3.
Compared with the WavLM baseline, our TSE-enhanced model achieved comparable
or slightly worse WER when the overlap ratio was low (below 50%). However, it
outperformed WavLM by a clear margin for higher overlap ratios. We conjecture
that for lower overlap ratios, WavLM is able to perform well because it
transcribes the majority speaker in the utterance. For higher overlap ratios,
it cannot predict the target speaker based on the majority duration, and TSE
provides this additional information for such cases.

We further conducted several experiments on Libri2Mix to analyze the impact of
different factors such as the upstream model, the embedding type and its
dimensionality, and the fusion technique used in the speaker adaptation layer.
These are described in the following sections.

####  5.1.1 Effect of upstream models

During pre-training, the upstream model learns generalizable task-agnostic
representations using SSL, the quality of which usually determines the
performance of downstream tasks. In Table 4, we compared the multi-talker ASR
performance for the three upstream models described earlier, using CLN-based
fusion with x-vector speaker embeddings.

First, we observe that the WavLM models significantly outperform HuBERT, both
with and without the TSE module, due to their enhanced speaker modeling
ability and robustness to noise and speaker overlaps. Pre-training on a large-
scale diverse dataset (WavLM Base+) provided the best WERs, as expected.
Nevertheless, adding TSE provides consistent improvements of 8-11% across all
models, indicating that explicit speaker enrollment is complementary to
mixture-based pre-training scheme. For the following experiments, we used
WavLM Base+ as the upstream model.

####  5.1.2 Effect of speaker embeddings

In this section, we investigate the impact of speaker embedding type and
dimensionality. For a fair comparison, we projected the speaker embeddings to
the same dimension using principal component analysis (PCA) [37]. From Table
4, we first observe that 64-dim embeddings outperform their 128-dim
counterparts, indicating that there may be extraneous information encoded in
higher dimensions. Next, even though ECAPA-TDNNs achieve SOTA performance on
speaker recognition, they are outperformed by x-vectors when used in the TSE
module (for both CLN and ADD fusion).

####  5.1.3 Effect of fusion methods

In § 3.1, we described four different fusion methods for the speaker
adaptation layer. Recall that ADD and CAT directly fuse the speaker embedding
$\mathbf{e}_{k}$ to the convolutional representation
$\mathbf{h}_{\mathrm{conv}}$, whereas FiLM and CLN estimate affine transforms
based on $\mathbf{e}_{k}$. Here, we compare their performance in both
utterance-based and utterance group-based evaluation settings, as shown in
Table 4. On utterance-based evaluation, all these four methods have similar
performance, with CLN slightly outperforming others. However, when the
segmentation is unknown (utterance group-based evaluation), CLN outperforms
other methods by a large margin.

###  5.2 Utterance group-based evaluation of unsegmented mixtures

Table 5: WER (%) on Libri2Mix and AMI IHM-mix for utterance group-based
evaluations of unsegmented mixtures.†PIT does not have a fixed speaker order,
therefore the results are shown using cpWER.

Model Libri2Mix AMI WavLM Base+ 104.44 100.26 \+ PIT† 18.45 27.02 \+ iterative
TSE 12.32 49.47 \+ TSE + JSM 10.68 28.40

From the above ablation experiments, we demonstrated that the combination of
WavLM Base+ upstream model, x-vector speaker embedding, and the CLN fusion
method provided the best WER performance. Next, we used this configuration to
conduct utterance group-based evaluations for unsegmented mixtures, and the
results are shown in Table 5.

It is immediately clear that the single-talker WavLM baseline, which performed
well on segmented utterances, failed miserably when the segmentation was not
known, resulting in WERs above 100%. This is because, without any additional
information, the model does not know which speaker to transcribe in the
unsegmented mixtures. Joint multi-talker inference using permutation invariant
training (PIT) helped the model overcome this limitation and the resulting
system obtained decent results on the task, achieving a cpWER of 18.45% on
Libri2Mix and 27.02% on AMI. Recall that since PIT does not have a
deterministic speaker order, we use cpWER instead of WER to evaluate its
performance.

Next, we evaluated an iterative decoding approach (described in § 2.2) based
on the TSE module. While this model conveniently outperformed PIT on Libri2Mix
with a WER of 12.32%, its performance on AMI IHM-Mix was sub-par. Our error
analysis revealed that this was because on AMI IHM-Mix, the model sometimes
mistakenly transcribed other speakers when the target speaker didn’t talk or
only talked for a short period in the utterance group. We had conjectured that
this problem of repeated decoding can be alleviated using the JSM module
described in § 3.2. As expected, the multi-talker ASR performance was
significantly improved after we jointly modeled the activities of all speakers
using the JSM module. WERs improved from 12.32% to 10.68% on Libri2Mix, and
from 49.47% to 28.40% on AMI IHM-mix. Although this WER may seem slightly
worse than PIT, it is simply because the evaluation of PIT through cpWER is
more forgiving. If we discard the speaker ordering in JSM and compute the best
permutation word error rate, JSM will outperform PIT with a cpWER of 25.88%.

##  6 Conclusion

In this paper, we investigated the adaptation of self-supervised models to the
multi-talker ASR task using speaker embeddings, for both segmented and
unsegmented inputs. For the former, we explored a target speaker extraction
module that was found to be complementary to mixture-aware pre-training. For
the latter, we proposed a novel joint speaker modeling approach to jointly
decode all speakers in the mixture. Furthermore, we performed controlled
experiments to study the impact of several modeling aspects in target speaker
extraction with SSL models. Our experimental results on LibriMix and AMI
demonstrated that self-supervised models can achieve substantially better
multi-talker ASR performance by utilizing information from speaker embeddings.
We will explore the possibility of applying our methods to other multi-talker
tasks such as speaker diarization and speech separation in the future.

## References

  * [1] Abdelrahman Mohamed et al.,  “Self-supervised speech representation learning: A review,”  IEEE Journal of Selected Topics in Signal Processing, 2022.
  * [2] Xiao Liu et al.,  “Self-supervised learning: Generative or contrastive,”  IEEE Transactions on Knowledge and Data Engineering, 2021.
  * [3] Shu-wen Yang et al.,  “SUPERB: Speech processing universal performance benchmark,”  in InterSpeech, 2021.
  * [4] Sanyuan Chen et al.,  “WavLM: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, 2022.
  * [5] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “Wav2vec 2.0: A framework for self-supervised learning of speech representations,”  NeurIPS, 2020.
  * [6] Wei-Ning Hsu et al.,  “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM TASLP, 2021.
  * [7] Zhengyang Chen et al.,  “Large-scale self-supervised speech representation learning for automatic speaker verification,”  in IEEE ICASSP, 2022.
  * [8] Zhiyun Fan et al.,  “Exploring wav2vec 2.0 on speaker verification and language identification,”  in InterSpeech, 2021.
  * [9] Vassil Panayotov et al.,  “Librispeech: An asr corpus based on public domain audio books,”  in IEEE ICASSP, 2015.
  * [10] Xuankai Chang et al.,  “End-to-end integration of speech recognition, speech enhancement, and self-supervised learning representation,”  in Interspeech, 2022.
  * [11] Özgür Çetin and Elizabeth Shriberg,  “Analysis of overlaps in meetings by dialog factors, hot spots, speakers, and collection site: Insights for automatic speech recognition,”  in InterSpeech, 2006.
  * [12] Marc Delcroix et al.,  “Single channel target speaker extraction and recognition with Speaker Beam,”  in IEEE ICASSP, 2018.
  * [13] Kateřina Žmolíková et al.,  “SpeakerBeam: Speaker aware neural network for target speaker extraction in speech mixtures,”  IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 4, 2019.
  * [14] Ivan Medennikov et al.,  “Target-speaker voice activity detection: a novel approach for multi-speaker diarization in a dinner party scenario,”  in InterSpeech, 2020.
  * [15] Quan Wang et al.,  “VoiceFilter: Targeted voice separation by speaker-conditioned spectrogram masking,”  in InterSpeech, 2019.
  * [16] Zewen Chi et al.,  “XLM-E: Cross-lingual language model pre-training via electra,”  in ACL, 2022.
  * [17] Alex Graves et al.,  “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,”  in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376.
  * [18] Ethan Perez, Florian Strub, et al.,  “FiLM: Visual reasoning with a general conditioning layer,”  in AAAI, 2018.
  * [19] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton,  “Layer normalization,”  ArXiv, 2016.
  * [20] Jonathan Pilault, Christopher Pal, et al.,  “Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data,”  in ICLR, 2020.
  * [21] Joris Cosentino et al.,  “LibriMix: An open-source dataset for generalizable speech separation,”  ArXiv, 2020.
  * [22] Jean Carletta et al.,  “The AMI meeting corpus: A pre-announcement,”  in International workshop on machine learning for multimodal interaction, 2005.
  * [23] J. Kahn et al.,  “Libri-Light: A benchmark for ASR with limited or no supervision,”  in IEEE ICASSP, 2020.
  * [24] Naoyuki Kanda et al.,  “Large-scale pre-training of end-to-end multi-talker ASR for meeting transcription with single distant microphone,”  in InterSpeech, 2021.
  * [25] Shinji Watanabe et al.,  “Chime-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings,”  in Proc. The 6th International Workshop on Speech Processing in Everyday Environments (CHiME 2020), 2020, pp. 1–7.
  * [26] Jacob Kahn et al.,  “Libri-light: A benchmark for asr with limited or no supervision,”  in IEEE ICASSP, 2020.
  * [27] Changhan Wang et al.,  “VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,”  in ACL/IJCNLP, 2021.
  * [28] Guoguo Chen et al.,  “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,”  in Interspeech, 2021.
  * [29] Najim Dehak et al.,  “Front-end factor analysis for speaker verification,”  IEEE TASLP, 2010.
  * [30] David Snyder et al.,  “X-vectors: Robust DNN embeddings for speaker recognition,”  in IEEE ICASSP, 2018.
  * [31] Brecht Desplanques et al.,  “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,”  InterSpeech, 2020.
  * [32] Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur,  “A time delay neural network architecture for efficient modeling of long temporal contexts,”  in InterSpeech, 2015.
  * [33] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu,  “Squeeze-and-excitation networks,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, 2020.
  * [34] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “VoxCeleb: A large-scale speaker identification dataset,”  in InterSpeech, 2017.
  * [35] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman,  “VoxCeleb2: Deep speaker recognition,”  in InterSpeech, 2018.
  * [36] Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen,  “Permutation invariant training of deep models for speaker-independent multi-talker speech separation,”  in IEEE ICASSP, 2017.
  * [37] Karl Pearson,  “On lines and planes of closest fit to systems of points in space,”  The London, Edinburgh, and Dublin philosophical magazine and journal of science, 1901.