#  UNSTABLE NEUTRINOS CAN RELAX COSMOLOGICAL MASS BOUNDS

S. SANDNER  Instituto de Física Corpuscular, Universidad de Valencia and CSIC,
Edificio Institutos Investigación, Catedrático José Beltrán 2, 46980 Spain

###### Abstract

The light neutrino masses are at present most stringently constraint via
cosmological probes. In particular the Planck collaboration reports $\sum
m_{\nu}\leq 0.12\,\mathrm{eV}$ at $95\%$ CL within the standard cosmological
model. This is more than one order of magnitude stronger than the one arising
from laboratory searches. The cosmological bound taken at face value excludes
a plethora of neutrino flavour models which can successfully explain the
neutrino oscillation data. The indirect nature of the cosmological bound,
however, allows to relax the bound to up to $\sum m_{\nu}\sim 1\,\mathrm{eV}$
if neutrinos decay on timescales shorter than the age of the Universe,
$\tau_{\nu}\leq t_{U}$. We present how a decay of the type
$\nu_{i}\to\nu_{4}\phi$ can be realized within general models of the minimal
extended seesaw framework. The idea is then explicitly realized within the
context of a $U(1)_{\mu-\tau}$ flavour model.

##  1 Introduction

Neutrino oscillation experiments provided valuable information not only on the
flavour mixing but also the neutrino mass square difference. Due to the
sensitivity to matter effects of the solar neutrinos we know that
$\sqrt{\Delta m_{21}^{2}}\simeq 0.0086\,\mathrm{eV}$ but for the atmospheric
neutrino mass splitting we only know $|\sqrt{\Delta
m_{32}^{2}}|\simeq|\sqrt{\Delta m_{21}^{2}}| \simeq 0.05\,\mathrm{eV}$.
Although the neutrino hierarchy can be resolved within near future neutrino
oscillation facilities, i.e. $m_{\nu_{3}}\leq m_{\nu_{1}}$ (inverted hierarchy
(IH)) or $m_{\nu_{1}}\leq m_{\nu_{3}}$ (normal hierarchy (NH)), the neutrino
absolute mass scale still remains undetermined and can not be addressed in
this type of experiments. The best laboratory constraint on the absolute
neutrino mass scale comes from the KATRIN experiment that reports an upper
bound of $\sum m_{\nu}\leq 2.7\,\mathrm{eV}$ at $95\%$ CL, but is expected to
increase its sensitivity to $\sum m_{\nu}\leq 0.6\,\mathrm{eV}$ in upcoming
runs. On the other hand, the cosmological neutrino mass bound is reported to
be $\sum m_{\nu}\leq 0.12\,\mathrm{eV}$ at $95\%$ CL within the standard
cosmological model $\Lambda CDM$. The main constraining power arises from the
contribution of the neutrino to the hot dark matter content of our Universe.
In figure 1 we summarize the evolutional history of the neutrinos over the
course of the cosmological expansion.

Figure 1:  Cosmological evolution of the normalized energy density for
different neutrino masses. In dashed the corresponding evolution for
cosmological fast decaying neutrinos is shown.

Measurements requiere the neutrinos to be relativistic at recombination,
making the evolutional history practically indistinguishable for $\sum
m_{\nu}\leq 1.8\,\mathrm{eV}$ before recombination. However, after
recombination the evolution strongly depends on the absolute mass scale as
this directly translates into a transition time between the relativistic and
non-relativistic propagation. Non-relativistic neutrinos, since they behave as
a matter component of our Universe, contribute to the hot dark matter content
and are severely constraint by the baryonic acoustic oscillations.
Furthermore, the current cosmological bound is robust against common
cosmological extension of the $\Lambda CDM$ model, as e.g. the inclusion of
dark radiation or non-standard dark energy dynamics. Nevertheless, a
modification of the particle physics properties of the neutrino, as e.g.
cosmological fast decays, can dilute the late time neutrino energy density and
hence evade the main constraining power, which is represented by the dashed
lines [1].

##  2 Categorizing Neutrino Decays

Neutrinos can decay already within the Standard Model (SM) of particle
physics, which was first investigated in Ref. [2]. However, it was found that
the decay rate exceeds by many orders of magnitude the current age of our
Universe, i.e. $\tau_{\nu}^{\mathrm{SM}}\geq(G_{F}^{2}m_{\nu}^{5})^{-1}\gg
t_{U}$. First investigation of a possible neutrino decay via a Beyond the
Standard Model (BSM) channel was done in Ref. [3] to explain the at the time
present solar neutrino problem, which then, however, was successfully
explained by corrections to the neutrino vacuum oscillation probability
induced by matter effects. The increasing experimental sensitivity and
especially the in future expected reach motivates a detailed study on the
field of neutrino decays to identify the required particle content, coupling
strength and mass scale of the involved particles needed to result in
$\tau_{\nu}\leq t_{U}$.

Let us take a first look into the simplest BSM realization of (fast) neutrino
decays, which is of the form $\nu_{i}\to\nu_{j}\phi/Z^{\prime}$. Here and in
the following we will denote with $\nu_{i},\nu_{j}$ active neutrino mass
eigenstates and with $\nu_{4}$ the newly introduced sterile neutrino. Angular
momentum conservation requieres the end product of the active neutrino decay
to contain a scalar ($\phi$) or vector ($Z^{\prime}$) boson. Assuming that the
exemplary decay is fast allows us to plot in figure 2 the actual physical mass
of the active neutrino against the one which would be inferred from
cosmological measurements [1]. In the stable neutrino scenario the
correspondence between both masses is one to one, represented by the black
dashed line. In contrary in the decaying neutrino scenario the physical
neutrino mass is generically higher than the cosmological one and the actual
relationship depends on the active neutrino hierarchy and is dictated purely
by energy and angular momentum conservation. In particular this can become
important if future experiments as DESI & EUCLID do not detect neutrino masses
as this would contradict the classical normal and inverted hierarchy
interpretation, whereas decaying neutrinos can bring both results and
interpretation in accordance.

In full generality, we can categorize neutrino decays according to two
criteria: i) the nature of the decay products and ii) the number of particles
in the final state. This leads us to

  1. i)

Nature of the decay products.

    1. a)

At least another active neutrino mass eigenstate $\nu_{j}$.
Since energy and angular momentum conservation ensure the lightest neutrino
state to be stable, the cosmological constraint on $\sum m_{\nu}$ cannot be
relaxed by more than $0.06\,\mathrm{eV}$ and $0.1\,\mathrm{eV}$ for NH and IH,
respectively. Although it can get very relevant in the future, as discussed
before, we will not focus on this possibility, since the relaxation is tightly
constraint.

    2. a)

Only BSM species.
In this case, the constraint on $\sum m_{\nu}$ can be significantly relaxed to
the level of $\sum m_{\nu}\leq 1\,\mathrm{eV}$ at 95% CL, provided that the
BSM particles are massless. This is frequently called neutrino decay into dark
radiation. The exact bound is subject to ongoing investigation and recently
updated in Ref. [4]. New experimental data sets, including the latest Planck
2018+BAO+Pantheon, and a refined analysis significantly strengthened the bound
on a possible cosmological neutrino decay while being non relativistic. The
main constraining power arises from the time modification of the gravitational
pull and the resulting modification of the Hubble parameter and hence a
modified late time large scale structure formation. On the other hand, if
$1/\tau_{\nu}\geq H_{0}\sqrt{\Omega_{m}}(\sum m_{\nu}/(9T_{\nu,0}))^{3/2}$,
neutrinos decay while being relativistic and the same reasoning can not be
applied. However, decays in such a regime can be constrained by noticing that
decay and inverse decay processes alter the neutrino free-streaming properties
to lower scales which results in a phase shift of the high multipole moments
in the cosmic microwave background power spectrum. The resulting bound in the
phase space of active neutrino lifetime and its mass is shown in figure 2.

  2. i)

Number of particles in the final state.

    1. 2)

2-body decays.
Angular momentum conservation requires the decay products to be one fermion
and one boson with spin $0$ (scalar $\phi$) or $1$ (vector $Z^{\prime}$).
Regarding the fermion, we will consider two possibilities: a light neutrino
mass eigenstate $\nu_{i}$ or a sterile neutrino $\nu_{4}$.

    2. 3)

3-body decays and beyond.

For sufficiently massive bosons, i.e. $m_{\phi,\,Z^{\prime}}>m_{\nu_{i}}$, any
2-body decay is kinematically closed. However, such a boson can mediate off-
shell a 3-body decay, which becomes the dominant channel. In a model
independent analysis only the decay of $\nu_{i}\to\nu_{4}\nu_{4}\nu_{4}$
allows to relax the cosmological neutrino mass bound in a broad region of
parameter space [1]. However, in realistic neutrino models the mixing between
active and sterile neutrino states will close the viable window completely.
Hence, we will not focus on 3-body decays and beyond here.

|
---|---
Figure 2:  Left: Relationship between the physical active neutrino mass and
the one inferred from cosmological measurements. The correspondence is $1:1$
in the case neutrinos are cosmological stable. If $\tau_{\nu}\leq t_{U}/10$
the cosmological mass is given as a result of energy and angular momentum
conservation by $\sum_{\mu}^{\mathrm{cosmo}}=3m_{\mathrm{lightest}}$. Right:
Constraints on invisible neutrino decays for massless final state particles
(dark radiation), see main text. In yellow dashed we represent the current
Planck bound on the neutrino mass.

Therefore, according to this classification and discussion therein, six
distinct neutrino decay topologies are possible out of which only the decay
$\nu_{i}\to\nu_{4}\phi/Z^{\prime}$ can relax the bound to up $\sum m_{\nu}\sim
1\,\mathrm{eV}$ in concrete model realizations. This will be the channel of
interest for the rest of the work. The effective Lagrangian describing the new
interaction with a scalar boson is

$\displaystyle\mathcal{L}^{\phi}$
$\displaystyle\supset-\frac{\phi}{2}\,\left[\overline{\nu_{i}}\left(h_{ij}+i\lambda_{ij}\gamma_{5}\right)\nu_{j}+\overline{\nu_{i}}\left(h_{i4}+i\lambda_{i4}\gamma_{5}\right)\nu_{4}+\overline{\nu_{4}}\left(h_{4i}+i\lambda_{4i}\gamma_{5}\right)\nu_{i}\right]+\mathrm{h.c.}\,,$
(1)

and with a vector boson

$\displaystyle\mathcal{L}^{Z^{\prime}}$
$\displaystyle\supset-\frac{Z^{\prime}_{\mu}}{2}\left[g_{ij}^{L}\,\bar{\nu}_{i}\gamma^{\mu}P_{L}\nu_{j}+g_{i4}^{L}\,\bar{\nu}_{i}\gamma^{\mu}P_{L}\nu_{4}+g_{4i}^{L}\,\bar{\nu}_{4}\gamma^{\mu}P_{L}\nu_{i}\right]+\mathrm{h.c.}\,,$
(2)

where $P_{L(R)}$ is the left (right) handed projection operator and $g^{L(R)}$
represent left (right) handed couplings. Note that we neglect the direct
coupling between the sterile neutrinos because it can only be phenomenological
relevant for $3-$body decays (and beyond) which, however, are strongly
disfavored as discussed above. The decay rate for the scalar interaction is
then given by

$\Gamma\simeq\frac{\lambda_{i4}^{2}}{16\pi}\,m_{\nu_{i}}\simeq
t_{U}^{-1}\left(\frac{m_{\nu_{i}}}{0.3\leavevmode\nobreak\
\mathrm{eV}}\right)\left(\frac{\lambda_{i4}}{5\times 10^{-16}}\right)^{2}\,,$
(3)

and for the vector interaction

$\Gamma\simeq\frac{g^{L}_{i4}{}^{2}}{16\pi}\,\frac{m_{\nu_{i}}^{3}}{m_{Z^{\prime}}^{2}}\simeq
t_{U}^{-1}\left(\frac{m_{\nu_{i}}}{0.3\leavevmode\nobreak\
\mathrm{eV}}\right)^{3}\left(\frac{400\,\mathrm{TeV}}{m_{Z^{\prime}}/g_{ij}^{L}}\right)^{2}\,.$
(4)

Note that the translation of the Lagrangians given in equations (1) and (2)
into the neutrino flavour basis, active ($\nu_{\alpha}$) and sterile
($\nu_{s}$), can be realized via a rotation of the type
$\nu_{\alpha}=U_{\alpha i}\,\nu_{i}+\theta_{\alpha 4}\,\nu_{4}$,
$\nu_{s}=\theta_{si}\,\nu_{i}+\theta_{s4}\,\nu_{4}$. Since the mixing between
the sterile and active neutrinos $\theta_{\alpha 4}$ and $\theta_{si}$ should
be small, $U_{\alpha i}$ with $\alpha=e,\mu,\tau$ and $i=1,2,3$ is given by
the PMNS matrix, up to subleading corrections driven by the active-sterile
neutrino mixing.

On the level of the effective Lagrangian couplings can be taken to be
unrelated but in UV complete theories couplings can be correlated via mixing
patterns which may have non-trivial origins. A first naive expectation is that
one should expect either one of the two

  * •

$(\lambda_{ij}:\lambda_{i4}:\lambda_{44})\simeq(1:\theta:\theta^{2})$: Only
active neutrinos interacting directly with a new force carrier and
$\nu_{4}-\phi/Z^{\prime}$ interaction arising via active-sterile neutrino
mixing $\theta$.

  * •

$(\lambda_{ij}:\lambda_{i4}:\lambda_{44})\simeq(\theta^{2}:\theta:1)$: Sterile
neutrinos interacting directly with a new force carrier while active neutrinos
interact only via the mixing $\theta$.

In the following section we will construct a model based on a minimal
extension of the generic seesaw neutrino mass framework which does not show
the above mentioned hierarchical pattern but leads to viable
$\nu_{i}\to\nu_{4}\phi$ decay.

##  3 Concrete Model Realization

Successful neutrino models have to show a mass and flavour structure in
accordance with the latest oscillation data [6]. Many of these models realize
such a pattern via imposed flavour symmetries which introduce symmetry
protected zero modes in the neutrino mass matrix. A commonly considered two-
flavour symmetry leads to two-zero textures which can successfully explain
neutrino oscillation data but predict $\sum m_{\nu}\geq
0.12\leavevmode\nobreak\ \mathrm{eV}$ in $5$ out of $7$ viable realizations
[7]. Furthermore, even though other models may not strictly predict neutrino
masses in conflict with the current cosmological bound many of them
accommodate for $\sum m_{\nu}\geq 0.12\leavevmode\nobreak\ \mathrm{eV}$ across
large regions of the parameter space. We will focus on the case of a neutrino
model with an underlying $U(1)_{\mu-\tau}$ flavour symmetry and incorporate an
invisible neutrino decay via a simple extension of the seesaw framework.

###  3.1 Neutrino Decay within the Seesaw Framework

The realization of the decay $\nu_{i} \to\nu_{4}\phi$ should not spoil the
active neutrino mass generation mechanism by the introduction of the new BSM
states. In particular, a large mixing between the active neutrino and the BSM
states should be prevented to be in accordance with current bounds on light
sterile neutrinos. Hence, the task is to generate a (nearly) massless sterile
neutrino.

This, for example, can be realized by adding an extra fermion singlet $S_{L}$
and one complex scalar singlet $\Phi$ to the usual seesaw model field content
consisting of $\geq 2$ Majorana type right-handed neutrinos. The stability of
the proposed model then is guarantied by an additional global symmetry
$U(1)_{X}$. Assigning the new fields $S_{L}$ and $\Phi$ with opposite
$U(1)_{X}$ charges while having the rest of the field content uncharged the
unique symmetry allowed term is $y\Phi\bar{N}_{R}S_{L}$. In the moment when
the global $U(1)_{X}$ symmetry will by dynamically broken the complex scalar
$\Phi$ obtains a vacuum expectation value $v_{\Phi}$, rendering a massless
goldstone boson – the Majoron – and a massive real part of order $v_{\Phi}$.
Working in the flavour basis, the $7\times 7$ neutrino mass matrix after
symmetry breaking is given by

$\displaystyle M_{\nu}=\begin{pmatrix}0&m_{D}&0\\\ m_{D}^{t}&M_{R}&\Lambda\\\
0&\Lambda^{t}&0\end{pmatrix},$ (5)

which is often referred to as the minimal extended seesaw. Here $m_{D}$ and
$M_{R}$ are the $3\times 3$ Dirac and Majorana mass matrices, representing the
usual seesaw model. The new piece is the $3\times 1$ matrix $\Lambda$ whose
elements are given by $\Lambda_{\alpha}=y_{\alpha}v_{\Phi}$. The eigenvalues
of the matrix represent the mass eigenstates of the different particles at
hand and the diagonalization under the assumption of a scale hierarchy of the
form $\Lambda\ll m_{D}\ll M_{R}$ leads to

$\displaystyle m_{i}$ $\displaystyle\simeq$ $\displaystyle
m_{D}^{2}/M_{R},\;\;\;\nu_{\alpha}\simeq\nu_{i}+\left(m_{D}/M_{R}\right)N_{i}-\left(\Lambda/m_{D}\right)\nu_{4}\,,$
$\displaystyle M_{i}$ $\displaystyle\simeq$ $\displaystyle
M_{R},\;\;\;\;\;\;\;\;\;N_{R}^{c}\simeq-(m_{D}/M_{R})\nu_{i}+N_{i}\,,$
$\displaystyle m_{4}$ $\displaystyle=$ $\displaystyle
0,\;\;\;\;\;\;\;\;\;\;\;\;\;\;S_{L}\simeq(\Lambda/m_{D})\nu_{i}+(\Lambda/M_{R})N_{i}+\nu_{4}\,.$
(6)

A striking feature is the immediate identification that the active neutrino
mass generation is dominantly achieved via the typical seesaw mechanism, while
the sterile state at tree level stays exactly massless. Corrections to the
active neutrino mass arising from the newly introduced sterile neutrino are
heavily suppressed and of order $\mathcal{O}(\Lambda^{2}/M_{R})$ as long as we
are in the limit of the assumed scale hierarchy. The flavour composition is
also such that mixings between different states are always suppressed by $x/y$
where $x$ is some low-scale and $y$ some high-scale. This allows the model to
be basically unaffected by BBN and neutrino oscillation constraints. The
strongest constraint on the active-to-sterile mixing is derived from
cosmological BBN data to be $|\theta_{\alpha 4}|^{2}\lesssim 4\times
10^{-6}\mathrm{eV}/\sqrt{|\Delta m_{4i}^{2}|}$, while laboratory neutrino
oscillation data constrain the mixing to be $|\theta_{\alpha
4}|^{2}\lesssim\mathcal{O}(0.01)$. Note that as a result of the assumed scale
hierarchy we have $m_{\nu_{4}}^{2}-m_{\nu_{i}}^{2}<0$, instead of the commonly
considered scenario of $m_{\nu_{4}}^{2}-m_{\nu_{i}}^{2}>0$ to explain neutrino
oscillation anomalies. Hence, in our model $\sqrt{|\Delta m_{4i}^{2}|}\simeq
m_{\nu_{i}}\lesssim 1\leavevmode\nobreak\ \mathrm{eV}$ which leads to
$|\theta_{\alpha 4}|^{2}\sim\Lambda/m_{D}\lesssim 10^{-3}$, which is
equivalent to requiring a minimal scale hierarchy for the model to work.

To connect to the discussion in the previous section we proceed by identifying
the BSM neutrino interactions in the mass basis which are given by expanding
the unique new term in the interaction Lagrangian

$\displaystyle\Delta\mathcal{L}$ $\displaystyle=$
$\displaystyle\frac{y}{2}\Phi\overline{N_{R}}S_{L}+h.c\supset-\frac{y}{2}\frac{\Lambda}{M_{R}}\overline{\nu_{i}^{c}}(\sigma-i\gamma_{5}\phi)\nu_{j}-\frac{y}{2}\frac{m_{D}}{M_{R}}\overline{\nu_{i}^{c}}(\sigma-i\gamma_{5}\phi)\nu_{4}$
(7) $\displaystyle+$
$\displaystyle\frac{y}{2}\overline{N_{i}}(\sigma-i\gamma_{5}\phi)\nu_{4}-\frac{y}{2}\frac{\Lambda}{m_{D}}\overline{N_{i}}(\sigma-i\gamma_{5}\phi)\nu_{j}+h.c\,,$

in which we denote with $\phi$ the Majoron and $\sigma$ the massive radial
component of $\Phi$. Now comparing to the phenomenological Lagrangian given in
eq. (1) it is easy to identify the relation

$\displaystyle\lambda_{ij}\simeq y\,\Lambda/M_{R},\,\,\,\,\lambda_{i4}\simeq
y\,m_{D}/M_{R},\,\,\,\,\lambda_{44}\simeq 0\,,$ (8)

which does not match any naive expectation for the coupling hierarchy as
presented in section 2. The sterile states among each other are non-
interacting, while active-active and active-sterile interactions are
effective. The dominant coupling is indeed the desired active-to-sterile
portal which can be parametrized using the typical seesaw scales as

$\lambda_{i4}\simeq y\cdot
10^{-13}\sqrt{\left(\frac{m_{\nu}}{0.1\rm{eV}}\right)\left(\frac{10^{16}\rm{GeV}}{M_{R}}\right)}\,.$
(9)

Note that although the model additionally acomodates for three body decays if
$m_{\sigma}\gtrsim m_{\nu}$ it can not be used to relax the cosmological
neutrino mass bound because $\lambda_{44}\simeq 0$. Having discussed the
general strategy to account for neutrino decay within minimal extended seesaw
model we will turn to a concrete model realization in the following.

###  3.2 Flavour Symmetric Model

An example for a neutrino model which satisfies all neutrino oscillation data
but predicts $\sum m_{\nu}=0.146\leavevmode\nobreak\
\text{eV}-0.227\leavevmode\nobreak\ \text{eV}\,(0.121\leavevmode\nobreak\
\text{eV}-2.7\leavevmode\nobreak\ \text{eV})\geq 0.12\leavevmode\nobreak\
\text{eV}$ at 1-$\sigma$ (3-$\sigma$) is the one based on a global
$U(1)_{\mu-\tau}$ flavour symmetry. These type of models are also trivially
anomaly free within the SM. We shall adapt this model in order to illustrate a
concrete UV-completion of the above discussed neutrino decay scenario within
the minimal extended seesaw framework. To this end we first investigate the
standard $U(1)_{\mu-\tau}$ scenario before we add the minimal extended
particle content.

Minimal $U(1)_{\mu-\tau}$ model allowing for $\nu_{i}\to\nu_{j}\phi$
The minimal $U(1)_{\mu-\tau}$ model is equipped with only one SM singlet
scalar $\varphi$ and $3$ heavy sterile neutrinos $N_{i}$ with
$U(1)_{\mu-\tau}$ charges $Q(\varphi)=+1$, $Q(N_{e})=0$ and
$Q(N_{\mu})=-Q(N_{\tau})=+1$. After the dynamical breaking of the global
$U(1)_{\mu-\tau}$ symmetry the CP-odd odd component of $\varphi$ appears a
massless Goldstone boson $\phi$ 111Note that when gravity is included into the
theory we may end up with $m_{\phi}>0$ but due to its weakness we can safely
expect $m_{\phi}\ll m_{\nu}$.. Hence, the model allows for decays of the type
$\nu_{i}\to\nu_{j}\phi$. The coupling $\lambda_{ij}$ controlling the decay
probability is generated via mixing between the heavy sterile and active
neutrinos through the interaction
$\mathcal{L}\supset-\mathcal{Y}_{e\mu}\varphi\bar{N_{e}^{c}}N_{\mu}-\mathcal{Y}_{e\tau}\varphi^{\dagger}\bar{N_{e}^{c}}N_{\tau}$
where $\mathcal{Y}$ denote the Yukawa couplings. To relate to the
phenomenological coupling as expressed in eq. (1) we use the standard seesaw
relation for the mixing $\theta_{N\nu}\simeq\sqrt{m_{\nu}/M_{N}}$ and assume
$\mathcal{Y}_{e\mu}\simeq\mathcal{Y}_{e\tau}$ to end up with

$\lambda_{ij}\simeq\mathcal{Y}_{e\mu}\theta_{N_{e}\nu_{i}}\theta_{N_{\mu}\nu_{j}}+\mathcal{Y}_{e\tau}\theta_{N_{e}\nu_{i}}\theta_{N_{\mu}\nu_{j}}\simeq\frac{\sqrt{m_{\nu_{i}}m_{\nu_{j}}}}{v_{\mu-\tau}}\simeq
10^{-11}\frac{m_{\nu_{i}}}{0.1\,\mathrm{eV}}\frac{10\,\mathrm{GeV}}{v_{\mu-\tau}}\,.$
(10)

The requirement of cosmological fast neutrino decay leads to

$\sqrt{\frac{16\pi}{f(m_{\nu_{i}},m_{\nu_{j}})}t_{U}}\lesssim\lambda_{ij}\lesssim\sqrt{\frac{32\pi}{f(m_{\nu_{i}},m_{\nu_{j}})}10^{-8}t_{U}\left(\frac{m_{\nu_{i}}}{0.1\,\text{eV}}\right)^{3}}\,,$
(11)

where we defined the $2-$body decay phase space function
$f(m_{\nu_{i}},m_{\nu_{j}})=(m_{\nu_{i}}-m_{\nu_{j}})^{3}(m_{\nu_{i}}+m_{\nu_{j}})/m_{\nu_{i}}^{3}$.
CMB constraints on the neutrino free streaming property sets the upper limit
on the coupling. Given the expression of the coupling above in eq. (10) this
maps to $1\leavevmode\nobreak\ \text{GeV}\lesssim v_{\mu-\tau}\lesssim
10\leavevmode\nobreak\ \text{TeV}$. Interestingly, the required scale is just
around the electroweak scale. Recall that the relaxation for decays of type
active-to-active is bounded by the mass splitting between the active neutrino
mass eigenstates and hence we need to extend the model to account for a
relaxation of $\sum m_{\nu}\sim 1\leavevmode\nobreak\ \text{eV}$.

Minimal Extended $U(1)_{\mu-\tau}$ model allowing for $\nu_{i}\to\nu_{4}\phi$
Let us now turn to discuss an extension of the analyzed $U(1)_{\mu-\tau}$
within the minimal extended seesaw framework. Following the general discussion
around eq. (5) we add one singlet fermion $S_{L}$ and one complex scalar
$\Phi$ to the model. Both are oppositely charged under some new global
symmetry $U(1)_{X}$, but only $\Phi$ can carry a $U(1)_{\mu-\tau}$ charge in
order to guarantee anomaly cancellations. In order to allow for active-to-
sterile neutrino conversion via the coupling $y\Phi\bar{N_{R}}S_{L}$ the
$\Phi$ charge under $U(1)_{\mu-\tau}$ has to one of the three options
$Q(\Phi)=0,\pm 1$, which leads to $\Lambda^{T}=\\{\Lambda_{e},0,0\\}$ or
$\Lambda^{T}=\\{0,\Lambda_{\mu},\Lambda_{\tau}\\}$ respectively. Combining the
model independent result for the decay rate of $\nu_{i}\to\nu_{4}\phi$, as
given in eq. (3), with the constraint on the coupling arising from the minimal
extended seesaw framework, as given in eq. (9), we arrive at

$\Gamma_{\nu_{i}\to\nu_{4}\phi}\simeq 10^{6}t_{U}^{-1}y^{2}\left(\frac{\sum
m_{\nu}}{1\leavevmode\nobreak\
\text{eV}}\right)^{2}\left(\frac{10^{14}\leavevmode\nobreak\
\text{GeV}}{M_{R}}\right)\,.$ (12)

We show in figure 3 the physical parameter space in which the cosmological
mass bound can be relaxed, together with the $1\sigma$ and $3\sigma$
prediction for the light neutrino mass within the considered model.

Figure 3:  Parameter space of the $U(1)_{\mu-\tau}$ model within the minimal
extended seesaw framework. The red bands indicate the $1\sigma$ and $3\sigma$
prediction for neutrino masses within de model, generically being higher than
the current Planck bound (yellow dashed). Neutrino decays within the model can
relax the tension within various orders of magnitude in Yukawa coupling and
Majorana mass scale.

In particular, for Yukawa couplings of order $y\sim 1$ the neutrino mass bound
can be relaxed for a Majorana scale of $10^{10}\,\text{GeV}\lesssim
M_{R}\lesssim 10^{19}\leavevmode\nobreak\ \text{GeV}$, which points to a
canonical realization of the seesaw mechanism. Thereby, potential cosmological
constraints on the right handed Majorana neutrino can be evaded as well as its
effects on low energy particle physics experiment would be suppressed in the
same manner.

##  4 Summary and Conclusion

Neutrino masses are at present best constraint via the indirect inference from
cosmological measurements, while direct laboratory bounds are currently one
order of magnitude weaker. However, the cosmological neutrino mass bound,
although stable against typical modification of the background evolution, is
strongly dependent on the underlying assumption of the particle physics nature
of the neutrino. The impact of cosmological fast neutrino decays onto the
cosmic microwave background has been investigated in Ref. [4]. This allows to
constraint the underlying particle physics parameters as i.e. the involved
couplings and its minimal beyond the Standard Model realization. General
constraints can be derived via a model independent analysis which can be used
to map to any concrete UV completed model realization. In particular, flavour
models of the type $U(1)_{\mu-\tau}$ can be embedded in the minimal extended
seesaw framework to allow to relax the cosmological neutrino mass bound $\sum
m_{\nu}\leq 0.12\leavevmode\nobreak\ \text{eV}\mapsto\sum m_{\nu}\lesssim
1\leavevmode\nobreak\ \text{eV}$. This is particularly interesting in the
light of the sensitivity of upcoming galaxy surveys as DESI and EUCLID. The
1-$\sigma$ sensitivity of $\sigma\left(\sum m_{\nu}\right)\simeq
0.02\leavevmode\nobreak\ \text{eV}$ is equivalent to a neutrino mass detection
if neutrinos are cosmological stable particles within the cosmological
standard model. A potential non-detection then would be in tension with well
established laboratory oscillation experiments, but could be resolved within
the neutrino decay scenario. Furthermore, if the KATRIN experiment reports a
neutrino mass detection within its expected sensitivity reach of $\sum
m_{\nu}\simeq 0.6\leavevmode\nobreak\ \text{eV}$, models of neutrino decays
are a possible explenation to reconcile cosmological and laboratory
measurements.

## Acknowledgments

The work of SS received the support of a fellowship from ”la Caixa” Foundation
(ID 100010434) with fellowship code LCF/BQ/DI19/11730034.

## References

## References

  * [1] M. Escudero, J. Lopez-Pavon, N. Rius and S. Sandner, JHEP 12 (2020), 119 doi:10.1007/JHEP12(2020)119 [arXiv:2007.04994 [hep-ph]].
  * [2] P. B. Pal and L. Wolfenstein, Phys. Rev. D 25 (1982), 766 doi:10.1103/PhysRevD.25.766
  * [3] J. N. Bahcall, N. Cabibbo and A. Yahil, Phys. Rev. D 28, 316 doi:10.1103/PhysRevLett.28.316
  * [4] G. Barenboim, J. Z. Chen, S. Hannestad, I. M. Oldengott, T. Tram and Y. Y. Y. Wong, JCAP 03 (2021), 087 doi:10.1088/1475-7516/2021/03/087 [arXiv:2011.01502 [astro-ph.CO]]; G. F. Abellán, Z. Chacko, A. Dev, P. Du, V. Poulin and Y. Tsai, [arXiv:2112.13862 [hep-ph]]; M. Escudero and M. Fairbairn, Phys. Rev. D 100 (2019) no.10, 103531 doi:10.1103/PhysRevD.100.103531 [arXiv:1907.05425 [hep-ph]]; Z. Chacko, A. Dev, P. Du, V. Poulin and Y. Tsai, Phys. Rev. D 103 (2021) no.4, 043519 doi:10.1103/PhysRevD.103.043519 [arXiv:2002.08401 [astro-ph.CO]];
  * [5] Z. Chacko, A. Dev, P. Du, V. Poulin and Y. Tsai, JHEP 04 (2020), 020 doi:10.1007/JHEP04(2020)020 [arXiv:1909.05275 [hep-ph]].
  * [6] I. Esteban, M. C. Gonzalez-Garcia, M. Maltoni, T. Schwetz and A. Zhou, JHEP 09 (2020), 178 doi:10.1007/JHEP09(2020)178 [arXiv:2007.14792 [hep-ph]].
  * [7] J. Alcaide, J. Salvado and A. Santamaria, JHEP 07 (2018), 164 doi:10.1007/JHEP07(2018)164 [arXiv:1806.06785 [hep-ph]].


# Enhancement of Over-the-Air Federated Learning by Using AI-based Fluid
Antenna System

IEEE Publication Technology  Mohsen Ahmadzadeh, Saeid Pakravan, Ghosheh Abed
Hodtani, Ming Zeng, and Jean-Yves Chouinard M. Ahmadzadeh and G. Abed Hodtani
are with the Department of Electric and Computer Engineering, Ferdowsi
University, Mashhad, Iran. email<EMAIL_ADDRESS><EMAIL_ADDRESS>Pakravan, M. Zeng, and J-Y. Chouinard are with the
Department of Electrical and Computer Engineering, Laval University, Quebec
City, QC, G1V 0A6, CA. email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This letter investigates an over-the-air federated learning (OTA-FL) system
that employs fluid antennas (FAs) at the access point (AP) to enhance learning
performance by leveraging the additional degrees of freedom provided by
antenna mobility. First, we analyze the convergence of the OTA-FL system and
derive the optimality gap to illustrate the influence of FAs on learning
performance. Then, we formulate a nonconvex optimization problem to minimize
the optimality gap by jointly optimizing the positions of the FAs and the
beamforming vector. To address the dynamic environment, we cast this
optimization problem as a Markov decision process (MDP) and propose the
recurrent deep deterministic policy gradient (RDPG) algorithm. Finally,
extensive simulations show that the FA-assisted OTA-FL system outperforms
systems with fixed-position antennas and that the RDPG algorithm surpasses the
existing methods.

###### Index Terms:

Optimality gap, Over-the-air federated learning, Fluid antenna, Deep
reinforcement learning.

##  I Introduction

Federated learning (FL) has gained significant prominence in the field of
machine learning in recent years due to its decentralized framework and robust
privacy protection measures. FL leverages the computational capabilities of
edge devices to collectively train a unified global model, all while ensuring
that sensitive locally stored data remains confidential. This approach is
particularly suitable for applications in various Internet of Mobile Things
(IoMT) scenarios, including the Internet of Drones (IoD)[1] and mobile crowd
sensing[2], among others. To mitigate communication latency and costs
associated with FL implementation, over-the-air computation (AirComp) for
model aggregation has emerged as a promising approach. It leverages the
superposition property of wireless multiple access channels (MAC) [3].
However, the performance of model aggregation in over-the-air FL (OTA-FL) is
severely affected by adverse wireless propagation channel conditions,
especially in massive mobile IoT scenarios.

To address this challenge, previous research has extensively explored the
integration of intelligent reflecting surfaces (IRSs) into OTA-FL to achieve
reliable model aggregation. This is accomplished by reconfiguring wireless
channels through adjusting the passive reflecting coefficients of the IRS [3].
Additionally, other studies have investigated beamforming techniques at the
receiver server to enhance OTA-FL performance by leveraging the degrees of
freedom (DoF) in the spatial domain [4]. However, the fixed positions of
receiver antennas can impose limitations on the potential gains from
beamforming. To overcome these limitations, we propose the use of fluid
antennas (FAs) in OTA-FL to dynamically manipulate wireless channel conditions
through adaptive movement of antennas. Unlike fixed-position antennas (FPA),
FAs offer the capability to reshape channel environments, thereby introducing
new DoFs to enhance OTA-FL performance [5]. Previous studies have demonstrated
the superior performance of FAs over traditional FPAs in various communication
systems, including AirComp systems [6], multi-user uplink communications [7,
5, 8], and mobile edge computing [9]. Moreover, FAs have been investigated to
maximize network sum-rate in multiple access communication systems through
deep reinforcement learning (DRL) [10]. However, the integration of FAs into
OTA-FL systems has not yet been addressed in current academic literature to
the best of our knowledge.

In this paper, we propose the integration of FA systems with OTA-FL to enhance
convergence performance. Our objective is to minimize the optimality gap
through joint optimization of the beamforming vector and the antenna position
vector at the access point (AP) under practical dynamic conditions [11].
First, we derive the optimality gap between the actual loss and the optimal
loss for OTA-FL to quantify the impact of the beamforming vector and antenna
positioning. Subsequently, based on the convergence analysis, we formulate a
non-convex optimization problem to enhance learning performance. We then
reformulate it as a Markov decision process (MDP) and leverage deep
reinforcement learning (DRL) techniques suitable for dynamic environments. We
introduce the recurrent deep deterministic policy gradient (RDPG) algorithm,
with actor and critic networks specifically designed to capture the temporal
correlation of state features, enabling real-time decisions under dynamic
conditions. Extensive simulations are undertaken to evaluate the performance
of the integrated FAs in OTA-FL systems compared to FPA, and the performance
of the proposed RDPG algorithm in comparison to standard DRL algorithms such
as soft actor-critic (SAC) and deep deterministic policy gradient (DDPG).
Simulation results indicate that the proposed RDPG algorithm outperforms both
SAC and DDPG methods in terms of performance and stability in both scenarios.
Furthermore, the FA-assisted OTA-FL system consistently outperforms the FPA
scenarios.

Notations: Italicized letters represent scalars, while boldface letters denote
vectors. $(\cdot)^{T}$ is the transpose, $(\cdot)^{H}$ the conjugate
transpose, and $\mathbb{E}[\cdot]$ the expectation operation. $|\cdot|$
signifies the magnitude of a scalar or the cardinality of a set. The Euclidean
norm of a vector is represented by $\left\|.\right\|$.

##  II System Model

We consider an OTA-FL system comprising $K$ single-antenna user equipment (UE)
devices, denoted as $\text{UE}_{k}$, $\forall
k\in\mathcal{K}\triangleq\\{1,2,\ldots,K\\}$. These devices could be include
mobile IoT devices, UAVs, drones, or similar entities. The UEs are randomly
distributed and move dynamically within a designated area of interest, where
they collect local data samples. These samples are collaboratively utilized to
train a global model at an AP equipped with $N$ FAs.

###  II-A OTA-FL Model

We consider an OTA-FL framework with full participation that executes
sequential actions at each iteration $t$ over $T$ training rounds as follows:

  * •

Global model broadcast: The AP broadcasts the current global model
$\boldsymbol{w}_{t}\in\mathbb{R}^{d}$ to all UEs, where $d$ represents the
dimensionality of the model parameter space.

  * •

Local model update: Each UE updates its local model using the gradient descent
algorithm as
$\boldsymbol{w}_{k,t}=\boldsymbol{w}_{t}-\gamma\nabla{F}(\boldsymbol{w}_{t},\mathbf{\mathcal{D}}_{k})$,
where $\gamma$ is the learning rate,
$\nabla{F}(\boldsymbol{w}_{t},\mathbf{\mathcal{D}}_{k})$ represents the
gradient of the local loss function, and $\mathbf{\mathcal{D}}_{k}$ is the
local dataset for $\text{UE}_{k}$ with a local dataset size denoted by
$|\mathbf{\mathcal{D}}_{k}|=D$.

  * •

Model aggregation: Each UE transmits its local model to the AP, which then
performs aggregation by averaging to update the global model as:

$\displaystyle\boldsymbol{w}_{t+1}$
$\displaystyle=\frac{1}{K}\sum_{k\in\mathcal{K}}\boldsymbol{w}_{k,t}.$ (1)

The procedure continues iteratively until reaching the maximum specified
number of outer iterations.

###  II-B Communication Model

We consider the uploading phase within the OTA-FL system, where each UE
synchronously transmits its updated model parameters to the AP. The AP is
equipped with an array of FAs, facilitating the adjustment of each FA along a
one-dimensional line segment that spans a length of $X$. Each FA’s position is
constrained within the interval $[0,X]$ with a minimum distance $X_{0}$
between adjacent FAs to prevent antenna coupling. The collective locations of
all $N$ FAs are represented as the vector
$\mathbf{x}=[x_{1},\ldots,x_{N}]^{T}$, with their movement along one dimension
restricted by $x_{1}<x_{2}<\ldots<x_{N}$. For the purposes of brevity and
clarity, time indices are omitted in this subsection. Assuming line-of-sight
(LoS) propagation conditions, the channel between the $\text{UE}_{k}$ and the
AP, represented by $\mathbf{h}_{k}[\mathbf{x}]\in\mathbb{C}^{N\times 1}$, is
given by

$\mathbf{h}_{k}[\mathbf{x}]=\sqrt{\frac{l_{0}}{{d^{\alpha}_{k}}}}\left[e^{j\frac{2\pi}{\lambda}x_{1}\cos(\phi_{k})},\ldots,e^{j\frac{2\pi}{\lambda}x_{N}\cos(\phi_{k})}\right]^{T},$
(2)

where $l_{0}$, $\lambda$, and $\alpha$ denote the path loss at the reference
distance, the wavelength, and the path loss exponent, respectively. $d_{k}$
and $\phi_{k}$ represent the distance between the FAs and $\text{UE}_{k}$ and
the angle of arrival (AoA) of the LoS path, respectively, which are determined
by the location of the UEs in each training round. In this system, we assume
each UE moves within an designated area and then transmits model parameters
from a stationary position [2]. Moreover, given that the signal path length
significantly exceeds the FA movement area, the far field condition between
the AP and UEs is assumed. Consequently, $\phi_{k}$ and $d_{k}$ are treated as
constants during transmission, regardless of FA positional changes [9, 7].

The AP receives the local model parameters from all UEs in the $t$-th training
round as:

$\mathbf{y}=\sum_{k\in\mathcal{K}}{p}_{k}\mathbf{{h}}_{k}[\mathbf{x}]\boldsymbol{w}_{k}+\mathbf{z},$
(3)

where, $p_{k}$ denotes the transmission power factor for the $k$-th UE, and
$\mathbf{z}\in\mathbb{C}^{N\times d}$ represents an additive white Gaussian
noise (AWGN) matrix with elements following a complex normal distribution
$\mathcal{CN}(0,\sigma^{2})$.

The aggregated model parameter vector, $\boldsymbol{\hat{w}}$, in the $t$-th
training round is estimated by conducting post-processing on the received
signal at the AP as follows:

$\displaystyle\boldsymbol{\hat{w}}$
$\displaystyle=\frac{1}{K}(\frac{1}{\sqrt{\eta}}\boldsymbol{m}^{H}\mathbf{y})$
(4)
$\displaystyle=\frac{1}{K}(\sum_{k\in\mathcal{K}}\frac{1}{\sqrt{\eta}}\boldsymbol{m}^{H}{p}_{k}\mathbf{h}_{k}[\mathbf{x}]\boldsymbol{w}_{k}+\frac{\boldsymbol{m}^{H}\mathbf{z}}{\sqrt{\eta}}),$

where, $\boldsymbol{m}\in\mathbb{C}^{N\times 1}$ denotes the beamforming
vector at the AP, and $\eta$ represents the scaling factor for signal
amplitude alignment.

According to [4, 3], in order to minimize the mean square error (MSE) in the
OTA-FL system, the transmit power factor in each client is designed as
follows:

$\displaystyle{p}_{k}=\frac{\sqrt{\eta}\left(\boldsymbol{m}^{H}\boldsymbol{h}_{k}[\boldsymbol{x}]\right)^{H}}{\left|\boldsymbol{m}^{H}\boldsymbol{h}_{k}[\boldsymbol{x}]\right|^{2}}.$
(5)

We consider that the transmission power allocated to each $\text{UE}_{k}$ does
not exceed the maximum transmission power limit $p_{\text{max}}$, expressed
as:

$\frac{1}{d}p_{k}^{2}\mathbb{E}\left[\|\boldsymbol{w}_{k}\|^{2}\right]\leq
p_{\max},\quad\forall k\in\mathcal{K}.$ (6)

Moreover, under the assumption of full participation in FL to adhere to the
maximum power budget for each client, the upper bound of $\eta$ must satisfy
the following condition:

$\displaystyle\eta\leq\frac{dp_{\text{max}}\left|\boldsymbol{m}^{H}\boldsymbol{h}_{k}[\boldsymbol{x}]\right|^{2}}{\mathbb{E}\left[\|\boldsymbol{w}_{k}\|^{2}\right]},\quad\forall
k\in\mathcal{K}.$ (7)

##  III Convergence Analysis

In order to facilitate our analysis of convergence, we incorporate the
following commonly accepted assumptions as discussed in [3, 1, 12]:

Assumption 1: The global loss function $F(\boldsymbol{w})$ is $\ell$-smooth.
Namely, for any given model parameters
$\boldsymbol{w},\boldsymbol{v}\in\mathbb{R}^{d}$, there exists a nonnegative
constant $\ell$, such that

$F(\boldsymbol{w})-F(\boldsymbol{v})\leq(\boldsymbol{w}-\boldsymbol{v})^{T}\nabla
F(\boldsymbol{v})+\frac{\ell}{2}\|\boldsymbol{w}-\boldsymbol{v}\|^{2}.$ (8)

Assumption 2: The loss function satisfies the Polyak-Lojasiewicz inequality,
where $F(\boldsymbol{\boldsymbol{w}}^{*})$ denotes the optimal global loss
value and $\mu>0$, as follows:

$\|\nabla F(\boldsymbol{w})\|^{2}\geq
2\mu[F(\boldsymbol{w})-F(\boldsymbol{w}^{*})].$ (9)

Assumption 3: The upper limit of the model parameter for $\text{UE}_{k}$ is
denoted as $\Gamma\geq 0$. This ensures:

$\mathbb{E}\left[\left\|\boldsymbol{w}_{k}\right\|^{2}\right]\leq\Gamma,\quad\forall
k\in\mathcal{K}.$ (10)

Theorm 1: Under the conditions outlined in Assumptions III, III, and III, and
setting the learning rate to $1/\ell$, the optimality gap after $T$ rounds of
training is bounded as follows:

$\displaystyle\mathbb{E}[F(\boldsymbol{w}_{T+1})]-F(\boldsymbol{w}^{*})$
$\displaystyle\leq\psi^{T}\left(\mathbb{E}[F(\boldsymbol{w}_{1})]-F(\boldsymbol{w}^{*})\right)$
(11)
$\displaystyle+\sum_{t=1}^{T}\psi^{T-t}\Theta_{t}=\Phi_{T}(\boldsymbol{m},\boldsymbol{x}),$

where, $\Theta_{t}$ and $\psi$ can be represented as follows:

$\displaystyle\Theta_{t}=\frac{l\sigma^{2}\Gamma}{2K^{2}p_{\max}}\max_{k\in\mathcal{K}}\frac{\|\boldsymbol{m}_{t}^{H}\|^{2}}{|\boldsymbol{m}_{t}^{H}\boldsymbol{h}_{k}[\boldsymbol{x}]|^{2}},$
(12) $\displaystyle\psi=1-\frac{\mu}{l}.$ (13)

###### Proof:

See Appendix. ∎

##  IV Problem Formulation

In this paper, our objective is to enhance learning performance in OTA-FL
through the design of FA systems within dynamic environments. According to
Theorem III, the optimality gap is influenced by the configuration of the
beamforming vector and the FA locations in the training iterations. Thus, we
formulate an optimization problem to jointly optimize
$\boldsymbol{m}=[m_{1},\ldots,m_{N}]^{T}$ and
$\boldsymbol{x}=[x_{1},\ldots,x_{N}]^{T}$ for all $t\in[1,\ldots,T]$, aiming
to minimize the total optimality gap as follows:

$\displaystyle\mathcal{P}_{1}:$
$\displaystyle\min_{\boldsymbol{m},\boldsymbol{x}}\hskip
5.69046pt\Phi_{T}(\boldsymbol{m},\boldsymbol{x})$ (14)
$\displaystyle\text{s.t.}\quad C_{1}:0\leq x_{n}\leq X,\ \ \ \ \ \ \forall
n\in[1,\ldots,N],$ $\displaystyle\ \ \ \ \ \ C_{2}:x_{1}<x_{2}<\ldots<x_{N},$
$\displaystyle\ \ \ \ \ \ C_{3}:x_{n}-x_{n-1}>X_{0},\ \forall
n\in[2,\ldots,N],$

where, $C_{1}$ constrains the permissible range for FA locations, $C_{2}$
determines the order of FA placements, and $C_{3}$ enforces a minimum
separation distance between adjacent FAs.

Due to the non-convex nature of the objective function and the stochasticity
inherent in the dynamic environment, particularly in massive mobile IoT
scenarios, traditional optimization methods face challenges in solving the
original intractable problem $\mathcal{P}_{1}$. In order to tackle this issue,
we propose transforming problem $\mathcal{P}_{1}$ into an online optimization
problem, subsequently reformulating it as an MDP. Based on Theorem III, the
optimality gap at the $t$-th training round, denoted as
$\Phi_{t}(\boldsymbol{m},\boldsymbol{x})$, is bounded as follows:

$\displaystyle\Phi_{t}(\boldsymbol{m},\boldsymbol{x})$
$\displaystyle\leq\text{$\Phi_{t-1}(\boldsymbol{m},\boldsymbol{x})$}$ (15)
$\displaystyle+(\psi^{t}-\psi^{t-1})(\mathbb{E}[F(\boldsymbol{w}_{1})]-F(\boldsymbol{w}^{*}))+\Theta_{t}(\boldsymbol{m},\boldsymbol{x}),$

where (15) indicates that when $\psi$ and the initial optimality gap
$(\mathbb{E}[F(\boldsymbol{w}_{1})]-F(\boldsymbol{w}^{*}))$ are known, the
optimality gap is determined by $\Theta_{t}$ and the previous optimality gap
$\Phi_{t-1}(\boldsymbol{m},\boldsymbol{x})$. Thus, the problem
$\mathcal{P}_{1}$ of minimizing the optimality gap after $T$ communication
rounds can be transformed into minimizing
$\Theta_{t}(\boldsymbol{m},\boldsymbol{x})$ in each round. This reformulation
is expressed as follows:

$\displaystyle\mathcal{P}_{2}:\min_{\boldsymbol{m},\boldsymbol{x}}\hskip
5.69046pt\Theta_{t}(\boldsymbol{m},\boldsymbol{x})$ (16) $\displaystyle\ \ \ \
\text{s.t. }C_{1},C_{2},C_{3}.$

##  V Proposed DRL Algorithm

To address Problem $\mathcal{P}_{2}$, we deploy a DRL agent on the AP to learn
an optimal decision policy that simultaneously optimizes the beamforming
vector $\boldsymbol{m}$ and the FA locations $\boldsymbol{x}$ in each training
round in order to minimize $\Theta_{t}(\boldsymbol{m},\boldsymbol{x})$.
Details of the MDP are provided below:

  * •

State Space: The state space at time slot $t$ consists of the distances
$d_{k}$ between the FAs and the $\text{UE}_{k}$, and the AoA of the LoS paths
$\phi_{k}$, $\forall k\in\mathcal{K}$. The state space can be expressed as:
$s_{t}=[[d_{1},\ldots,d_{K}],[\phi_{1},\ldots,\phi_{K}]]$.

  * •

Action space: The action space at each time slot $t$ comprises the beamforming
vector and the locations of the FAs. Consequently, the action space at time
slot $t$ can be expressed as:

$a_{t}=[[m_{1},\ldots,m_{N}],[x_{1},\ldots,x_{N}]].$ (17)

  * •

Reward function: Based on (12), in order to minimize
$\Theta_{t}(\boldsymbol{m},\boldsymbol{x})$, the reward function can be
formulated as:

$r(s_{t},a_{t})=\begin{cases}r_{1},\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
\ \ \ \ \ \ \ \ \ \ \ \|\boldsymbol{m}\|=0,\\\
r_{2}\max_{k\in\mathcal{K}}\left(\frac{\|\boldsymbol{m}\|^{2}}{|\boldsymbol{m}^{H}\mathbf{h}_{k}[\boldsymbol{x}]|^{2}}\right),\
\ \ \ \ \text{otherwise},\end{cases}$ (18)

where, the constants $r_{1}$ and $r_{2}$ are negative values that require
tuning during the simulation process to achieve better convergence. Notably,
the reward function is formulated as a negative value. Therefore, by
maximizing this (negative) reward, the agent effectively minimizes
$\Theta_{t}(\boldsymbol{m},\boldsymbol{x})$.

Since the action space is continuous, we cannot use model-free value-based DRL
algorithms such as deep Q-network (DQN), as they can only handle discrete
action spaces. Instead, we utilize policy gradient-based reinforcement
learning methods. The DDPG algorithm is a suitable off-policy actor-critic
approach capable of managing continuous action spaces. However, the fully-
connected deep neural networks (DNNs) employed in conventional DDPG are
inadequate for capturing the temporal patterns of environmental dynamics, such
as user mobility [13, 11]. Therefore, we adjust the RDPG approach by
incorporating long short-term memory (LSTM) into the DDPG architecture to
exploit temporal state patterns and adapt continuously to environmental
dynamics.

The proposed RDPG algorithm utilizes four neural networks: an Actor Network
(policy network) denoted by $\pi_{\phi}$ with parameter $\phi$, which
determines actions $a_{t}=\pi_{\phi}(s_{t})+\xi$ based on states $s_{t}$,
where $\xi$ is a random process added to actions for exploration; a Critic
Network (Q-network) with parameters $\theta$ that computes Q-values
$Q_{\theta}(s_{t},a_{t};\theta)$ for state-action pairs; a Target Actor
Network, which is an older version of the actor network; and a Target Critic
Network, which is an older version of the critic network.

In our optimization problem, the objective is to minimize the optimality gap
by maximizing the expected reward $r(s_{t},a_{t})$ in each training round. The
goal of the RDPG, given the state $s_{t}$ and action $a_{t}$, is to identify a
policy that maximizes the expected cumulative reward, defined as:

$\pi^{*}=\arg\max_{\pi}\mathbb{E}_{s_{t},a_{t}}\left[\sum_{t=0}^{\infty}r(s_{t},a_{t})\right],$
(19)

to achieve this, the Actor Network is optimized based on the gradient of the
objective function $J(\phi)$ as follows:

$\nabla_{\phi}J(\phi)=\mathbb{E}\left[\nabla_{a_{t}}Q_{\theta_{1}}(s_{t},a_{t})\bigg{|}_{a_{t}=\pi_{\phi}(s_{t})}\nabla_{\phi}\pi_{\phi}(s_{t})\right],$
(20)

while the critic network is trained to minimize the loss function relative to
the target value $Y_{t}$, defined as:

$Y_{t}=r_{t}+\gamma
Q_{\theta_{i}^{\prime}}(s_{t+1},\pi_{\phi^{\prime}}(s_{t+1})+{\xi}).$ (21)

The proposed RDPG method is described in Algorithm 1.

Initialize: experience replay memory $M$, mini-batch size $H$, the actor
network $\pi_{\phi}$, the critic network $Q_{\theta}$ with random values, and
create the target networks by setting $\theta^{\prime}\leftarrow\theta$ and
$\phi^{\prime}\leftarrow\phi$.

Set: Set $E$ and $T$ as the maximum number of episodes and episode length,
respectively.

for  _each episode $e:E{}$_ do

Initialize the environment state $s_{0}$, and the exploration noise $\xi$;

for  _$t=1:T$_ do

Receive $s_{t}$ from the environment;

Obtain $a_{t}=\pi_{\phi}(s_{t})+\xi$ from the actor network and re-shape it;

Obtain $r_{t}$ based on equation (18);

Observe the new state, $s_{t+1}$;

Store transition $(s_{t},a_{t},r_{t},s_{t+1})$ into $M$;

end for

Randomly sample a $H$ mini-batch of transitions from $M$;

Compute the target function $Y_{t}$ according to (21);

Update the actor and critic networks using the Adam optimizer.

Soft update the target actor and target critics with $\tau\in[0,1]$, as the
soft update coefficient:
$\phi^{\prime}\leftarrow\tau\phi+(1-\tau)\phi^{\prime},\quad\theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta^{\prime}$

end for

Algorithm 1 The RDPG Algorithm

##  VI Simulation Results

In this section, we provide numerical results illustrating how combining FA
arrays with the proposed RDPG algorithm can improve OTA-FL learning
performance. We consider the distance between users and the AP to be uniformly
distributed in the range $[20,100]$ m, and the AoAs to be uniformly
distributed over $[-\pi/2,\pi/2]$ rad. The parameters for the FA arrays are
set with $X_{0}=0.5\lambda$ and $X=8\lambda$. The RDPG algorithm is configured
with a learning rate of 0.0005, a replay buffer size of $10^{4}$, a batch size
of 64, a soft update parameter of 0.001, and a discount factor of 0.9. For
performance evaluation, we consider two comparisons. Initially, we evaluate
the performance of FA relative to FPA with a predetermined location vector
$\boldsymbol{x}=\left[\frac{X}{N+1},\ldots,\frac{NX}{N+1}\right]^{T}$.
Subsequently, we compare the proposed RDPG algorithm with SAC [14] and DDPG
[13], which are conventional DRL algorithms. Learning performance is evaluated
by computing the average rewards over 100 episodes, which is determined at
episode $e$ by employing the formula
${R_{\text{avg}}(e)}=\frac{1}{100}\sum_{i=e-100}^{e}R_{i}$, where $R_{i}$
signifies the mean reward of episode $i$.

Figure 1: Comparing DRL Algorithms in FA and FPA systems across (a) training
episodes; (b) antenna numbers; and (c) client numbers.

Fig. 1 (a) demonstrates the convergence characteristics of different DRL
algorithms, depicting the average rewards with solid curves and showing the
standard deviations as shaded regions. The RDPG algorithm exhibits higher
average rewards and lower variance compared to standard DRLs, demonstrating
superior performance and improved stability in dynamic environments.

To assess the performance of the proposed algorithms across varying numbers of
antennas, we kept the number of clients fixed and varied the antenna count in
both the FA and FPA scenarios. As depicted in Fig. 1 (b), the average reward
performance of all DRL methods improves with increasing $N$, although this
improvement diminishes as $N$ continues to increase. Furthermore, due to the
increased DoF provided by antenna adjustments in FA systems, FAs consistently
outperform FPAs at all values $N$. Moreover, RDPG demonstrating superior
performance over other DRL algorithms.

Fig. 1(c) provides a detailed comparison of the performance of FAs and the
RDPG algorithm across varying numbers of users. It is evident from the results
that as the number of users increases, there is a noticeable decrease in
performance for both FA and FPA scenarios. This decline is attributed to the
increased challenge of optimizing the beamforming vector and the antenna
position vector of the AP in the presence of more dynamic users. Despite these
challenges, FAs consistently outperform FPAs in all tested scenarios,
highlighting the efficacy of FAs in enhancing OTA-FL system performance.
Moreover, the RDPG algorithm consistently exhibits superior performance
compared to other optimization methods in mitigating the adverse effects of
dynamic user dynamics on system performance.

##  VII Conclusion

In this study, we investigated the integration of FAs into AP to improve the
performance of OTA-FL systems. Our convergence analysis highlighted the
significant impact of FA positions and the beamforming vector on the
optimality gap. We addressed this issue with a non-convex optimization problem
and proposed the RDPG algorithm for real-time optimization. Through
simulations, we demonstrated that the OTA-FL system enhanced by FAs
outperformed conventional FPA systems. Moreover, RDPG demonstrates superior
performance and stability compared to existing methods, validating its
effectiveness in dynamic environments.

## Proof of Theorem III

In the $t$-th communication round, the global model update is formulated by
integrating (5) into (4), as follows:

$\displaystyle\boldsymbol{w}_{t+1}$
$\displaystyle=\frac{1}{M}\sum_{m\in\mathcal{M}}\boldsymbol{w}_{m,t}+\frac{1}{M\sqrt{\eta_{t}}}\boldsymbol{m}_{t}^{H}\mathbf{z}_{t}$
(22)
$\displaystyle=\frac{1}{M}\sum_{m\in\mathcal{M}}(\boldsymbol{w}_{t}-\gamma\nabla{F}(\boldsymbol{w}_{t},\mathbf{\mathcal{D}}_{m}))+\frac{1}{M\sqrt{\eta_{t}}}\boldsymbol{m}_{t}^{H}\mathbf{z}_{t}$
$\displaystyle=\boldsymbol{w}_{t}-\gamma(\nabla{F}(\boldsymbol{w}_{t})-\mathbf{e}_{t}),$

where,
$\nabla{F}(\boldsymbol{w}_{t})=\frac{1}{M}\sum_{m\in\mathcal{M}}{\nabla{F}_{m}(\boldsymbol{w}_{t},\mathbf{\mathcal{D}}_{m})}$
is the global gradient and
$\mathbf{e}_{t}=\frac{\boldsymbol{m}_{t}^{H}\mathbf{z}_{t}}{\gamma
M\sqrt{\eta_{t}}}$ is the model aggregation error caused by wireless
communication in the $t$-th communication round. By applying (22) in
Assumption III, and setting $\eta=\frac{1}{l}$, after taking expectation, we
have:

$\displaystyle\mathbb{E}[{F}(\boldsymbol{w}_{t+1})]$
$\displaystyle\leq\mathbb{E}[{F}(\boldsymbol{w}_{t})]-\frac{1}{2l}\|\nabla{F}(\boldsymbol{w}_{t})\|_{2}^{2}+\frac{1}{2l}\mathbb{E}[\|\mathbf{e}_{t}\|_{2}^{2}]$
(23)
$\displaystyle\leq\mathbb{E}[{F}(\boldsymbol{w}_{t})]-\frac{1}{2l}\|\nabla{F}(\boldsymbol{w}_{t})\|_{2}^{2}+\frac{l}{2}\frac{d\sigma^{2}\|\boldsymbol{m}_{t}^{H}\|_{2}^{2}}{\eta
M^{2}}.$

By applying the upper bound of $\eta_{t}$ based on (LABEL:eta) and employing
Assumption III and III, then subtracting $F(\boldsymbol{w}^{*})$ from both
sides of (23), we have:

$\mathbb{E}[F(w_{t+1})]-F(w^{*})\leq(1-\frac{\mu}{l})(\mathbb{E}[F(w_{t})]-F(w^{*}))\\\
+\frac{l\sigma^{2}\Gamma}{2M^{2}p_{\max}}\max_{k\in\mathcal{K}}\frac{\|m_{t}^{H}\|_{2}^{2}}{|m_{t}^{H}h_{m}[x]|^{2}}.$
(24)

By recursively operating on (24) and using the definitions of $\Theta_{t}$ and
$\psi$ in (12) and (13), the cumulative optimality gap is calculated as
follows:

$\displaystyle\mathbb{E}[F(\boldsymbol{w}_{T+1})$
$\displaystyle]-F(\boldsymbol{w}^{*})\leq\psi(\mathbb{E}[F(\boldsymbol{w}_{T})]-F(\boldsymbol{w}^{*})+\Theta_{T}$
(25)
$\displaystyle\leq\psi(\psi(\mathbb{E}[F(\boldsymbol{w}_{T-1})]-F(\boldsymbol{w}^{*}))+\Theta_{T-1})+\Theta_{T})$
$\displaystyle\leq\ldots\leq\psi^{T}(\mathbb{E}[F(\boldsymbol{w}_{1})]-F(\boldsymbol{w}^{*}))+\sum_{t=1}^{T}\psi^{T-t}\Theta_{t}.$

Proof of Theorem III is complete.

##  Appendix A References Section

## References

  * [1] J. Yao and et al., “Secure federated learning by power control for internet of drones,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 7, no. 4, pp. 1021–1031, Apr. 2021.
  * [2] Y. Wang and et al., “Learning in the air: Secure federated learning for UAV-assisted crowdsensing,” _IEEE Transactions on Network Science and Engineering_ , vol. 8, no. 2, pp. 1055–1069, Aug. 2020.
  * [3] Z. Wang and et al., “Federated learning via intelligent reflecting surface,” _IEEE Transactions on Wireless Communications_ , vol. 21, no. 2, pp. 808–822, Jul. 2021.
  * [4] C. Chen and et al., “Joint client selection and receive beamforming for over-the-air federated learning with energy harvesting,” _IEEE Open Journal of the Communications Society_ , vol. 4, pp. 1127–1140, May. 2023.
  * [5] G. Hu and et al., “Fluid Antennas-Enabled Multiuser Uplink: A Low-Complexity Gradient Descent for Total Transmit Power Minimization,” _IEEE Communications Letters_ , vol. 28, no. 3, pp. 602–606, Jan. 2024.
  * [6] D. Zhang and et al., “Fluid antenna array enhanced over-the-air computation,” _IEEE Wireless Communications Letters_ , vol. 13, no. 6, pp. 1541–1545, Mar. 2024.
  * [7] H. Qin and et al., “Antenna positioning and beamforming design for fluid antenna-assisted multi-user downlink communications,” _IEEE Wireless Communications Letters_ , vol. 13, pp. 1073–1077, Jan. 2024.
  * [8] Z. Cheng and et al., “Sum-rate maximization for fluid antenna enabled multiuser communications,” _IEEE Communications Letters_ , vol. 28, no. 5, pp. 1206–1210, Mar. 2024.
  * [9] Y. Zuo and et al., “Fluid Antenna for Mobile Edge Computing,” _IEEE Communications Letters_ , May. 2024.
  * [10] N. Waqar and et al., “Opportunistic fluid antenna multiple access via team-inspired reinforcement learning,” _IEEE Transactions on Wireless Communications_ , Apr. 2024.
  * [11] X. Fan and et al., “UAV-Enabled Federated Learning in Dynamic Environments: Efficiency and Security Trade-off,” _IEEE Transactions on Vehicular Technology_ , vol. 73, no. 5, pp. 6993–7006, Dec. 2023.
  * [12] X. Cao and et al., “Transmission power control for over-the-air federated averaging at network edge,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 5, pp. 1571–1586, Jan. 2022.
  * [13] S. Sheikhzadeh and et al., “AI-based secure NOMA and cognitive radio-enabled green communications: Channel state information and battery value uncertainties,” _IEEE Transactions on Green Communications and Networking_ , vol. 6, no. 2, pp. 1037–1054, Dec. 2021.
  * [14] V. Konda and et al., “Actor-critic algorithms,” _Advances in neural information processing systems_ , vol. 12, pp. 1008–1014, Dec. 1999.