# An example of non-Kähler Calabi–Yau fourfold

Nam-Hoon Lee  Department of Mathematics Education, Hongik University 42-1,
Sangsu-Dong, Mapo-Gu, Seoul 121-791, Korea  School of Mathematics, Korea
Institute for Advanced Study, Dongdaemun-gu, Seoul 130-722, South Korea
<EMAIL_ADDRESS>
###### Abstract.

We show that there exists a non-Kähler Calabi–Yau fourfold, constructing an
example by smoothing a normal crossing variety.

###### Key words and phrases:

non-Kähler Calabi–Yau fourfold, semistable smoothing, semistable degeneration

###### 2010 Mathematics Subject Classification:

14J32, 32Q25,14J35, 32J18, 14D06

##  1\. Introduction

In this note, a _Calabi–Yau manifold_ is a simply-connected compact complex
manifold with trivial canonical class and $H^{i}(M,\mathcal{O}_{M})=0$ for
$0<i<\dim M$. $K3$ surfaces are Calabi–Yau twofolds in this definition and
they are all Kähler ([24]), forming a single irreducible smooth family of
dimension $20$ ([14, 15]). Hence, they are all diffeomorphic. The moduli
spaces of projective $K3$ surfaces are a countable union of analytic subspaces
inside of the family of all $K3$ surfaces.

In higher dimensions, the situation is a bit different. Kähler Calabi–Yau
manifolds are necessarily projective when their dimensions are greater than
two and there exist non-Kähler Calabi–Yau threefolds. A number of non-
homeomorphic Kähler Calabi–Yau threefolds have been constructed ([1, 2, 17])
and it is still an open problem whether there are finitely many non-
homeomorphic Kähler Calabi–Yau threefolds or not. On the other hand, there are
found infinitely many non-Kähler, non-homeomorphic Calabi–Yau threefolds ([5,
8, 9, 12]). In an effort to understand the situation in a similar way to that
of the $K3$ surfaces, Reid conjectured that there still may be a single
irreducible moduli space of Calabi–Yau threefolds, such that any Kähler (thus,
projective) Calabi–Yau threefold is the small resolution of a degeneration of
this family and that any two Kähler Calabi–Yau threefolds may be related by
deformations, small resolutions and their inverses through non-Kähler
Calabi–Yau threefolds, although they are non-homeomorphic ([22]). —
figuratively speaking, projective Calabi–Yau threefolds may appear as
scattered islands in the sea of (generically non-projective) Calabi–Yau
threefolds just as projective $K3$ surfaces appear as scattered islands in the
sea of (generically non-projective) $K3$ surfaces. This speculation
demonstrates a role of non-Kähler Calabi–Yau manifolds in understanding Kähler
ones.

In dimension four, the situation is even more obscure. A huge number of non-
homeomorphic Kähler Calabi–Yau fourfolds can be constructed as complete
intersections in toric varieties. However, to the best of the author’s
knowledge, not a single example of non-Kähler Calabi–Yau fourfold has been
found. The purpose of this note is to construct an example of non-Kähler
Calabi–Yau fourfold. Hence, we establish the following:

######  Theorem 1.1.

There exists a non-Kähler Calabi–Yau fourfold.

We note that some examples of simply-connected non-Kähler compact holomorphic
symplectic fourfolds with trivial canonical class and
$H^{2}(M,\mathcal{O}_{M})\neq 0$ were constructed ([4, 11]). We also remark
that interests in non-Kähler Calabi–Yau manifolds in other directions also
have been growing rapidly ([6, 10, 21, 25]).

We shall construct our example by smoothing a normal crossing variety. By
smoothing, we mean the reverse process of the semistable degeneration of a
manifold to a normal crossing variety. If a normal crossing variety is the
central fiber of a semistable degeneration of Calabi–Yau manifolds, it can be
regarded as a member in a deformation family of those Calabi–Yau manifolds. So
building a normal crossing variety smoothable to a Calabi–Yau manifold can be
regarded as building a deformation type of Calabi–Yau manifolds. The
construction by smoothing is intrinsically up to deformation.

The structure of this note is as follows.

We start Section 2 by introducing two background materials – a smoothing
theorem and a Calabi–Yau threefold. The smoothing theorem will be used to
smooth a normal crossing variety to a non-Kähler Calabi–Yau fourfold and the
Calabi–Yau threefold will be used as a building block in constructing the
normal crossing variety in the next sections.

In Section 3, we build a smoothable normal crossing variety. The construction,
starting from the Calabi–Yau threefold introduced in Section 2, involves
several steps of taking quotients and blow-ups of varieties.

In Section 4, we showed that the fourfold, which is a smoothing of the normal
crossing variety constructed in Section 3, is a non-Kähler Calabi–Yau fourfold
and calculate its topological Euler number.

After finishing the manuscript, the author received an e-mail from Taro Sano,
informing that he constructed non-Kähler Calabi–Yau manifolds of dimension
higher than three by smoothing method ([23]).

##  2\. A smoothing theorem and a Calabi–Yau threefold

By a variety, we mean a reduced complex analytic space. Let
$\mathcal{X}=X_{1}\cup X_{2}$ be a variety whose irreducible components are
two smooth varieties $X_{1}$ and $X_{2}$. $\mathcal{X}$ is called a _normal
crossing variety_ if, near any point $p\in X_{1}\cap X_{2}$, $\mathcal{X}$ is
locally isomorphic to

$\\{(x_{0},x_{1},\cdots,x_{n})\in\mathbb{C}^{n+1}|x_{n-1}x_{n}=0\\}$

with $p$ corresponding to the origin and $X_{1}$, $X_{2}$ locally
corresponding to the hypersurfaces $x_{n-1}=0,x_{n}=0$ respectively in
$\mathbb{C}^{n+1}$. Note that the variety $D_{\mathcal{X}}:=X_{1}\cap X_{2}$
is smooth. Suppose that there is a proper map
$\varsigma:\mathfrak{X}\rightarrow\Delta$ from a complex manifold
${\mathfrak{X}}$ onto the unit disk $\Delta=\\{t\in\mathbb{C}|\|t\|\leq 1\\}$
such that the fiber ${\mathfrak{X}}_{t}=\varsigma^{-1}(t)$ is a smooth
manifold for every $t\neq 0$ and ${\mathfrak{X}}_{0}=\mathcal{X}$. We denote a
generic fiber ${\mathfrak{X}}_{t}$ ( $t\neq 0$) by $M_{\mathcal{X}}$ and we
say that $\mathcal{X}$ is a semistable degeneration of a smooth manifold
$M_{\mathcal{X}}$ and that $M_{\mathcal{X}}$ is a _semistable smoothing_
(simply smoothing) of $\mathcal{X}$.

We will use the following result from [12] and [7] as a generalization of
results in [13].

######  Theorem 2.1.

Let $\mathcal{X}=X_{1}\cup X_{2}$ be a normal crossing variety of dimension
$n\geq 3$ whose irreducible components are two smooth compact varieties
$X_{1}$ and $X_{2}$ such that $D_{\mathcal{X}}$ is smooth. Assume that

  1. (1)

$\omega_{\mathcal{X}}\simeq{\mathcal{O}_{\mathcal{X}}}$,

  2. (2)

$H^{n-1}({\mathcal{X}},{\mathcal{O}}_{\mathcal{X}})=0$,
$H^{n-2}(X_{i},{\mathcal{O}}_{X_{i}})=0$ for $i=1,2$,

  3. (3)

$N_{D_{\mathcal{X}}/X_{1}}\otimes N_{D_{\mathcal{X}}/X_{2}}$ on
$D_{\mathcal{X}}$ is trivial.

Then $\mathcal{X}$ is smoothable to a complex manifold $M_{\mathcal{X}}$.

This smoothing theorem was proved firstly in [13] with an assumption of
Kählerness condition and the condition was removed in [12] and [7]. We will
also need the following lemma (Lemma 3.14 in [12]) in the proof of Theorem
4.1.

######  Lemma 2.2.

With conditions in Theorem 2.1, assume further that $X_{1},X_{2}$ are
projective and $M_{\mathcal{X}}$, $D_{\mathcal{X}}$ are a projective
Calabi–Yau $n$-fold and a Calabi–Yau $(n-1)$-fold respectively. Then there
exists a big line bundle on $\mathcal{X}$.

We will build a normal crossing variety and smooth it to a Calabi–Yau fourfold
by applying Theorem 2.1 and show, using Lemma 2.2, that the Calabi–Yau
fourfold is non-Kähler. To make the normal crossing variety
$\mathcal{X}=X_{1}\cup X_{2}$, we need to construct the varieties
$X_{1},X_{2}$ first. $X_{1}$ and $X_{2}$ will be built from Beauville’s
Calabi–Yau threefold and two involutions on it.

We briefly recall Beauville’s Calabi–Yau threefold. Let
$\zeta=e^{2\pi\frac{\sqrt{-1}}{3}}$. By $E_{\zeta}$, we denote the elliptic
curve whose period is $\zeta$ and by $E_{\zeta}^{3}/\langle\zeta\rangle$ the
quotient of the product manifold $E_{\zeta}^{3}$ by the scalar multiplication
by $\zeta$. Let

$Q_{0}=0,\,\,Q_{1}=\frac{1-\zeta}{3},\,\,Q_{2}=\frac{-1+\zeta}{3}$

in $E_{\zeta}$. These are exactly the fixed points of the scalar
multiplication by $\zeta$ on $E_{\zeta}$. For $i,j,k=0,1,2$, let

$Q_{i,j,k}=(Q_{i},Q_{j},Q_{k})\in E_{\zeta}^{3}$

and let $\overline{Q}_{i,j,k}$ be its image in
$E_{\zeta}^{3}/\langle\zeta\rangle$. Then
$\overline{Y}=E_{\zeta}^{3}/\langle\zeta\rangle$ has singularities of type
$\frac{1}{3}(1,1,1)$ at $\overline{Q}_{i,j,k}$’s and the blow-up
$Y\rightarrow\overline{Y}$ at these 27 singular points gives a Calabi–Yau
threefold $Y$. This is a Kähler Calabi–Yau threefold with Hodge numbers
$h^{1,2}(Y)=0$ and $h^{1,1}(Y)=36$, which was originally found by Beauville
([3]).

##  3\. Construction of a normal crossing variety $\mathcal{X}=X_{1}\cup
X_{2}$

Consider two $3\times 3$ matrices in ${\rm
GL}_{3}\left(\mathbb{Z}[\zeta]\right)$

$A_{1}=\left(\begin{array}[]{rrr}-1&0&0\\\ 0&1&0\\\ 0&0&1\\\
\end{array}\right),\,\,\,A_{2}=\left(\begin{array}[]{rrr}1&0&0\\\ -1&0&1\\\
1&1&0\\\ \end{array}\right).$

Note that $A_{i}^{2}$ is the $3\times 3$ identity matrix and $\det(A_{i})=-1$
for $i=1,2$. The matrix $A_{i}$ induces an involution $\sigma_{i}$ on
$E_{\zeta}^{3}$. Note that $\sigma_{i}^{*}$ acts as multiplication by $-1$ on
$H^{3,0}\left(E_{\zeta}^{3}\right)$. Noting that the subgroup of ${\rm
GL}_{3}\left(\mathbb{Z}[\zeta]\right)$ that is generated by $A_{1}$, $A_{2}$
is infinite, we make the following remark which will play a key role in
showing the non-Kählerness in the proof of Theorem 4.1.

######  Remark 3.1.

The group of automorphisms of $E_{\zeta}^{3}$ that is generated by
$\sigma_{1}$, $\sigma_{2}$ is infinite.

There is a unique involution $\rho_{i}$ on $Y$ such that the following diagram
commutes:

$\begin{CD}E_{\zeta}^{3}@>{\sigma_{i}}>{}>E_{\zeta}^{3}\\\ @V{}V{}V@V{}V{}V\\\
\overline{Y}\overline{Y}\\\ @A{}A{}A@A{}A{}A\\\ Y@>{\rho_{i}}>{}>Y\end{CD}$

We note that $\rho_{i}^{*}$ also acts as multiplication by $-1$ on
$H^{3,0}(Y)$ and the fixed locus $S_{i}$ of $\rho_{i}$ is a smooth surface (a
disjoint union of irreducible surfaces). Now we move on to the construction of
smooth varieties $X_{1},X_{2}$. We borrow a method of construction from [16]
(§4 in [16]) that makes use of non-symplectic involutions on $K3$ surfaces.

Let $\psi:\mathbb{P}^{1}\rightarrow\mathbb{P}^{1}$ be any involution fixing
two distinct points and consider a quotient
$\overline{X}_{i}=(Y\times\mathbb{P}^{1})/(\rho_{i}\times\psi$). Then the
singular locus of $\overline{X}_{i}$ is a product of smooth surfaces and
ordinary double points, resulting from the fixed locus of $\rho_{i}$. Let
$X_{i}\rightarrow\overline{X}_{i}$ be the blow-up along the singular locus of
$\overline{X}_{i}$. It is elementary to check that ${X_{i}}$ is smooth. Choose
a point $p\in\mathbb{P}^{1}$ such that $p\neq\psi(p)$. Let $D^{\prime}_{i}$ be
the image of $Y\times\\{p\\}$ in $\overline{X}_{i}$ and $D_{i}$ be the inverse
image of $D^{\prime}_{i}$ in ${X_{i}}$. Then $D_{i}$ is isomorphic to $Y$ and
it is an anticanonical divisor of ${X_{i}}$ whose normal bundle
$N_{D_{i}/{X_{i}}}$ in $X_{i}$ is trivial. Since $Y$ is projective, all the
varieties $Y\times\mathbb{P}^{1}$, $\overline{X}_{i}$ and $X_{i}$ are
projective.

We summarize our notations, including ones to be defined:

  * •

$\zeta=e^{2\pi\frac{\sqrt{-1}}{3}}$.

  * •

$E_{\zeta}=\mathbb{C}/(\mathbb{Z}\oplus\mathbb{Z}\zeta)$ is the elliptic curve
with period $\zeta$.

  * •

$\overline{Y}=E_{\zeta}^{3}/\langle\zeta\rangle$.

  * •

$\phi:\widetilde{Y}\rightarrow E_{\zeta}^{3}$ is the blow-up at the 27 points
of $Q_{i,j,k}$’s.

  * •

$\eta:\widetilde{Y}\rightarrow Y$ is the map induced by
$E_{\zeta}^{3}\rightarrow\overline{Y}$.

  * •

$Y\rightarrow\overline{Y}$ is the blow-up of $\overline{Y}$ at its singular
points. $Y$ is a Kähler Calabi–Yau threefold.

  * •

$\sigma_{i}:E_{\zeta}^{3}\rightarrow E_{\zeta}^{3}$ is the involution induced
by the matrix $A_{i}$ for $i=1,2$.

  * •

$\rho_{i}:Y\rightarrow Y$ is the involution induced by $\sigma_{i}$ for
$i=1,2$.

  * •

$S_{i}=Y^{\rho_{i}}$ is the fixed locus of $\rho_{i}$ for $i=1,2$.

  * •

$\check{S}_{i}=\phi(\eta^{-1}(S_{i}))$, the fixed locus of $\sigma_{i}$ for
$i=1,2$.

  * •

$\psi:\mathbb{P}^{1}\rightarrow\mathbb{P}^{1}$ is an involution fixing two
distinct points $q_{1},q_{2}\in\mathbb{P}^{1}$.

  * •

$\overline{X}_{i}=(Y\times\mathbb{P}^{1})/(\rho_{i}\times\psi)$ for $i=1,2$.

  * •

$X_{i}\rightarrow\overline{X}_{i}$ is the blow-up along the singular locus of
$\overline{X}_{i}$ for $i=1,2$.

  * •

$\widetilde{X_{i}}\rightarrow Y\times\mathbb{P}^{1}$ is the blow-up of
$Y\times\mathbb{P}^{1}$ along the surface $S_{i}\times\\{q_{1},q_{2}\\}$.

  * •

$D^{\prime}_{i}$: the image of $Y\times\\{p\\}$ in $\overline{X}_{i}$ for a
point $p\in\mathbb{P}^{1}$ with $p\neq\psi(p)$ for $i=1,2$.

  * •

$D_{i}$ : the inverse image of $D^{\prime}_{i}$ in $X_{i}$ for $i=1,2$.
$D_{i}\simeq Y$ and $D_{i}\in\left|-K_{X_{i}}\right|$.

  * •

$X_{i}^{*}=X_{i}-D_{i}$.

  * •

$\mathcal{X}=X_{1}\cup X_{2}$ is the normal crossing variety of $X_{1},X_{2}$,
made by gluing along their isomorphic smooth anticanonical sections
$D_{1},D_{2}$.

  * •

$D_{\mathcal{X}}=X_{1}\cap X_{2}$.

  * •

$M_{\mathcal{X}}$ is a smoothing of a normal crossing variety
$\mathcal{X}=X_{1}\cup X_{2}$.

We make a normal crossing variety $\mathcal{X}=X_{1}\cup X_{2}$ by gluing
transversally along $D_{1}$ and $D_{2}$, (see §2 (especially Corollary 2.4) of
[12] for details of the gluing process). Then $D_{\mathcal{X}}:=X_{1}\cap
X_{2}$ is a copy of $Y$. Since $D_{\mathcal{X}}=X_{1}\cap X_{2}$ is an
anticanonical divisor of both $X_{1}$ and $X_{2}$,
$\omega_{\mathcal{X}}\simeq\mathcal{O}_{\mathcal{X}}$. Note that
$N_{D_{\mathcal{X}}/X_{1}}\otimes N_{D_{\mathcal{X}}/X_{2}}$ is trivial. Let
$X_{i}^{*}=X_{i}-D_{i}$. For varieties $X_{1}$, $X_{2}$ and
$\mathcal{X}=X_{1}\cup X_{2}$ constructed in this section, we gather some of
their properties:

######  Proposition 3.2.

  1. (1)

$X_{1}$ and $X_{2}$ are projective.

  2. (2)

$\omega_{\mathcal{X}}\simeq{\mathcal{O}_{\mathcal{X}}}$.

  3. (3)

$N_{D_{\mathcal{X}}/X_{1}}\otimes N_{D_{\mathcal{X}}/X_{2}}$ on
$D_{\mathcal{X}}$ is trivial.

  4. (4)

Both $X_{i}$ and $X_{i}^{*}$ are simply-connected for $i=1,2$.

  5. (5)

$H^{k}(X_{i},{\mathcal{O}}_{X_{i}})=0$ for $i=1,2$, $k=1,2,3,4$.

  6. (6)

$H^{k}({\mathcal{X}},{\mathcal{O}}_{\mathcal{X}})=0$ for $k=1,2,3$.

###### Proof.

The properties (1), (2), (3) are already shown.

We show the property (4). One can obtain ${X_{i}}$ differently. Let
${q_{1},q_{2}}$ be the fixed points of $\psi$. Let $\widetilde{X_{i}}$ be the
blow-up of $Y\times\mathbb{P}^{1}$ along the surface
$S_{i}\times\\{q_{1},q_{2}\\}$. Then the involution on $Y\times\mathbb{P}^{1}$
induces an involution on $\widetilde{X_{i}}$, whose fixed locus is the
exceptional divisor over $S_{i}\times\\{q_{1},q_{2}\\}$. The quotient of
$\widetilde{X_{i}}$ by the involution is isomorphic to ${X_{i}}$. This may be
summarized by the diagram,

$\begin{CD}\widetilde{{X_{i}}}@>{}>{}>{{X_{i}}}\\\ @V{}V{}V@V{}V{}V\\\
Y\times\mathbb{P}^{1}@>{}>{}>\overline{X}_{i}\end{CD}$

Let $\widehat{X_{i}}\rightarrow{X_{i}}$ be the universal covering. The
fourfold $\widetilde{X_{i}}$ is simply-connected, therefore the quotient map
$\widetilde{X_{i}}\to{X_{i}}$ lifts to a map
$\widetilde{X_{i}}\rightarrow\widehat{X_{i}}$ so there is a commutative
diagram:

$\xymatrix{&\widehat{X_{i}}\ar[d]\\\ \widetilde{X_{i}}\ar[r]\ar[ur]{X_{i}}.}$

Since the map $\widetilde{X_{i}}\rightarrow{X_{i}}$ is a double covering, the
degree of the map $\widetilde{X_{i}}\rightarrow\widehat{X_{i}}$ is one or two.
Suppose that the degree of this map is one. Then it is an isomorphism and so
the universal covering $\widehat{X_{i}}\rightarrow{X_{i}}$ is of degree two,
having the same branch locus with that of the map
$\widetilde{X_{i}}\rightarrow{X_{i}}$. Since the involution
$\rho_{i}\times\psi$ has fixed points, the induced involution on
$\widetilde{X_{i}}$ also has fixed points and hence the branch locus of the
map $\widetilde{X_{i}}\rightarrow{X_{i}}$ is not empty. This means that the
universal covering $\widehat{X_{i}}\rightarrow{X_{i}}$ has non-empty branch
locus, which is impossible. Hence, we conclude that the degree of the map
$\widetilde{X_{i}}\rightarrow\widehat{X_{i}}$ is two and, accordingly, the
universal covering $\widehat{X_{i}}\rightarrow{X_{i}}$ is of degree one, which
implies that it is an isomorphism. Therefore, $\widehat{X_{i}}$ is necessarily
isomorphic to ${X_{i}}$ and so ${X_{i}}$ is simply-connected.

There is a natural projection:
$\overline{X}_{i}=(Y\times\mathbb{P}^{1})/({\rho_{i}}\times\psi)\rightarrow
Y/{\rho_{i}}$. Let $\nu$ be the composition of
${X_{i}}\rightarrow\overline{X}_{i}$ and
$(Y\times\mathbb{P}^{1})/({\rho_{i}}\times\psi)\rightarrow Y/{\rho_{i}}$. Let
$x\in Y/{\rho_{i}}$ be a point in the branch locus of the map $Y\rightarrow
Y/{\rho_{i}}$. Then $\nu^{-1}(x)$ is a union of three smooth rational curves,
one of which (denoted by $l$) crosses ${D_{i}}$ transversely at a single point
and the other two are disjoint from ${D_{i}}$, resulting from the blow-up.
Since ${X_{i}}$ and ${D_{i}}$ are simply connected, the fundamental group
$\pi_{1}(X_{i}^{*})$ of $X_{i}^{*}$ is generated by a loop around ${D_{i}}$.
We can assume that the loop is contained in $l^{*}=l-{D_{i}}$. Since the loop
can be contracted to a point in $l^{*}$, $X_{i}^{*}$ is simply-connected.

We move on to the property (5). Since

$\dim H^{k}(X_{i},\mathcal{O}_{X_{i}})\leq\dim
H^{k}(\widetilde{X_{i}},\mathcal{O}_{\widetilde{X_{i}}})=\dim
H^{k}(Y\times\mathbb{P}^{1},\mathcal{O}_{Y\times\mathbb{P}^{1}})=0$

for $k=1,2$, we have

$H^{k}(X_{i},\mathcal{O}_{X_{i}})=0$

for $k=1,2$.

Note that $X_{i}$ has an effective anticanonical divisor $D_{i}$, which is a
Calabi–Yau threefold. Hence, we have

$H^{4}(X_{i},\mathcal{O}_{X_{i}})\simeq H^{0}(X_{i},\Omega_{X_{i}}^{4})=0.$

Taking the cohomology of the structure sheaf sequence,

$0\rightarrow\mathcal{O}_{X_{i}}(K_{X_{i}})\rightarrow\mathcal{O}_{X_{i}}\rightarrow\mathcal{O}_{D_{i}}\rightarrow
0,$

we obtain an exact sequence

$\displaystyle H^{3}({X_{i}},\mathcal{O}_{X_{i}}(K_{{X_{i}}}))\rightarrow
H^{3}({X_{i}},\mathcal{O}_{X_{i}})\rightarrow
H^{3}({D_{i}},\mathcal{O}_{D_{i}})\rightarrow$ $\displaystyle
H^{4}({X_{i}},\mathcal{O}_{X_{i}}(K_{{X_{i}}}))\rightarrow
H^{4}({X_{i}},\mathcal{O}_{X_{i}})=0.$

Since, by Serre duality,

$H^{3}({X_{i}},\mathcal{O}_{X_{i}}(K_{{X_{i}}}))\simeq
H^{1}({X_{i}},\mathcal{O}_{X_{i}})=0,$ $\dim
H^{4}({X_{i}},\mathcal{O}_{X_{i}}(K_{{X_{i}}}))=\dim
H^{0}({X_{i}},\mathcal{O}_{X_{i}})=1$

and

$\dim H^{3}({D_{i}},\mathcal{O}_{D_{i}})=1,$

we have $\dim H^{3}({X_{i}},\mathcal{O}_{X_{i}})=0$.

Finally, we show the property (6). From the exact sequence of sheaves

$0\rightarrow\mathcal{O}_{\mathcal{X}}\rightarrow\mathcal{O}_{X_{1}}\oplus\mathcal{O}_{X_{2}}\rightarrow\mathcal{O}_{D_{\mathcal{X}}}\rightarrow
0,$

we obtain an exact sequence

$H^{k-1}({D_{\mathcal{X}}},\mathcal{O}_{D_{\mathcal{X}}})\rightarrow
H^{k}({\mathcal{X}},\mathcal{O}_{\mathcal{X}})\rightarrow
H^{k}({X_{1}},\mathcal{O}_{X_{1}})\oplus H^{k}({X_{2}},\mathcal{O}_{X_{2}}).$

Since

$H^{k-1}({D_{\mathcal{X}}},\mathcal{O}_{D_{\mathcal{X}}})=H^{k}({X_{1}},\mathcal{O}_{X_{1}})=H^{k}({X_{2}},\mathcal{O}_{X_{2}})=0$

for $k=2,3$, we have

$H^{2}({\mathcal{X}},\mathcal{O}_{\mathcal{X}})=H^{3}({\mathcal{X}},\mathcal{O}_{\mathcal{X}})=0.$

Moreover, the exact sequence

$\displaystyle 0\rightarrow
H^{0}({\mathcal{X}},\mathcal{O}_{\mathcal{X}})\rightarrow
H^{0}({X_{1}},\mathcal{O}_{X_{1}})\oplus
H^{0}({X_{2}},\mathcal{O}_{X_{2}})\rightarrow
H^{0}({D_{\mathcal{X}}},\mathcal{O}_{D_{\mathcal{X}}})$
$\displaystyle\rightarrow
H^{1}({\mathcal{X}},\mathcal{O}_{\mathcal{X}})\rightarrow
H^{1}({X_{1}},\mathcal{O}_{X_{1}})\oplus H^{1}({X_{2}},\mathcal{O}_{X_{2}})=0$

gives $H^{1}({\mathcal{X}},\mathcal{O}_{\mathcal{X}})=0$.

∎

##  4\. The example

By Theorem 2.1 with the properties in Proposition 3.2, one can show that the
normal crossing variety $\mathcal{X}$, constructed in Section 3, is smoothable
to a smooth fourfold $M_{\mathcal{X}}$ with trivial canonical class. We can
also check that $H^{i}(M_{\mathcal{X}},\mathcal{O}_{\mathcal{X}})=0$ for
$i=1,2,3$ by the upper semicontinuity theorem with the property (6) in
Proposition 3.2.

######  Theorem 4.1.

$M_{\mathcal{X}}$ is a non-Kähler Calabi–Yau fourfold.

###### Proof.

We only need to show that $M_{\mathcal{X}}$ is simply-connected and non-
Kähler.

First, we show the simply-connectedness. We can obtain the topological type of
$M_{\mathcal{X}}$ by pasting $X_{1}^{*}$ and $X_{2}^{*}$. One can regard the
normal bundle $N_{{D_{\mathcal{X}}}/X_{i}}$ as a complex manifold containing
$D_{\mathcal{X}}$. Then

$N_{{D_{\mathcal{X}}}/X_{i}}^{*}:=N_{{D_{\mathcal{X}}}/X_{i}}-D_{\mathcal{X}}$

is a $\mathbb{C}^{*}$-bundle over ${D_{\mathcal{X}}}$, where
$\mathbb{C}^{*}:=\mathbb{C}-\\{0\\}$. The triviality property on
$N_{D_{\mathcal{X}}/X_{1}}\otimes N_{D_{\mathcal{X}}/X_{2}}$ implies the map

$\varphi:N_{{D_{\mathcal{X}}}/X_{1}}^{*}\rightarrow
N_{{D_{\mathcal{X}}}/X_{2}}^{*},$

locally defined by

$(x\in\mathbb{C}^{*},y\in{D_{\mathcal{X}}})\mapsto\left({1}/{x},y\right),$

is globally well-defined and an isomorphism. Note that ${D_{\mathcal{X}}}$ in
$X_{i}$ has a neighborhood $U_{i}$ that is homeomorphic to
$N_{{D_{\mathcal{X}}}/X_{i}}$. Let $U_{i}^{*}=U_{i}-{D_{\mathcal{X}}}$. Then
the map $\varphi$ induces a homeomorphism between $U_{1}^{*}$ and $U_{2}^{*}$.
One can construct a manifold $M^{\prime}$ by pasting together $X_{1}^{*}$ and
$X_{2}^{*}$ along $U_{1}^{*}$ and $U_{2}^{*}$ with the homeomorphism. The
manifold $M^{\prime}$ is homeomorphic to $M_{\mathcal{X}}$. Note that
$X_{1}^{*}$, $X_{2}^{*}$ are simply-connected (the property (4) in Proposition
3.2). Hence, by Seifert–van Kampen theorem, $M^{\prime}$ is simply-connected.

For the non-Kählerness, suppose that $M_{\mathcal{X}}$ is Kähler, then it is
necessarily projective. Note that $D_{\mathcal{X}}$, $X_{1}$ and $X_{2}$ are
all projective (the property (1) in Proposition 3.2). Hence, by Lemma 2.2,
there exists a big line bundle $\mathcal{L}$ on $\mathcal{X}$. Let $h_{i}$ be
the big divisor class in ${\rm{Pic}}(X_{i})$ corresponding to
$\mathcal{L}|_{X_{i}}$, then $h_{1}|_{D_{\mathcal{X}}}$ is linearly equivalent
to $h_{2}|_{D_{\mathcal{X}}}$. Note that $D_{\mathcal{X}}$ is a copy of $Y$.
Let us denote the divisor class in ${\rm{Pic}}(Y)$ of
$h_{1}|_{D_{\mathcal{X}}}$, $h_{2}|_{D_{\mathcal{X}}}$ by $\hat{h}$. Chasing
the construction of $X_{1}$ and $X_{2}$, one can check that $\hat{h}$ belongs
to ${\rm{Pic}}(Y)^{\rho_{1}^{*}}\cap{\rm{Pic}}(Y)^{\rho_{2}^{*}}$, where
${\rm{Pic}}(Y)^{\rho_{i}^{*}}$ is the subgroup of ${\rm{Pic}}(Y)$ that
consists of the classes invariant under ${\rho_{i}^{*}}$.

The linear system $\left|D_{\mathcal{X}}\right|$ is base-point free and it
gives a fibration $X_{i}\rightarrow\mathbb{P}^{1}$ and $D_{\mathcal{X}}$ is
one of its generic fibers. Hence $\hat{h}$ is a big divisor of $Y$ (see, for
example, Corollary 2.2.11 in [18]). Let $\phi:\widetilde{Y}\rightarrow
E_{\zeta}^{3}$ be the blow-up at the 27 points of $Q_{i,j,k}$’s and
$\eta:\widetilde{Y}\rightarrow Y$ be the map induced by
$E_{\zeta}^{3}\rightarrow\overline{Y}$ such that the diagram commutes:

$\begin{CD}{\widetilde{Y}}@>{\eta}>{}>{Y}\\\ @V{\phi}V{}V@V{}V{}V\\\
E_{\zeta}^{3}@>{}>{}>\overline{Y}\end{CD}$

It is not hard to check that $\check{h}=\phi_{*}(\eta^{*}(\hat{h}))$ is a big
divisor of $E_{\zeta}^{3}$ and the class $\check{h}$ belongs to ${\rm
NS}(E_{\zeta}^{3})^{\sigma_{1}^{*}}\cap{\rm
NS}(E_{\zeta}^{3})^{\sigma_{2}^{*}}$. Note that any big divisor of the abelian
variety $E_{\zeta}^{3}$ is ample. However, the group of automorphisms of
$E_{\zeta}^{3}$ that is generated by $\sigma_{1},\sigma_{2}$ is infinite
(Remark 3.1) and so ${\rm NS}(E_{\zeta}^{3})^{\sigma_{1}^{*}}\cap{\rm
NS}(E_{\zeta}^{3})^{\sigma_{2}^{*}}$ does not contain an ample class.
Therefore, we have a contradiction and $M_{\mathcal{X}}$ should be non-Kähler.

∎

Topological invariants of $M_{\mathcal{X}}$ can be calculated from the
topological manifold $M^{\prime}$ in the proof of Theorem 4.1. For example,
the topological Euler number $\chi(M_{\mathcal{X}})$ of $M_{\mathcal{X}}$ is

$\chi(M_{\mathcal{X}})=\chi(M^{\prime})=\chi(X_{1}^{*})+\chi(X_{2}^{*})-\chi(U_{1}).$

Note

$\chi(X_{i})=\chi(X_{i}^{*})+\chi(D_{i})=\chi(X_{i}^{*})+\chi(Y)$

and $\chi(U_{1})=\chi(S^{1})\chi(D_{1})=0$. Hence,

$\chi(M_{\mathcal{X}})=\chi(X_{1})+\chi(X_{2})-2\chi(Y).$

The topological Euler characteristic of ${\widetilde{X}_{i}}$ is

$\chi({\widetilde{X}_{i}})=2\chi(Y)+2\chi(S_{i})$

On the other hand,

$2\chi({X_{i}})-4\chi(S_{i})=\chi({\widetilde{X}_{i}})$

and so

$\displaystyle\chi({X_{i}})$
$\displaystyle=\frac{1}{2}\left(4\chi(S_{i})+\chi({\widetilde{X}_{i}})\right)$
$\displaystyle=2\chi(S_{i})+\chi(Y)+\chi(S_{i})$
$\displaystyle=\chi(Y)+3\chi(S_{i}).$

Let $\check{S}_{i}=\phi(\eta^{-1}(S_{i}))$. Note that $\check{S}_{i}$ is the
fixed locus of $\sigma_{i}$. Let $\Theta=\\{Q_{i,j,k}\mid i,j,k=0,1,2\\}$. One
can easily check

$\chi(S_{i})=2\left|\check{S}_{i}\cap\Theta\right|=18,$

where $\left|\check{S}_{i}\cap\Theta\right|$ is the number of points in
$\check{S}_{i}\cap\Theta$.

Therefore, the topological Euler number $\chi(M_{\mathcal{X}})$ of
$M_{\mathcal{X}}$ is

$\displaystyle\chi(M_{\mathcal{X}})$
$\displaystyle=\chi(X_{1})+\chi(X_{2})-2\chi(Y)$
$\displaystyle=\chi(Y)+3\chi(S_{1})+\chi(Y)+3\chi(S_{2})-2\chi(Y)$
$\displaystyle=3\chi(S_{1})+3\chi(S_{2})$ $\displaystyle=108.$

The pair of matrices $A_{1}$, $A_{2}$ in Section 2, which can be used in the
construction, is obviously not unique. Any pair of $3\times 3$ matrices
$A_{1}$, $A_{2}$ that satisfies the following conditions,

  1. (1)

$A_{i}\in{\rm GL}_{3}(\mathbb{Z}[\zeta])$,

  2. (2)

$A_{i}^{2}$ is the $3\times 3$ identity matrix,

  3. (3)

$\det(A_{i})=-1$,

  4. (4)

$\rho_{i}$ is not fixed-free and its fixed locus is a smooth surface and

  5. (5)

The subgroup of ${\rm GL}_{3}(\mathbb{Z}[\zeta])$ containing both
$A_{1},A_{2}$ is infinite,

gives rise to a non-Kähler Calabi–Yau fourfolds through the construction of
Sections 3 and 4, where the conditions guarantee, respectively,

  1. (1)

$A_{i}$ induces an automorphism of $E_{\zeta}^{3}$ which induces an
automorphism of $Y$.

  2. (2)

$A_{i}$ induces an involution of $E_{\zeta}^{3}$ which induces an involution
of $Y$,

  3. (3)

$\rho_{i}^{*}$ acts as multiplication by $-1$ on $H^{3,0}(Y)$ so that $X_{i}$
has an anticanonical section isomorphic to $Y$,

  4. (4)

both $X_{i}$ and $X_{i}^{*}$ are simply-connected, which leads to the simply-
connectedness of $M_{\mathcal{X}}$ and

  5. (5)

the group of automorphisms of $E_{\zeta}^{3}$ that is generated by
$\sigma_{1}$, $\sigma_{2}$ is infinite, which eventually leads to the non-
Kählerness of $M_{\mathcal{X}}$.

The author could not obtain other non-Kähler Calabi–Yau fourfolds of different
topological Euler numbers although he tried many pairs of such matrices. The
author suspects that all the pairs of such matrices may give rise to non-
Kähler Calabi–Yau fourfolds of the same topological Euler number ($=108$).

## References

  * [1] Altman, Ross; Gray, James; He, Yang-Hui; Jejjala, Vishnu; Nelson, Brent D. A Calabi–Yau database: threefolds constructed from the Kreuzer-Skarke list. J. High Energy Phys. 2015, no. 2, 158.
  * [2] Batyrev, Victor V. Dual polyhedra and mirror symmetry for Calabi–Yau hypersurfaces in toric varieties. J. Algebraic Geom. 3 (1994), no. 3, 493–535.
  * [3] Beauville, Arnaud Some remarks on Kähler manifolds with $c_{1}=0$.  Classification of algebraic and analytic manifolds (Katata, 1982), 1–26, Progr. Math., 39, Birkhäuser Boston, Boston, MA, 1983.
  * [4] Bogomolov, Fedor A. On Guan’s examples of simply connected non-Kähler compact complex manifolds. Amer. J. Math. 118 (1996), no. 5, 1037–1046.
  * [5] Clemens, C. Herbert Double solids. Adv. in Math. 47 (1983), no. 2, 107–230.
  * [6] Fei, Teng A construction of non-Kähler Calabi–Yau manifolds and new solutions to the Strominger system. Adv. Math. 302 (2016), 529–550.
  * [7] Felten, Simon; Filip, Matej; Ruddat, Helge Smoothing toroidal crossing spaces. arXiv:1908.11235
  * [8] Friedman, Robert Simultaneous resolution of threefold double points. Math. Ann. 274 (1986), no4̇, 671–689.
  * [9] Friedman, Robert On threefolds with trivial canonical bundle. Complex geometry and Lie theory, 103–134, Proc. Sympos. Pure Math., 53, Amer. Math. Soc., Providence, RI, 1991.
  * [10] Fu, Jixiang; Li, Jun; Yau, Shing-Tung Balanced metrics on non-Kähler Calabi–Yau threefolds. J. Differential Geom. 90 (2012), no. 1, 81–129.
  * [11] Guan, Daniel Examples of compact holomorphic symplectic manifolds which are not Kählerian. II. Invent. Math. 121 (1995), no. 1, 135–145.
  * [12] Hashimoto, Kenji; Sano, Taro Examples of non-Kähler Calabi–Yau 3-folds with arbitrarily large $b_{2}$, arXiv:1902.01027
  * [13] Kawamata, Yujiro; Namikawa, Yoshinori Logarithmic deformations of normal crossing varieties and smoothing of degenerate Calabi–Yau varieties. Invent. Math. 118 (1994), no. 3, 395–409.
  * [14] Kodaira, K. On the structure of compact complex analytic surfaces. I. Amer. J. Math. 86 (1964), 751–798.
  * [15] Kodaira, K.; Nirenberg, L.; Spencer, D. C. On the existence of deformations of complex analytic structures. Ann. of Math. 68 (1958), 450–459.
  * [16] Kovalev, Alexei; Lee, Nam-Hoon $K3$ surfaces with non-symplectic involution and compact irreducible G2-manifolds. Math. Proc. Cambridge Philos. Soc. 151 (2011), no. 2, 193–218.
  * [17] Kreuzer, Maximilian; Skarke, Harald  Calabi–Yau data http://hep.itp.tuwien.ac.at/$\sim$kreuzer/CY/
  * [18] Lazarsfeld, Robert Positivity in algebraic geometry. I. Classical setting: line bundles and linear series. A Series of Modern Surveys in Mathematics, 48. Springer-Verlag, Berlin, 2004.
  * [19] Lee, Nam-Hoon Calabi–Yau construction by smoothing normal crossing varieties. Internat. J. Math. 21 (2010), no. 6, 701–725.
  * [20] Lee, Nam-Hoon Mirror pairs of Calabi–Yau threefolds from mirror pairs of quasi-Fano threefolds. J. Math. Pures Appl. (9) 141 (2020), 195–219.
  * [21] Qin, Lizhen; Wang, Botong A family of compact complex and symplectic Calabi–Yau manifolds that are non-Kähler. Geom. Topol. 22 (2018), no. 4, 2115–2144.
  * [22] Reid, Miles The moduli space of 3-folds with $K=0$ may nevertheless be irreducible. Math. Ann. 278 (1987), no. 1–4, 329–334.
  * [23] Sano, Taro Examples of non-Kähler Calabi-Yau manifolds with arbitrarily large $b_{2}$, arXiv:2102.13437
  * [24] Siu, Y. T. Every $K3$ surface is Kähler. Invent. Math. 73 (1983), no. 1, 139–150.
  * [25] Tosatti, Valentino Non-Kähler Calabi–Yau manifolds. Analysis, complex geometry, and mathematical physics: in honor of Duong H. Phong, 261–277, Contemp. Math., 644, Amer. Math. Soc., Providence, RI, 2015.


# Deep Reinforcement Learning for Multi-User RF Charging with Non-linear
Energy Harvesters
††thanks: This work is partially supported by the Research Council of Finland
(Grants 348515 and 346208 (6G Flagship)); by the European Commission through
the Horizon Europe/JU SNS project Hexa-X-II (Grant no. 101095759); by the
Finnish-American Research & Innovation Accelerator; and by the Villum
Investigator Grant ”WATER” from the Velux Foundation, Denmark.

Amirhossein Azarbahram∗, Onel L. A. López∗, Petar Popovski†, Shashi Raj
Pandey†, and Matti Latva-aho∗
∗Centre for Wireless Communications (CWC), University of Oulu, Finland
†Department of Electronic Systems, Aalborg University, Denmark
Emails: {amirhossein.azarbahram<EMAIL_ADDRESS>{petarp,
<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
###### Abstract

Radio frequency (RF) wireless power transfer (WPT) is a promising technology
for sustainable support of massive Internet of Things (IoT). However, RF-WPT
systems are characterized by low efficiency due to channel attenuation, which
can be mitigated by precoders that adjust the transmission directivity. This
work considers a multi-antenna RF-WPT system with multiple non-linear energy
harvesting (EH) nodes with energy demands changing over discrete time slots.
This leads to the charging scheduling problem, which involves choosing the
precoders at each slot to minimize the total energy consumption and meet the
EH requirements. We model the problem as a Markov decision process and propose
a solution relying on a low-complexity beamforming and deep deterministic
policy gradient (DDPG). The results show that the proposed beamforming
achieves near-optimal performance with low computational complexity, and the
DDPG-based approach converges with the number of episodes and reduces the
system’s power consumption, while the outage probability and the power
consumption increase with the number of devices.

###### Index Terms:

Radio frequency wireless charging, energy beamforming, near-field channels,
charging scheduling.

##  I Introduction

The massive growth of the Internet of Things (IoT) networks threatens to
skyrocket the maintenance cost of these systems, especially those related to
battery replacement and waste. This calls for practical alternatives such as
relying on radio frequency (RF) wireless power transfer (WPT) technology. RF-
WPT can provide wireless charging capabilities to prevent IoT devices from
battery depletion and increase their lifespan [1, 2]. Also, it can potentially
charge multiple devices over large distances and use the same infrastructure
as wireless communications by utilizing the broadcast nature of the wireless
channel. However, the inherently low efficiency of RF-WPT systems is still an
open challenge, demanding significant attention [3, 4]. One of the main
inefficiency sources is channel attenuation, which can be mitigated by energy
beamforming (EB). Furthermore, there is a need to properly allocate charging
resources in a multi-node IoT system to avoid interruptions during the system
operation time.

EB techniques can compensate for the channel losses by focusing the energy
beams on the devices [3]. For example, the authors in [5] propose EB designs
to power multiple devices in a radio stripe system, while the deployment
problem of the radio stripes using maximum ratio transmission beamformers is
investigated in [6]. In [7, 8], the authors propose EB approaches for near-
field RF-WPT with dynamic metasurface antennas as the transmitter. Moreover, a
low-complexity EB design is proposed in [9] to power single-antenna devices.

There are some works in the literature on improving the performance of energy
harvesting (EH)-assisted wireless systems over a time horizon. For example,
the authors in [10] consider a wireless-powered sensor network and utilize
deep reinforcement learning (DRL) to select a node to charge and its
corresponding allocated power at each time slot to minimize the packet loss
caused by insufficient energy. In [11], deep deterministic policy gradient
(DDPG) is used to tune the transmit power to minimize long-term energy
consumption. The authors in [12] utilize DDPG to improve the energy efficiency
in a heterogeneous network by tuning transmit/harvest strategies for macro or
small cells at each time slot. Interestingly, DRL is used in [13] to split the
channel resources between EH and information transmission using DRL aiming to
minimize the outage probability of the information transmission phase.
Moreover, in [14], the authors consider a WPCN with reflective intelligent
surfaces and unmanned aerial vehicles and jointly optimize the transmit power,
trajectory, and phase shifts using DRL.

Although there are many works considering DRL-based approaches for intelligent
charging, no work has yet addressed the joint beamforming and charging
scheduling problem in RF-WPT systems. Herein, we aim to precisely fill this
research gap by considering a multi-antenna RF-WPT system with multiple non-
linear EH devices with energy requirements at each time slot. Our main
contributions are: i) we formulate the charging scheduling problem, which
consists of choosing precoders at each time slot aiming to minimize the
average power consumption and meet the EH requirements of the nodes; ii) we
propose a solution relying on DDPG and a low-complexity beamforming design;
iii) the results evince that the proposed beamforming achieves near-optimal
performance with much less complexity, while the power consumption decreases
and the reward function converges to a suboptimal solution with the number of
episodes in the DDPG algorithm.

Structure: Section II introduces the system model and the problem formulation.
The proposed optimization framework is discussed in Section III, while Section
IV provides the numerical analysis, and Section V concludes the paper.
Notations: Bold-face and non-Bold-face characters refer to vectors and
scalars, respectively, $(\cdot)^{H}$ denotes the Hermitian operator,
$\Re\\{\cdot\\}$ is the real operator, $\mathbbm{1}(\cdot)$ is the indicator
function, while $\mathbf{0}$ and $\mathbf{1}$ refer to all-zeros and all-ones
vector, respectively.

##  II System Model

The system operates in $T$ time slots, where a fully-digital antenna array
with $N$ elements charges $K$ EH nodes with $d_{i,k}$ being the energy demand
of the $k$th device in the $i$th slot, which is unknown to the transmitter.

Figure 1: The abstract system (left) and the buffer model (right). Case I:
there is not sufficient energy to satisfy the EH requirement; case II: the
device is in idle mode; and case III: all the incident power cannot be
harvested due to buffer size limitation ($B_{\max})$.

###  II-A Signal & Channel Model

We consider $M\leq\min(N,K)$ energy symbols with $x_{i,m}$ being the $m$th
normalized energy symbol at slot $i$ such that
$\mathbb{E}\\{x_{i,m}x^{H}_{i,n}\\}=0$,
$\mathbb{E}\\{x_{i,m}x^{H}_{i,m}\\}=1$, and
$\mathbf{w}_{i,m}\in\mathbb{C}^{N\times 1}$ being the corresponding digital
precoder. Thus, the transmit signal at time slot $i$ is
$\mathbf{z}^{\textrm{tx}}_{i}=\sum_{m=1}^{M}\mathbf{w}_{i,m}x_{i,m}$.
Motivated by the common use of RF-WPT in indoor environments with line-of-
sight (LOS) and near-field communication, as in smart homes, automated
warehousing, and medical instruments [1], we utilize the near-field LOS
channel model from [8].

Let us define the Cartesian coordinate of the $l$th radiating element as
$\mathbf{g}_{l}$. The channel coefficient between user $k$ and the $l$th
element is given by $a_{l,k}=A_{l,k}e^{\frac{-j2\pi}{\lambda}h_{l,k}}$, where
${\frac{2\pi}{\lambda}h_{l,k}}$ is the phase shift caused by the propagation
distance, with wavelength $\lambda$, from the transmitter to the receiver and
$h_{l,k}=||\mathbf{e}_{k}-\mathbf{g}_{l}||$ is the Euclidean distance between
the element and the user located at $\mathbf{e}_{k}$. Moreover,

$A_{l,k}=\sqrt{F(\Theta_{l,k})}\frac{\lambda}{4\pi h_{l,k}}$ (1)

is the channel gain coefficient. Here,
$\Theta_{l,k}=(\theta_{l,k},\psi_{l,k})$ is the elevation-azimuth angle pair,
and $F(\Theta_{l,k})$ is the corresponding radiation profile of each element,
given by [15]

$F(\Theta_{l,k})=\begin{cases}G_{t}\cos{(\theta_{l,k})}^{\frac{G_{t}}{2}-1},&\theta_{l,k}\in[0,\pi/2],\\\
0,&\text{otherwise},\end{cases}$ (2)

where $G_{t}=2(g+1)$ is the transmit antenna gain, and $g$ denotes the
boresight gain, which depends on the antenna technology. Notably, the channel
becomes $A_{k}e^{-j\psi_{l,k}}$ for far-field communications, where $A_{k}$
depends on the $k$th user distance and $\psi_{l,k}$ is determined by the user
direction and the relative disposition of the array elements.

###  II-B Received Signal & EH Device Model

We consider $\mathbf{a}_{k}=[a_{1,k},\ldots,a_{N,k}]^{T}$ is the channel
vector between user $k$ and the transmitter. Then, the $k$th device’s received
signal at time slot $i$ is
$z^{\textrm{rx}}_{i,k}=\sum_{m=1}^{M}\mathbf{a}^{H}_{k}\mathbf{w}_{i,m}x_{i,m}$,
and the corresponding RF power is given by

$P^{\textrm{rf}}_{i,k}=\mathbb{E}_{m}\\{z^{\textrm{rx}}_{i,k}{z^{\textrm{rx}}_{i,k}}^{H}\\}=\sum_{m=1}^{M}||\mathbf{a}^{H}_{k}\mathbf{w}_{i,m}||^{2},$
(3)

while $P_{i}^{\textrm{tx}}=\sum_{m=1}^{M}||\mathbf{w}_{i,m}||^{2}$ is the
$i$th slot’s transmit power. In practice, the EH device is not ideal and this
RF power cannot be fully utilized due to RF-to-direct current (DC) conversion
inefficiencies. Specifically, there are saturation and decrements in
conversion efficiency as we move toward higher input powers. Thus, we utilize
the practical non-linear model presented in [16] with the harvested DC power
of the $k$th device at slot $i$ given by

$P^{\textrm{dc}}_{i,k}=\frac{\Psi_{k}-P^{\textrm{sat}}_{k}/(1+e^{\varphi_{k}\omega_{k}})}{1-1/(1+e^{\varphi_{k}\omega_{k}})},$
(4)

where
$\Psi_{k}=P^{\textrm{sat}}_{k}/\bigl{(}1+e^{-\varphi_{k}(P^{rf}_{i,k}-\omega_{k})}\bigr{)}$,
$P^{\textrm{sat}}_{k}$ is the saturation power, while $\varphi_{k}$ and
$\omega_{k}$ are the parameters related to the EH circuit obtained by curve
fitting on the measurement data.

Notably, each device has a buffer to store the harvested energy. The amount of
available energy in the buffer of user $k$ at the beginning of time slot $i$
is written as

$b_{i,k}=\min\bigl{\\{}b_{i-1,k}+P^{\textrm{dc}}_{i-1,k}\\\
-(d_{i-1,k}+P_{o})\mathbbm{1}{(b_{i-1,k}\geq
d_{i-1,k}+P_{o})},B_{\max}\bigr{\\}},$ (5)

where $B_{\max}$ is the maximum buffer size and $P_{o}$ is the fixed power
required for keeping the EH device’s circuit operational in idle mode.
Equation (5) indicates that devices need $P_{o}$ to keep functionality in the
idle mode (${d}_{i,k}=0$). Moreover, the energy demand is deducted from the
buffer if there is sufficient stored energy. Without loss of generality, we
have normalized the time slot duration, which leads to dealing with the
average harvested power during each time slot. Fig. 1 further clarifies the
system and buffer model.

###  II-C Problem Formulation

The goal is to design the beamformers at each time slot to meet the EH
requirements while minimizing the system’s energy consumption. Thus, the
optimization problem is

$\displaystyle\operatorname*{minimize}_{\mathbf{w}_{i,m}}\quad$
$\displaystyle\frac{1}{T}\sum_{m=1}^{M}\sum_{i=1}^{T}||\mathbf{w}_{i,m}||^{2}$
(6a) subject to $\displaystyle B_{\max}\geq b_{i,k}\geq
d_{i,k}+P_{o},\quad\forall k,i,$ (6b) $\displaystyle P_{i}^{\textrm{tx}}\leq
P_{\max},\quad\forall i,$ (6c)

where $P_{\max}$ is the maximum transmit power at each time slot. This problem
captures the charging scheduling by tuning the beamformers’ direction and
power toward the devices at each time slot, leading to different charging
priorities and selections. Notably, (6) is a complex decision-making problem
due to the EH non-linearity, the randomness of the EH requirements, and the
correlation between the decisions in different time slots.

##  III Optimization Framework

The Markov decision process (MDP) is used to model a problem of such nature,
which leads to efficiently handling the complexity caused by the correlation
between decisions in different time slots. MDP models can be optimized to
derive suboptimal decisions using DRL-based approaches. For this, MDP models
rely on three main components: the action space, the state space, and the
reward function. Notably, for a high-dimensional action space, one may
encounter the ‘curse of dimensionality’ when optimizing the model using DRL,
i.e., the inability of the algorithm to find proper solutions [17]. This may
also happen in our scenario if the action space consists of precoders
($\mathbf{w}_{i,m}$), leading to a drastic increase in the dimension of the
action space, scaling with the array size.

###  III-A MDP Model

We proceed by defining $\alpha_{i,k}$ as a charging weight for the $k$th
device and $P_{i}^{\textrm{tx}}$ as the transmit power of the $i$th slot.
Hereby and by considering given values for $\alpha_{i,k}$ and
$P_{i}^{\textrm{tx}}$, we formulate the beamforming problem at slot $i$ as

$\displaystyle\operatorname*{maximize}_{\mathbf{w}_{i,m},\forall m}\quad$
$\displaystyle\sum_{k=1}^{K}\alpha_{i,k}P^{\textrm{dc}}_{i,k}$ (7a) subject to
$\displaystyle\sum_{m=1}^{M}||\mathbf{w}_{i,m}||^{2}\leq P_{i}^{\textrm{tx}}.$
(7b)

Now, the action space of the MDP model only consists of the charging weights,
i.e., $\alpha_{i,k}$, and the transmit power, i.e., $P^{\textrm{tx}}_{i}$, at
each time slot. Considering this action space leads our proposed MDP model to
inherently capture the beamforming design. Let us proceed by defining the main
components of the MDP model in our setup: i) Action Space: the action at time
slot $i$ is denoted by
$\boldsymbol{\alpha}_{i}=[\alpha_{i,1},\ldots,\alpha_{i,K},\alpha_{i,K+1}]$,
where $\alpha_{i,K+1}$ is the action that determines the transmit power, such
that $P_{i}^{\textrm{tx}}=\alpha_{i,K+1}P_{\max}$, and the action space is
$\mathcal{A}_{i}=\bigl{\\{}\boldsymbol{\alpha}_{i}\in\mathbb{R}^{K+1}|\mathbf{0}\leq\boldsymbol{\alpha}_{i}\leq\mathbf{1}\bigr{\\}}$;
ii) State Space: the system state at time slot $i$ is determined by the
available energy in the buffers and is given by
$\mathbf{s}_{i}=[b_{i,1},\ldots,b_{i,K}]$ and the state space can be written
as
$\mathcal{S}_{i}=\bigl{\\{}\mathbf{s}_{i}\in\mathbb{R}^{K}|\mathbf{0}\leq\mathbf{s}_{i}\leq
B_{\max}\mathbf{1}\bigr{\\}}$; iii) Reward Function: the reward function
should be able to capture (6a) and (6b). Thus, the immediate reward at time
slot $i$ is

${r}_{i}=-\rho_{1}e^{\bar{N}_{i}+\rho_{2}\Delta_{B}}-e^{\frac{P_{i}^{\textrm{tx}}}{P_{\max}}},$
(8)

where $\rho_{1}\geq 1$ and $0\leq\rho_{2}\leq 1$ are the weighting terms,

$\Delta_{B}=\frac{1}{B_{\max}}\sum_{k=1}^{K}{(d_{i,k}+P_{o}-b_{i,k})}\mathbbm{1}{(d_{i,k}+P_{o}>b_{i,k})},$
(9)

is the normalized total deficiency of energy in the devices, and
$\bar{N}_{i}=\sum_{k=1}^{K}\mathbbm{1}{\bigl{(}d_{i,k}+P_{o}>b_{i,k}\bigr{)}}$
is the number of devices that were unable to meet their requirements at the
$i$th slot. This reward indicates that the system is penalized based on the
number of unsatisfied devices and their buffer state, and compels the system
to reduce energy outage before minimizing the transmit power. Hereby, the
discounted cumulative reward at the end of the time slot $i$ is
${R}_{i}=\sum_{u=i}^{\infty}\gamma^{u-i}{r}_{u}$, where $0\leq\gamma\leq 1$ is
the discount factor, i.e., a lower $\gamma$ puts more emphasis on the
immediate reward.

###  III-B Beamforming Design

It is important to solve problem (7) given a specific action with low
computation complexity since the system is time-sensitive. Let
$\mathbf{w}^{\star}_{i,m}$ be the optimal solution of (7). It has been proven
[18, 19] that there exists $\mu_{k}^{\star}$ and $\beta_{k}^{\star}$ values
such that $\mathbf{w}^{\star}_{i,m}$ is the optimal solution to the following
problem:

$\displaystyle\operatorname*{argmax}_{\mathbf{w}_{i,m}\in\mathcal{F},\forall
m}\quad$
$\displaystyle\sum_{k=1}^{K}\alpha_{i,k}\mu_{k}^{\star}\bigl{[}P^{\textrm{sat}}_{k}-\beta_{k}^{\star}(1+e^{-\varphi_{k}({P^{\textrm{rf}}_{i,k}}-\omega_{k})})\bigr{]},$
(10)

where $\mathcal{F}$ is the set of feasible precoders satisfying (7b), while
${P^{\textrm{rf}}_{i,k}}=\sum_{m=1}^{M}||\mathbf{a}^{H}_{k}\mathbf{w}_{i,m}||^{2}$
satisfies the following equations:

$\displaystyle\beta^{\star}_{k}(1+e^{-\varphi_{k}({P^{\textrm{rf}}_{i,k}}^{\star}-\omega_{k})})-P^{\textrm{sat}}_{k}$
$\displaystyle=0,$ (11a)
$\displaystyle\mu^{\star}_{k}(1+e^{-\varphi_{k}({P^{\textrm{rf}}_{i,k}}^{\star}-\omega_{k})})-1$
$\displaystyle=0.\vspace{-2mm}$ (11b)

This problem can be solved by an iterative algorithm including an inner loop
for finding $\mathbf{w}_{i,m}$ given $\mu_{k}$ and $\beta_{k}$, and an outer
loop for updating $\mu_{k}$ and $\beta_{k}$. One possible approach to solve
the inner loop’s problem is to reformulate the problem as a semi-definite
program (SDP), as in [16]. However, this is not computationally/time
efficient, especially for large $N$ since SDP deals with Hermitian matrix
subspaces with size $N^{2}$.

####  III-B1 Suboptimal precoders given $\mu_{k}$ & $\beta_{k}$

We consider $\mathbf{w}_{i,m}^{(l)}$ as the initial point and approximate (10)
by defining $P^{\textrm{rf}}_{i,k}$ as auxiliary variable and using the first-
order Taylor expansion of the right-hand side of (3) as

$\displaystyle\operatorname*{minimize}_{{\mathbf{w}_{i,m}\in\mathcal{F},P^{\textrm{rf}}_{i,k}}}\quad$
$\displaystyle\Omega=-\sum_{k=1}^{K}\alpha_{i,k}\mu_{k}^{\star}\bigl{[}P^{\textrm{sar}}_{k}-\beta_{k}^{\star}(1+e^{-\varphi_{k}(P^{\textrm{rf}}_{i,k}-\omega_{k})})\bigr{]}$
(12a) subject to $\displaystyle
P_{i,k}^{\textrm{rf}}\leq\sum_{m=1}^{M}\biggl{[}2\Re\bigl{\\{}\mathbf{w}_{i,m}^{H}\mathbf{a}_{k}\mathbf{a}_{k}^{H}\mathbf{w}_{i,m}^{(l)}\bigr{\\}}-$
$\displaystyle\hskip
71.13188pt{\mathbf{w}_{i,m}^{(l)}}^{H}\mathbf{a}_{k}\mathbf{a}_{k}^{H}\mathbf{w}_{i,m}^{(l)}\biggr{]},$
(12b)

which is convex in the neighborhood of the initial point. Then, the problem
can be solved using successive convex approximation [20]. Note that (12) deals
with complex vector subspace with size $N$, leading to less computational
complexity in a single iteration compared to SDP. However, it still might
challenge the system’s real-time operation since the transmitter has to solve
multiple subproblems in the inner loop.

Let us proceed by writing the Lagrangian for (12) as

$\mathcal{L}_{i}=-\sum_{k=1}^{K}\alpha_{i,k}\mu_{k}^{\star}\bigl{[}P^{\textrm{sat}}_{k}-\beta_{k}^{\star}(1+e^{-\varphi_{k}(P^{\textrm{rf}}_{i,k}-\omega_{k})})\bigr{]}+\\\
\bar{\nu}_{i}(\sum_{m=1}^{M}|\mathbf{w}_{i,m}|^{2}-P_{i}^{\textrm{tx}})+\sum_{k=1}^{M}\nu_{i,k}\biggl{(}P_{i,k}^{\textrm{rf}}-\\\
\sum_{m=1}^{M}\bigl{[}2\Re\bigl{\\{}\mathbf{w}_{i,m}^{H}\mathbf{a}_{k}\mathbf{a}_{k}^{H}\mathbf{w}_{i,m}^{(l)}\bigr{\\}}-{\mathbf{w}_{i,m}^{(l)}}^{H}\mathbf{a}_{k}\mathbf{a}_{k}^{H}\mathbf{w}_{i,m}^{(l)}\bigr{]}\biggr{)},$
(13)

where $\nu_{i,k}$ and $\bar{\nu}_{i}$ are the associated dual variables. Then,
using Karush–Kuhn–Tucker conditions [20] and setting the derivative of (13)
w.r.t. $\mathbf{w}_{i,m}$ equal to zero, we can write

$\mathbf{w}_{i,m}^{\star}=\frac{1}{\bar{\nu}^{(l-1)}}\sum_{k=1}^{K}\nu_{i,k}\mathbf{a}_{i,k}\mathbf{a}_{k}^{H}\mathbf{w}_{i,m}^{(l-1)}.$
(14)

Notably, we introduce an additional parameter $\kappa$ to control the
convergence of the proposed iterative method. Hereby, the precoders at the
$l$th iteration are updated using
$\mathbf{w}_{i,m}^{(l)}=\mathbf{w}_{i,m}^{(l-1)}+\kappa(\mathbf{w}_{i,m}^{\star}-\mathbf{w}_{i,m}^{(l-1)})$.
Obviously, the optimal solution of (7) utilizes the total amount of transmit
power such that $\sum_{m=1}^{M}||\mathbf{w}_{i,m}||^{2}=P^{\textrm{tx}}_{i}$,
thus, $\bar{\nu}_{i}$ can be updated using

$\displaystyle\bar{\nu}^{(l)}_{i}=\sqrt{\frac{1}{P_{i}^{\textrm{tx}}}\sum_{m=1}^{M}\bigl{|}\bigl{|}\sum_{k}\nu_{i,k}\mathbf{a}_{k}\mathbf{a_{k}}^{H}\mathbf{w}_{i,m}^{(l)}\bigr{|}\bigr{|}^{2}}.$
(15)

Moreover, setting the derivative of (13) w.r.t. $P^{\textrm{rf}}_{i,k}$ equal
to zero gives us

$\displaystyle\nu_{i,k}^{(l)}=\alpha_{i,k}\mu_{k}\beta_{k}\varphi_{k}e^{-\varphi_{k}({P_{i,k}^{\textrm{rf}}}^{(l)}-\omega_{k})},$
(16)

and ${P_{i,k}^{\textrm{rf}}}^{(l)}$ is obtained by substituting
$\mathbf{w}_{i,m}^{(l)}$ in (3). These updating rules lead to low complexity
since the beamformers can be derived using closed-form expressions.

####  III-B2 Updating $\mu_{k}$ & $\beta_{k}$ given $\mathbf{w}_{i,m}$

For this, we rely on the well-known damped Newton method [19, 16]. Let us
define
$\phi_{k}=\beta_{k}(1+e^{-\varphi_{k}(P_{i,k}^{\textrm{rf}}-\omega_{k})})-P^{sat}_{k}$
and
$\phi_{M+k}=\mu_{k}(1+e^{-\varphi_{k}(P_{i,k}^{\textrm{rf}}-\omega_{k})})-1$.
It has been proven in [18, 19] that the unique optimal solution
$\mu_{k}^{\star}$ and $\beta_{k}^{\star}$ is obtained if and only if
$\boldsymbol{\phi}(\boldsymbol{\mu,\boldsymbol{\beta}})=[\phi_{1},\ldots,\phi_{2K}]=\mathbf{0}$,
where $\boldsymbol{\mu}=[\mu_{1},\ldots,\mu_{K}]$ and
$\boldsymbol{\beta}=[\beta_{1},\ldots,\beta_{K}]$. Therefore, $\mu_{k}$ and
$\beta_{k}$ can be updated at the $n$th outer iteration using

$\boldsymbol{\mu}^{(n+1)}\\!\\!=\\!\boldsymbol{\mu}^{(n)}\\!\\!+\\!\zeta^{(n)}\mathbf{q}^{(n)}_{[K\\!+\\!1:2K]},\
\boldsymbol{\beta}^{(n+1)}\\!\\!=\\!\boldsymbol{\beta}^{(n)}\\!\\!+\\!\zeta^{(n)}\mathbf{q}_{[1:K]}^{(n)},$
(17)

where $\mathbf{q}_{[1:K]}^{(n)}$ refers to the first to $K$th element of
$\mathbf{q}^{(n)}$,
$\mathbf{q}^{(n)}=[\boldsymbol{\phi}^{\prime}(\boldsymbol{\mu},\boldsymbol{\beta})]^{-1}\boldsymbol{\phi}(\boldsymbol{\mu,\boldsymbol{\beta}})$
, $\boldsymbol{\phi}^{\prime}(\boldsymbol{\mu},\boldsymbol{\beta})$ is the
Jacobian matrix of $\boldsymbol{\phi}(\boldsymbol{\mu},\boldsymbol{\beta})$,
and $\zeta^{(n)}$ is the largest $\epsilon^{t}$ satisfying

$||\boldsymbol{\phi}(\boldsymbol{\mu}\\!+\\!\epsilon^{t}\mathbf{q}^{(n)}_{[K\\!+\\!1:2K]},\boldsymbol{\beta}\\!+\\!\epsilon^{t}\mathbf{q}_{[1:K]}^{(n)})||\\!\leq\\!(1\\!-\\!\sigma\epsilon^{t})||\boldsymbol{\phi}(\boldsymbol{\mu},\boldsymbol{\beta})||,$
(18)

where $t\in\\{1,2,\ldots\\}$, $\epsilon\in[0,1]$, and $\sigma\in[0,1]$.

Algorithm 1 Iterative beamforming for the $i$th slot (IT-BF).

1:Input:$\\{\alpha_{i,m}\\}_{\forall m},P_{i}^{\textrm{tx}}$, $\epsilon$,
$\zeta$, $\sigma$, $\Phi$, $\rho$ Output: $\mathbf{w}_{i,m}^{(l)},\forall m$

2:Initialize: $\mathbf{w}^{(l)}_{i,m},\forall m$, $\bar{\nu}$,
$\nu_{k},\forall k$, $\xi=0$

3:repeat (outer loop)

4: repeat (inner loop)

5: $\xi^{\star}\leftarrow\xi$

6: Obtain $\mathbf{w}_{i,m}^{\star},\forall m$ using (14)

7:
$\mathbf{w}_{i,m}^{(l)}\leftarrow\mathbf{w}_{i,m}^{(l)}+\rho(\mathbf{w}_{i,m}^{\star}-\mathbf{w}_{i,m}^{(l)}),\forall
m$

8: Update $\bar{\nu}$ and $\nu_{k},\forall k$ using (15) and (16)

9:
${P_{i,k}^{\textrm{rf}}}^{(l)}=\sum_{m=1}^{M}||\mathbf{a}^{H}_{k}\mathbf{w}^{(l)}_{i,m}||^{2}$,
compute $\xi$ using (12a)

10: until $|1-\frac{\xi}{\xi^{\star}}|<\Phi$

11: Obtain $\zeta^{(n)}$, update $\boldsymbol{\mu}^{(n)}$ and
$\boldsymbol{\beta}^{(n)}$ using (18) and (17)

12:until convergence

Algorithm 1 illustrates the proposed beamforming approach at time slot $i$.
Therein, the derived closed-form solutions are used to update the beamformers
and the dual variables in the inner loop (lines 4-10) until (12a) converges,
while $\mu_{k}$ and $\beta_{k}$ are updated in the outer loop (lines 3-12).
This procedure is repeated until a convergence criterion is satisfied.

###  III-C DDPG-based Joint Beamforming & Charging Scheduling

The goal is to maximize the long-term commutative reward by training a policy.
For this, we utilize DDPG [21], a model-free off-policy algorithm for learning
continuous actions, as in our case. Let us denote the Q-value function as the
expected commutative reward given by
$\mathcal{Q}(\mathbf{s}_{i},\boldsymbol{\alpha}_{i})=\mathbb{E}\bigl{\\{}{R}_{i}\bigr{\\}}$,
while the target Q-network is $\mathcal{Q}^{\star}$. DDPG approximates the
policy and Q-value functions utilizing two neural networks (NNs), defined as
the actor and the critic (see details on the operation of the actor-critic
approach in [21]). The following outlines our DDPG implementation of the
formulated MPD.

Let us define the parameterized actor function $\mu$ as the current policy by
deterministically mapping states to a specific action. By defining
$\theta^{\mu}$ and $\theta^{\mathcal{Q}}$ as the parameters of the actor and
critic NNs, respectively, the actor NN parameters are updated by applying the
chain rule to the expected return from the start distribution ($J$) such that
[21]

$\displaystyle\nabla_{\\!\theta^{\mu}}\\!J\\!\approx\\!\mathbb{E}\biggl{\\{}\\!\\!\nabla_{\boldsymbol{\alpha}}\mathcal{Q}(\mathbf{s},\boldsymbol{\alpha}|{\theta^{Q}})\bigl{|}_{\mathbf{s}\\!=\\!\mathbf{s}_{i},\boldsymbol{\alpha}\\!=\\!\mu(\mathbf{s}_{i})}\\!\\!\\!\nabla_{\\!\theta^{\mu}}\mu(\mathbf{s}|\theta^{\mu})|_{\mathbf{s}=\mathbf{s}_{i}}\\!\\!\biggr{\\}}.$
(19)

Meanwhile, the critic NN aims to learn
$\mathcal{Q}(\mathbf{s}_{i},\alpha_{i})$. It has been proven that each
Q-function for some policy obeys the Bellman equation, thus, we write [17]

$y_{i}={r}_{i}+\gamma\mathcal{Q}^{\star}\bigl{(}s_{i+1},\mu^{\star}(s_{i+1}|\theta^{\mu^{\star}})|\theta^{Q^{\star}}\bigr{)},$
(20)

where $\mu^{\star}$ is the target actor. The critic NN aims to minimize a loss
function of the temporal difference error given by

$\mathcal{L}_{i}=\frac{1}{|\mathcal{B}_{i}|}\sum_{(\mathbf{s}_{i},\boldsymbol{\alpha}_{i},\mathbf{s}_{i+1},{r}_{i})\in\mathcal{B}_{i}}[\mathcal{Q}(\mathbf{s}_{i},\boldsymbol{\alpha}_{i})-y_{i}]^{2},$
(21)

where $\mathcal{B}_{i}\subseteq\mathcal{B}$ is the minibatch of transitions
and $\mathcal{B}$ is the memory buffer. Notice that the system includes a
replay memory to store the transitions for future use.

To make the system observable, the active devices send a cost-free
reliable111In practice, the feedback comes with an associated transmission
error and cost depending on the modulation and the message length. However,
for simplicity and without loss of generality, we assume successful message
delivery and that the devices always save energy for feedback. feedback
message at the end of each time slot declaring their past required energy and
whether they had sufficient energy to meet that. If the transmitter does not
receive any feedback, it assumes that the device is in idle mode. Hereby, the
system becomes observable and the transmitter can update the states after each
time slot.

Algorithm 2 illustrates the proposed DDPG-based joint beamforming and charging
scheduling. At first, the system state and a random process $\mathcal{N}$ for
the exploration noise are initialized. Each episode consists of $T$ time slots
and at the beginning of each time slot, an action is selected using the actor
NN and a generated exploration noise. This noise leads to a trade-off between
exploration and exploration, leading to better convergence. Then, Algorithm 1
is run to obtain the beamformers for the selected action. Finally, the actor
and critic NNs are updated, and the corresponding target NNs’ parameters are
softly updated using $\tau$, which leads to small changes in target NNs; thus,
offering stability in training [21].

Algorithm 2 DDPG-based joint beamforming and charging scheduling (DDPG-BCS).

1:Input: $\gamma$, $\theta^{\mu}$, $\theta^{\mu^{\star}}$,
$\theta^{\mathcal{Q}}$, $N_{ep}$, $\theta^{\mathcal{Q}^{\star}}$, $\tau$
Output: $\mu^{\star}$

2:Get the initial system state $\mathbf{s}_{1}$

3:for $q=1,2,\ldots,N_{ep}$ do

4: Initialize a random process $\mathcal{N}$ for exploration

5: for $i=1,2,\ldots,T$ do

6: Generate a random noise $\mathbf{n}$ using $\mathcal{N}$

7: Select an action using
$\boldsymbol{\alpha}_{i}=\mu(\mathbf{s}_{i}|\theta^{\mu})+\mathbf{n}$

8: Run IT-BF to obtain $\mathbf{w}_{i,m}$

9: Compute $r_{i}$ and $\mathbf{s}_{i+1}$ using (5), and (8)

10: Store the transition
$(\mathbf{s}_{i},\boldsymbol{\alpha}_{i},\mathbf{s}_{i+1},{r}_{i})$ in
$\mathcal{B}$

11: Sample a random minibatch $\mathcal{B}_{i}\subseteq\mathcal{B}$ with size
$N_{b}$

12: Update the critic NN by minimizing $\mathcal{L}_{i}$ in (21)

13: Update the actor NN by sampling (19) for $\mathcal{B}_{i}$

14:
$\theta^{\mu^{\star}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\star}}$,
$\theta^{\mathcal{Q}^{\star}}\leftarrow\tau\theta^{\mathcal{Q}}+(1-\tau)\theta^{\mathcal{Q}^{\star}}$

15: end for

16: $\mathbf{s}_{1}\leftarrow\mathbf{s}_{T}$

17:end for

##  IV Numerical Analysis

We consider a square uniform planar array with $N=64$, inter-element distance
$\frac{\lambda}{2}$, and $\lambda=12.5$ cm, corresponding to operation at 2.4
GHz. The transmitter is at the ceiling’s center of a $5\times 5\times 5$ m3
area, while the users are uniformly positioned on a circle along the $xy$
dimension with a 2 m radius such that the position of $k$th device is
$[2.5+2\cos{(2\pi/K)},2.5+2\sin(2\pi/K),2]$. Moreover, $\mathcal{N}$ follows a
Ornstein-Uhlenbeck process with parameters $\theta^{\mathcal{N}}=0.15$,
$\sigma^{\mathcal{N}}=0.2$ [21]. For device activation, we utilize an alarm
generation process based on the spatial location of the devices, in which an
alarm activates a set of devices based on the activation probability function
$f(\chi_{k})=e^{-\chi_{k}}$ with $\chi_{k}$ being the distance of the $k$th
device from the event epicenter. Notably, the epicenter is generated using a
uniform distribution across the area. Moreover, when a device is activated,
the required amount of energy is selected using a geometric distribution with
parameter $\vartheta=0.5$. Thus, there is an energy demand burst $nd_{b}$ for
an activated device with probability $\vartheta^{n-1}(1-\vartheta)$, while the
maximum possible demand is 50 mW, $d_{b}=10$ mW, and $P_{o}=10$ $\mu$W [1].
The initial battery level of the devices is 50 mW, $B=200\leavevmode\nobreak\
mW$, and $P_{\max}=10$ W. The EH device parameters are $\phi_{k}=6400$,
$\omega_{k}=0.003$, $P_{k}^{\textrm{ sat}}=20$ mW [16].

The actor NN consists of two fully connected layers with 128 neurons and a
Rectified Linear Unit (ReLU) activation function. Moreover, the output
activation function is tangent hyperbolic, leading to the range $[-1,1]$,
thus, the outputs are scaled to fit in the desired range $[0,1]$. The critic
NN has two hidden layers with 64 neurons and ReLU activation function, while
the output utilizes linear activation. The IT-BF algorithm convergence
criterion is reaching 20 outer loop iterations or
$\boldsymbol{\phi}(\boldsymbol{\mu},\boldsymbol{\beta})\leq
10^{-4}\mathbf{1}$. The optimization parameters are $T=100$, $\kappa=0.05$,
$g=2$, $\tau=0.001$, $N_{b}=64$, $|B|=10^{6}$, $\gamma=0.99$, $\Phi=10^{-6}$,
$\rho_{1}=\rho_{2}=1$, $\sigma=0.5$, $\epsilon=0.5$, while actor and critic
learning rates are $10^{-4}$ and $2\times 10^{-4}$, respectively. We compare
the performance of our proposed DDPG-BCS approach with a heuristic aiming to
keep the buffer state of the devices above the maximum energy demand (50 mW).
Thus, at the $i$th time slot, the set of devices with the buffer value below
50 mW, denoted as $\mathcal{K}_{i}$, is charged with
$P_{i}^{\textrm{tx}}=P_{\max}$,
$\alpha_{i,k}=\bar{b}_{i,k}/\sum_{k\in\mathcal{K}_{i}}\bar{b}_{i,k}$, and
$\bar{b}_{i,k}=50-b_{i,k}$.

Fig. 2 illustrates the average performance of the proposed IT-BF using Monte
Carlo simulations. It is seen that the IT-BF performs near the optimal SDP
solution and the bound becomes tighter as $N$ increases. Meanwhile, IT-BF
requires considerably less time 222Note that the convergence time is obtained
using an average-performance computer, and the purpose is to only show the
relative time difference. However, the convergence time can become
significantly lower by using proper hardware resources. to converge and the
complexity increases with $N$ and $K$. Note that the convergence performance
of IT-BF, presented in Fig. 3, shows that the algorithm converges toward a
suboptimal solution with outer iterations.

Fig. 4 showcases the reward function of DDPG-BCS over learning episodes. It is
seen that the reward gradually converges toward a suboptimal solution for each
$K$. Meanwhile, Fig. 5 shows the average outage probability, i.e.,
$(\sum_{i=1}^{T}\bar{N}_{i})/KT$, and the average transmit power over
episodes. Observe that the DDPG-BCS outperforms the heuristic in terms of
power consumption with a considerable gap. However, as the number of devices
increases, the heuristic leads to fewer outages, but with much higher transmit
power. Instead, the proposed approach aims for the trade-off between outage
and transmit power, which can be modified by shaping the reward function. In
general, it is seen that the power consumption increases with $K$, while more
outages may happen as $K$ becomes larger. Notably, even for $K=6$ the outage
probability is relatively small ($\leq 10^{-2}$), while for lower $K$, this
value is near zero.

Figure 2: (a) The average sum harvested power (top) and (b) the convergence
time (bottom) as a function of $N$. Figure 3: The $\Omega$ value as a function
of the inner loop iterations for $K=6$, $P^{\textrm{tx}}=10$ W, and random
$\alpha_{k}$. Figure 4: The average reward as a function of the number of
episodes for different numbers of devices.

Figure 5: (a) The average transmit power (top), (b) the average outage
probability (bottom) as a function of the number of episodes for different
numbers of devices.

##  V Conclusion and Future Work

In this paper, we considered a multi-antenna RF-WPT system to charge multiple
non-linear EH devices over time. Moreover, we formulated the joint beamforming
and charging scheduling problem to minimize the average transmit power and
meet the EH requirements and proposed a solution relying on DDPG and a low-
complexity beamforming design. The results demonstrated that our proposed
beamforming approach achieves a near-optimal solution and that the DDPG-based
optimization converges after some episodes. It was shown that the outage
probability and the energy consumption increase and decrease with the number
of devices, respectively.

As a prospect for future research, one may investigate the system performance
in a partially observable environment or consider channel variations due to
device mobility.

## References

  * [1] 3GPP, “Study on Ambient IoT (Internet of Things) in RAN,” Tech. Rep. 38.848, 2023.
  * [2] O. López  _et al._ , “Zero-energy Devices for 6G: Technical Enablers at a Glance,” 2024.
  * [3] O. L. A. López  _et al._ , “Massive wireless energy transfer: enabling sustainable IoT toward 6G era,” IEEE Internet Things J., vol. 8, no. 11, pp. 8816–8835, 2021.
  * [4] O. L. A. López  _et al._ , “High-Power and Safe RF Wireless Charging: Cautious Deployment and Operation,” 2023.
  * [5] O. L. A. López  _et al._ , “Massive MIMO with radio stripes for indoor wireless energy transfer,” IEEE Trans. Wirel. Commun., vol. 21, no. 9, pp. 7088–7104, 2022.
  * [6] A. Azarbahram  _et al._ , “On the Radio Stripe Deployment for Indoor RF Wireless Power Transfer,” WCNC 2024.
  * [7] A. Azarbahram  _et al._ , “Energy Beamforming for RF Wireless Power Transfer With Dynamic Metasurface Antennas,” IEEE Wireless Commun. Lett., vol. 13, no. 3, pp. 781–785, 2024.
  * [8] H. Zhang  _et al._ , “Near-field wireless power transfer with dynamic metasurface antennas,” in IEEE SPAWC, pp. 1–5, 2022.
  * [9] O. L. A. López  _et al._ , “A Low-Complexity Beamforming Design for Multiuser Wireless Energy Transfer,” IEEE Wireless Commun. Lett., vol. 10, no. 1, pp. 58–62, 2021.
  * [10] K. Li, W. Ni, M. Abolhasan, and E. Tovar, “Reinforcement Learning for Scheduling Wireless Powered Sensor Communications,” IEEE Trans. Green Commun. Netw., vol. 3, no. 2, pp. 264–274, 2019.
  * [11] J. Yao and N. Ansari, “Wireless Power and Energy Harvesting Control in IoD by Deep Reinforcement Learning,” IEEE Trans. Green Commun. Netw., vol. 5, no. 2, pp. 980–989, 2021.
  * [12] J. Ye and H. Gharavi, “Deep Reinforcement Learning-Assisted Energy Harvesting Wireless Networks,” IEEE Trans. Green Commun. Netw., vol. 5, no. 2, pp. 990–1002, 2021.
  * [13] J.-M. Kang, “Reinforcement Learning Based Adaptive Resource Allocation for Wireless Powered Communication Systems,” IEEE Communications Letters, vol. 24, no. 8, pp. 1752–1756, 2020.
  * [14] K. Nguyen  _et al._ , “RIS-Assisted UAV Communications for IoT With Wireless Power Transfer Using Deep Reinforcement Learning,” IEEE J Sel Top Signal Process, vol. 16, no. 5, pp. 1086–1096, 2022.
  * [15] S. W. Ellingson, “Path Loss in Reconfigurable Intelligent Surface-Enabled Channels,” in IEEE PIMRC, pp. 829–835, 2021.
  * [16] E. Boshkovska  _et al._ , “Practical Non-Linear Energy Harvesting Model and Resource Allocation for SWIPT Systems,” IEEE Commun. Lett., vol. 19, no. 12, pp. 2082–2085, 2015.
  * [17] K. Arulkumaran  _et al._ , “Deep Reinforcement Learning: A Brief Survey,” IEEE Signal Process Mag, vol. 34, no. 6, pp. 26–38, 2017.
  * [18] Y. Jong, “An efficient global optimization algorithm for nonlinear sum-of-ratios problem,” Optimization Online, vol. 32, pp. 38–39, 2012.
  * [19] G. Yu  _et al._ , “Joint Downlink and Uplink Resource Allocation for Energy-Efficient Carrier Aggregation,” IEEE Trans. Wirel. Commun., vol. 14, no. 6, pp. 3207–3218, 2015.
  * [20] S. Boyd and L. Vandenberghe, Convex optimization.  Cambridge university press, 2004.
  * [21] P. L. Timothy  _et al._ , “Continuous control with deep reinforcement learning,” 2019.