# Rate-Splitting Multiple Access for Satellite-Terrestrial Integrated
Networks: Benefits of Coordination and Cooperation

Longfei Yin, Bruno Clerckx  L. Yin and B. Clerckx are with the Communications
and Signal Processing Group, Department of Electrical and Electronic
Engineering, Imperial College London, London, SW7 2AZ, U.K (e-mail:
<EMAIL_ADDRESS>b.clerckx@imperial.ac.uk). This work has been
partially supported by the U.K. Engineering and Physical Sciences Research
Council (EPSRC) under grant EP/R511547/1.

###### Abstract

This work studies the joint beamforming design problem of achieving max-min
rate fairness in a satellite-terrestrial integrated network (STIN) where the
satellite provides wide coverage to multibeam multicast satellite users (SUs),
and the terrestrial base station (BS) serves multiple cellular users (CUs) in
a densely populated area. Both the satellite and BS operate in the same
frequency band. Since rate-splitting multiple access (RSMA) has recently
emerged as a promising strategy for non-orthogonal transmission and robust
interference management in multi-antenna wireless networks, we present two
RSMA-based STIN schemes, namely the coordinated scheme relying on channel
state information (CSI) sharing and the cooperative scheme relying on CSI and
data sharing. Our objective is to maximize the minimum fairness rate amongst
all SUs and CUs subject to transmit power constraints at the satellite and the
BS. A joint beamforming algorithm is proposed to reformulate the original
problem into an approximately equivalent convex one which can be iteratively
solved. Moreover, an expectation-based robust joint beamforming algorithm is
proposed against the practical environment when satellite channel phase
uncertainties are considered. Simulation results demonstrate the effectiveness
and robustness of our proposed RSMA schemes for STIN, and exhibit significant
performance gains compared with various traditional transmission strategies.

###### Index Terms:

Rate-splitting multiple access (RSMA), max-min fairness, satellite-terrestrial
integrated network (STIN), beamforming design, multibeam multicast
transmission, channel phase uncertainty

##  I Introduction

In recent years, due to the explosive growth of wireless applications and
multimedia services, satellite-terrestrial integrated network (STIN) has
gained a tremendous amount of attention in both academia and industry as it
can provide ubiquitous coverage and convey rich multimedia services, e.g.,
video on demand (VoD) streaming and TV broadcasting, etc. to users in both
densely and sparsely populated areas[1]. The integration of terrestrial and
satellite networks is of great potential in achieving geographic coverage,
especially for remote areas where no terrestrial base station (BS)
infrastructure can be employed [2]. It is envisaged that the C-band ($4-8\
\mathrm{GHz}$) and S-band ($2-4\ \mathrm{GHz}$) can be shared between the
terrestrial and satellite networks. In addition, the spectrum above Ka band
from $20\ \mathrm{GHz}$ to $90\ \mathrm{GHz}$ is foreseen to be the most
promising candidate radio band for the next generation terrestrial cellular
networks, and part of this band has already been allocated to the satellite
networks [3, 4]. The concept of STIN has been proposed in the literature [5,
6, 7]. The satellite sub-network shares the same radio frequency band with the
terrestrial sub-network through dynamic spectrum access technology to enhance
spectrum utilization, thereby achieving higher spectrum efficiency and
throughput. However, aggressive frequency reuse can induce severe interference
in and between the sub-networks. In this work, we will concentrate on RSMA-
based joint beamforming schemes to efficiently mitigate the interference of
STIN.

###  I-A Related Works

A number of research efforts have investigated STIN systems. A coexistence
framework of the satellite and terrestrial network was presented in [8] with
the satellite link as primary and the terrestrial link as secondary. Transmit
beamforming techniques were studied to maximize the signal to interference
plus noise ratio (SINR) towards secondary terrestrial users and minimize the
interference towards primary satellite users. In [9], closed-form outage
probabilities of the terrestrial users were derived while satisfying the
interference constraints of satellite users. [10] studied a time division
cooperative STIN, where a weighted max–min fair (MMF) beamforming design
problem was formulated to jointly optimize the beamforming vectors of BSs and
the satellite. A multicast beamforming STIN system was investigated in [11]
with the aim of maximizing the sum of user minimum ratio under constraints of
backhaul links and quality of service (QoS). All aforementioned works
generally assumed the satellite channels as Rician channels. The effects of
satellite antenna gain, path loss and atmospheric attenuation can be taken
into account to model more practical satellite channels so as to evaluate the
system performance more accurately. In this regard, [4] investigated a joint
beamforming scheme for secure communication of STIN operating in mmWave
frequencies. Total transmit power was minimized while satisfying both the QoS
constraints of a single terrestrial user and the secrecy rate (SR)
requirements of multiple satellite users. [12] focused on the joint
optimization for wireless information and power transfer (WIPT) technique in
STINs. In [13], the cache-enabled LEO satellite network was introduced, and
the scheme of STIN was proposed to enable an energy-efficient radio access
network (RAN) by offloading traffic from BSs through satellite’s broadcast
transmission. [14] investigated a novel secure 3D mobile unmanned aerial
vehicle (UAV) relaying for STIN considering three opportunistic UAV relay
selection strategies.

The above works consider conventional space division multiple access (SDMA)
based on linear precoding and assume perfect channel state information at the
transmitters (CSIT). Each user decodes its desired stream while treating all
the other interference streams as noise. The spatial degrees of freedom
provided by multiple antennas are exploited, however the effectiveness of
beamforming design relies on the accuracy of CSIT significantly. In the real
satellite communication (SatCom) environment, one practical issue is that
accurate CSI is very difficult to acquire at the gateway (GW) because of the
long-distance propagation delay and device mobility. Thus, robust design in
the presence of imperfect CSIT has been widely studied in the literature [15,
16, 17, 18, 19, 20]. [15, 16, 17] assumed the satellite channel uncertainty as
additive estimation error located in a bounded error region. Robust
beamforming was designed based on the optimization of the worst case
situation. Yet, due to the special characteristics of satellite channels, the
channel magnitude does not vary significantly due to the fact that the channel
propagation is dominated by the line-of-sight component. The phase variations
constitute the major source of channel uncertainty [21]. Therefore, in [18,
19, 20], robust beamforming was studied when considering constant channel
amplitudes within the coherence time interval and independent time varying
phase components. Apart from the difficulties in obtaining perfect CSIT,
another consideration for beamformer design in STIN is the frame-based
structure of multibeam satellite standards such as DVB-S2 [22] and DVB-S2X
[23]. Each spot beam of the satellite serves more than one user simultaneously
by transmitting a single coded frame. Multiple users within the same beam
share the same precoding vector, and leads to multibeam multicast
transmission, which is a promising solution for the rapidly growing content-
centric applications including video streaming, advertisements, large scale
system updates and localized services, etc.

Motivated by the above practical issues in STIN, advanced multiple access
techniques and interference management strategies are required. Rate-splitting
multiple access (RSMA) has recently emerged as a promising non-orthogonal
transmission for downlink multi-antenna wireless networks owning to its
capability to enhance the system performance in a wide range of network loads,
user deployments and CSIT qualities. RSMA was shown analytically in [24] to
generalize four seemingly different strategies, namely space division multiple
access (SDMA), power-domain non-orthogonal multiple access (NOMA), orthogonal
multiple access (OMA) and physical-layer multicasting. The key behind the
flexibility and robust manner of RSMA is to split each message into a common
part and a private part. All common parts are jointly encoded into a common
stream to be decoded by all users, while the private parts are individually
encoded into private streams. The common stream is decoded at first and
subtracted by successive interference cancellation (SIC). Each user then
decodes its desired private stream and treats the remaining interference as
noise. Such framework enables RSMA to partially decode interference and
partially treat interference as noise. The benefits achieved by RSMA have been
demonstrated in various multi-antenna terrestrial scenarios, such as multiuser
unicast systems with perfect CSIT [24, 25, 26, 27] and imperfect CSIT [28, 29,
30, 31, 32], and multigroup multicast systems [33, 34, 35]. The superior
performance of RSMA can also be seen in massive MIMO systems with residual
transceiver hardware impairments [36], mmWave communications [37],
simultaneous wireless information and power transfer (SWIPT) networks [38] and
radar-communication (RadCom) systems [39].

More recently, the use of RSMA in multibeam SatCom or integrated satellite
systems has been investigated. [40] studied RSMA in a two-beam satellite
system adopting time division multiplexing (TDM) in each beam. [41] focused on
the sum-rate optimization and low complexity RSMA precoding design by
decoupling the design of common stream and private streams. [42, 43] proposed
a RSMA-based multibeam multicast beamforming scheme and formulated a per-feed
power constrained max-min fair (MMF) problem with different CSIT qualities.
Simulation results demonstrated the effectiveness of adopting RSMA to manage
inter-beam interference, taking into account various practical challenges. In
[44], RSMA was proven to be promising for multigateway multibeam satellite
systems with feeder link interference. [45] considered a satellite and aerial
integrated network comprising a satellite and a unmanned aerial vehicle (UAV).
The satellite employed multicast transmission, while the UAV used RSMA to
improve spectral efficiency. In [46], a secure beamforming scheme for STIN was
presented, where the satellite served one earth station (ES) with $K$
eavesdroppers (Eves). RSMA was employed at the BS to achieve higher spectral
efficiency. A robust beamforming scheme was proposed to maximize the secrecy
energy efficiency of the ES considering Euclidean norm bounded channel
uncertainty.

###  I-B Contributions

Motivated by the prior works, we further investigate the application of RSMA
into STIN consisting of a satellite sub-network and a terrestrial sub-network
to manage the interference in and between both sub-networks. Practical
challenges are taken into account, such as the per-feed constraints, CSIT
uncertainty, and multibeam multicast transmission due to the existing
satellite communication standards [22, 23]. The contributions of this article
are summarized as follows.

  * •

First, we present a general framework of STIN where the satellite provides
wide coverage to multibeam multicast satellite users (SUs), while the
terrestrial BS serves multiple cellular users (CUs) in a densely populated
area. The satellite and BS share the same radio spectrum resources, and the GW
operates as a control center to collect and manage various kinds of
information, implement centralized processing and control the whole system
through resource allocation and interference management. The BS is connected
to the GW through optical links. We introduce two scenarios of RSMA-based
STIN, namely the coordinated scheme with CSI of the whole system available at
the GW, and the cooperative scheme with both CSI and transmit data information
of the whole system available at the GW. This framework differs from the prior
RSMA and STIN papers, where the benefits of coordination and cooperation are
not investigated.

  * •

Second, based on the coordinated scheme and cooperative scheme, we
respectively formulate optimization problems to maximize the minimum fairness
rate amongst all SUs and CUs subject to the per-feed power constraints at the
satellite and sum power constraint at the BS. Perfect CSI is assumed at the
GW. This is the first work on the joint beamforming design for RSMA-based
coordinated STIN and cooperative STIN with the objective of achieving max-min
fairness. Since the formulated problem is nonconvex, we exploit the sequential
convex approximation (SCA) method to reformulate the original problem into an
approximately equivalent convex one. Simulation results confirm the
superiority of the cooperative scheme over the coordinated scheme, and
significant performance gains of RSMA-based beamforming compared with
traditional techniques.

  * •

Third, since it is very challenging to obtain accurate CSI at the GW in the
practical STIN environment, we investigate expectation-based robust
beamforming design for both RSMA-based coordinated STIN and cooperative STIN
with the satellite channel phase uncertainty taken into account. Nonconvex MMF
problems are formulated. A novel iterative algorithm is proposed using the SCA
together with a penalty function method to tackle the optimization. Compared
with existing literatures, this is the first work to design a robust RSMA-
based joint beamforming scheme in STIN considering the existence of satellite
channel phase uncertainty and multibeam multicast transmission. The
effectiveness of our proposed RSMA schemes for STIN is verified through
simulations.

The rest of this paper is organized as follows. The system model, channel
models and two schemes of RSMA-based STIN are introduced in Section II.
Section III investigates the design of a joint beamforming scheme to achieve
max-min fairness in the STIN with the assumption of perfect CSIT. Section IV
concentrates on a robust jont beamforming scheme in the presence of satellite
channel phase uncertainty. Simulation results are given in Section V to
evaluate the effectiveness of our proposed algorithms. Finally, we conclude
this work in Section VI.

Notations: In the remainder of this paper, boldface uppercase, boldface
lowercase and standard letters denote matrices, column vectors, and scalars
respectively. $\mathbb{R}$ and $\mathbb{C}$ denote the real and complex
domains. $\mathbb{E}\left(\cdot\right)$ is the expectation of a random
variable. The operators $\left(\cdot\right)^{T}$ and $\left(\cdot\right)^{H}$
denote the transpose and the Hermitian transpose. $\left|\cdot\right|$ and
$\left\|\cdot\right\|$ denote the absolute value and Euclidean norm
respectively.

##  II System Model

As illustrated in Fig. 1, we consider a STIN system employing full frequency
reuse where all SUs and CUs operate in the same frequency band. A
geostationary orbit (GEO) satellite is equipped with an array fed reflector
antenna. It provides services to the SUs that lack terrestrial access in
sparsely populated or remote areas. By assuming a single feed per beam (SFPB)
architecture, the array fed reflector antenna comprises a feed array with
$N_{S}$ feeds and generates $N_{S}$ adjacent beams. Within the multibeam
coverage area, we assume $K_{S}$ SUs, and $\rho=\frac{K_{s}}{N_{s}}$ users in
each beam. Since the SUs of one beam are served simultaneously by transmitting
a single coded frame following the DVB-S2X technology, the GEO satellite
implements multibeam multicast transmission. Meanwhile, the terrestrial BS
equipped with $N_{t}$-antenna uniform planar array (UPA) serves densely
populated areas in the same frequency band. $K_{t}$ unicast cellular users are
assumed. Spectrum sharing is able to improve spectrum efficiency, which also
leads to interference in and between the terrestrial and satellite sub-
networks. As shown in Fig. 1, the GW acts as a control center to collect and
manage various kinds of information, implement centralized processing and
control the whole STIN. Optimal resource allocation and interference
management on the satellite and BS can be jointly implemented at the GW to
improve system performance.

Figure 1: Model of a satellite-terrestrial integrated network.

###  II-A Satellite Channel Model

Considering the free space loss, radiation pattern and rain attenuation of
satellite channels, the downlink channel vector from the satellite to
SU-$k_{s}$ can be expressed as

$\displaystyle\mathbf{f}_{k_{s}}=\mathbf{b}_{k_{s}}\circ\mathbf{q}_{k_{s}}\circ\mathrm{exp}\left\\{j\bm{\phi}_{k_{s}}\right\\},$
(1)

where $\circ$ is the Hadamard product. $\mathbf{b}_{k_{s}}$ is composed of
free space loss and satellite beam radiation pattern. The $n_{s}$-th element
of $\mathbf{b}_{k_{s}}$ is modeled as

$\displaystyle
b_{k_{s},n_{s}}=\frac{\sqrt{G_{R}G_{k_{s},n_{s}}}}{4\pi\frac{d_{k_{s}}}{\lambda}\sqrt{\kappa
T_{sys}B_{w}}},$ (2)

where $G_{R}$ is the user terminal antenna gain, $d_{k_{s}}$ is the distance
between SU-$k_{s}$ and the satellite, $\lambda$ is the carrier wavelength,
$\kappa$ is the Boltzmann constant, $T_{sys}$ is the receiving system noise
temperature and $B_{w}$ denotes the bandwidth. In this regard, all channels
are normalized with respect to the noise standard deviation. The beam gain
from the feed-$n_{s}$ to SU-$k_{s}$ can be approximated by

$\displaystyle
G_{k_{s},n_{s}}=G_{max}\Big{[}\frac{J_{1}\left(u_{k_{s},n_{s}}\right)}{2u_{k_{s},n_{s}}}+36\frac{J_{3}\left(u_{k_{s},n_{s}}\right)}{u_{k_{s},n_{s}}^{3}}\Big{]}^{2},$
(3)

where
$u_{k_{s},n_{s}}=2.07123\sin\left(\theta_{k_{s},n_{s}}\right)/\sin\left(\theta_{\mathrm{3dB}}\right)$.
Given the $k_{s}$-th SU position, $\theta_{k_{s},n_{s}}$ is the angle between
it and the center of beam-$n_{s}$ with respect to the satellite, and
$\theta_{\mathrm{3dB}}$ is the $3\ \mathrm{dB}$ loss angle compared with the
beam center. The maximum beam gain observed at each beam center is denoted by
$G_{max}$. $J_{1}$ and $J_{3}$ are respectively the first-kind Bessel
functions with order $1$ and $3$. The rain attenuation effect is characterized
in
$\mathbf{q}_{k_{s}}=\left[q_{k_{s},1},q_{k_{s},2},\cdots,q_{k_{s},N_{S}}\right]^{T}$
with elements $q_{k_{s},n_{s}}=\chi_{k_{s},n_{s}}^{1/2}$, where the dB form
$\chi_{k_{s},n_{s}}^{dB}=20\log_{10}\left(\chi_{k_{s},n_{s}}\right)$ is
commonly modeled as a log-normal random variable, i.e.,
$\ln\small(\chi_{k_{s},n_{s}}^{dB}\small)\sim\mathcal{N}\left(\mu,\sigma\right)$.
Moreover,
$\bm{\phi}_{k_{s}}=\left[\phi_{k_{s},1},\phi_{k_{s},2},\cdots,\phi_{k_{s},N_{S}}\right]^{T}$
is a $N_{s}$-dimensional phase vector with each element uniformly distributed
between $0$ and $2\pi$. The satellite channel matrix between the satellite and
all SUs is denoted by
$\mathbf{F}=\left[\mathbf{f}_{1},\cdots,\mathbf{f}_{K_{s}}\right]\in\mathbb{C}^{N_{s}\times
K_{s}}$. Similarly, when we consider $n_{s}\in\left\\{1,\cdots,N_{s}\right\\}$
and $k_{t}\in\left\\{1,\cdots,K_{t}\right\\}$, the interfering channel matrix
between the satellite and all CUs is formulated by
$\mathbf{Z}=\left[\mathbf{z}_{1},\cdots,\mathbf{z}_{K_{t}}\right]\in\mathbb{C}^{N_{s}\times
K_{t}}$.

Figure 2: Geometry of uniform planar array employed at the BS.

###  II-B Terrestrial Channel Model

As illustrated in Fig. 2, we assume UPA at the BS with dimension
$N_{t}=N_{1}\times N_{2}$. $N_{1}$ and $N_{2}$ are respectively the number of
array elements uniformly employed along the X-axis and the Z-axis. Due to the
characteristic of radio wave propagation at high frequency band, the
terrestrial channels can be expressed by a model consisting of $L$ scatters.
Each scatter contributes to a single propagation path. Mathematically, the
terrestrial channel vector between the BS and CU
$k_{t}\in\left\\{1,\cdots,K_{t}\right\\}$ is given by

$\displaystyle\mathbf{h}_{k_{t}}=\sqrt{\frac{1}{L}}\sum_{l=1}^{L}\alpha_{k_{t},l}\mathbf{a}_{h}\left(\theta_{k_{t},l},\varphi_{k_{t},l}\right)\otimes\mathbf{a}_{v}\left(\theta_{k_{t},l}\right),$
(4)

where $\alpha_{k_{t},l}$ is the complex channel gain of the $l$-th path. Each
$\alpha_{k_{t},l}$ is assumed to follow independent and identical distribution
(i.i.d.) $\mathcal{C}\mathcal{N}\left(0,1\right)$. By denoting
$\theta_{k_{t},l}$ and $\varphi_{k_{t},l}$ as the vertical and horizontal AoD
angles of the $l$-th path, and
$\mathbf{r}_{n_{1},n_{2}}=\left[x_{n_{1}},0,z_{n_{2}}\right]^{T}$ as the
location vector of the $\small(n_{1},n_{2}\small)$-th element, the horizontal
steering vector
$\mathbf{a}_{h}\left(\theta_{k_{t},l},\varphi_{k_{t},l}\right)$ and the
vertical steering vector $\mathbf{a}_{v}\small(\theta_{k_{t},l}\small)$ can be
written as

$\displaystyle\mathbf{a}_{h}\left(\theta_{k_{t},l},\varphi_{k_{t},l}\right)=\big{[}$
$\displaystyle
e^{-j\frac{2\pi}{\lambda}\small(\frac{N_{1}-1}{2}\small)d_{1}\sin\theta_{k_{t},l}\cos\varphi_{k_{t},l}},\cdots,e^{+j\frac{2\pi}{\lambda}\small(\frac{N_{1}-1}{2}\small)d_{1}\sin\theta_{k_{t},l}\cos\varphi_{k_{t},l}}\big{]}^{T},$
(5) $\displaystyle\mathbf{a}_{v}\left(\theta_{k_{t},l}\right)=\big{[}$
$\displaystyle
e^{-j\frac{2\pi}{\lambda}\small(\frac{N_{2}-1}{2}\small)d_{2}\cos\theta_{k_{t},l}},\cdots,e^{+j\frac{2\pi}{\lambda}\small(\frac{N_{2}-1}{2}\small)d_{2}\cos\theta_{k_{t},l}}\big{]}^{T},$
(6)

The terrestrial channel matrix between the BS and all CUs is denoted by
$\mathbf{H}=\left[\mathbf{h}_{1},\cdots,\mathbf{h}_{K_{t}}\right]\in\mathbb{C}^{N_{t}\times
K_{t}}$.

###  II-C Coordinated scheme and Cooperative Scheme

In this work, we consider two levels of integration between the satellite and
terrestrial BS.

####  II-C1 Coordinated Scheme

First, we consider the basic level of integration where the CSI of both direct
and interfering links of the whole network is available at the GW, while the
transmitted data information is not shared between the satellite and BS at the
GW. We call such scheme a coordinated scheme. It allows the satellite and BS
to coordinate power allocation and beamforming directions to suppress
interference. Different multiple access strategies can be exploited at the
satellite and BS, such as RSMA, SDMA, NOMA, etc. Here, we elaborate on the
scenario where RSMA111RSMA has been shown analytically as a general multiple
access strategy, which boils down to SDMA and NOMA when allocating powers to
the different types of message streams [24]. is used at both the BS and
satellite. To that end, the unicast messages $W_{1},\cdots,W_{K_{t}}$ intended
for CUs indexed by $\mathcal{K}_{t}=\left\\{1,\cdots,K_{t}\right\\}$ are split
into common parts and private parts, i.e.,
$W_{k_{t}}\rightarrow\left\\{W_{c,k_{t}},W_{p,k_{t}}\right\\},\forall
k_{t}\in\mathcal{K}_{t}$. All common parts are combined into $W_{c}$ and
encoded into a common stream $s_{c}$ to be decoded by all CUs. The private
parts are independently encoded into private streams $s_{1},\cdots,s_{K_{t}}$.
The vector of BS streams
$\mathbf{s}=\left[s_{c},s_{1},\cdots,s_{K_{t}}\right]^{T}\in\mathbb{C}^{\left(K_{t}+1\right)\times
1}$ is therefore created, and we suppose it obeying
$\mathbb{E}\left\\{\mathbf{s}\mathbf{s}^{H}\right\\}=\mathbf{I}$. For the
satellite, multicast messages $M_{1},\cdots M_{N_{s}}$ are intended to the
beams indexed by $\mathcal{N}_{s}=\left\\{1,\cdots,N_{s}\right\\}$. Each
message $M_{n_{s}},\forall n_{s}\in\mathcal{N}_{s}$ is split into a common
part $M_{c,n_{s}}$ and a private part $M_{p,n_{s}}$. All common parts are
combined as $M_{c}$ and encoded into $m_{c}$, while the private parts are
encoded independently into $m_{1},\cdots,m_{N_{s}}$. The vector of satellite
streams
$\mathbf{m}=\left[m_{c},m_{1},\cdots,m_{N_{s}}\right]^{T}\in\mathbb{C}^{\left(N_{s}+1\right)\times
1}$ is obtained, and we assume it satisfying
$\mathbb{E}\left\\{\mathbf{m}\mathbf{m}^{H}\right\\}=\mathbf{I}$. Both
$\mathbf{s}$ and $\mathbf{m}$ are linearly precoded. The transmitted signals
at the satellite and BS are respectively

$\displaystyle\mathbf{x}^{sat}=\mathbf{w}_{c}m_{c}+\sum_{n_{s}=1}^{N_{s}}\mathbf{w}_{n_{s}}m_{n_{s}}\quad\mathrm{and}\quad\mathbf{x}^{bs}=\mathbf{p}_{c}s_{c}+\sum_{k_{t}=1}^{K_{t}}\mathbf{p}_{k_{t}}s_{k_{t}},$
(7)

where
$\mathbf{W}=\left[\mathbf{w}_{c},\mathbf{w}_{1},\cdots,\mathbf{w}_{N_{s}}\right]\in\mathbb{C}^{N_{s}\times\left(N_{s}+1\right)}$
and
$\mathbf{P}=\left[\mathbf{p}_{c},\mathbf{p}_{1},\cdots,\mathbf{p}_{K_{t}}\right]\in\mathbb{C}^{N_{t}\times\left(K_{t}+1\right)}$
are defined as the beamforming matrices at the satellite and BS. $m_{c}$ and
$s_{c}$ are superimposed on top of the private signals. Due to the lack of
flexibility in sharing energy resources among satellite feeds, per-feed
transmit power constraints of the satellite are given by
$\small(\mathbf{W}\mathbf{W}^{H}\small)_{n_{s},n_{s}}\leq\frac{P_{s}}{N_{s}},\
\forall n_{s}\in\mathcal{N}_{s}$. The sum transmit power constraint of BS is
given by $\mathrm{tr}\small(\mathbf{P}\mathbf{P}^{H}\small)\leq P_{t}$. Based
on the channel models defined above, the received signal at each SU-$k_{s}$
writes as

$y_{k_{s}}^{sat}=\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{c}m_{c}+\mathbf{f}_{k_{s}}^{H}\sum_{i=1}^{N_{s}}\mathbf{w}_{i}m_{i}+n_{k_{s}}^{sat}.$
(8)

Each SU sees multibeam interference but no interference coming form the BS
because we assume all SUs locate out of the BS serving area. The received
signal at each CU-$k_{t}$ writes as

$y_{k_{t}}^{bs}=\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{c}s_{c}+\mathbf{h}_{k_{t}}^{H}\sum_{j=1}^{K_{t}}\mathbf{p}_{j}s_{j}+\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{c}m_{c}+\mathbf{z}_{k_{t}}^{H}\sum_{i=1}^{N_{s}}\mathbf{w}_{i}m_{i}+n_{k_{t}}^{bs}.$
(9)

Each CU suffers intra-cell interference and interference from the satellite.
$\mathbf{z}_{1},\cdots,\mathbf{z}_{K{t}}$ represent satellite interfering
channels. $n_{k_{s}}^{sat}$ and $n_{k_{t}}^{bs}$ are the additive Gaussian
white noises (AWGN) with zero mean and variance $\sigma_{k_{s}}^{sat2}$ and
$\sigma_{k_{t}}^{bs2}$ respectively. For both SUs and CUs, the common stream
is first decoded while treating the other interference as noise. The SINRs of
decoding the common stream at SU-$k_{s}$ and CU-$k_{t}$ are given by

$\displaystyle\gamma_{c,k_{s}}^{sat}$
$\displaystyle=\frac{\left|\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{c}\right|^{2}}{\sum_{i=1}^{N_{s}}\left|\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{i}\right|^{2}+\sigma_{k_{s}}^{sat2}},$
(10) $\displaystyle\gamma_{c,k_{t}}^{bs}$
$\displaystyle=\frac{\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{c}\right|^{2}}{\sum_{j=1}^{K_{t}}\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{j}\right|^{2}+\left|\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{c}\right|^{2}+\sum_{i=1}^{N_{s}}\left|\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{i}\right|^{2}+\sigma_{k_{t}}^{bs2}}.$
(11)

The corresponding rates are
$R_{c,k_{s}}^{sat}=\log_{2}\small(1+\gamma_{c,k_{s}}^{sat}\small)$ and
$R_{c,k_{t}}^{bs}=\log_{2}\small(1+\gamma_{c,k_{t}}^{bs}\small)$. To guarantee
that each SU is capable of decoding $m_{c}$, and each CU is capable of
decoding $s_{c}$, we define the common rates as

$\displaystyle
R_{c}^{sat}=\min_{k_{s}\in\mathcal{K}_{s}}\left\\{R_{c,k_{s}}^{sat}\right\\}=\sum_{n_{s}=1}^{N_{s}}C_{n_{s}}^{sat}\quad\mathrm{and}\quad
R_{c}^{bs}=\min_{k_{t}\in\mathcal{K}_{t}}\left\\{R_{c,k_{t}}^{bs}\right\\}=\sum_{k_{t}=1}^{K_{t}}C_{k_{t}}^{bs},$
(12)

where $C_{n_{s}}^{sat}$ is the rate of the common part of the $n_{s}$-th
beam’s message. $C_{k_{t}}^{bs}$ is the rate of the common part of the
$k_{t}$-th CU’s message. After the common stream is re-encoded, precoded and
subtracted from the received signal through SIC, each user then decodes its
desired private stream. We define
$\mu:\mathcal{K}_{s}\rightarrow\mathcal{N}_{s}$ as mapping an SU to its
corresponding beam. The SINRs of decoding $m_{\mu\left(k_{s}\right)}$ at
SU-$k_{s}$ and decoding $s_{k_{t}}$ at CU-$k_{t}$ are given by

$\displaystyle\gamma_{k_{s}}^{sat}$
$\displaystyle=\frac{\left|\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{\mu\left(k_{s}\right)}\right|^{2}}{\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\left|\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{i}\right|^{2}+\sigma_{k_{s}}^{sat2}},$
(13) $\displaystyle\gamma_{k_{t}}^{bs}$
$\displaystyle=\frac{\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{k_{t}}\right|^{2}}{\sum_{j=1,j\neq
k_{t}}^{K_{t}}\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{j}\right|^{2}+\left|\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{c}\right|^{2}+\sum_{i=1}^{N_{s}}\left|\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{i}\right|^{2}+\sigma_{k_{t}}^{bs2}}.$
(14)

The private rates are respectively
$R_{k_{s}}^{sat}=\log_{2}\small(1+\gamma_{k_{s}}^{sat}\small)$ and
$R_{k_{t}}^{bs}=\log_{2}\small(1+\gamma_{k_{t}}^{bs}\small)$. Thus, the
$n_{s}$-th beam rate and the $k_{t}$-th CU rate write as

$\displaystyle
R^{sat}_{\mathrm{tot},n_{s}}=C^{sat}_{n_{s}}+\min_{i\in\mathcal{G}_{n_{s}}}R^{sat}_{i}\quad\mathrm{and}\quad
R^{bs}_{\mathrm{tot},k_{t}}=C^{bs}_{k_{t}}+R^{bs}_{k_{t}},$ (15)

where $\mathcal{G}_{n_{s}}$ denote the set of SUs belonging to the $n_{s}$-th
beam.

####  II-C2 Cooperative Scheme

Second, we consider a higher level of integration, i.e., cooperative scheme
where the satellite and BS share both CSI and data information at the GW. In
this scenario, all downlink messages $W_{1},\cdots,W_{K_{t}}$ intended for CUs
and multicast messages $M_{1},\cdots,M_{N_{s}}$ intended for SUs are
transmitted at both the satellite and BS. All propagation links (including
interfering ones) are exploited to carry useful data upon appropriate
beamforming. We still consider RSMA to manage interference in this cooperative
STIN, including inter-beam interference, intra-cell interference and
interference between the satellite and terrestrial sub-networks. Each message
is split into a common part and a private part. All common parts are encoded
together into a super common stream shared by all users in the system. As a
result, the symbol stream to be transmitted is given by
$\acute{\mathbf{s}}=\left[\acute{s}_{c},\acute{m}_{1},\cdots,\acute{m}_{N_{s}},\acute{s}_{1},\cdots,\acute{s}_{K_{t}}\right]^{T}\in\mathcal{C}^{N_{s}+K_{t}+1}$.
Throughout this work, we use $``\ \acute{}\ "$ to differentiate notations in
the cooperative scheme and the above coordinated scheme. The transmitted
signals at the satellite writes as

$\acute{\mathbf{x}}^{sat}=\acute{\mathbf{w}}_{c}\acute{s}_{c}+\sum_{i=1}^{N_{s}}\acute{\mathbf{w}}_{i}^{sat}\acute{m}_{i}+\sum_{j=1}^{K_{t}}\acute{\mathbf{w}}_{j}^{bs}\acute{s}_{j},$
(16)

where
$\acute{\mathbf{W}}=\left[\acute{\mathbf{w}}_{c},\acute{\mathbf{w}}_{1}^{sat},\cdots,\acute{\mathbf{w}}_{N_{s}}^{sat},\acute{\mathbf{w}}_{1}^{bs},\cdots,\acute{\mathbf{w}}_{K_{t}}^{bs}\right]$
is the beamforming matrix, and the superscripts of
$\acute{\mathbf{w}}_{i}^{sat}$ and $\acute{\mathbf{w}}_{j}^{bs}$ are used to
differentiate the precoder of satellite data and BS data. Per-feed transmit
power constraints write as
$\small(\acute{\mathbf{W}}\acute{\mathbf{W}}^{H}\small)_{n_{s},n_{s}}\leq\frac{P_{s}}{N_{s}},\
\forall n_{s}\in\mathcal{N}_{s}$. Similarly, the transmitted signal at the BS
writes as

$\acute{\mathbf{x}}^{bs}=\acute{\mathbf{p}}_{c}\acute{s}_{c}+\sum_{i=1}^{N_{s}}\acute{\mathbf{p}}_{i}^{sat}\acute{m}_{i}+\sum_{j=1}^{K_{t}}\acute{\mathbf{p}}_{j}^{bs}\acute{s}_{j},$
(17)

where
$\acute{\mathbf{P}}=\left[\acute{\mathbf{p}}_{c},\acute{\mathbf{p}}_{1}^{sat},\cdots,\acute{\mathbf{p}}_{N_{s}}^{sat},\acute{\mathbf{p}}_{1}^{bs},\cdots,\acute{\mathbf{p}}_{K_{t}}^{bs}\right]$
is the beamforming matrix, and the sum transmit power constraint of the BS is
$\mathrm{tr}\small(\acute{\mathbf{P}}\acute{\mathbf{P}}^{H}\small)\leq P_{t}$.
Accordingly, the signal received at SU-$k_{s}$ becomes

$\acute{y}_{k_{s}}^{sat}=\mathbf{f}_{k_{s}}^{H}\acute{\mathbf{w}}_{c}\acute{s}_{c}+\mathbf{f}_{k_{s}}^{H}\sum_{i=1}^{N_{s}}\acute{\mathbf{w}}_{i}^{sat}\acute{m}_{i}+\mathbf{f}_{k_{s}}^{H}\sum_{j=1}^{K_{t}}\acute{\mathbf{w}}_{j}^{bs}\acute{s}_{j}+\acute{n}_{k_{s}}^{sat}.$
(18)

The received signal at CU-$k_{t}$ is given by

$\displaystyle\acute{y}_{k_{t}}^{bs}$
$\displaystyle=\mathbf{h}_{k_{t}}^{H}\acute{\mathbf{p}}_{c}\acute{s}_{c}+\mathbf{h}_{k_{t}}^{H}\sum_{j=1}^{K_{t}}\acute{\mathbf{p}}_{j}^{bs}\acute{s}_{j}+\mathbf{h}_{k_{t}}^{H}\sum_{i=1}^{N_{s}}\acute{\mathbf{p}}_{i}^{sat}\acute{m}_{i}+\mathbf{z}_{k_{t}}^{H}\acute{\mathbf{w}}_{c}\acute{s}_{c}+\mathbf{z}_{k_{t}}^{H}\sum_{i=1}^{N_{s}}\acute{\mathbf{w}}_{i}^{sat}\acute{m}_{i}+\mathbf{z}_{k_{t}}^{H}\sum_{j=1}^{K_{t}}\acute{\mathbf{w}}_{j}^{bs}\acute{s}_{j}+\acute{n}_{k_{t}}^{bs}.$
(19)

To simplify (19), aggregate channels and aggregate beamforming vectors are
defined by

$\displaystyle\mathbf{g}_{k_{t}}$
$\displaystyle=\left[\mathbf{z}_{k_{t}}^{H},\mathbf{h}_{k_{t}}^{H}\right]^{H}\in\mathbb{C}^{\left(N_{s}+N_{t}\right)\times
1},\ \forall k_{t}\in\mathcal{K}_{t},$ (20) $\displaystyle\mathbf{v}_{c}$
$\displaystyle=\left[\mathbf{w}_{c}^{*H},\mathbf{p}_{c}^{*H}\right]^{H}\in\mathbb{C}^{\left(N_{s}+N_{t}\right)\times
1},$ (21) $\displaystyle\mathbf{v}_{n_{s}}^{sat}$
$\displaystyle=\left[\acute{\mathbf{w}}_{n_{s}}^{satH},\acute{\mathbf{p}}_{n_{s}}^{satH}\right]^{H}\in\mathbb{C}^{\left(N_{s}+N_{t}\right)\times
1},\ \forall n_{s}\in\mathcal{N}_{s},$ (22)
$\displaystyle\mathbf{v}_{k_{t}}^{bs}$
$\displaystyle=\left[\acute{\mathbf{w}}_{k_{t}}^{bsH},\acute{\mathbf{p}}_{k_{t}}^{bsH}\right]^{H}\in\mathbb{C}^{\left(N_{s}+N_{t}\right)\times
1},\ \forall k_{t}\in\mathcal{K}_{t}.$ (23)

The received signal at the $k_{t}$-th CU can be rewritten as

$\acute{y}_{k_{t}}^{bs}=\mathbf{g}_{k_{t}}^{H}\mathbf{v}_{c}\acute{s}_{c}+\mathbf{g}_{k_{t}}^{H}\sum_{j=1}^{K_{t}}\mathbf{v}_{j}^{bs}\acute{s}_{j}+\mathbf{g}_{k_{t}}^{H}\sum_{i=1}^{N_{s}}\mathbf{v}_{i}^{sat}\acute{m}_{i}+\acute{n}_{k{t}}^{bs}.$
(24)

From this expression, we can regard the satellite and BS working together as a
super $``\mathrm{BS}"$ but subject to their respective power constraints to
serve the CUs and SUs. Satellite interfering links are exploited to carry
terrestrial data so as to improve the performance of STIN. The aggregate
beamforming vectors are collected into a matrix
$\mathbf{V}=\small[\mathbf{v}_{c},\mathbf{v}_{1}^{sat},\cdots,\mathbf{v}_{N_{s}}^{sat},\mathbf{v}_{1}^{bs},\cdots,\mathbf{v}_{K_{t}}^{bs}\small]\in\mathbb{C}^{\left(N_{s}+N_{t}\right)\times\left(N_{s}+K_{t}+1\right)}$.
It can also be denoted by
$\mathbf{V}=\small[\acute{\mathbf{W}}^{H},\acute{\mathbf{P}}^{H}\small]^{H}$.

For both SUs and CUs, the common stream is firstly decoded and removed from
the received signal through SIC. The SINRs of decoding $\acute{s}_{c}$ at the
$k_{s}$-th SU and the $k_{t}$-th CU are respectively

$\displaystyle\acute{\gamma}_{c,k_{s}}^{sat}$
$\displaystyle=\frac{\left|\mathbf{f}_{k_{s}}^{H}\acute{\mathbf{w}}_{c}\right|^{2}}{\sum_{i=1}^{N_{s}}\left|\mathbf{f}_{k_{s}}^{H}\acute{\mathbf{w}}^{sat}_{i}\right|^{2}+\sum_{j=1}^{K_{t}}\left|\mathbf{f}_{k_{s}}^{H}\acute{\mathbf{w}}^{bs}_{j}\right|^{2}+\sigma_{k_{s}}^{sat2}},$
(25) $\displaystyle\acute{\gamma}_{c,k_{t}}^{bs}$
$\displaystyle=\frac{\left|\mathbf{g}_{k_{t}}^{H}\mathbf{v}_{c}\right|^{2}}{\sum_{j=1}^{K_{t}}\left|\mathbf{g}_{k_{t}}^{H}\mathbf{v}_{j}^{bs}\right|^{2}+\sum_{i=1}^{N_{s}}\left|\mathbf{g}_{k_{t}}^{H}\mathbf{v}_{i}^{sat}\right|^{2}+\sigma_{k_{t}}^{bs2}}.$
(26)

The corresponding rates are
$\acute{R}_{c,k_{s}}^{sat}=\log_{2}\small(1+\acute{\gamma}_{c,k_{s}}^{sat}\small)$
and
$\acute{R}_{c,k_{t}}^{bs}=\log_{2}\small(1+\acute{\gamma}_{c,k_{t}}^{bs}\small)$.
Since $\acute{s}_{c}$ is decoded by all users in the system, we define the
common rate as

$\displaystyle\acute{R}_{c}=\min_{k_{s}\in\mathcal{K}_{s},k_{t}\in\mathcal{K}_{t}}\big{\\{}\acute{R}_{c,k_{s}}^{sat},\acute{R}_{c,k_{t}}^{bs}\big{\\}}=\sum_{n_{s}=1}^{N_{s}}\acute{C}_{n_{s}}^{sat}+\sum_{k_{t}=1}^{K_{t}}\acute{C}_{k_{t}}^{bs}.$
(27)

Note that $\acute{s}_{c}$ is shared among all satellite beams and cellular
users. $\acute{C}_{n_{s}}^{sat}$ and $\acute{C}_{k_{t}}^{bs}$ respectively
correspond to the beam-$n_{s}$’s and CU-$k_{t}$’s portion of common rate.
After removing $\acute{s}_{c}$ using SIC, each user then decodes its desired
private stream. The SINRs of decoding private streams are

$\displaystyle\acute{\gamma}_{k_{s}}^{sat}$
$\displaystyle=\frac{\left|\mathbf{f}_{k_{s}}^{H}\acute{\mathbf{w}}^{sat}_{\mu\left(k_{s}\right)}\right|^{2}}{\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\left|\mathbf{f}_{k_{s}}^{H}\acute{\mathbf{w}}^{sat}_{i}\right|^{2}+\sum_{j=1}^{K_{t}}\left|\mathbf{f}_{k_{s}}^{H}\acute{\mathbf{w}}^{bs}_{j}\right|^{2}+\sigma_{k_{s}}^{sat2}},$
(28) $\displaystyle\acute{\gamma}_{k_{t}}^{bs}$
$\displaystyle=\frac{\left|\mathbf{g}_{k_{t}}^{H}\mathbf{v}_{k_{t}}^{bs}\right|^{2}}{\sum_{j=1,j\neq
k_{t}}^{K_{t}}\left|\mathbf{g}_{k_{t}}^{H}\mathbf{v}_{j}^{bs}\right|^{2}+\sum_{i=1}^{N_{s}}\left|\mathbf{g}_{k_{t}}^{H}\mathbf{v}_{i}^{sat}\right|^{2}+\sigma_{k_{t}}^{bs2}}.$
(29)

$\acute{R}_{k_{s}}^{sat}=\log_{2}\small(1+\acute{\gamma}_{k_{s}}^{sat}\small)$
and
$\acute{R}_{k_{t}}^{bs}=\log_{2}\small(1+\acute{\gamma}_{k_{t}}^{bs}\small)$
are the private rates. The $n_{s}$-th beam rate and the $k_{t}$-th CU rate
write as

$\displaystyle\acute{R}^{sat}_{\mathrm{tot},n_{s}}=\acute{C}^{sat}_{n_{s}}+\min_{i\in\mathcal{G}_{n_{s}}}\acute{R}^{sat}_{i}\quad\mathrm{and}\quad\acute{R}^{bs}_{\mathrm{tot},k_{t}}=\acute{C}^{bs}_{k_{t}}+\acute{R}^{bs}_{k_{t}}.$
(30)

##  III Proposed Joint Beamforming Scheme

In this section, the problem of interest is to design a joint beamforming
scheme to maximize the minimum fairness rate amongst all unicast CUs and
multibeam multicast SUs subject to transmit power constraints. We respectively
consider the scenarios of RSMA-based coordinated STIN and cooperative STIN
with perfect CSI at the GW.

###  III-A Joint Beamforming Design for Coordinated STIN

For RSMA-based coordinated STIN, the optimization problem of achieving max-min
rate fairness among all users can be formulated as

$\displaystyle\mathcal{P}_{1}:\
\max_{\mathbf{W},\mathbf{P},\mathbf{c}^{sat},\mathbf{c}^{bs}}$
$\displaystyle\min_{n_{s}\in\mathcal{N}_{s},k_{t}\in\mathcal{K}_{t}}\big{\\{}R^{bs}_{\mathrm{tot},k_{t}},R^{sat}_{\mathrm{tot},n_{s}}\big{\\}}$
(31) $\displaystyle s.t.\qquad$ $\displaystyle
R_{c,k_{t}}^{bs}\geq\sum_{j=1}^{K_{t}}C_{j}^{bs},\quad\forall
k_{t}\in\mathcal{K}_{t}$ (32) $\displaystyle C_{k_{t}}^{bs}\geq 0,\quad\forall
k_{t}\in\mathcal{K}_{t}$ (33)
$\displaystyle\mathrm{tr}\small(\mathbf{P}\mathbf{P}^{H}\small)\leq P_{t}$
(34) $\displaystyle
R_{c,k_{s}}^{sat}\geq\sum_{j=1}^{N_{s}}C_{j}^{sat},\quad\forall
k_{s}\in\mathcal{K}_{s}$ (35) $\displaystyle C_{n_{s}}^{sat}\geq
0,\quad\forall n_{s}\in\mathcal{N}_{s}$ (36)
$\displaystyle\small(\mathbf{W}\mathbf{W}^{H}\small)_{n_{s},n_{s}}\leq\frac{P_{s}}{N_{s}},\quad\forall
n_{s}\in\mathcal{N}_{s}$ (37)

where $\mathbf{c}^{sat}=\small[C_{1}^{sat},\cdots,C_{N_{s}}^{sat}\small]^{T}$,
$\mathbf{c}^{bs}=\small[C_{1}^{bs},\cdots,C_{K_{t}}^{bs}\small]^{T}$ are the
vectors of common rate portions. (32) guarantees that the common stream
$s_{c}$ can be decoded by all CUs. (34) is the sum transmit power constraint
of the BS. Similarly, (35) ensures the common stream $m_{c}$ to be decoded by
all SUs. (37) represents per-feed power budgets of the satellite. (33) and
(36) guarantee the non-negativity of all common rate portions. Note that the
formulated problem is nonconvex, we exploit an SCA-based method to convexify
the nonconvex constraints and iteratively approximate the nonconvex problem to
a convex one. First, we introduce an equivalent reformulation of
$\mathcal{P}_{1}$, which is

$\displaystyle\mathcal{E}_{1}:\qquad$
$\displaystyle\max_{\mathbf{W},\mathbf{P},\mathbf{c}^{sat},\mathbf{c}^{bs},q,\mathbf{r},\bm{\alpha}}q$
(38) $\displaystyle s.t.\qquad$ $\displaystyle
C_{k_{t}}^{bs}+\alpha_{k_{t}}\geq q,\quad\forall k_{t}\in\mathcal{K}_{t}$ (39)
$\displaystyle R_{k_{t}}^{bs}\geq\alpha_{k_{t}},\quad\forall
k_{t}\in\mathcal{K}_{t}$ (40) $\displaystyle C_{n_{s}}^{sat}+r_{k_{s}}\geq
q,\quad\forall k_{s}\in\mathcal{G}_{n_{s}}$ (41) $\displaystyle
R_{k_{s}}^{sat}\geq r_{k_{s}},\quad\forall k_{s}\in\mathcal{K}_{s}$ (42)
$\displaystyle(\ref{p1_constraint1})-(\ref{p1_constraint6})$

where $q$, $\bm{\alpha}=\left[\alpha_{1},\cdots,\alpha_{K_{t}}\right]^{T}$,
$\mathbf{r}=\left[r_{1},\cdots,r_{K_{s}}\right]^{T}$ are newly introduced
auxiliary variables. To deal with the nonconvexity of constraint (32), (35),
(40), (42), we further introduce new auxiliary variables
$\mathbf{a}=\left[a_{1},\cdots,a_{K_{t}}\right]^{T}$,
$\mathbf{a}_{c}=\left[a_{c,1},\cdots,a_{c,K_{t}}\right]^{T}$,
$\mathbf{b}=\left[b_{1},\cdots,b_{K_{s}}\right]^{T}$ and
$\mathbf{b}_{c}=\left[b_{c,1},\cdots,b_{c,K_{s}}\right]^{T}$. The problem
$\mathcal{E}_{1}$ is rewritten by

$\displaystyle\mathcal{S}_{1}:\quad$
$\displaystyle\max_{q,\mathbf{W},\mathbf{P},\mathbf{c}^{sat},\mathbf{c}^{bs},\mathbf{r},\bm{\alpha},\mathbf{a},\mathbf{a}_{c},\mathbf{b},\mathbf{b}_{c}}q$
(43) $\displaystyle s.t.\qquad$
$\displaystyle\log\left(1+a_{k_{t}}\right)\geq\alpha_{k_{t}}\log
2,\quad\forall k_{t}\in\mathcal{K}_{t}$ (44)
$\displaystyle\gamma_{k_{t}}^{bs}\geq a_{k_{t}},\quad\forall
k_{t}\in\mathcal{K}_{t}$ (45) $\displaystyle\log\left(1+b_{k_{s}}\right)\geq
r_{k_{s}}\log 2,\quad\forall k_{s}\in\mathcal{K}_{s}$ (46)
$\displaystyle\gamma_{k_{s}}^{sat}\geq b_{k_{s}},\quad\forall
k_{s}\in\mathcal{K}_{s}$ (47)
$\displaystyle\log\left(1+a_{c,k_{t}}\right)\geq\sum_{j=1}^{K_{t}}C_{j}^{bs}\log
2,\quad\forall k_{t}\in\mathcal{K}_{t}$ (48)
$\displaystyle\gamma_{c,k_{t}}^{bs}\geq a_{c,k_{t}},\quad\forall
k_{t}\in\mathcal{K}_{t}$ (49)
$\displaystyle\log\left(1+b_{c,k_{s}}\right)\geq\sum_{j=1}^{N_{s}}C_{j}^{sat}\log
2,\quad\forall k_{s}\in\mathcal{K}_{s}$ (50)
$\displaystyle\gamma_{c,k_{s}}^{sat}\geq b_{c,k_{s}},\quad\forall
k_{s}\in\mathcal{K}_{s}$ (51)
$\displaystyle(\ref{p1_constraint2}),(\ref{p1_constraint3}),(\ref{p1_constraint5}),(\ref{p1_constraint6}),(\ref{e1_constraint1}),(\ref{e1_constraint3})$

where (44) - (51) are obtained by expanding the nonconvex rate constraints in
$\mathcal{E}_{1}$. Since the constraints of $\mathcal{S}_{1}$ hold with
equality at optimality, the equivalence between $\mathcal{P}_{1}$ and
$\mathcal{S}_{1}$ can be guaranteed. Now, we observe that the nonconvexity of
$\mathcal{S}_{1}$ is due to (45), (47), (49) and (51) which contain SINR
expressions. The constraint (45) can be expanded as

$\displaystyle\sum_{j=1,j\neq
k_{t}}^{K_{t}}\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{j}\right|^{2}+\left|\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{c}\right|^{2}+\sum_{i=1}^{N_{s}}\left|\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{i}\right|^{2}+\sigma_{k_{t}}^{bs2}\leq\frac{\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{k_{t}}\right|^{2}}{a_{k_{t}}},$
(52)

where the right-hand side (RHS) quadratic-over-linear function is convex, thus
(52) is nonconvex. Next, we need to approximate
$\frac{\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{k_{t}}\right|^{2}}{a_{k_{t}}}$
with its lower bound, which is obtained by the first-order Taylor series
approximation around the point
$\big{(}\mathbf{p}_{k_{t}}^{\left[n\right]},a_{k_{t}}^{\left[n\right]}\big{)}$.
Then, we have

$\displaystyle\frac{\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{k_{t}}\right|^{2}}{a_{k_{t}}}$
$\displaystyle\geq\frac{2\mathcal{R}\big{(}\mathbf{p}_{k_{t}}^{\left[n\right]H}\mathbf{h}_{k_{t}}\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{k_{t}}\big{)}}{a_{k_{t}}^{\left[n\right]}}-\frac{\mathbf{p}_{k_{t}}^{\left[n\right]H}\mathbf{h}_{k_{t}}\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{k_{t}}^{\left[n\right]}}{\big{(}a_{k_{t}}^{\left[n\right]}\big{)}^{2}}a_{k_{t}}\triangleq\widehat{f_{1}}\big{(}\mathbf{p}_{k_{t}},a_{k_{t}};\mathbf{p}_{k_{t}}^{\left[n\right]},a_{k_{t}}^{\left[n\right]}\big{)}$
(53)

where $n$ represents the $n$-th SCA iteration. Replacing the linear
approximation
$\widehat{f_{1}}\big{(}\mathbf{p}_{k_{t}},a_{k_{t}};\mathbf{p}_{k_{t}}^{\left[n\right]},a_{k_{t}}^{\left[n\right]}\big{)}$
with the RHS of (52) yields

$\displaystyle\sum_{j=1,j\neq
k_{t}}^{K_{t}}\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{j}\right|^{2}+\left|\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{c}\right|^{2}+\sum_{i=1}^{N_{s}}\left|\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{i}\right|^{2}+\sigma_{k_{t}}^{bs2}-\widehat{f_{1}}\big{(}\mathbf{p}_{k_{t}},a_{k_{t}};\mathbf{p}_{k_{t}}^{\left[n\right]},a_{k_{t}}^{\left[n\right]}\big{)}\leq
0.$ (54)

Similarly, the constraint (47) can be expanded as

$\displaystyle\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\left|\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{i}\right|^{2}+\sigma_{k_{s}}^{sat2}\leq\frac{\left|\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{\mu\left(k_{s}\right)}\right|^{2}}{b_{k_{s}}}.$
(55)

We approximate its RHS around the point
$\big{(}\mathbf{w}_{\mu\left(k_{s}\right)}^{\left[n\right]},b_{k_{s}}^{\left[n\right]}\big{)}$,
and obtain

$\displaystyle\frac{\left|\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{\mu\left(k_{s}\right)}\right|^{2}}{b_{k_{s}}}\geq\frac{2\mathcal{R}\big{(}\mathbf{w}_{\mu\left(k_{s}\right)}^{\left[n\right]H}\mathbf{f}_{k_{s}}\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{\mu\left(k_{s}\right)}\big{)}}{b_{k_{s}}^{\left[n\right]}}$
$\displaystyle-\frac{\mathbf{w}_{\mu\left(k_{s}\right)}^{\left[n\right]H}\mathbf{f}_{k_{s}}\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{\mu\left(k_{s}\right)}^{\left[n\right]}}{\big{(}b_{k_{s}}^{\left[n\right]}\big{)}^{2}}b_{k_{s}}\triangleq\widehat{f_{2}}\big{(}\mathbf{w}_{\mu\left(k_{s}\right)},b_{k_{s}};\mathbf{w}_{\mu\left(k_{s}\right)}^{\left[n\right]},b_{k_{s}}^{\left[n\right]}\big{)}.$
(56)

We replace
$\widehat{f_{2}}\big{(}\mathbf{w}_{\mu\left(k_{s}\right)},b_{k_{s}};\mathbf{w}_{\mu\left(k_{s}\right)}^{\left[n\right]},b_{k_{s}}^{\left[n\right]}\big{)}$
with the RHS of (55) and yield

$\displaystyle\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\left|\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{i}\right|^{2}+\sigma_{k_{s}}^{sat2}-\widehat{f_{2}}\big{(}\mathbf{w}_{\mu\left(k_{s}\right)},b_{k_{s}};\mathbf{w}_{\mu\left(k_{s}\right)}^{\left[n\right]},b_{k_{s}}^{\left[n\right]}\big{)}\leq
0.$ (57)

Following the same logic, (49) and (51) are respectively approximated by

$\displaystyle\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{k_{t}}\right|^{2}+\sum_{j=1,j\neq
k_{t}}^{K_{t}}\left|\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{j}\right|^{2}+\left|\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{c}\right|^{2}+\sum_{i=1}^{N_{s}}\left|\mathbf{z}_{k_{t}}^{H}\mathbf{w}_{i}\right|^{2}+\sigma_{k_{t}}^{bs2}-\widehat{f_{3}}\big{(}\mathbf{p}_{c},a_{c,k_{t}};\mathbf{p}_{c}^{\left[n\right]},a_{c,k_{t}}^{\left[n\right]}\big{)}\leq
0,$ (58)
$\displaystyle\left|\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{\mu\left(k_{s}\right)}\right|^{2}+\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\left|\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{i}\right|^{2}+\sigma_{k_{s}}^{sat2}-\widehat{f_{4}}\big{(}\mathbf{w}_{c},b_{c,k_{s}};\mathbf{w}_{c}^{\left[n\right]},b_{c,k_{s}}^{\left[n\right]}\big{)}\leq
0,$ (59)

where
$\widehat{f_{3}}\big{(}\mathbf{p}_{c},a_{c,k_{t}};\mathbf{p}_{c}^{\left[n\right]},a_{c,k_{t}}^{\left[n\right]}\big{)}$
and
$\widehat{f_{4}}\big{(}\mathbf{w}_{c},b_{c,k_{s}};\mathbf{w}_{c}^{\left[n\right]},b_{c,k_{s}}^{\left[n\right]}\big{)}$
are linear lower bound expressions

$\displaystyle\widehat{f_{3}}\big{(}\mathbf{p}_{c},a_{c,k_{t}};\mathbf{p}_{c}^{\left[n\right]},a_{c,k_{t}}^{\left[n\right]}\big{)}\triangleq\frac{2\mathcal{R}\big{(}\mathbf{p}_{c}^{\left[n\right]H}\mathbf{h}_{k_{t}}\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{c}\big{)}}{a_{c,k_{t}}^{\left[n\right]}}-\frac{\mathbf{p}_{c}^{\left[n\right]H}\mathbf{h}_{k_{t}}\mathbf{h}_{k_{t}}^{H}\mathbf{p}_{c}^{\left[n\right]}}{\big{(}a_{c,k_{t}}^{\left[n\right]}\big{)}^{2}}a_{c,k_{t}},$
(60)
$\displaystyle\widehat{f_{4}}\big{(}\mathbf{w}_{c},b_{c,k_{s}};\mathbf{w}_{c}^{\left[n\right]},b_{c,k_{s}}^{\left[n\right]}\big{)}\triangleq\frac{2\mathcal{R}\big{(}\mathbf{w}_{c}^{\left[n\right]H}\mathbf{f}_{k_{s}}\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{c}\big{)}}{b_{c,k_{s}}^{\left[n\right]}}-\frac{\mathbf{w}_{c}^{\left[n\right]H}\mathbf{f}_{k_{s}}\mathbf{f}_{k_{s}}^{H}\mathbf{w}_{c}^{\left[n\right]}}{\big{(}b_{c,k_{s}}^{\left[n\right]}\big{)}^{2}}b_{c,k_{s}}.$
(61)

Although (44), (46), (48) and (50) are convex constraints and solvable through
the CVX toolbox, the $\log$ terms belong to generalized nonlinear convex
program, which leads to high computational complexity. Aiming at more
efficient implementation, [47] approximates the $\log$ constraints to a set of
second-order cone (SOC) constraints, which introduce a great number of slack
variables and result in an increase of per-iteration complexity. Here, we use
the property that $x\log\left(1+x\right)$ is convex as in [48], and
approximate (44), (46), (48), (50) without additional slack variables. Since
$a_{k_{t}}\geq 0$, the constraint (44) can be rewritten as

$\displaystyle a_{k_{t}}\log\left(1+a_{k_{t}}\right)\geq
a_{k_{t}}\alpha_{k_{t}}\log 2.$ (62)

Its left hand side (LHS) is convex, so we compute the first-order Taylor
series approximation of $a_{k_{t}}\log\left(1+a_{k_{t}}\right)$ around the
point $a_{k_{t}}^{\left[n\right]}$ as

$\displaystyle a_{k_{t}}\log\left(1+a_{k_{t}}\right)\geq
a_{k_{t}}^{\left[n\right]}\log\big{(}1+a_{k_{t}}^{\left[n\right]}\big{)}+\big{(}a_{k_{t}}-a_{k_{t}}^{\left[n\right]}\big{)}$
$\displaystyle\Big{[}\frac{a_{k_{t}}^{\left[n\right]}}{1+a_{k_{t}}^{\left[n\right]}}+\log\big{(}1+a_{k_{t}}^{\left[n\right]}\big{)}\Big{]}=a_{k_{t}}v_{k_{t}}^{\left[n\right]}-u_{k_{t}}^{\left[n\right]},$
(63)

where $v_{k_{t}}^{\left[n\right]}$ and $u_{k_{t}}^{\left[n\right]}$ are
expressions of $a_{k_{t}}^{\left[n\right]}$ given by

$v_{k_{t}}^{\left[n\right]}=\frac{a_{k_{t}}^{\left[n\right]}}{a_{k_{t}}^{\left[n\right]}+1}+\log\big{(}1+a_{k_{t}}^{\left[n\right]}\big{)}\
\mathrm{and}\
u_{k_{t}}^{\left[n\right]}=\frac{\big{(}a_{k_{t}}^{\left[n\right]}\big{)}^{2}}{a_{k_{t}}^{\left[n\right]}+1}.$
(64)

Now, (62) can be rewritten by
$a_{k_{t}}v_{k_{t}}^{\left[n\right]}-u_{k_{t}}^{\left[n\right]}\geq
a_{k_{t}}\alpha_{k_{t}}\log 2$, which is SOC representable as

$\Big{\|}\big{[}a_{k_{t}}+\alpha_{k_{t}}\log 2-v_{k_{t}}^{\left[n\right]}\quad
2\sqrt{u_{k_{t}}^{\left[n\right]}}\big{]}\Big{\|}_{2}\leq
a_{k_{t}}-\alpha_{k_{t}}\log 2+v_{k_{t}}^{\left[n\right]}.$ (65)

Similarly, the constraint (46), (48), (50) can be replaced by

$\displaystyle\Big{\|}\big{[}b_{k_{s}}+r_{k_{s}}\log
2-\bar{v}_{k_{s}}^{\left[n\right]}\quad
2\sqrt{\bar{u}_{k_{s}}^{\left[n\right]}}\big{]}\Big{\|}_{2}\leq
b_{k_{s}}-r_{k_{s}}\log 2+\bar{v}_{k_{s}}^{\left[n\right]},$ (66)
$\displaystyle\Big{\|}\big{[}a_{c,k_{t}}+\sum_{j=1}^{K_{t}}C_{j}^{bs}\log
2-v_{c,k_{t}}^{\left[n\right]}\quad
2\sqrt{u_{c,k_{t}}^{\left[n\right]}}\big{]}\Big{\|}_{2}\leq
a_{c,k_{t}}-\sum_{j=1}^{K_{t}}C_{j}^{bs}\log 2+v_{c,k_{t}}^{\left[n\right]},$
(67)
$\displaystyle\Big{\|}\big{[}b_{c,k_{s}}+\sum_{j=1}^{N_{s}}C_{j}^{sat}\log
2-\bar{v}_{c,k_{s}}^{\left[n\right]}\quad
2\sqrt{\bar{u}_{c,k_{s}}^{\left[n\right]}}\big{]}\Big{\|}_{2}\leq
b_{c,k_{s}}-\sum_{j=1}^{N_{s}}C_{j}^{sat}\log
2+\bar{v}_{c,k_{s}}^{\left[n\right]}.$ (68)

The expressions of $\bar{v}_{k_{s}}^{\left[n\right]}$,
$\bar{u}_{k_{s}}^{\left[n\right]}$, $v_{c,k_{t}}^{\left[n\right]}$,
$u_{c,k_{t}}^{\left[n\right]}$, $\bar{v}_{c,k_{s}}^{\left[n\right]}$,
$\bar{u}_{c,k_{s}}^{\left[n\right]}$ are respectively

$\displaystyle\bar{v}_{k_{s}}^{\left[n\right]}=\frac{b_{k_{s}}^{\left[n\right]}}{b_{k_{s}}^{\left[n\right]}+1}+\log\big{(}1+b_{k_{s}}^{\left[n\right]}\big{)}\
\mathrm{and}\
\bar{u}_{k_{s}}^{\left[n\right]}=\frac{\big{(}b_{k_{s}}^{\left[n\right]}\big{)}^{2}}{b_{k_{s}}^{\left[n\right]}+1},$
$\displaystyle
v_{c,k_{t}}^{\left[n\right]}=\frac{a_{c,k_{t}}^{\left[n\right]}}{a_{c,k_{t}}^{\left[n\right]}+1}+\log\big{(}1+a_{c,k_{t}}^{\left[n\right]}\big{)}\
\mathrm{and}\
u_{c,k_{t}}^{\left[n\right]}=\frac{\big{(}a_{c,k_{t}}^{\left[n\right]}\big{)}^{2}}{a_{c,k_{t}}^{\left[n\right]}+1},$
$\displaystyle\bar{v}_{c,k_{s}}^{\left[n\right]}=\frac{b_{c,k_{s}}^{\left[n\right]}}{b_{c,k_{s}}^{\left[n\right]}+1}+\log\big{(}1+b_{c,k_{s}}^{\left[n\right]}\big{)}\
\mathrm{and}\
\bar{u}_{c,k_{s}}^{\left[n\right]}=\frac{\big{(}b_{c,k_{s}}^{\left[n\right]}\big{)}^{2}}{b_{c,k_{s}}^{\left[n\right]}+1}.$
(69)

Above all, the approximate convex problem at iteration-$n$ is given by

$\displaystyle\mathcal{A}_{1}:\quad$
$\displaystyle\max_{q,\mathbf{W},\mathbf{P},\mathbf{c}^{sat},\mathbf{c}^{bs},\mathbf{r},\bm{\alpha},\mathbf{a},\mathbf{a}_{c},\mathbf{b},\mathbf{b}_{c}}q$
(70) $\displaystyle s.t.\qquad$
$\displaystyle(\ref{f1}),(\ref{f3}),(\ref{s1_constraint2.1_transform}),(\ref{s1_constraint5.1_transform}),\quad\forall
k_{t}\in\mathcal{K}_{t}$
$\displaystyle(\ref{f2}),(\ref{f4}),(\ref{s1_constraint4.1_transform}),(\ref{s1_constraint8.1_transform}),\quad\forall
k_{s}\in\mathcal{K}_{s}$
$\displaystyle(\ref{p1_constraint2}),(\ref{p1_constraint3}),(\ref{p1_constraint5}),(\ref{p1_constraint6}),(\ref{e1_constraint1}),(\ref{e1_constraint3})$

where
$\big{(}\mathbf{W}^{\left[n\right]},\mathbf{P}^{\left[n\right]},\mathbf{a}^{\left[n\right]},\mathbf{a}_{c}^{\left[n\right]},\mathbf{b}^{\left[n\right]},\mathbf{b}_{c}^{\left[n\right]}\big{)}$
obtained from the previous iteration are treated as constants when solving
$\mathcal{A}_{1}$. Variables are updated iteratively until a stopping
criterion is satisfied. We summarize the procedure of this RSMA-based joint
beamforming scheme in Algorithm 1. $\varepsilon$ is the tolerance value. The
convergence of Algorithm 1 is guaranteed since the solution of Problem
$\mathcal{A}_{1}$ at iteration-$n$ is a feasible solution of the problem at
iteration-$n+1$. As a consequence, the objective variable $q$ increases
monotonically and it is bounded above by the transmit power constraints. The
solution obtained at each iteration satisfies the KKT optimality conditions of
$\mathcal{A}_{1}$, which are indeed identical to those of $\mathcal{P}_{1}$ at
convergence [49].

Algorithm 1 Proposed Joint Beamforming Scheme

Initialize: $n\leftarrow 0,\
\mathbf{W}^{\left[n\right]},\mathbf{P}^{\left[n\right]},\mathbf{a}^{\left[n\right]},\mathbf{a}_{c}^{\left[n\right]},\mathbf{b}^{\left[n\right]},\mathbf{b}_{c}^{\left[n\right]},q^{\left[n\right]}$;

repeat

Solve $\mathcal{A}_{1}$ at
$\big{(}\mathbf{W}^{\left[n\right]},\mathbf{P}^{\left[n\right]},\mathbf{a}^{\left[n\right]},\mathbf{a}_{c}^{\left[n\right]},\mathbf{b}^{\left[n\right]},\mathbf{b}_{c}^{\left[n\right]}\big{)}$
to get the optimal solution
$\big{(}\breve{\mathbf{W}},\breve{\mathbf{P}},\breve{\mathbf{a}},\breve{\mathbf{a}}_{c},\breve{\mathbf{b}},\breve{\mathbf{b}}_{c},\breve{q}\big{)}$;

$n\leftarrow n+1$;

Update
$\mathbf{W}^{\left[n\right]}\leftarrow\breve{\mathbf{W}},\mathbf{P}^{\left[n\right]}\leftarrow\breve{\mathbf{P}},\mathbf{a}^{\left[n\right]}\leftarrow\breve{\mathbf{a}},\mathbf{a}_{c}^{\left[n\right]}\leftarrow\breve{\mathbf{a}}_{c},\mathbf{b}^{\left[n\right]}\leftarrow\breve{\mathbf{b}},\mathbf{b}_{c}^{\left[n\right]}\leftarrow\breve{\mathbf{b}}_{c},q^{\left[n\right]}\leftarrow\breve{q};$

until $\left|q^{\left[n\right]}-q^{\left[n-1\right]}\right|<\varepsilon$ ;

###  III-B Joint Beamforming Design for Cooperative STIN

When RSMA-based cooperative STIN is considered, the optimization problem of
max-min rate fairness among all users is given by

$\displaystyle\mathcal{P}_{2}:\quad\max_{\acute{\mathbf{W}},\acute{\mathbf{P}},\acute{\mathbf{c}}}$
$\displaystyle\min_{n_{s}\in\mathcal{N}_{s},k_{t}\in\mathcal{K}_{t}}\big{\\{}\acute{R}_{k_{t}}^{bs},\acute{R}_{k_{s}}^{sat}\big{\\}}$
(71) $\displaystyle s.t.\qquad$
$\displaystyle\acute{R}_{c,k_{t}}^{bs}\geq\sum_{j=1}^{K_{t}}\acute{C}_{j}^{bs}+\sum_{j=1}^{N_{s}}\acute{C}_{j}^{sat},\quad\forall
k_{t}\in\mathcal{K}_{t}$ (72) $\displaystyle\acute{C}_{k_{t}}^{bs}\geq
0\quad\forall k_{t}\in\mathcal{K}_{t}$ (73)
$\displaystyle\mathrm{tr}\big{(}\acute{\mathbf{P}}\acute{\mathbf{P}}^{H}\big{)}\leq
P_{t}$ (74)
$\displaystyle\acute{R}_{c,k_{s}}^{sat}\geq\sum_{j=1}^{K_{t}}\acute{C}_{j}^{bs}+\sum_{j=1}^{N_{s}}\acute{C}_{j}^{sat},\quad\forall
k_{s}\in\mathcal{K}_{s}$ (75) $\displaystyle\acute{C}_{n_{s}}^{sat}\geq
0,\quad\forall n_{s}\in\mathcal{N}_{s}$ (76)
$\displaystyle\big{(}\acute{\mathbf{W}}\acute{\mathbf{W}}^{H}\big{)}_{n_{s},n_{s}}\leq\frac{P_{s}}{N_{s}},\quad\forall
n_{s}\in\mathcal{N}_{s}$ (77)

where
$\acute{\mathbf{c}}=\small[\acute{C}_{1}^{sat},\cdots,\acute{C}_{N_{s}}^{sat},\acute{C}_{1}^{bs},\cdots,\acute{C}_{K_{t}}^{bs}\small]^{T}$
is the vector of all common rate portions. (72) and (75) guarantee that the
common stream of the whole system $\acute{s}_{c}$ can be decoded by all SUs
and CUs. (73) and (76) ensure non-negativity of each element in
$\acute{\mathbf{c}}$. (74) and (77) are respectively the sum transmit power
constraint of the BS and per-feed transmit power constraints of the satellite.
The formulated MMF problem for cooperative STIN is also nonconvex. Note that
the main difference between $\mathcal{P}_{1}$ and $\mathcal{P}_{2}$ lies in
the transmit data information sharing in $\mathcal{P}_{2}$. One super common
stream is transmitted at both the satellite and BS instead of transmitting
individual common streams. The expressions of rates and beamforming matrices
for cooperative STIN has been given in Section III. We can still use the SCA-
based algorithm to solve $\mathcal{P}_{2}$. Here, we omit the detailed problem
transformation and optimization framework, which follow the same procedure as
that for $\mathcal{P}_{1}$.

##  IV Robust Joint Beamforming Scheme

In this section, we further investigate the robust joint beamforming scheme
for RSMA-based coordinated STIN and cooperative STIN considering satellite
channel phase uncertainty. The CSIT of terrestrial channels is assumed to be
perfect.

From the satellite channel model given in Section II, we can observe that the
amplitudes of the channel vector components are determined by some constant
coefficients during the coherence time interval, including the free space
loss, satellite antenna gain and rain attenuation [19]. However, the satellite
channel phases vary rapidly due to a series of time-varying factors, such as
the use of different local oscillators (LO) on-board, the rain, cloud and
gaseous absorption, and the use of low-noise block (LNB) at receive sides [19,
21]. Therefore, within a coherence time interval, the phase of the channel
vector from the satellite to SU-$k_{s}$ at time instant $t_{1}$ can be modeled
as

$\displaystyle\bm{\phi}_{k_{s}}\left(t_{1}\right)=\bm{\phi}_{k_{s}}\left(t_{0}\right)+\mathbf{e}_{k_{s}},$
(78)

where $\bm{\phi}_{k_{s}}\left(t_{0}\right)$ represents the phase vector which
is estimated at the previous time instant $t_{0}$ and fed back to the GW.
$\mathbf{e}_{k_{s}}=\left[e_{k_{s},1},e_{k_{s},2},\cdots,e_{k_{s},N_{s}}\right]^{T}$
is the phase uncertainty following the distribution
$\mathbf{e}_{k_{s}}\sim\mathcal{N}\left(\mathbf{0},\delta^{2}\mathbf{I}\right)$,
with i.i.d Gaussian random entries. For ease of notation, we can generally
indicate $\bm{\phi}_{k_{s}}\left(t_{1}\right)$ and
$\bm{\phi}_{k_{s}}\left(t_{0}\right)$ by $\bm{\phi}_{k_{s}}$ and
$\widehat{\bm{\phi}}_{k_{s}}$ respectively. Since we assume identical channel
amplitudes within the coherence time interval, the channel vector from the
satellite to SU-$k_{s}$ can be written as

$\displaystyle\mathbf{f}_{k_{s}}=\widehat{\mathbf{f}}_{k_{s}}\circ\mathbf{x}_{k_{s}}=\mathrm{diag}\big{(}\widehat{\mathbf{f}}_{k_{s}}\big{)}\mathbf{x}_{k_{s}},$
(79)

where $\mathbf{x}_{k_{s}}=\mathrm{exp}\left\\{j\mathbf{e}_{k_{s}}\right\\}$ is
a random vector. We further assume that the channel estimate
$\widehat{\mathbf{f}}_{k_{s}}$ and the correlation matrix of
$\mathbf{x}_{k_{s}}$ denoted by
$\mathbf{X}_{k_{s}}=\mathbb{E}\left\\{\mathbf{x}_{k_{s}}\mathbf{x}_{k_{s}}^{H}\right\\}$
are known at the GW [19, 50]. For the interfering channels, by defining
$\mathbf{y}_{k_{t}}=\mathrm{exp}\left\\{j{\mathbf{e}}^{\prime}_{k_{t}}\right\\}$
and
${\mathbf{e}}^{\prime}_{k_{t}}=\left[{e}^{\prime}_{k_{t},1},{e}^{\prime}_{k_{t},2},\cdots,{e}^{\prime}_{k_{t},N_{s}}\right]^{T}$
following
${\mathbf{e}}^{\prime}_{k_{t}}\sim\mathcal{N}\left(\mathbf{0},\delta^{2}\mathbf{I}\right)$,
the channel vector from the satellite to CU-$k_{t}$ write as

$\displaystyle\mathbf{z}_{k_{t}}=\widehat{\mathbf{z}}_{k_{t}}\circ\mathbf{y}_{k_{t}}=\mathrm{diag}\big{(}\widehat{\mathbf{z}}_{k_{t}}\big{)}\mathbf{y}_{k_{t}},$
(80)

where the channel estimate $\widehat{\mathbf{z}}_{k_{t}}$ and the correlation
matrix
$\mathbf{Y}_{k_{t}}=\mathbb{E}\left\\{\mathbf{y}_{k_{t}}\mathbf{y}_{k_{t}}^{H}\right\\}$
are available at the GW. Hence, we concentrate on the expectation-based robust
beamforming design. The MMF optimization problem for RSMA-based coordinated
STIN considering satellite phase uncertainty remains the same as
$\mathcal{P}_{1}$ in Section III, By introducing auxiliary variables $q$,
$\bm{\alpha}=\left[\alpha_{1},\cdots,\alpha_{K_{t}}\right]^{T}$,
$\mathbf{r}=\left[r_{1},\cdots,r_{K_{s}}\right]^{T}$,
$W=\left\\{\mathbf{W}_{c},\mathbf{W}_{1},\cdots,\mathbf{W}_{N_{s}}\right\\}$
and
$P=\left\\{\mathbf{P}_{c},\mathbf{P}_{1},\cdots,\mathbf{P}_{K_{t}}\right\\}$,
the original $\mathcal{P}_{1}$ can be equivalently transformed into a semi-
definite programming (SDP) problem with rank-one constraints

$\displaystyle\mathcal{D}_{1}:\quad$
$\displaystyle\max_{W,P,\mathbf{c}^{sat},\mathbf{c}^{bs},q,\mathbf{r},\bm{\alpha}}q$
(81) $\displaystyle s.t.\qquad$
$\displaystyle\mathrm{tr}\left(\mathbf{P}_{c}\right)+\sum_{k_{t}=1}^{K_{t}}\mathrm{tr}\left(\mathbf{P}_{k_{t}}\right)\leq
P_{t}$ (82)
$\displaystyle\big{[}\mathbf{W}_{c}+\sum_{i=1}^{N_{s}}\mathbf{W}_{i}\big{]}_{n_{s},n_{s}}\leq\frac{P_{s}}{N_{s}},\quad
n_{s}\in\mathcal{N}_{s}$ (83) $\displaystyle\mathbf{W}_{c}\succeq 0,\
\mathbf{W}_{n_{s}}\succeq 0,\quad\forall n_{s}\in\mathcal{N}_{s}$ (84)
$\displaystyle\mathbf{P}_{c}\succeq 0,\ \mathbf{P}_{k_{t}}\succeq
0,\quad\forall k_{t}\in\mathcal{K}_{t}$ (85)
$\displaystyle\mathrm{rank}\left(\mathbf{W}_{c}\right)=1,\
\mathrm{rank}\left(\mathbf{W}_{n_{s}}\right)=1,\ \forall
n_{s}\in\mathcal{N}_{s}$ (86)
$\displaystyle\mathrm{rank}\left(\mathbf{P}_{c}\right)=1,\
\mathrm{rank}\left(\mathbf{P}_{k_{t}}\right)=1,\quad\forall
k_{t}\in\mathcal{K}_{t}$ (87)
$\displaystyle(\ref{p1_constraint1}),(\ref{p1_constraint2}),(\ref{p1_constraint4}),(\ref{p1_constraint5}),(\ref{e1_constraint1})-(\ref{e1_constraint4})$

where $\mathbf{W}_{c}=\mathbf{w}_{c}\mathbf{w}_{c}^{H}$,
$\left\\{\mathbf{W}_{n_{s}}=\mathbf{w}_{n_{s}}\mathbf{w}_{n_{s}}^{H}\right\\}_{n_{s}=1}^{N_{s}}$,
$\mathbf{P}_{c}=\mathbf{p}_{c}\mathbf{p}_{c}^{H}$,
$\left\\{\mathbf{P}_{k_{t}}=\mathbf{p}_{k_{t}}\mathbf{p}_{k_{t}}^{H}\right\\}_{k_{t}=1}^{K_{t}}$.
(82) and (82) are the equivalent transmit power constraints. All the rate
expressions in this section are redefined by the general form
$R\triangleq\mathbb{E}\left\\{\log_{2}\left(1+\mathrm{SINR}\right)\right\\}$,
which is averaged with respect to the satellite channel phase uncertainty to
guarantee the robustness. It has been verified in [20, 51] that for
nonnegative random variables $A$ and $B$, we can use the following
approximation

$\displaystyle\mathbb{E}\Big{\\{}\log_{2}\Big{(}1+\frac{A}{B}\Big{)}\Big{\\}}\approx\log_{2}\Big{(}1+\frac{\mathbb{E}\left(A\right)}{\mathbb{E}\left(B\right)}\Big{)}.$
(88)

By taking (42) as an example, $R_{k_{s}}^{sat}$ can be approximated by

$\displaystyle R_{k_{s}}^{sat}$
$\displaystyle=\mathbb{E}\left\\{\log_{2}\left(1+\gamma_{k_{s}}^{sat}\right)\right\\}$
$\displaystyle\approx\log_{2}\left(\frac{\mathbb{E}\left\\{\mathrm{tr}\left(\mathbf{F}_{k_{s}}\mathbf{W}_{\mu\left(k_{s}\right)}\right)\right\\}+\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\mathbb{E}\left\\{\mathrm{tr}\left(\mathbf{F}_{k_{s}}\mathbf{W}_{i}\right)\right\\}+\sigma_{k_{s}}^{sat2}}{\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\mathbb{E}\left\\{\mathrm{tr}\left(\mathbf{F}_{k_{s}}\mathbf{W}_{i}\right)\right\\}+\sigma_{k_{s}}^{sat2}}\right)$
$\displaystyle=\log_{2}\left(\frac{\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{\mu\left(k_{s}\right)}\right)+\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{i}\right)+\sigma_{k_{s}}^{sat2}}{\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{i}\right)+\sigma_{k_{s}}^{sat2}}\right).$
(89)

Specifically,
$\mathbf{F}_{k_{s}}=\mathrm{diag}\big{(}\widehat{\mathbf{f}}_{k_{s}}\big{)}\mathbf{x}_{k_{s}}\mathbf{x}^{H}_{k_{s}}\mathrm{diag}\big{(}\widehat{\mathbf{f}}^{H}_{k_{s}}\big{)}$,
and
$\overline{\mathbf{F}}_{k_{s}}=\mathbb{E}\left\\{\mathbf{F}_{k_{s}}\right\\}=\mathrm{diag}\big{(}\widehat{\mathbf{f}}_{k_{s}}\big{)}\mathbf{X}_{k_{s}}\mathrm{diag}\big{(}\widehat{\mathbf{f}}^{H}_{k_{s}}\big{)}$
is defined as the channel correlation matrix which can capture the expectation
over the distribution of phase uncertainty. Based on the approximated rate
expressions, $\mathcal{D}_{1}$ can be rewritten as $\mathcal{F}_{1}$.

$\displaystyle\mathcal{F}_{1}:\quad$
$\displaystyle\max_{W,P,\mathbf{c}^{sat},\mathbf{c}^{bs},q,\mathbf{r},\bm{\alpha},\eta,\xi}q$
(90) $\displaystyle s.t.\qquad$
$\displaystyle\eta^{bs}_{k_{t}}-\xi^{bs}_{k_{t}}\geq\alpha_{k_{t}}\log 2,\
\forall k_{t}\in\mathcal{K}_{t}$ (91) $\displaystyle
e^{\eta^{bs}_{k_{t}}}\leq\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{k_{t}}\right)+\sum_{j=1,j\neq
k_{t}}^{K_{t}}\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{j}\right)+\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{c}\right)+\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{i}\right)+\sigma_{k_{t}}^{bs2},\
\forall k_{t}\in\mathcal{K}_{t}$ (92) $\displaystyle
e^{\xi^{bs}_{k_{t}}}\geq\sum_{j=1,j\neq
k_{t}}^{K_{t}}\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{j}\right)+\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{c}\right)+\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{i}\right)+\sigma_{k_{t}}^{bs2},\
\forall k_{t}\in\mathcal{K}_{t}$ (93)
$\displaystyle\eta^{sat}_{k_{s}}-\xi^{sat}_{k_{s}}\geq r_{k_{s}}\log 2,\
\forall k_{s}\in\mathcal{K}_{s}$ (94) $\displaystyle
e^{\eta^{sat}_{k_{s}}}\leq\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{\mu\left(k_{s}\right)}\right)+\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{i}\right)+\sigma_{k_{s}}^{sat2},\
\forall k_{s}\in\mathcal{K}_{s}$ (95) $\displaystyle
e^{\xi^{sat}_{k_{s}}}\geq\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{i}\right)+\sigma_{k_{s}}^{sat2},\
\forall k_{s}\in\mathcal{K}_{s}$ (96)
$\displaystyle\eta^{bs}_{c,k_{t}}-\xi^{bs}_{c,k_{t}}\geq\sum_{j=1}^{K_{t}}C_{j}^{bs}\log
2,\ \forall k_{t}\in\mathcal{K}_{t}$ (97) $\displaystyle
e^{\eta^{bs}_{c,k_{t}}}\leq\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{c}\right)+\sum_{j=1}^{K_{t}}\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{j}\right)+\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{c}\right)+\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{i}\right)+\sigma_{k_{t}}^{bs2},\
\forall k_{t}\in\mathcal{K}_{t}$ (98) $\displaystyle
e^{\xi^{bs}_{c,k_{t}}}\geq\sum_{j=1}^{K_{t}}\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{j}\right)+\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{c}\right)+\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{i}\right)+\sigma_{k_{t}}^{bs2},\
\forall k_{t}\in\mathcal{K}_{t}$ (99)
$\displaystyle\eta^{sat}_{c,k_{s}}-\xi^{sat}_{c,k_{s}}\geq\sum_{j=1}^{N_{s}}C_{j}^{sat}\log
2,\ \forall k_{s}\in\mathcal{K}_{s}$ (100) $\displaystyle
e^{\eta^{sat}_{c,k_{s}}}\leq\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{c}\right)+\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{i}\right)+\sigma_{k_{s}}^{sat2},\
\forall k_{s}\in\mathcal{K}_{s}$ (101) $\displaystyle
e^{\xi^{sat}_{c,k_{s}}}\geq\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{i}\right)+\sigma_{k_{s}}^{sat2},\
\forall k_{s}\in\mathcal{K}_{s}$ (102)
$\displaystyle(\ref{p1_constraint2}),(\ref{p1_constraint5}),(\ref{e1_constraint1}),(\ref{e1_constraint3}),(\ref{d1_constraint1})-(\ref{d1_constraint6})$

where $\eta$ and $\xi$ are the sets of introduced slack variables. The
constraints (91)-(93), (94)-(96), (97)-(99), and (100)-(102) are respectively
the expansions of the rate constraints (40), (42), (32) and (35). Note that
(93), (96), (99) and (102) are nonconvex with convex LHSs which can be
approximated by the first-order Taylor approximation. Hence, we obtain these
approximated linear constraints

$\displaystyle\sum_{j=1,j\neq
k_{t}}^{K_{t}}\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{j}\right)+\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{c}\right)+\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{i}\right)+\sigma_{k_{t}}^{bs2}\leq
e^{\xi^{bs\left[n\right]}_{k_{t}}}\big{(}\xi^{bs}_{k_{t}}-\xi^{bs\left[n\right]}_{k_{t}}+1\big{)},$
(103)
$\displaystyle\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{i}\right)+\sigma_{k_{s}}^{sat2}\leq
e^{\xi^{sat\left[n\right]}_{k_{s}}}\big{(}\xi^{sat}_{k_{s}}-\xi^{sat\left[n\right]}_{k_{s}}+1\big{)},$
(104)
$\displaystyle\sum_{j=1}^{K_{t}}\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{j}\right)+\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{c}\right)+\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{i}\right)+\sigma_{k_{t}}^{bs2}\leq
e^{\xi^{bs\left[n\right]}_{c,k_{t}}}\big{(}\xi^{bs}_{c,k_{t}}-\xi^{bs\left[n\right]}_{c,k_{t}}+1\big{)},$
(105)
$\displaystyle\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{i}\right)+\sigma_{k_{s}}^{sat2}\leq
e^{\xi^{sat\left[n\right]}_{c,k_{s}}}\big{(}\xi^{sat}_{c,k_{s}}-\xi^{sat\left[n\right]}_{c,k_{s}}+1\big{)}.$
(106)

where $n$ represents the $n$-th SCA iteration. The constraints (93), (96),
(99) and (102) belong to generalized nonlinear convex program with high
computational complexity. Following the same method introduced in Section III,
they can be represented in linear and SOC forms given by

$\displaystyle
t^{bs}_{k_{t}}\leq\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{k_{t}}\right)+\sum_{j=1,j\neq
k_{t}}^{K_{t}}\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{j}\right)+\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{c}\right)+\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{i}\right)+\sigma_{k_{t}}^{bs2},$
(107)
$\displaystyle\Big{\|}t^{bs}_{k_{t}}+\eta^{bs}_{k_{t}}-\big{(}\log\small(t^{bs\left[n\right]}_{k_{t}}\small)+1\big{)}\quad
2\sqrt{t^{bs\left[n\right]}_{k_{t}}}\Big{\|}^{2}\leq
t^{bs}_{k_{t}}-\eta^{bs}_{k_{t}}+\big{(}\log\small(t^{bs\left[n\right]}_{k_{t}}\small)+1\big{)},$
(108) $\displaystyle
t^{sat}_{k_{s}}\leq\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{\mu\left(k_{s}\right)}\right)+\sum_{i=1,i\neq\mu\left(k_{s}\right)}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{i}\right)+\sigma_{k_{s}}^{sat2},$
(109)
$\displaystyle\Big{\|}t^{sat}_{k_{s}}+\eta^{sat}_{k_{s}}-\big{(}\log\small(t^{sat\left[n\right]}_{k_{s}}\small)+1\big{)}\quad
2\sqrt{t^{sat\left[n\right]}_{k_{s}}}\Big{\|}^{2}\leq
t^{sat}_{k_{s}}-\eta^{sat}_{k_{s}}+\big{(}\log\small(t^{sat\left[n\right]}_{k_{s}}\small)+1\big{)},$
(110) $\displaystyle
t^{bs}_{c,k_{t}}\leq\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{c}\right)+\sum_{j=1}^{K_{t}}\mathrm{tr}\left(\mathbf{H}_{k_{t}}\mathbf{P}_{j}\right)+\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{c}\right)+\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{Z}}_{k_{t}}\mathbf{W}_{i}\right)+\sigma_{k_{t}}^{bs2},$
(111)
$\displaystyle\Big{\|}t^{bs}_{c,k_{t}}+\eta^{bs}_{c,k_{t}}-\big{(}\log\small(t^{bs\left[n\right]}_{c,k_{t}}\small)+1\big{)}\quad
2\sqrt{t^{bs\left[n\right]}_{c,k_{t}}}\Big{\|}^{2}\leq
t^{bs}_{c,k_{t}}-\eta^{bs}_{c,k_{t}}+\big{(}\log\small(t^{bs\left[n\right]}_{c,k_{t}}\small)+1\big{)},$
(112) $\displaystyle
t^{sat}_{c,k_{s}}\leq\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{c}\right)+\sum_{i=1}^{N_{s}}\mathrm{tr}\left(\overline{\mathbf{F}}_{k_{s}}\mathbf{W}_{i}\right)+\sigma_{k_{s}}^{sat2},$
(113)
$\displaystyle\Big{\|}t^{sat}_{c,k_{s}}+\eta^{sat}_{c,k_{s}}-\big{(}\log\small(t^{sat\left[n\right]}_{c,k_{s}}\small)+1\big{)}\quad
2\sqrt{t^{sat\left[n\right]}_{c,k_{s}}}\Big{\|}^{2}\leq
t^{sat}_{c,k_{s}}-\eta^{sat}_{c,k_{s}}+\big{(}\log\small(t^{sat\left[n\right]}_{c,k_{s}}\small)+1\big{)}.$
(114)

Since rank-one implies only one nonzero eigenvalue, the nonconvex constraints
(86) and (87) can be rewritten by

$\displaystyle\mathrm{tr}\left(\mathbf{W}_{c}\right)-\lambda_{\max}\left(\mathbf{W}_{c}\right)=0,\
\mathrm{tr}\left(\mathbf{W}_{n_{s}}\right)-\lambda_{\max}\left(\mathbf{W}_{n_{s}}\right)=0,\
\forall n_{s}\in\mathcal{N}_{s},$ (115)
$\displaystyle\mathrm{tr}\left(\mathbf{P}_{c}\right)-\lambda_{\max}\left(\mathbf{P}_{c}\right)=0,\
\mathrm{tr}\left(\mathbf{P}_{k_{t}}\right)-\lambda_{\max}\left(\mathbf{P}_{k_{t}}\right)=0,\
\forall k_{t}\in\mathcal{K}_{t},$ (116)

where $\lambda_{\max}\left(\mathbf{X}\right)$ denotes the maximum eigenvalue
of $\mathbf{X}\succeq 0$. Then, we build a penalty function to insert these
constraints into the objective function (90) and obtain

$\displaystyle\max_{W,P,\mathbf{c}^{sat},\mathbf{c}^{bs},q,\mathbf{r},\mathbf{\alpha},\eta,\xi}q$
$\displaystyle-\beta\Big{(}\left[\mathrm{tr}\left(\mathbf{W}_{c}\right)-\lambda_{\max}\left(\mathbf{W}_{c}\right)\right]+\sum_{n_{s}=1}^{N_{s}}\left[\mathrm{tr}\left(\mathbf{W}_{n_{s}}\right)-\lambda_{\max}\left(\mathbf{W}_{n_{s}}\right)\right]$
$\displaystyle+\left[\mathrm{tr}\left(\mathbf{P}_{c}\right)-\lambda_{\max}\left(\mathbf{P}_{c}\right)\right]+\sum_{k_{t}=1}^{K_{t}}\left[\mathrm{tr}\left(\mathbf{P}_{k_{t}}\right)-\lambda_{\max}\left(\mathbf{P}_{k_{t}}\right)\right]\Big{)}.$
(117)

$\beta$ is a proper penalty factor to guarantee the penalty function as small
as possible. (117) is nonconcave due to the existence of the penalty function.
To tackle this issue, we adopt an iterative method [20]. By taking
$\mathrm{tr}\left(\mathbf{W}_{c}\right)-\lambda_{\max}\left(\mathbf{W}_{c}\right)$
as an example, we have following inequality

$\displaystyle\mathrm{tr}\small(\mathbf{W}_{c}\small)-\small(\mathbf{v}_{c,\max}^{\left[n\right]}\small)^{H}\mathbf{W}_{c}\mathbf{v}_{c,\max}^{\left[n\right]}\geq\mathrm{tr}\small(\mathbf{W}_{c}\small)-\lambda_{\max}\small(\mathbf{W}_{c}\small)\geq
0,$ (118)

where $\mathbf{v}_{c,\max}$ is the normalized eigenvector corresponding to the
maximum eigenvalue $\lambda_{\max}\left(\mathbf{W}_{c}\right)$. Furthermore,
we define $\mathbf{v}_{n_{s},\max}$ as the corresponding eigenvector of
$\lambda_{\max}\left(\mathbf{W}_{n_{s}}\right)$, and so does
$\mathbf{b}_{c,\max}$ for $\lambda_{\max}\left(\mathbf{P}_{c}\right)$ and
$\mathbf{b}_{k_{t},\max}$ for $\lambda_{\max}\left(\mathbf{P}_{k_{t}}\right)$.
Let $\mathrm{PF}$ denote the iterative penalty function

$\displaystyle\mathrm{PF}$
$\displaystyle=\beta\Big{(}\big{[}\mathrm{tr}\left(\mathbf{W}_{c}\right)-\left(\mathbf{v}_{c,\max}^{\left[n\right]}\right)^{H}\mathbf{W}_{c}\mathbf{v}_{c,\max}^{\left[n\right]}\big{]}+\sum_{n_{s}=1}^{N_{s}}\big{[}\mathrm{tr}\left(\mathbf{W}_{n_{s}}\right)-\left(\mathbf{v}_{n_{s},\max}^{\left[n\right]}\right)^{H}\mathbf{W}_{n_{s}}\mathbf{v}_{n_{s},\max}^{\left[n\right]}\big{]}$
$\displaystyle+\big{[}\mathrm{tr}\left(\mathbf{P}_{c}\right)-\left(\mathbf{b}_{c,\max}^{\left[n\right]}\right)^{H}\mathbf{P}_{c}\mathbf{b}_{c,\max}^{\left[n\right]}\big{]}+\sum_{k_{t}=1}^{K_{t}}\big{[}\mathrm{tr}\left(\mathbf{P}_{k_{t}}\right)-\left(\mathbf{b}_{k_{t},\max}^{\left[n\right]}\right)^{H}\mathbf{P}_{c}\mathbf{b}_{k_{t},\max}^{\left[n\right]}\big{]}\Big{)}.$
(119)

Then, the problem $\mathcal{F}_{1}$ can be reformulated as

$\displaystyle\mathcal{G}_{1}:\ $
$\displaystyle\max_{W,P,\mathbf{c}^{sat},\mathbf{c}^{bs},q,\mathbf{r},\mathbf{\alpha},\eta,\xi,t}q-\mathrm{PF}$
(120) $\displaystyle s.t.\ $
$\displaystyle(\ref{p1_constraint2}),(\ref{p1_constraint5}),(\ref{e1_constraint1}),(\ref{e1_constraint3}),(\ref{d1_constraint1})-(\ref{d1_constraint4}),(\ref{f1_constraint1}),(\ref{f1_constraint4}),(\ref{f1_constraint7}),(\ref{f1_constraint10})$
$\displaystyle(\ref{g1_constraint1}),(\ref{g1_constraint3}),(\ref{g1_constraint5}),(\ref{g1_constraint6}),(\ref{g1_constraint9}),(\ref{g1_constraint10}),\
\forall k_{t}\in\mathcal{K}_{t}$
$\displaystyle(\ref{g1_constraint2}),(\ref{g1_constraint4}),(\ref{g1_constraint7}),(\ref{g1_constraint8}),(\ref{g1_constraint11}),(\ref{g1_constraint12}),\
\forall k_{s}\in\mathcal{K}_{s}$

Now, the problem is convex and can be effectively solved by the CVX toolbox.
The results
$\small(W^{\left[n\right]},P^{\left[n\right]},\eta^{\left[n\right]},\xi^{\left[n\right]},t^{\left[n\right]}\small)$
obtained from the $n$-th iteration are treated as constants while solving
$\mathcal{G}_{1}$. The objective function is guaranteed to converge by the
existence of lower bounds, i.e., (118). In other words, the rank-one
constraints can be satisfied [20]. We summarize the procedure of this robust
joint beamforming scheme in Algorithm 2. Finally, eigenvalue decomposition
(EVD) can be used to obtain the optimized beamforming vectors of both
satellite and BS. The MMF optimization problem for RSMA-based cooperative STIN
with satellite phase uncertainty taken into account remains the same as
$\mathcal{P}_{2}$ in Section III. Here, we still omit the detailed
optimization framework. The process keeps the same as that for the coordinated
STIN scenario.

Algorithm 2 Robust Joint Beamforming Scheme

Initialize: $n\leftarrow
0,W^{\left[n\right]},P^{\left[n\right]},\eta^{\left[n\right]},\xi^{\left[n\right]},t^{\left[n\right]}$;

repeat

Solve $\mathcal{G}_{1}$ at
$\small(W^{\left[n\right]},P^{\left[n\right]},\eta^{\left[n\right]},\xi^{\left[n\right]},t^{\left[n\right]}\small)$
to get the optimal solution
$\small(\breve{W},\breve{P},\breve{\eta},\breve{\xi},\breve{t},\breve{\mathrm{objective}}\small)$;

$n\leftarrow n+1$;

Update
$W^{\left[n\right]}\leftarrow\breve{W},P^{\left[n\right]}\leftarrow\breve{P},\eta^{\left[n\right]}\leftarrow\breve{\eta},\xi^{\left[n\right]}\leftarrow\breve{\xi},t^{\left[n\right]}\leftarrow\breve{t},\mathrm{objective}^{\left[n\right]}\leftarrow\breve{\mathrm{objective}};$

until
$\big{|}\mathrm{objective}^{\left[n\right]}-\mathrm{objective}^{\left[n-1\right]}\big{|}<\varepsilon$
;

##  V Simulation Results

In this section, simulation results are provided to evaluate the performance
of our proposed joint beamforming algorithms. Both perfect CSIT and imperfect
CSIT with satellite channel phase uncertainties are considered. The tolerance
of accuracy is set to be $\varepsilon=10^{-5}$. Channel models have been
introduced in Section II, and the simulation parameters are listed in Table I.
Specifically, we assume the satellite is equipped with $N_{s}=3$ antennas, and
$\rho=2$ multicasting SUs locate uniformly in each beam coverage area.
According to the architecture of SFPB, the number of SUs is $K_{s}=\rho
N_{s}=6$. Meanwhile, the BS is deployed with $N_{t}=4\times 4=16$ UPAs, and
there are $K_{t}=4$ CUs uniformly distributed within the BS coverage. Since
the noise power is normalized by $\kappa T_{sys}B_{w}$ in (2), we set unit
noise variance, i.e., $\sigma_{k_{s}}^{sat2}=\sigma_{k_{t}}^{bs2}=1,\ \forall
k_{s}\in\mathcal{K}_{s},\ \forall k_{t}\in\mathcal{K}_{t}$. All MMF rate
curves throughout the simulation are calculated by averaging $100$ channel
realizations.

At first, we assume that perfect CSI is available at the GW. Fig. 4 compare
the MMF rate performance of RSMA-based coordinated and cooperative scheme. The
label “coordinated rsma rsma” means RSMA is adopted at both the satellite and
BS, while “cooperative rsma” means the satellite and BS work cooperatively as
a super transmitter and RSMA is adopted. As the BS transmit power budget
$P_{t}$ grows, we can see that the MMF rates of both schemes increase and tend
to saturate at large $P_{t}$ region. The cooperative scheme outperforms the
coordinated scheme apparently at low $P_{t}$ region. The gap between two
schemes decreases gradually as $P_{t}$ grows, and finally converge to the same
value when $P_{t}$ is sufficiently large. The reasons are as follows. When
$P_{t}$ is relatively small, the STIN’s performance is restricted in the
coordinated scheme because the SINRs of CUs are much lower than the SINRs of
SUs. Joint beamforming is designed to achieve optimal MMF rates. However, in
the cooperative scheme, transmit data sharing is assumed and the satellite can
complement the services of BS to CUs, thereby remaining the optimized MMF rate
at a higher level than that in the coordinated scheme. As $P_{t}$ grows, the
benefit of the cooperative scheme compared with coordinated scheme decreases.
Finally, when $P_{t}$ is sufficiently large, the MMF rates of both schemes are
restricted and converge to the save value due to the fixed satellite transmit
power budget $P_{s}$. We also investigate the influence of different $P_{s}$
values. Apparently, the larger $P_{s}$ is, the better MMF rate performance can
be achieved.

TABLE I: Simulation Parameters Parameter | Value
---|---
Frequency band (carrier frequency) | Ka $\left(28\ \mathrm{GHz}\right)$
Satellite height | $35786\ \mathrm{km}\left(\mathrm{GEO}\right)$
Bandwidth | 500 MHz
3 dB angle | $0.4\degree$
Maximum beam gain | 52 dBi
User terminal antenna gain | 42.7 dBi
System noise temperature | 300 K
Rain fading parameters | $\left(\mu,\sigma\right)=\left(-3.125,1.591\right)$
UPA inter-element spacing | $d_{1}=d_{2}=\frac{\lambda}{2}$
Number of NLoS paths | $3$

Figure 3: MMF rate versus $P_{t}$ for different $P_{s}$ setups. $N_{t}=16$,
$K_{t}=4$, $N_{s}=3$, $K_{s}=6$.

Figure 4: MMF rate versus $P_{t}$ for different transmission strategies.
$N_{t}=16$, $K_{t}=4$, $N_{s}=3$, $K_{s}=6$, $P_{s}=120\mathrm{W}$.

Fig. 4 depicts the MMF rates versus $P_{t}$ for different transmission
strategies. Here, we fix $P_{s}$ to be $120\ \mathrm{Watts}$. Both RSMA and
conventional SDMA are considered in the cooperative scheme. SDMA is a special
case of RSMA which can be simulated by discarding the common stream and
allocating all transmit power to the private streams. Different combinations
of transmission strategies are considered in the coordinated scheme, e.g., the
label “coordinated rsma sdma” means RSMA is exploited at the satellite while
SDMA is used at the BS. It has been shown in [43] that adopting RSMA compared
with SDMA in an overloaded system can provide more gains than in an
underloaded system. Therefore, in this STIN, the performance improvement
obtained by using RSMA compared with SDMA at the satellite is more obvious
than at the BS. As a consequence, the “coordinated rsma rsma” successively
outperforms “coordinated rsma sdma”, “coordinated sdma rsma” and “coordinated
sdma sdma” . For the “coordinated noma noma”, SC-SIC is implemented at both
the satellite and BS. The decoding order of NOMA at the satellite is decided
by the ascending order of the weakest user’s channel strength in each beam. We
can observe that the MMF rate achieved by NOMA is the worst compared with RSMA
and SDMA. The low performance of NOMA in multi-antenna settings is inline with
the observations in [52] and the references therein. As discussed in Fig. 4,
cooperative schemes can provide higher MMF rates than the corresponding
coordinated schemes. Thus, the “cooperative rsma” outperforms “coordinated
rsma rsma” in Fig. 4, and finally they tend to reach the same MMF rate
restricted by the fixed $P_{s}$ at very large $P_{t}$ region. Similarly, the
“cooperative sdma” outperforms “coordinated sdma sdma”. As $P_{t}$ increases,
they converge to the same value which is lower than that of RSMA. We do not
consider “cooperative noma” in this STIN because NOMA leads to not only a
waste of spatial resources (and therefore rate loss) in multi-antenna settings
but also additional decoding order selection complexity and large receiver
complexity [52].

Two baseline scenarios are also evaluated. “Baseline 1” represents a two-step
beamforming design. Neither CSI or data information is exchanged between the
satellite and BS. The beamforming for the satellite is at first optimized.
Then, the beamforming for the BS is optimized. We consider RSMA at both the
satellite and the BS. Based on such two-step beamfoming, we calculate the
minimum rate among all SUs and CUs. The rate curve is still generated by
averaging $100$ channel realizations. Since the satellite beamfoming vectors
are not jointly designed with the BS beamforming vectors, CUs will see serious
interference from the satellite. At low $P_{t}$ region, the extremely poor
minimum rate performance comes from the rates of CUs. At high $P_{t}$ region,
the rates of CUs are relatively large even if they see severe interference
from the satellite, and the minimum rate is decided by the rates of SUs.
Therefore, as $P_{t}$ grows, the value of minimum rate tends to reach the
saturation MMF rate of RSMA-based coordinated and cooperative schemes. In
“Baseline 2”, the satellite and BS operate on different frequency bands. We
still consider RSMA at both the satellite and BS. This traditional baseline
strategy cannot effectively make use of the spectrum, and thus results in poor
MMF rate performance.

Figure 5: MMF rate versus $P_{t}$ for different transmission strategies.
$N_{t}=4$, $K_{t}=4$, $N_{s}=3$, $K_{s}=6$, $P_{s}=120\mathrm{W}$.

Figure 6: MMF rate versus $P_{t}$ for different transmission strategies.
$N_{t}=64$, $K_{t}=4$, $N_{s}=3$, $K_{s}=6$, $P_{s}=120\mathrm{W}$.

Fig. 6 and Fig. 6 illustrate the influence of the number of BS transmit
antennas. In Fig. 6, we set $N_{t}=2\times 2=4$ UPAs, which is not enough to
support effective beamforming at the BS so as to mitigate the intra-cell
interference and the interference from the satellite. Compared with Fig. 4
with $16$ UPAs, the MMF rates of all strategies are suppressed. Specifically,
the performance of “coordinated sdma rsma” becomes better than the
“coordinated rsma sdma”. It implies that when $N_{t}$ is not sufficient to
suppress the interference suffered by the CUs, the gains obtained by using
RSMA compared with SDMA at the BS can become more obvious than at the
satellite. In Fig. 6, we assume $N_{t}=8\times 8=64$ UPAs, which is
sufficiently large to support effective beamforming at the BS to eliminate the
interference suffered by the CUs. The benefit of using RSMA compared with SDMA
at the BS can be ignored as the terrestrial sub-network is adequately
underloaded. As a result, the curve of “coordinated rsma rsma” and
“coordinated rsma sdma” overlap, and so do the the curve of “coordinated sdma
rsma” and “coordinated sdma sdma”. By comparing Fig. 4, Fig. 6 and Fig. 6, we
can conclude that the larger $N_{t}$ is, the better MMF rate performance can
be achieved, however different setups of $N_{t}$ will not affect the
saturation values of each transmission strategy. In other words, as $N_{t}$
increases, less $P_{t}$ is required to reach the same MMF rate performance.

Figure 7: MMF rate versus $P_{t}$ for different satellite phase uncertainties.
RSMA is used at the transmitters. $N_{t}=16$, $K_{t}=4$, $N_{s}=3$, $K_{s}=6$,
$P_{s}=120\mathrm{W}$.

Figure 8: MMF rate versus $P_{t}$ for different satellite phase uncertainties.
SDMA is used at the transmitters. $N_{t}=16$, $K_{t}=4$, $N_{s}=3$, $K_{s}=6$,
$P_{s}=120\mathrm{W}$.

Furthermore, we assume imperfect satellite CSI at the GW with phase
uncertainties. Fig. 8 shows the MMF rate performance of the proposed robust
joint beamforming scheme in both RSMA-based coordinated STIN and cooperative
STIN. As the variance of phase uncertainty $\delta^{2}$ increases, the MMF
rates of both scenarios decrease gradually. When $P_{t}=20\ \mathrm{dB}$, from
perfect CSIT to the assumptions when $\delta^{2}=5^{\circ}$,
$\delta^{2}=15^{\circ}$, and $\delta^{2}=45^{\circ}$, the corresponding MMF
rate decreases by $10.74\%$, $17.50\%$, $23.22\%$ respectively in the
coordinated STIN, and decreases by $10.37\%$, $17.69\%$, $25.04\%$
respectively in the cooperative STIN. For comparison, SDMA is adopted at the
transmitters, i.e., we consider the relatively competitive transmission
strategy “cooperative sdma” and “coordinated sdma sdma” to evaluate the MMF
rate performance with satellite channel phase uncertainty. From Fig.8, we can
observe that the gaps between perfect CSIT curves and imperfect CSIT curves
become larger compared with the RSMA results in Fig.8. When $P_{t}=20\
\mathrm{dB}$, from perfect CSIT to the assumptions when
$\delta^{2}=5^{\circ}$, $\delta^{2}=15^{\circ}$, and $\delta^{2}=45^{\circ}$,
the corresponding MMF rate decreases by $16.14\%$, $26.96\%$, $35.94\%$
respectively in the coordinated STIN, and decreases by $16.29\%$, $29.41\%$,
$39.18\%$ respectively in the cooperative STIN. Therefore, RSMA is more robust
to the channel phase uncertainty than SDMA due to its more flexible
architecture to partially decoding the interference and partially treat the
interference as noise.

##  VI Conclusion

In this work, we investigate the application of RSMA to STIN considering
either perfect CSI or imperfect CSI with satellite channel phase uncertainties
at the GW. Two RSMA-based STIN schemes are presented, namely the coordinated
scheme relying on CSI sharing and the cooperative scheme relying on both CSI
and data sharing. MMF optimization problems are formulated while satisfying
transmit power budgets at both the BS and the satellite. To tackle the
optimization, two iterative algorithms are respectively proposed for the joint
beamforming design considering either perfect or imperfect CSIT. Through
simulation results, the superiority of our proposed RSMA-based schemes for
STIN is demonstrated compared with various benchmark strategies. The
robustness of RSMA is verified in the presence of channel phase uncertainties.
In conclusion, RSMA is shown very promising for STINs to manage the
interference in and between the satellite and terrestrial sub-networks.

## References

  * [1] Y. Kawamoto, Z. M. Fadlullah, H. Nishiyama, N. Kato, and M. Toyoshima, “Prospects and challenges of context-aware multimedia content delivery in cooperative satellite and terrestrial networks,” _IEEE Communications Magazine_ , vol. 52, no. 6, pp. 55–61, 2014.
  * [2] H. Zhang, C. Jiang, J. Wang, L. Wang, Y. Ren, and L. Hanzo, “Multicast beamforming optimization in cloud-based heterogeneous terrestrial and satellite networks,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 2, pp. 1766–1776, 2019.
  * [3] S. Maleki, S. Chatzinotas, B. Evans, K. Liolis, J. Grotz, A. Vanelli-Coralli, and N. Chuberre, “Cognitive spectrum utilization in ka band multibeam satellite communications,” _IEEE Communications Magazine_ , vol. 53, no. 3, pp. 24–29, 2015.
  * [4] M. Lin, Z. Lin, W.-P. Zhu, and J.-B. Wang, “Joint beamforming for secure communication in cognitive satellite terrestrial networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 36, no. 5, pp. 1017–1029, 2018\.
  * [5] S. Vassaki, M. I. Poulakis, A. D. Panagopoulos, and P. Constantinou, “Power allocation in cognitive satellite terrestrial networks with qos constraints,” _IEEE Communications Letters_ , vol. 17, no. 7, pp. 1344–1347, 2013.
  * [6] M. Jia, X. Gu, Q. Guo, W. Xiang, and N. Zhang, “Broadband hybrid satellite-terrestrial communication systems based on cognitive radio toward 5G,” _IEEE Wireless Communications_ , vol. 23, no. 6, pp. 96–106, 2016\.
  * [7] J. P. Choi and C. Joo, “Challenges for efficient and seamless space-terrestrial heterogeneous networks,” _IEEE Communications Magazine_ , vol. 53, no. 5, pp. 156–162, 2015.
  * [8] S. K. Sharma, S. Chatzinotas, and B. Ottersten, “Transmit beamforming for spectral coexistence of satellite and terrestrial networks,” in _8th International Conference on Cognitive Radio Oriented Wireless Networks_. IEEE, 2013, pp. 275–281.
  * [9] K. An, M. Lin, W.-P. Zhu, Y. Huang, and G. Zheng, “Outage performance of cognitive hybrid satellite–terrestrial networks with interference constraint,” _IEEE Transactions on Vehicular Technology_ , vol. 65, no. 11, pp. 9397–9404, 2016.
  * [10] X. Zhu, C. Jiang, L. Yin, L. Kuang, N. Ge, and J. Lu, “Cooperative multigroup multicast transmission in integrated terrestrial-satellite networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 36, no. 5, pp. 981–992, 2018.
  * [11] Y. Zhang, L. Yin, C. Jiang, and Y. Qian, “Joint beamforming design and resource allocation for terrestrial-satellite cooperation system,” _IEEE Transactions on Communications_ , vol. 68, no. 2, pp. 778–791, 2019\.
  * [12] Z. Lin, M. Lin, J.-B. Wang, X. Wu, and W.-P. Zhu, “Joint optimization for secure wipt in satellite-terrestrial integrated networks,” in _2018 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 2018, pp. 1–6.
  * [13] J. Li, K. Xue, D. S. Wei, J. Liu, and Y. Zhang, “Energy efficiency and traffic offloading optimization in integrated satellite/terrestrial radio access networks,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 4, pp. 2367–2381, 2020.
  * [14] P. K. Sharma and D. I. Kim, “Secure 3d mobile uav relaying for hybrid satellite-terrestrial networks,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 4, pp. 2770–2784, 2020.
  * [15] W. Lu, K. An, and T. Liang, “Robust beamforming design for sum secrecy rate maximization in multibeam satellite systems,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 55, no. 3, pp. 1568–1572, 2019.
  * [16] X. Guo, D. Yang, Z. Luo, H. Wang, and J. Kuang, “Robust THP design for energy efficiency of multibeam satellite systems with imperfect CSI,” _IEEE Communications Letters_ , vol. 24, no. 2, pp. 428–432, 2019.
  * [17] Z. Lin, M. Lin, J. Ouyang, W.-P. Zhu, A. D. Panagopoulos, and M.-S. Alouini, “Robust secure beamforming for multibeam satellite communication systems,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 6, pp. 6202–6206, 2019.
  * [18] A. Gharanjik, M. B. Shankar, P.-D. Arapoglou, M. Bengtsson, and B. Ottersten, “Robust precoding design for multibeam downlink satellite channel with phase uncertainty,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2015, pp. 3083–3087.
  * [19] W. Wang, L. Gao, R. Ding, J. Lei, L. You, C. A. Chan, and X. Gao, “Resource efficiency optimization for robust beamforming in multi-beam satellite communications,” _IEEE Transactions on Vehicular Technology_ , 2021.
  * [20] J. Chu, X. Chen, C. Zhong, and Z. Zhang, “Robust design for noma-based multibeam LEO satellite internet of things,” _IEEE Internet of Things Journal_ , vol. 8, no. 3, pp. 1959–1970, 2020.
  * [21] M. Á. Vázquez, A. Perez-Neira, D. Christopoulos, S. Chatzinotas, B. Ottersten, P.-D. Arapoglou, A. Ginesi, and G. Taricco, “Precoding in multibeam satellite communications: Present and future challenges,” _IEEE Wireless Communications_ , vol. 23, no. 6, pp. 88–95, 2016.
  * [22] “Second generation framing structure, channel coding modulation systems broadcasting, interactive services, news gathering other broad-band satellite applications (DVB-S2),” European Broadcasting Union (EBU), document ETSI EN 302 307–1 V1.4.1, Standard, Nov. 2014.
  * [23] “Second generation framing structure, channel coding and modulation systems for broadcasting, interative services, news gathering and other broadband satellite applications; Part 2: DVB-S2 Extensions (DVB-S2X),” European Broadcasting Union (EBU), document ETSI EN 302-307-2 V1.1.1, Standard, Oct. 2014\.
  * [24] B. Clerckx, Y. Mao, R. Schober, and H. V. Poor, “Rate-splitting unifying SDMA, OMA, NOMA, and multicasting in MISO broadcast channel: A simple two-user rate analysis,” _IEEE Wireless Communications Letters_ , vol. 9, no. 3, pp. 349–353, 2019.
  * [25] Y. Mao, B. Clerckx, and V. O. Li, “Rate-splitting multiple access for downlink communication systems: bridging, generalizing, and outperforming SDMA and NOMA,” _EURASIP journal on wireless communications and networking_ , vol. 2018, no. 1, p. 133, 2018.
  * [26] A. A. Ahmad, H. Dahrouj, A. Chaaban, A. Sezgin, and M.-S. Alouini, “Interference mitigation via rate-splitting and common message decoding in cloud radio access networks,” _IEEE Access_ , vol. 7, pp. 80 350–80 365, 2019.
  * [27] J. Zhang, B. Clerckx, J. Ge, and Y. Mao, “Cooperative rate-splitting for MISO broadcast channel with user relaying, and performance benefits over cooperative NOMA,” _IEEE Signal Processing Letters_ , vol. 26, no. 11, pp. 1678–1682, 2019.
  * [28] H. Joudeh and B. Clerckx, “Sum-rate maximization for linearly precoded downlink multiuser MISO systems with partial CSIT: A rate-splitting approach,” _IEEE Transactions on Communications_ , vol. 64, no. 11, pp. 4847–4861, 2016.
  * [29] C. Hao, Y. Wu, and B. Clerckx, “Rate analysis of two-receiver MISO broadcast channel with finite rate feedback: A rate-splitting approach,” _IEEE Transactions on Communications_ , vol. 63, no. 9, pp. 3232–3246, 2015.
  * [30] H. Joudeh and B. Clerckx, “Robust transmission in downlink multiuser MISO systems: A rate-splitting approach,” _IEEE Transactions on Signal Processing_ , vol. 64, no. 23, pp. 6227–6242, 2016.
  * [31] E. Piovano and B. Clerckx, “Optimal DoF region of the $k$-user MISO BC with partial CSIT,” _IEEE Communications Letters_ , vol. 21, no. 11, pp. 2368–2371, 2017.
  * [32] Y. Mao and B. Clerckx, “Beyond dirty paper coding for multi-antenna broadcast channel with partial CSIT: A rate-splitting approach,” _IEEE Transactions on Communications_ , vol. 68, no. 11, pp. 6775–6791, 2020.
  * [33] H. Joudeh and B. Clerckx, “Rate-splitting for max-min fair multigroup multicast beamforming in overloaded systems,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 11, pp. 7276–7289, 2017.
  * [34] A. Z. Yalcin, M. Yuksel, and B. Clerckx, “Rate splitting for multi-group multicasting with a common message,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 10, pp. 12 281–12 285, 2020.
  * [35] O. Tervo, L.-N. Trant, S. Chatzinotas, B. Ottersten, and M. Juntti, “Multigroup multicast beamforming and antenna selection with rate-splitting in multicell systems,” in _2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_. IEEE, 2018, pp. 1–5.
  * [36] A. Papazafeiropoulos, B. Clerckx, and T. Ratnarajah, “Rate-splitting to mitigate residual transceiver hardware impairments in massive MIMO systems,” _IEEE Transactions on Vehicular Technology_ , vol. 66, no. 9, pp. 8196–8211, 2017.
  * [37] M. Dai and B. Clerckx, “Multiuser millimeter wave beamforming strategies with quantized and statistical CSIT,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 11, pp. 7025–7038, 2017.
  * [38] Y. Mao, B. Clerckx, and V. O. Li, “Rate-splitting for multi-user multi-antenna wireless information and power transfer,” in _2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_. IEEE, 2019, pp. 1–5.
  * [39] C. Xu, B. Clerckx, S. Chen, Y. Mao, and J. Zhang, “Rate-splitting multiple access for multi-antenna joint communication and radar transmissions,” in _2020 IEEE International Conference on Communications Workshops (ICC Workshops)_. IEEE, 2020, pp. 1–6.
  * [40] M. Caus, A. Pastore, M. Navarro, T. Ramírez, C. Mosquera, N. Noels, N. Alagha, and A. I. Perez-Neira, “Exploratory analysis of superposition coding and rate-splitting for multibeam satellite systems,” in _2018 15th International Symposium on Wireless Communication Systems (ISWCS)_. IEEE, 2018, pp. 1–5.
  * [41] M. Vazquez, M. Caus, and A. Perez-Neira, “Rate-splitting for MIMO multibeam satellite systems,” in _WSA 2018; 22nd International ITG Workshop on Smart Antennas_. VDE, 2018, pp. 1–6.
  * [42] L. Yin and B. Clerckx, “Rate-splitting multiple access for multibeam satellite communications,” in _2020 IEEE International Conference on Communications Workshops (ICC Workshops)_. IEEE, 2020, pp. 1–6.
  * [43] ——, “Rate-splitting multiple access for multigroup multicast and multibeam satellite systems,” _IEEE Transactions on Communications_ , vol. 69, no. 2, pp. 976–990, 2020.
  * [44] Z. W. Si, L. Yin, and B. Clerckx, “Rate-splitting multiple access for multigateway multibeam satellite systems with feeder link interference,” _arXiv preprint arXiv:2102.05792_ , 2021.
  * [45] Z. Lin, M. Lin, T. de Cola, J.-B. Wang, W.-P. Zhu, and J. Cheng, “Supporting IoT with rate-splitting multiple access in satellite and aerial integrated networks,” _IEEE Internet of Things Journal_ , 2021.
  * [46] Z. Lin, M. Lin, B. Champagne, W.-P. Zhu, and N. Al-Dhahir, “Secure and energy efficient transmission for RSMA-based cognitive satellite-terrestrial networks,” _IEEE Wireless Communications Letters_ , vol. 10, no. 2, pp. 251–255, 2020.
  * [47] O. Tervo, L.-N. Tran, and M. Juntti, “Optimal energy-efficient transmit beamforming for multi-user MISO downlink,” _IEEE Transactions on Signal Processing_ , vol. 63, no. 20, pp. 5574–5588, 2015.
  * [48] K.-G. Nguyen, Q.-D. Vu, M. Juntti, and L.-N. Tran, “Distributed solutions for energy efficiency fairness in multicell MISO downlink,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 9, pp. 6232–6247, 2017\.
  * [49] B. R. Marks and G. P. Wright, “A general inner approximation algorithm for nonconvex mathematical programs,” _Operations research_ , vol. 26, no. 4, pp. 681–683, 1978.
  * [50] A. Gharanjik, M. B. Shankar, P.-D. Arapoglou, M. Bengtsson, and B. Ottersten, “Precoding design and user selection for multibeam satellite channels,” in _2015 IEEE 16th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_. IEEE, 2015, pp. 420–424.
  * [51] M. Shao and W.-K. Ma, “A simple way to approximate average robust multiuser miso transmit optimization under covariance-based csit,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 3504–3508.
  * [52] B. Clerckx, Y. Mao, R. Schober, E. Jorswieck, D. J. Love, J. Yuan, L. Hanzo, G. Y. Li, E. G. Larsson, and G. Caire, “Is NOMA efficient in multi-antenna networks? a critical look at next generation multiple access techniques,” _IEEE Open Journal of the Communications Society_ , 2021.