# GPatch: Patching Graph Neural Networks for Cold-Start Recommendations

Hao Chen<EMAIL_ADDRESS>Tencent Inc. ,  Zefan Wang
<EMAIL_ADDRESS>Jinan University. ,  Yue Xu<EMAIL_ADDRESS>Alibaba
Group. ,  Xiao Huang<EMAIL_ADDRESS>The Hong Kong Polytechnic
University. and  Feiran Huang<EMAIL_ADDRESS>Jinan University.

###### Abstract.

Cold start is an essential and persistent problem in recommender systems.
State-of-the-art solutions rely on training hybrid models for both cold-start
and existing users/items, based on the auxiliary information. Such a hybrid
model would compromise the performance of existing users/items, which might
make these solutions not applicable in real-worlds recommender systems where
the experience of existing users/items must be guaranteed. Meanwhile, graph
neural networks (GNNs) have been demonstrated to perform effectively warm
(non-cold-start) recommendations. However, they have never been applied to
handle the cold-start problem in a user-item bipartite graph. This is a
challenging but rewarding task since cold-start users/items do not have links.
Besides, it is nontrivial to design an appropriate GNN to conduct cold-start
recommendations while maintaining the performance for existing users/items. To
bridge the gap, we propose a tailored GNN-based framework (GPatch) that
contains two separate but correlated components. First, an efficient GNN
architecture — GWarmer, is designed to model the warm users/items. Second, we
construct correlated Patching Networks to simulate and patch GWarmer by
conducting cold-start recommendations. Experiments on benchmark and large-
scale commercial datasets demonstrate that GPatch is significantly superior in
providing recommendations for both existing and cold-start users/items.

††conference: 4th Workshop on Deep Learning Practice and Theory for High-
Dimensional Sparse and Imbalanced Data with KDD 2022; August 14, 2022; Virtual
Conference, Online††price: 15.00††isbn: 978-1-4503-XXXX-X/22/08

##  1\. Introduction

Recommender systems aim to retrieve deserving information for users out of
massive online content and provide personalized recommendations. There are
thousands of user registrations and item productions in various recommender
systems every second, introducing cold-start problems. That is how to
recommend for the users and items without historical interactions. In a
typical user-item cold-start recommendation scene, recommender systems can
access the auxiliary information for both existing and cold-start users/items
to alleviate the cold-start problem.

Considering the large proportion of existing users and items, one main
challenge of cold-start recommendation is to serve cold-start recommendations
without affecting the recommendations for existing users and items. Given
trained warm embeddings and the auxiliary information, state-of-the-art
solutions such as DropoutNet (volkovs2017dropoutnet, ) and Heater
(zhu2020heater, ) train hybrid models to recommend for both cold-start
users/items and existing users/items. Such hybrid models have to tackle two
totally different types of inputs simultaneously: 1. for warm (non-cold-start)
recommendations, feeding with warm embeddings and the auxiliary information to
generate the warm representation; and 2. for cold-start recommendations,
feeding with vacant embeddings and the auxiliary information to generate the
cold-start representation. Although showing certain cold-start ability, such
hybrid cold-start models would compromise the performance of existing
users/items due to accommodating the two entirely different inputs: vacant
embeddings and the trained warm embeddings. The hybrid models cannot provide
satisfying recommendations for existing users and items, restricting these
solutions cannot be applied in real-world recommender systems, where the
experience of existing users/items must be guaranteed.

The core of the warm recommender system is how to model the interactions of
existing users and items. Accordingly, Graph Convolutional Networks (GCNs),
skilled in modeling graph-structured data, have achieved the state-of-the-art
performance in many recommendation applications (fu2020magnn, ;
chen2022neighbor, ; lu2020meta, ; chen2021non, ; liu2020heterogeneous, ;
chen2020label, ; fan2019meirec, ; zhao2019intentgc, ; chen2022generative, ;
chen2020graph, ). NGCF (wang2019ngcf, ) and LightGCN (he2020lightgcn, )
construct bipartite graphs from historical user-item interactions and then
embed the users and items with the help of graph convolutional networks
(kipf2016gcn, ). However, current GCN models cannot be directly applied to
deal with the cold-start problem due to three major challenges as follows.
First, there are no historical interactions for the cold-start users/items. It
means that they do not have neighbors. GCN models could not perform
neighborhood aggregations to refine the embedding representations of cold-
start users/items. They do not have exact representations either. Second, it
is nontrivial to design a suitable GCN to maintain the performance of
recommendation on existing users/items, while performing cold-start
recommendations. Current cold-start models are designed to tackle the non-
graph recommendation models. Third, due to the recursive message passing
structure of GCN models, GCN models are much time consuming than simple inner-
product functions. Performing cold-start GCN models may encounter
computational burdens than the current inner-product-based cold-start models.

To overcome the above challenges, this paper proposes Graph Patching Networks
(GPatch) to provide satisfying recommendations for both existing and cold-
start users and items. GPatch contains two separative but correlated models:
GWarmer and Patching Networks. GWarmer is a specially designed efficient GCN
Model which enhances the recommendation for existing users/items. It is
impressive that GWarmer is as simple as the inner product during inferring,
making GPatch more efficient than SOTA cold-start models. Besides, serving as
patches for GWarmer, Patching Networks afford cold-start recommendations by
generating GWarmer-friendly embeddings from the auxiliary information. Unlike
traditional cold-start models, Patching Networks only serve cold-start
recommendations and thus will not influence the warm recommendation models.
Extensive experiments on two benchmark and one 15-day WeChat commercial
datasets, with three different type of embedding methods, present that GPatch
statistically significantly outperforms other recommendation models on both
hybrid recommendation performance (including warm/cold-start recommendation
tasks) and pure-warm recommendation performance.

Figure 1. Sketch of GPatch. GWarmer provides recommendation for existing users
and items, while Patching Networks recommend for cold-start users and items.

In summary, our contributions are listed as below:

  * •

This paper at the first time enables the cold-start recommendation of GCN
models, resulting in better recommendation performance for both existing/cold-
start users and items.

  * •

This paper first notices the compromising caused by the hybrid cold-start
models and proposed Patching Networks that serve cold recommendations without
compromising the performance of existing users/items.

  * •

Experiments on two benchmark and one online commercial datasets, with three
different embeddings, show that GPatch significantly outperforms other cold-
start models on all situations with statistically significant margins.

##  2\. Our Method: GPatch

This section first presents the whole framework of our GPatch, and then we
introduce the details of the proposed GWarmer and Patching Networks.

###  2.1. Overall Framework

Let $u\in{\mathcal{U}}$ denote the user index, and $i\in{\mathcal{I}}$ denote
the item index. ${\mathcal{G}}$ denotes the user-item bipartite graph
constructed from the user-item interactions ${\mathcal{R}}$.
${\mathcal{U}}_{w}$ and ${\mathcal{I}}_{w}$ denote the existing (warm)
user/item set. ${\bm{C}}_{U}$ and ${\bm{C}}_{I}$ denote the auxiliary
information for all users/items. ${\bm{E}}_{U}$ and ${\bm{E}}_{I}$ denote the
user/item embedding matrix trained by any given user-item embedding model.

We first define the general form of the warm/cold-start recommendation task.
_Warm recommendations_ , by definition, mean that recommend warm items to warm
users. For warm users and items, we can access the warm embeddings
${\bm{E}}_{U}$ and ${\bm{E}}_{I}$ and the historical interaction graph
${\mathcal{G}}$. Formally, GWarmer predicts the relevance of a warm user-item
pair $(u,i)$ with
${\mathcal{W}}(u,i;{\mathcal{G}},{\bm{E}}_{U},{\bm{E}}_{I})$. By contrast,
_Cold-start Recommendations_ refer to the recommendation for either cold-start
user or cold-start item, since the recommendation model cannot access the
historical interaction graph and the embeddings for both users and items
simultaneously. Without the graph and the embeddings, Patching Networks
generate their cold-start embeddings from their auxiliary information
${\bm{c}}_{u}$ and ${\bm{c}}_{i}$, namely,
${\mathcal{P}}(u,i;{\bm{c}}_{u},{\bm{c}}_{i})$. After defining GWarmer and
Patching Networks, for an arbitrary user-item pair, GPatch computes its
relevance scores with the following formula,

(1) $\displaystyle\hat{y}_{ui}=\left\\{\begin{aligned}
&{\mathcal{W}}(u,i;{\mathcal{G}},{\bm{E}}_{U},{\bm{E}}_{I}),\quad&u\in{\mathcal{U}}_{w}\text{
and }i\in{\mathcal{I}}_{w},\\\
&{\mathcal{P}}(u,i;{\bm{c}}_{u},{\bm{c}}_{i}),&u\notin{\mathcal{U}}_{w}\text{
or }i\notin{\mathcal{I}}_{w},\end{aligned}\right.$

where ${\mathcal{W}}(\cdot)$ denotes GWarmer, and ${\mathcal{P}}(\cdot)$
denotes Patching Networks. We will detail these two parts in the next two
subsections.

###  2.2. GWarmer

Common models (rendle2012bprmf, ; rao2015grmf, ; huang2017label, ;
he2020lightgcn, ) usually focus on investigating more powerful user/item
embeddings and then predict the relevance scores with inner product function.

(2) $\hat{y}_{ui}^{(w)}={\bm{e}}_{u}^{\top}{\bm{e}}_{i}.$

Correspondingly, an ideal warm recommendation model for real-world recommender
systems, where user-item embeddings have already been trained by various warm
embedding models, should be able to enhance any given embeddings with graph
structure ${\mathcal{G}}$, without specifying the embedding models (such as MF
(koren2009MF, ), MetaPath2Vec(M2V) (dong2017metapath2vec, ) or LightGCN
(he2020lightgcn, )). Besides, the warm recommendation model should be
efficient in training and inferring.

As illustrated in Figure 1, GWarmer first constructs the $K$-layer subgraphs
for warm users/items with random-walk and then aggregates the random-walk set
to represent the graph topology of warm users/items. Finally, a self-adaptive
inner product relevance function is designed to perform the warm
recommendation.

Random-Walk Step. Random-walk is an efficient method in extracting the graph
topology (perozzi2014deepwalk, ; grover2016node2vec, ). As shown in Figure 1,
for any given warm user/item $t$, we repetitively generate $K$-length walks
$S$ times with setting $t$ as the root node, where $S$ denotes the sampling
size. Here we denote the random-walk set as ${\mathcal{T}}_{K}(t)$. Note that
in bipartite user-items graphs, user nodes connect to item nodes while item
nodes connect to user nodes. In Figure 1, we use green circles to denote user
nodes and yellow circles to denote item nodes.

Walk Pooling Step. After obtaining the random-walk set, we substitute the node
ids by their corresponding embedding vectors, then we aggregate the random-
walk set with mean pooling aggregator (gunets, ). Mean pooling is permutation-
invariant on set elements, such that the order of the walks will not influence
the aggregation. After the mean pooling aggregation, for node $t$ we get its
layer-wise representation for each layer of the subgraph. Formally, the layer-
wise representation can be calculated with the following formula,

(3)
${\bm{x}}_{t}^{(k)}=\frac{1}{S}\sum_{{\bm{T}}_{i}\in\mathcal{T}_{K}(v)}{\bm{T}}_{i}^{(k)},$

where $k$ denotes the layer number of the subgraph and $S$ denotes the
sampling size of the random-walk. Specifically, ${\bm{x}}_{t}^{(0)}=E_{t}$.

Self-adaptive Inner Product With the above two steps, we have $K$
representation vectors for each warm user or item $t$. Taken user $u$ as an
example, ${\bm{x}}_{u}^{(0)}$ is her own embeddings, while
${\bm{x}}_{u}^{(1)}$ contains the mean embedding of her interacted items, and
${\bm{x}}_{u}^{(2)}$ aggregates the embeddings of the users that have shared
interacted items with $u$. From this point of view, the representations of
different layers profile user $u$ from various aspects. Motivated by this, we
introduce the self-adaptive weighted sum which assigns different weights for
different layers. This further enhances the flexibility and extensibility of
the inner product. By specifying the user representations and item
representations with different weights, the relevance function is given as,

(4) $\displaystyle{\bm{x}}_{u}=\sum_{k}w_{u}^{(k)}{\bm{x}}_{u}^{(k)},\
{\bm{x}}_{i}=\sum_{k}w_{i}^{(k)}{\bm{x}}_{i}^{(k)},$ (5)
$\displaystyle\hat{y}_{ui}^{(w)}={\mathcal{W}}(u,i;{\mathcal{G}},{\bm{E}}_{U},{\bm{E}}_{I})={\bm{x}}_{u}^{\top}{\bm{x}}_{i},$

where $\\{w_{u}^{(0)},\cdots,w_{u}^{(K)}\\}$ and
$\\{w_{i}^{(0)},\cdots,w_{i}^{(K)}\\}$ denotes the self-adaptive weights for
users’ and items’ layer-wise representations, respectively. ${\bm{x}}_{u}$ and
${\bm{x}}_{i}$ denotes the GWarmer representation of $u$ and $i$.

Comparing with other recursive message passing models, GWarmer is much
suitable for large-scale online recommender systems. First, the random walk
step and the walk pooling step are parameter-free, which could be pre-computed
and pre-stored offline in a parallel manner. Second, in large commercial
systems, the historical user-item interactions are stored in distributed
databases. Comparing with traditional recursive GCNs which cost an extensive
query time for the recursive neighbor expanding, random walk policy needs
significantly lower I/O times. Finally, for the online inferring process,
after storing the GWarmer representations, GWarmer is as simple as inner
product functions, which can be easily adapted with online recommender
frameworks (_e.g._ , Faiss (johnson2019faiss, )).

###  2.3. Patching Networks

Severing as patches of GWarmer, Patching Networks have to be consistent with
GWarmer when recommending for cold-start recommendation tasks. To this end, as
shown in Figure 1, we bridge GWarmer and Patching Networks by using GWarmer
representation instead of raw embeddings to train the Patching Networks. As
shown in Figure 1, we employ the popular cold-start dropout mechanism
(volkovs2017dropoutnet, ; zhu2020heater, ) to optimize the Patching Networks
by randomly masking the GWarmer representations of warm users/items with ratio
$\tau$. Given any warm node $t$, we sample a random variable $p\in\\{0,1\\}$
from $Bernoulli(\tau)$. Then the masked GWarmer representation is:

(6)
${\mathcal{D}}({\bm{x}}_{t},\tau)=p\cdot\mathbf{0}+(1-p)\cdot{\bm{x}}_{t},$

where $\mathbf{0}$ is a zero vector that has the same dimension as
${\bm{x}}_{t}$.

Given the warm user-item pair $(u,i)$, the Patching Networks predict its
relevance scores with following formula,

(7)
$\displaystyle{\bm{x}}_{uc}=f_{U}({\mathcal{D}}({\bm{x}}_{u},\tau)\parallel{\bm{c}}_{u}),\
{\bm{x}}_{ic}=f_{I}({\mathcal{D}}({\bm{x}}_{i},\tau)\parallel{\bm{c}}_{i}),$
(8)
$\displaystyle\hat{y}_{ui}^{(c)}={\mathcal{P}}(u,i;{\bm{c}}_{u},{\bm{c}}_{i})={\bm{x}}_{uc}^{\top}{\bm{x}}_{ic},$

where $f_{U}$ and $f_{I}$ denotes the mapping function for users and items,
respectively. Note that, $p$ is set as 1 during the inference of cold-start
recommendation tasks when we set the embeddings of cold-start users/items as
vacant embedding 0.

In GPatch framework, the self-adaptive weights in Eq. (4) and the mapping
functions in Eq. (7) require optimization. Thus, we employ the Mean Squared
Error (MSE) loss (volkovs2017dropoutnet, ; zhu2020heater, ) to optimize both
GWarmer and Patching Networks simultaneously. Formally, the loss function
contains two parts and we present it as:

(9)
${\mathcal{L}}=\sum_{{\mathcal{R}}^{+}\cup{\mathcal{R}}^{-}}(y_{ui}-{\mathcal{W}}(u,i;{\mathcal{G}},{\bm{E}}_{U},{\bm{E}}_{I}))^{2}+(y_{ui}-{\mathcal{P}}(u,i;{\bm{c}}_{u},{\bm{c}}_{i}))^{2},$

where ${\mathcal{R}}^{+}$ denotes the observed user-item interactions, while
${\mathcal{R}}^{-}$ denotes the sampling negative interactions. Thus,
$y_{ui}=1$ for $(u,i)\in{\mathcal{R}}^{+}$ and $y_{ui}=0$ for
$(u,i)\in{\mathcal{R}}^{-}$.

##  3\. Experiments

Table 1. Statistics of the three datasets Dataset | User # | Item # | Interaction # | Density
---|---|---|---|---
CiteULike | 5,551 | 16,980 | 204,986 | 0.22%
XING | 106,881 | 20,519 | 3,856,580 | 0.18%
WeChat | 87,772 | 446,408 | 17,768,997 | 0.05%
Table 2. Comparison of hybrid recommendation performance. The percentages give
the relative improvement over the best baseline (underlined). The superscript
* denotes statistically significance against the best baseline with p¡0.05.

Embeddings | Models | CiteULike(REC@20) | CiteULike(PRE@20) | CiteULike(NDCG@20) | XING(REC@20) | XING(PRE@20) | XING(NDCG@20) | WeChat(REC@100) | WeChat(PRE@100) | WeChat(NDCG@100)
---|---|---|---|---|---|---|---|---|---|---
MF+ | DropoutNet | 0.0780 | 0.0359 | 0.0654 | 0.1988 | 0.0712 | 0.1686 | 0.0519 | 0.0108 | 0.0315
| Heater | 0.0968 | 0.0479 | 0.0849 | 0.2190 | 0.0792 | 0.1921 | 0.0640 | 0.0130 | 0.0403
| GPatch | 0.1404(45.04%)∗ | 0.0668(39.46%)∗ | 0.1187(39.81%)∗ | 0.2346(7.12%)∗ | 0.0866(9.34%)∗ | 0.2055(6.98%)∗ | 0.0659(2.97%)∗ | 0.0133(2.31%)∗ | 0.0443(9.93%)∗
M2V+ | DropoutNet | 0.1019 | 0.0443 | 0.0798 | 0.2159 | 0.0782 | 0.1764 | 0.0685 | 0.0126 | 0.0401
| Heater | 0.0779 | 0.0406 | 0.0687 | 0.1946 | 0.0716 | 0.1658 | 0.0770 | 0.0139 | 0.0481
| GPatch | 0.1292(26.79%)∗ | 0.0565(27.54%)∗ | 0.1172(46.87%)∗ | 0.2541(17.69%)∗ | 0.0937(19.82%)∗ | 0.2204(24.94%)∗ | 0.0804(4.42%)∗ | 0.0154(10.79%)∗ | 0.0535(11.23%)∗
LightGCN+ | DropoutNet | 0.0933 | 0.0429 | 0.0795 | 0.2072 | 0.0756 | 0.1796 | - | - | -
| Heater | 0.1136 | 0.0534 | 0.1018 | 0.2411 | 0.0873 | 0.2057 | - | - | -
| GPatch | 0.1631(43.57%)∗ | 0.0763(42.88%)∗ | 0.1411(38.61%)∗ | 0.2621(8.71%)∗ | 0.0954(9.28%)∗ | 0.2296(11.62%)∗ | - | - | -

Table 3. Comparison on warm recommendation tasks.

Embeddings | Models | CiteULike(NDCG@20) | XING(NDCG@20) | Wechat(NDCG@100)
---|---|---|---|---
MF+ | DropoutNet | 0.0737 | 0.2167 | 0.0394
| Heater | 0.0934 | 0.2653 | 0.0489
| GPatch |  0.1428(52.89%)* |  0.2757(3.92%)* |  0.0533(9.00%)*
M2V+ | DropoutNet | 0.0765 | 0.2211 | 0.0567
| Heater | 0.0857 | 0.2393 | 0.0629
| GPatch |  0.1652(92.77%)* |  0.3157(31.93%)* |  0.0696(10.65%)*
LightGCN+ | DropoutNet | 0.0796 | 0.2506 | -
| Heater | 0.1232 | 0.2930 | -
| GPatch |  0.1800(46.10%)* |  0.3398(15.97%)* | -

Table 4. Comparison of inferring time.

Model | CiteULike | XING | WeChat
---|---|---|---
DropoutNet | 56 s | 79 s | 1259 s
Heater | 73 s | 97 s | 1483 s
GPatch | 43 s | 64 s | 1054 s

In this section, we aim to answer the following two research questions: RQ1:
How does GPatch perform compared with SOTA cold-start recommendation models in
terms of effectiveness and efficiency? RQ2: How do the hyper-parameters, such
as the dropout ratio $\tau$, affect GPatch’s performance?

#### Dataset Description.

We conduct our experiments on two cold-start benchmark datasets
(zhu2020heater, ) (CiteULike and XING) and one online commercial dataset
(WeChat). CiteULike is a user-article dataset, where each article has a
300-dimension tf-idf vector. XING is a user-view-job dataset where each job is
described by a 2738-dimension vector. WeChat is a 15-day online user-video
dataset, where each video can be represented by their NLP descriptions, a
200-dimension vector. We summarize the three datasets in Table 1. Here we
mainly study the cold-start problem of items. For CiteULike and XING, 20% of
items are selected to be the cold-start items. Excluding the cold-start items,
65% of historical interactions are used for embedding training, while 15% of
historical interactions are used for training GPatch, and the remaining 20%
interactions are used for validation/testing. For WeChat, we split the dataset
according to its timeline, where we collect the newly produced videos as the
cold-start items. We adopt the popular all-ranking evaluation protocol
(wang2019ngcf, ; he2020lightgcn, ; zhu2020heater, ). Specifically, we report
Precision@20, Recall, and NDCG metrics for the hybrid recommendation task, and
we report the NDCG metric for the pure-warm recommendation task.

#### Baselines & Embedding Methods.

We consider two SOTA cold-start recommendation models: DropoutNet
(volkovs2017dropoutnet, ) and Heater (zhu2020heater, ) as our baselines. To
evaluate the universality of GPatch, we choose three different types of
embedding methods:

  * •

MF (rendle2012bprmf, ) optimizes the MF (koren2009MF, ) with a pairwise
ranking loss, which is tailored to learn from implicit feedback.

  * •

M2V (dong2017metapath2vec, ) formalizes metapath-based random walks on user-
item graphs as corpus and then leverages skip-gram (mikolov2013word2vec, )
models to compute node embeddings.

  * •

LightGCN (he2020lightgcn, ) utilizes GCN layers to aggregate neighbors
information, which presents to be currently the most powerful embedding model
for user-item recommendation.

The embedding size is fixed to 200 for all models, and all the embedding
methods are implemented with the official codes. We skip LightGCN embeddings
on WeChat dataset, since LightGCN reports out of CUDA memory error on a NVIDIA
2080Ti GPU (11G).

#### Implementation.

In terms of GPatch, we restrict the number of subgraph layers as 3. For
computational efficiency, we sample the neighbors to accelerate the training
and inference process. We set the sampling size of trajectory size as 25. We
use Adam optimizer with a learning rate of 0.001, where the batch size is
1024. The coefficient of $\ell_{2}$ normalization is set as $1e^{-5}$, and the
randomize dropout ratio $\tau$ is set as 0.5. Tanh is used as the activation
function. The size of the hidden layers and the output layer is set as 200,
while the depth of MLP is set as 2 by default. Moreover, early stopping
strategy is performed by observing the AUC scores.

###  3.1. Main Results (Q1)

Table 2 reports the performance comparison of GPatch and baselines on the
hybrid recommendation tasks, which evaluates the ability of cold-start models
to rank the top-20(100 for WeChat) warm and cold-start items out of all items.
We conduct our experiments on three different warm embeddings. From this
table, we observe that GPatch statistically outperforms all other baselines on
all embeddings and all datasets with $p<0.05$, demonstrating the consistent
superiority of our GPatch on hybrid recommendations. This further verifies
that separating warm recommendation and cold-start recommendation models can
exploit more potentialities from warm/cold-start recommendation models at the
same time, resulting in better hybrid recommendation performance.

To verify the pure-warm recommendation performance, in Table 3, we present how
powerful are GPatch and other baselines in recommending the top-20(100 for
WeChat) warm items out of all warm items. From this table, we highlight that
GPatch outperforms DropoutNet and Heater with statistically significant
margins on all embeddings and all datasets. This further presents the
necessity of solving cold-start problems with powerful GCN models.

In terms of efficiency, we present the average inferring time of DropoutNet,
Heater, and GPatch on all three datasets. As shown in Table 4, GPatch is more
efficient than DropoutNet and Heater. The reason is that GWarmer is as simple
as the inner product during inferring, while DropoutNet and Heater still have
to generate the embeddings by the MLP function.

Figure 2. Comparison of different training dropout ratio $\tau$.

###  3.2. Ablation Study (Q2)

In Figure 2, we compare the hybrid recommendation performance of GPatch,
DropoutNet, and Heater by training with different dropout ratio $\tau$.
Observing from this table, GPatch consistently outperforms DropoutNet and
Heater on most situations. This verifies the consistently superior of GPatch.
Especially, since Patching Network is only used for cold-start recommendation.
GPatch may achieve the best performance when set $\tau$ as 1.

##  4\. Conclusion

In this paper, we propose a general cold-start framework named GPatch, which
at the first time solves the cold-start problem for GCN models. As a
consequence, GPatch is generally powerful on all datasets and embeddings in
both the hybrid recommendation task and the warm recommendation task. Besides,
the proposed Patching Networks offer researchers an efficient framework to
patch powerful warm recommendation models with other patching cold-start
models. Since then, future studies can be conducted in 1. designing more
powerful patching cold-start models, 2. combing with SOTA warm recommendation
models, and 3. investigating specially designed patching models.

## References

  * (1) Maksims Volkovs, Guangwei Yu, and Tomi Poutanen.  Dropoutnet: Addressing cold start in recommender systems.  In NIPS, 2017.
  * (2) Ziwei Zhu, Shahin Sefati, Parsa Saadatpanah, and James Caverlee.  Recommendation for new users and new items via randomized training and mixture-of-experts transformation.  In SIGIR, pages 1121–1130, 2020.
  * (3) Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King.  Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding.  In Proceedings of The Web Conference 2020, pages 2331–2341, 2020\.
  * (4) Hao Chen, Zhong Huang, Yue Xu, Zengde Deng, Feiran Huang, Peng He, and Zhoujun Li.  Neighbor enhanced graph convolutional networks for node classification and recommendation.  Knowledge-Based Systems, 246:108594, 2022.
  * (5) Yuanfu Lu, Yuan Fang, and Chuan Shi.  Meta-learning on heterogeneous information networks for cold-start recommendation.  In KDD, 2020.
  * (6) Hao Chen, Zengde Deng, Yue Xu, and Zhoujun Li.  Non-recursive graph convolutional networks.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3205–3209. IEEE, 2021.
  * (7) Siwei Liu, Iadh Ounis, Craig Macdonald, and Zaiqiao Meng.  A heterogeneous graph neural model for cold-start recommendation.  In SIGIR, pages 2029–2032, 2020.
  * (8) Hao Chen, Yue Xu, Feiran Huang, Zengde Deng, Wenbing Huang, Senzhang Wang, Peng He, and Zhoujun Li.  Label-aware graph convolutional networks.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1977–1980, 2020.
  * (9) Shaohua Fan, Junxiong Zhu, Xiaotian Han, Chuan Shi, Linmei Hu, Biyu Ma, and Yongliang Li.  Metapath-guided heterogeneous graph neural network for intent recommendation.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2478–2486, 2019.
  * (10) Jun Zhao, Zhou Zhou, Ziyu Guan, Wei Zhao, Wei Ning, Guang Qiu, and Xiaofei He.  Intentgc: a scalable graph convolution framework fusing heterogeneous information for recommendation.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2347–2357, 2019.
  * (11) Hao Chen, Zefan Wang, Feiran Huang, Xiao Huang, Yue Xu, Yishi Lin, Peng He, and Zhoujun Li.  Generative adversarial framework for cold-start item recommendation.  In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2565–2571, 2022.
  * (12) Hao Chen, Wenbing Huang, Yue Xu, Fuchun Sun, and Zhoujun Li.  Graph unfolding networks.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1981–1984, 2020.
  * (13) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua.  Neural graph collaborative filtering.  In Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval, pages 165–174, 2019.
  * (14) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang.  Lightgcn: Simplifying and powering graph convolution network for recommendation.  Proceedings of the 43rd international ACM SIGIR conference on Research and development in Information Retrieval, 2020.
  * (15) Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * (16) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme.  Bpr: Bayesian personalized ranking from implicit feedback.  arXiv preprint arXiv:1205.2618, 2012.
  * (17) Nikhil Rao, Hsiang-Fu Yu, Pradeep K Ravikumar, and Inderjit S Dhillon.  Collaborative filtering with graph information: Consistency and scalable methods.  In NIPS, 2015.
  * (18) Xiao Huang, Jundong Li, and Xia Hu.  Label informed attributed network embedding.  In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pages 731–739, 2017.
  * (19) Yehuda Koren, Robert Bell, and Chris Volinsky.  Matrix factorization techniques for recommender systems.  Computer, 42(8):30–37, 2009.
  * (20) Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami.  metapath2vec: Scalable representation learning for heterogeneous networks.  In KDD, 2017.
  * (21) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena.  Deepwalk: Online learning of social representations.  In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701–710, 2014.
  * (22) Aditya Grover and Jure Leskovec.  node2vec: Scalable feature learning for networks.  In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855–864, 2016.
  * (23) Hao Chen, Wenbing Huang, Yue Xu, Fuchun Sun, and Zhoujun Li.  Graph unfolding networks.  In CIKM, 2020.
  * (24) Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with gpus.  IEEE Transactions on Big Data, 2019.
  * (25) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.


# Multi-Frequency Channel Modeling for
Millimeter Wave and THz Wireless Communication
via Generative Adversarial Networks

Yaqi Hu, Mingsheng Yin, William Xia, Sundeep Rangan, Marco Mezzavilla The
authors were supported by NSF grants 1952180, 1925079, 1564142, 1547332, the
SRC, OPPO, and the industrial affiliates of NYU WIRELESS. The work was also
supported by Remcom that provided the Wireless Insite software. NYU Tandon
School of Engineering, Brooklyn, NY, USA

###### Abstract

Modern cellular systems rely increasingly on simultaneous communication in
multiple discontinuous bands for macro-diversity and increased bandwidth.
Multi-frequency communication is particularly crucial in the millimeter wave
(mmWave) and Terahertz (THz) frequencies, as these bands are often coupled
with lower frequencies for robustness. Evaluation of these systems requires
statistical models that can capture the joint distribution of the channel
paths across multiple frequencies. This paper presents a general neural
network based methodology for training multi-frequency double directional
statistical channel models. In the proposed approach, each is described as a
multi-clustered set, and a generative adversarial network (GAN) is trained to
generate random multi-cluster profiles where the generated cluster data
includes the angles and delay of the clusters along with the vectors of random
received powers, angular, and delay spread at different frequencies. The model
can be readily applied for multi-frequency link or network layer simulation.
The methodology is demonstrated on modeling urban micro-cellular links at
$28\text{\,}$ and $140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ trained from
extensive ray tracing data. The methodology makes minimal statistical
assumptions and experiments show the model can capture interesting statistical
relationships between frequencies.

###### Index Terms:

Channel modeling, millimeter wave, sub-terahertz, neural networks, GANs

##  I Introduction

Cellular wireless systems often operate over multiple frequency bands
simultaneously for increased bandwidth and macro-diversity. Commercial
cellular system support several mechanisms for multi-band operation including
carrier aggregation [1], dual connectivity [2], and multi-connectivity [3].
The simultaneous use of multiple frequencies is particularly important in the
millimeter wave (mmWave) and terahertz (THz) bands where lower frequency
carriers are needed for robustness and control signaling [4, 5]. There has
also been active research in leveraging information from lower frequencies for
communication in the mmWave and THz bands for procedures such as beam search
[6, 7, 8], and localization and sensing [9].

Evaluation of these systems, necessitates channel models that can describe the
behavior of links across multiple frequency bands. In this work, we seek to
develop _statistical channel models_ that can generate random instances of the
channel parameters following the observed statistical distribution of channel
parameters in some environment (e.g. urban micro). Statistical models such as
those by 3GPP [10] are the mainstay for commercial cellular evaluation and are
a critical initial step in any simulation evaluation.

Developing accurate statistical channel models for mmWave systems is well
known to be challenging, even for a single band. Since mmWave systems operate
at very high bandwidths with highly directional beams, statistical models must
capture the full _double directional_ nature of the channel meaning the angles
of arrival and departure, delays and gains of all the paths [11]. This
difficulty is compounded in the multi-frequency setting, where one must
generate gains of the paths across multiple frequencies. These parameters can
have complex relationships as factors that influence propagation, such as
transmission and reflection losses, vary considerably across frequency [12].

Due to the difficulty in modeling wireless channels from first principles,
there has been a growing interest in using data driven, neural networks (NN)
based techniques [13, 14, 15, 16, 17, 18]. The key benefit of these approaches
is that they make minimal assumptions and can thus learn complex statistical
relationships between variables. In this line, the work in [18] used NNs for
air-to-ground channel models at $28\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$.
The approach was able to model the full double-directional nature of the
channel and demonstrated the ability to learn non-obvious relationships. In
addition, the trained models considerably outperformed standard 3GPP models
such as [10], which assume very specific structures to be tractable.

In this work, we wish to model the _joint_ distribution of the paths between a
transmitter (TX) and receiver (RX) across multiple frequencies. The channel is
modeled by a set of clusters and a generative model is then used to generate
the cluster parameters from the distance between the TX and RX. The paper then
develops and evaluates several new features as compared to the modelling
methodology of [18] to support multiple frequencies:

  * •

_Multi-frequency cluster models:_ Most importantly, the model represents the
channel in a compact manner that can capture the key commonalities and
differences between frequencies. Specifically, the channel is represented by a
set of clusters whose mean angles of arrivals and departure and delay are
common among the frequencies, reflecting that macro-level path of propagation
are not frequency dependent. On the other hand, the power, and the angular and
delay spread are modeled as frequency dependent to capture the variations in
scattering and transmission losses across the frequencies, which is
particularly vital to model in the mmWave and THz regimes [19, 20, 21, 22].

  * •

_Ray tracing processing:_ There are limited number of multi-frequency channel
measurements [23, 24] and these generally have an insufficient numbers of data
points for training large NNs. Similar to [18, 25, 26], we thus employ a
powerful ray tracing package, Wireless InSite by Remcom [27], to obtain high
volume training data.

  * •

_Cluster identification and pre-processing:_ When using the ray tracing data,
we propose a novel clustering algorithm as a pre-processing step that
identifies common clusters across frequencies but captures the delay and
angular variations in different frequencies. The methods can be considered as
a multi-frequency extension of clustering techniques proposed in [28].

  * •

_Evaluation on micro-cellular urban environment_ : The method is evaluated on
micro-cellular terrestrial links in a region($\approx 0.6\times 0.4$
$\mathrm{k}\mathrm{m}$2) of Beijing, China. It is shown that the method can
capture important correlations in the channel characteristics between
frequencies.

##  II Multi-Frequency Modeling Problem

We consider the modeling of channels on $M$ frequencies, $f_{1},\ldots,f_{M}$.
For example, in the examples below, we will look at two frequencies: $f_{1}=$
$28\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ and $f_{2}=$
$140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$. We model a single link, meaning
the channel from a transmitter (TX) to receiver (RX). For the cellular
applications below, the TX will be the base station (gNB) and the RX will be
the mobile user equipment (UE). However, due to reciprocity, the TX and RX can
be reversed. We model the channel across the $M$ frequencies by a common set
of $L$ clusters where each cluster $\ell$ is described by a vector of
parameters,

${\mathbf{x}}^{(\ell)}=\left[\tilde{\tau}_{\ell},\bar{\theta}^{\rm
rx}_{\ell},\bar{\phi}^{\rm rx}_{\ell},\bar{\theta}^{\rm
tx}_{\ell},\bar{\phi}^{\rm tx}_{\ell},F_{\ell 1},\ldots,F_{\ell M}\right],$
(1)

where $\tilde{\tau}_{\ell}$ is the minimum path delay within the cluster,
$\bar{\theta}^{\rm rx}_{\ell}$, $\bar{\phi}^{\rm rx}_{\ell}$ are the average
intra-cluster inclination and azimuth angles of arrival (AoA),
$\bar{\theta}^{\rm tx}_{\ell}$, $\bar{\phi}^{\rm tx}_{\ell}$ are the average
intra-cluster inclination and azimuth angles of departure (AoD). These
parameters are common across the clusters, reflecting that the angles and
delay are dependent on the large scale path routes and do not depend on the
frequency. In contrast, the parameters $F_{\ell m}$, $m=1,\ldots,M$, are the
frequency-dependent components given by

$F_{\ell m}=\left[\Delta\theta^{\rm rx}_{\ell m},\Delta\phi^{\rm rx}_{\ell
m},\Delta\theta^{\rm tx}_{\ell m},\Delta\phi^{\rm tx}_{\ell
m},\Delta\tau_{\ell m},P_{\ell m}\right],$ (2)

where $\Delta\theta^{\rm rx}_{\ell m}$, $\Delta\phi^{\rm rx}_{\ell m}$ are
root mean square (RMS) inclination and azimuth angles of arrival spread,
$\Delta\theta^{\rm tx}_{\ell m}$, $\Delta\phi^{\rm tx}_{\ell m}$ are the RMS
inclination and azimuth angles of departure spread, $\Delta\tau_{\ell m}$ is
the RMS delay spread, $P_{\ell m}$ is the total received power. The multi-
frequency channel can then be described by the set of $L$ clusters

${\mathbf{x}}=\left[{\mathbf{x}}^{(1)},\ldots,{\mathbf{x}}^{(L)}\right].$ (3)

We will call ${\mathbf{x}}$ the _cluster vector_.

To simplify the modeling, we will fix the number of clusters at $L=10$. When
there are less than $L$ clusters at a certain frequency, say $L_{0}<L$, we
simply set $P_{\ell m}=0$ (in linear scale) for all $\ell>L_{0}$ at frequency
m. Similarly, if a path cluster exist in one frequency $m$ but not at another
frequency $m^{\prime}$, we take $P_{\ell m}>0$ and $P_{\ell m^{\prime}}=0$.
This scenario can occur, for example, where a path can penetrate materials at
a lower frequency, but be blocked at a millimeter wave frequency.

The above parameterization captures the fact that the geometric path route is
independent of frequency. However, the angular spread from scattering as well
as the transmission and reflection losses are well-known to be frequency
dependent – for example, see the scattering studies in THz in [19, 20, 21]. We
also note that the 3GPP procedure for multi-frequency models also generates
paths with common cluster mean angles of arrival and departure, but frequency
dependent gains and angular spread [10]. However, the frequency-dependent
components are assumed to be indepdent in the 3GPP model. In contrast, our
procedure is general and can, in principle, learn statistical relationships
with sufficient data.

Since the model (1) has $5+6M$ parameters per cluster, and $L$ clusters, there
are a total of $(5+6M)L$ parameters in (3). For example, in the example below,
we will consider $M=2$ frequencies and $L=10$ clusters for a total of
$p=(5+6M)L=170$ parameters.

Each link also has a condition vector, ${\mathbf{u}}$. For our case, we will
simply take

${\mathbf{u}}=(d_{x},d_{y},d_{z}),$ (4)

the 3D vector between the TX and RX. Other parameters could be added to
${\mathbf{u}}$ such as the antenna heights or cell types.

The statistical channel modeling problem is to model the distribution of the
cluster vector ${\mathbf{x}}$ as a function of the link conditions
${\mathbf{u}}$. That is, we want to model the conditional probability
distribution $p({\mathbf{x}}|{\mathbf{u}})$. In this work, we will be
interested in so-called generative models where the conditional distribution
is represented as a mapping

${\mathbf{x}}=g({\mathbf{u}},{\mathbf{z}}),$ (5)

where ${\mathbf{z}}$ is some random vector with a known distribution.

Generative models (5) are the basis for cellular evaluations such as [10]. For
example, the gNBs and UEs are first randomly placed in an environment
following some deployment model assumption (e.g. urban micro with some inter-
site distance and density). The deployment determines the link conditions
${\mathbf{u}}$ between each gNB and UE. Using a generative model (5), we can
then randomly generate path parameters between all the gNBs and UEs. Once we
combine these path parameters with the antenna array assumptions on both
sides, we can determine all the MIMO channels in the network. One can then run
any algorithm of interest to evaluate relevant system performance statistics
such as the SNR or rate distribution.

##  III Urban Cellular Ray Tracing Data and Scattering Modeling

Figure 1: Ray tracing simulation area representing a 550 $\times$ 380
$\text{m}^{2}$ region of Beijing, China.

Although our methodology is general, we will demonstrate the methods on two
frequencies: $f_{1}=$ $28\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$, as used in
5G systems today [29], and $f_{2}=$
$140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$, a key sub-THz frequency targeted
for potential 6G use cases [30, 31]. While there have been recent multi-
frequency measurement campaigns [23, 24], these measurements are enormously
time-consuming and there are no current datasets with sufficient data points
for training complex neural networks. Thus, similar to several recent works
[18, 26, 25] we use a powerful ray tracing tool, Wireless InSite by Remcom
[27] to generate data for training. Specifically, an area of Beijing, China,
consisting of 550 $\times$ 380 square meters as shown in Fig. 1, is imported.
Using similar parameters to 3GPP report [10], we manually place 100
terrestrial receiving gNBs on the outside of buildings with
$10\text{\,}\mathrm{m}$ height, and transmitting UEs are placed at 320
locations on streets $1.5\text{\,}\mathrm{m}$. The deployment results in 100
$\times$ 320 $=$ 32000 links (i.e. gNB-UE pairs). We run independent ray
tracing simulations on these the two frequencies. We set the maximum number of
reflections as six and the maximum number of diffraction as one, and we
limited the maximum number of paths of a link to 20. In addition, the ray
tracing tool only collects paths with a power of at least -250 dBm.

Critical to modeling the frequency dependence is the effect of diffuse
scattering, which is particularly prominent in the mmWave and THz frequencies
[20, 21, 22]. As part of our simulation, we enable the diffusing scattering
model in the ray tracing tool which is based on the directive model developed
by [19]. A key parameter is the scattering coefficient, $S$, which is the
ratio of the scattered to incident electric field. Following measurements in
[19], we set the scattering coefficient to $S=0.273$ for
$28\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$, and $S=0.4$ for
$140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$, in which case, rays in
$140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ disperse more power on the
forward lobe of diffuse scattered. A diffuse scattering interaction is limited
to the last interaction along a path before it reaches the receiver. This
helps reduce the run time for diffuse scattering calculations.

##  IV Paths Cluster and Generative Neural Network Modeling

Figure 2: The architecture for the K-Means path clusters generator and the
generative adversarial neural network model

###  IV-A Path Clusters Identification by $K$-means

The ray tracing data produces rays at each frequency. Our first step is to
pre-preocess the rays to identify the clusters and determine the cluster
parameters. Similar to [28], we use a $k$-means clustering procedure on the
inclination and azimuth angles of arrival, inclination and azimuth angles of
departure, and delay for each link of the set. Silhouette scores [32] are used
to determine the best number of clusters for each link. A typical
identification of the clusters is shown in the map in (2) where the circles
indicate the identified clusters.

After identifying the clusters, we can determine the angular and delay spread
and power in each cluster to to generate the cluster vector ${\mathbf{x}}$ as
indicated in (3) for each link. To limit the size of the cluster vector, we
use $L=10$ as the maximum number of clusters.

We obtain one data record ${\mathbf{x}}$ for each link. Each link also has
condition vector ${\mathbf{u}}$ as in (4). The condition vector ${\mathbf{u}}$
is first converted to a 3D distance and the logarithm of the 3D distance,
denoted ${\mathbf{u}}=(d_{3D},\log_{10}d_{3D})$.

###  IV-B Cluster Generative Network

The goal of the network is to generate random cluster vectors ${\mathbf{x}}$
in (3) following the conditional distribution $p({\mathbf{x}}|{\mathbf{u}})$
as observed in the data. For this purpose, we use a conditional Wasserstein
generative adversarial network with gradient penalty (CWGAN-GP) [33, 34] that
is relatively simple to implement while being effective in complex datasets.

The CWGAN-GP can be summarized as follows: There are two components: a
_generator_ ${\mathbf{G}}$ and _critic_ ${\mathbf{C}}$. The generator takes
the condition vector ${\mathbf{u}}$ and some random input ${\mathbf{z}}\sim
p({\mathbf{z}})$ and generates a random path data vector
${\mathbf{x}}={\mathbf{G}}({\mathbf{u}},{\mathbf{z}})$. The vector
${\mathbf{z}}$ is called the _latent vector_. As is commonly used, we take its
distribution $p({\mathbf{z}})$ to be a unit variance Gaussian vector. We set
the latent dimension to 20. To ensure the generator matches the data
distribution, one also trains a critic function ${\mathbf{C}}$ that attempt to
discriminate between the generated and true samples. The critic ${\mathbf{C}}$
and generator ${\mathbf{G}}$ networks are trained via a loss:

$\displaystyle L=$
$\displaystyle\underbrace{\underset{\tilde{\boldsymbol{x}}\sim\mathbb{P}_{g}}{\mathbb{E}}[C(\tilde{\boldsymbol{x}})]-\underset{\boldsymbol{x}\sim\mathbb{P}_{r}}{\mathbb{E}}[C(\boldsymbol{x})]}_{\text{Original
critic loss }}+$
$\displaystyle\underbrace{\lambda\underset{\hat{\boldsymbol{x}}\sim\mathbb{P}_{\hat{\boldsymbol{x}}}}{\mathbb{E}}\left[\left(\left\|\nabla_{\hat{\boldsymbol{x}}}D(\hat{\boldsymbol{x}})\right\|_{2}-1\right)^{2}\right].}_{\text{Gradient
penalty }}$ (6)

where $\mathbb{P}_{g}$ is the generator distribution implicitly defined by
$\tilde{\boldsymbol{x}}={\mathbf{G}}(\boldsymbol{u},\boldsymbol{z}),\quad\boldsymbol{z}\sim
p(\boldsymbol{z})$, and $\mathbb{P}_{r}$ is the distribution of the data
points. The loss (IV-B) is optimized via a minimax: The critic ${\mathbf{C}}$
attempts to minimize the loss to discriminate between the generated and true
samples, while the generators attempts to maximize the loss to fake the
critic. The minimax is similar to other GANs, but the key concept in the WGAN-
GP is that the gradient penalty term avoids mode collapse – see [33, 34] for
details. For our application, both the generator and critic are realized with
as fully connected neural networks with parameters shown in Table I.

One important detail for the wireless channel modeling is that the condition
vector ${\mathbf{u}}$ and cluster data ${\mathbf{x}}$ have heterogeneous data
types and need to be brought to a uniform format. To overcome this challenge,
we align the angles of each path to the LOS direction and measure excess
signal delay relative to the LOS distance delay. We then scale all the values
using either min-max or standard scalers. Details are in the code [35].

TABLE I: Generative model configuration |  | Cluster GAN
---
Critic
| Cluster GAN
---
Generator
Num. of inputs | 170 $+$ 2 | 20 $+$ 2
Hidden units | [1120,560,280] | [280,560,1120]
Num. of outputs | 1 | 140
Optimizer | Adam
Learning rate | 0.0001
Epochs | 10000
Batch size | 1024
Num. of parameters | 1167551 | 1142290*
∗Different from the traditional parameters calculation method

##  V Modeling Results

As described in Section III, the data consists of 32000 links, with each link
having the paths at both $28\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ and
$140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$. We show the path clusters
following on a GAN framework is able to capture interesting joint statistics
between these two frequencies. In all the experiments below, we use 80% of the
links for training and 20% for test. The generative network is implemented in
Tensorflow 2, and the data set, code, and pre-train models can be accessed
through [35].

###  V-A Received Power with Omnidirectional Antennas

Figure 3: The CDF of the total received power of links on the test data versus
the CDF of randomly generated links’ total received power from the trained
model. Figure 4: The KDE plot of bivariate distributions of the total received
power in $28\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ and
$140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$.

We first evaluate the ability of the model to capture the joint distribution
of the total received power at the two frequencies. The comparison can be
performed as follows: Let
$\tilde{\boldsymbol{x}}={\mathbf{G}}(\boldsymbol{u},\boldsymbol{z})$ denote
the learned generative model and let $({\mathbf{u}}_{i},{\mathbf{x}}_{i})$,
$i=1,\ldots,N_{\rm ts}$ denote the test data, where each sample contains the
link condition ${\mathbf{u}}_{i}$ and the corresponding cluster data vector
${\mathbf{x}}_{i}$. For each test sample, we can compute the vector

${\mathbf{v}}_{i}=(v_{i1},v_{i2})=(\phi_{1}({\mathbf{x}}_{1}),\phi_{2}({\mathbf{x}}_{2})),$
(7)

where $v_{ij}$ is the omni-directional received power on sample $i$ at
frequency $j$ where $j=1,2$. Here, $\phi_{j}({\mathbf{x}})$ is the function
that computes the omni-directional received power at frequency $j$ from the
cluster data ${\mathbf{x}}$. The omni-directional received power is simply the
received power that would be experienced if the gNB and UE had omni-
directional antennas.

To compare these values with the generated distribution, for each test sample
$i$, we also generate a random sample
${\mathbf{x}}_{i}^{\text{rnd}}={\mathbf{G}}({\mathbf{u}}_{i},{\mathbf{z}}_{i})$
using the trained generator ${\mathbf{G}}$ and a random ${\mathbf{z}}_{i}$
with the same conditions ${\mathbf{u}}_{i}$ as the test data. We can then
compute a set of generated received powers

${\mathbf{v}}_{i}^{\rm rnd}=(v_{i1}^{\rm rnd},v_{i2}^{\rm
rnd})=(\phi_{1}({\mathbf{x}}_{1}^{\rm rnd}),\phi_{2}({\mathbf{x}}_{2}^{\rm
rnd})),$ (8)

Ideally, the samples ${\mathbf{v}}_{i}$ and ${\mathbf{v}}_{i}^{\rm rnd}$
should have a similar distribution.

Fig. 3 plots the empirical cumulative distribution functions (CDFs) for the
_marginal_ distributions for the data and model at the two frequencies.
Specifically, the left plot shows the CDF of the data $v_{ij}$ and generated
samples $v_{ij}^{\rm rnd}$ at frequency $j=1$. The right plot shows the data
and generated CDF for frequency $j=2$. We see in both cases, these two
marginal CDFs match almost perfectly.

But, what is most interesting is that the generator can also capture the joint
statistics. The left panel of Fig. 4 shows a kernel density estimation (KDE)
plot of the test data points $(v_{i1},v_{i2})$ and the right panel shows a KDE
plot of the trained model $(v_{i1}^{\rm rnd},v_{i2}^{\rm rnd})$. We see that
the two _joint_ distributions match well.

###  V-B Delay Distribution and RMS Delay Spread

Figure 5: The CDF of the paths minimum delay in clusters. Figure 6: The CDF of
the RMS cluster delay spread per link, where we only considering the path loss
of the clusters are smaller than 160 dB.

To evaluate the model’s ability to generate the delay, we first plot the CDF
of the minimum path delay over the clusters, as shown in Fig. 5. There is an
almost identical delay between clusters generated by the network and the real
ray tracing value, and the maximum delay is also around 4000 nanoseconds. For
each link, we also compute the RMS delay spread in order to gain an intuitive
understanding of how the model fits the delays across the different clusters
within a link. Fig. 6 shows that the RMS delay spread at
$140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ is smaller than at
$28\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ overall. As the frequency
increases, the RMS delay spread becomes smaller due to the increase in power
loss by diffuse scattering. This conclusion is consistent with the mmWave and
sub-Terahertz measurements in [22].

###  V-C Angles Distribution and RMS Angles Spread

Figure 7: Azimuth angles vs. inclination angles, where we only consider the
path loss of the clusters that are smaller than 160 dB. Figure 8: RMS azimuth
AoA spread, including the zero values. Figure 9: RMS azimuth AoD spread,
including the zero values.

As we mentioned above, to overcome the challenge of heterogeneous data types,
we align the angles of each path to the direction of LOS. More precisely, when
preprocessing the angles of arrival, we choose the LOS AoA direction of a link
as the z-axis of the new spherical coordinate system and calculate the
associated azimuth and inclination transformation angles. Similarly, in the
case of angles of departure, the AoD direction of LOS is used. In this way,
the network can more effectively learn a statistical model of the angle
relative to the condition vector. As illustrated in Fig.7, both AoA and AoD
distributions are similar to those observed in real ray tracing samples. On
the other hand, the RMS angle spread has important implications for evaluating
channel models, especially when considering mmWave and sub-Terahertz bands.
Fig.8 and Fig.9 show CDF images of AoA and AoD RMS spread, respectively.
Overall, the model is able to capture the properties of the RMS angle spread
at both frequencies. As a result of our testing, the RMS angle spread at
$28\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ is slightly greater than at
$140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$. When there is only one cluster
on a link at a given frequency, the RMS angle spread is zero. Based on Figs. 8
and 9, we can see that at $140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$, the
proportion of RMS spread value of zero is higher than that at
$28\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$, owing to higher diffuse scattered
power and increased path loss, which reduces the number of paths that can be
detected by the receiver. The above observations are consistent with the
conclusions obtained from actual measurements at
$28\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ and
$140\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ in [22].

##  VI Conclusion

Wireless multi-path channels can exhibit complex statistical relationships
across different frequencies which are difficult to capture via standard
models such as [10], particularly for mmWave and THz channels. In this work,
we have presented a general modeling methodology for deriving full double
directional models at multiple frequencies. The methods use state-of-the-art
neural networks and make minimal a priori assumptions in the data. As such, we
show they are able to capture interesting cross-frequency relations. The
current method is based on extensive ray tracing data. A natural extension,
left as future work, will be to augment the data with real data as these
become available.

## References

  * [1] Z. Shen, A. Papasakellariou, J. Montojo, D. Gerstenberger, and F. Xu, “Overview of 3GPP LTE-Advanced Carrier Aggregation for 4G Wireless Communications,” _IEEE Communications Magazine_ , vol. 50, no. 2, pp. 122–130, 2012.
  * [2] O. N. Yilmaz, O. Teyeb, and A. Orsino, “Overview of LTE-NR dual connectivity,” _IEEE Communications Magazine_ , vol. 57, no. 6, pp. 138–144, 2019.
  * [3] A. Ghosh, A. Maeder, M. Baker, and D. Chandramouli, “5G evolution: A view on 5G cellular technology beyond 3GPP release 15,” _IEEE access_ , vol. 7, pp. 127 639–127 651, 2019.
  * [4] A. V. Lopez, A. Chervyakov, G. Chance, S. Verma, and Y. Tang, “Opportunities and challenges of mmWave NR,” _IEEE Wireless Communications_ , vol. 26, no. 2, pp. 4–6, 2019.
  * [5] G. Liu, Y. Huang, Z. Chen, L. Liu, Q. Wang, and N. Li, “5G deployment: Standalone vs. Non-standalone from the operator perspective,” _IEEE Communications Magazine_ , vol. 58, no. 11, pp. 83–89, 2020.
  * [6] N. González-Prelcic, A. Ali, V. Va, and R. W. Heath, “Millimeter-wave communication with out-of-band information,” _IEEE Communications Magazine_ , vol. 55, no. 12, pp. 140–146, 2017.
  * [7] M. Hashemi, C. E. Koksal, and N. B. Shroff, “Out-of-band millimeter wave beamforming and communications to achieve low latency and high energy efficiency in 5G systems,” _IEEE transactions on communications_ , vol. 66, no. 2, pp. 875–888, 2017.
  * [8] M. Giordani, M. Polese, A. Roy, D. Castor, and M. Zorzi, “Standalone and non-standalone beam management for 3GPP NR at mmWaves,” _IEEE Communications Magazine_ , vol. 57, no. 4, pp. 123–129, 2019.
  * [9] J. Yang, X. Yang, C.-K. Wen, and S. Jin, “Integrated sensing and communication with multi-domain cooperation,” _arXiv preprint arXiv:2105.03065_ , 2021\.
  * [10] 3GPP Technical Report 38.901, “Study on channel model for frequencies from 0.5 to 100 GHz (Release 16),” Dec. 2019.
  * [11] R. W. Heath Jr. and A. Lozano, _Foundations of MIMO Communication_. Cambridge University Press, 2018.
  * [12] T. S. Rappaport, R. W. Heath Jr., R. C. Daniels, and J. N. Murdock, _Millimeter Wave Wireless Communications_. Pearson Education, 2014.
  * [13] K. Stocker, B. Gschwendtner, and F. Landstorfer, “Neural network approach to prediction of terrestrial wave propagation for mobile radio,” in _IEE Proceedings H (Microwaves, Antennas and Propagation)_ , vol. 140, no. 4, 1993, pp. 315–320.
  * [14] P.-R. Chang and W.-H. Yang, “Environment-adaptation mobile radio propagation prediction using radial basis function neural networks,” _IEEE Trans. Veh. Techn._ , vol. 46, no. 1, pp. 155–160, 1997.
  * [15] L. Bai, C.-X. Wang, J. Huang, Q. Xu, Y. Yang, G. Goussetis, J. Sun, and W. Zhang, “Predicting wireless mmWave massive MIMO channel characteristics using machine learning algorithms,” _Wireless Commun. and Mobile Computing_ , 2018.
  * [16] J. Huang, C.-X. Wang, L. Bai, J. Sun, Y. Yang, J. Li, O. Tirkkonen, and M. Zhou, “A big data enabled channel model for 5G wireless communication systems,” _IEEE Trans. Big Data_ , vol. 6, no. 2, pp. 211–222, 2018.
  * [17] X. Zhao, F. Du, S. Geng, Z. Fu, Z. Wang, Y. Zhang, Z. Zhou, L. Zhang, and L. Yang, “Playback of 5G and beyond measured MIMO channels by an ANN-based modeling and simulation framework,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 9, pp. 1945–1954, 2020.
  * [18] W. Xia, S. Rangan, M. Mezzavilla, A. Lozano, G. Geraci, V. Semkin, and G. Loianno, “Millimeter wave channel modeling via generative neural networks,” in _Proc. IEEE Globecom Workshops._ , 2020, pp. 1–6.
  * [19] V. Degli-Esposti, F. Fuschini, E. M. Vitucci, and G. Falciasecca, “Measurement and modelling of scattering from buildings,” _IEEE Transactions on Antennas and Propagation_ , vol. 55, no. 1, pp. 143–153, 2007.
  * [20] S. Schecklman, L. M. Zurk, S. Henry, and G. P. Kniffin, “Terahertz material detection from diffuse surface scattering,” _Journal of Applied Physics_ , vol. 109, no. 9, p. 094902, 2011.
  * [21] S. Ju, S. H. A. Shah, M. A. Javed, J. Li, G. Palteru, J. Robin, Y. Xing, O. Kanhere, and T. S. Rappaport, “Scattering mechanisms and modeling for terahertz wireless communications,” in _ICC 2019-2019 IEEE International Conference on Communications (ICC)_. IEEE, 2019, pp. 1–7.
  * [22] Y. Xing and T. S. Rappaport, “Millimeter wave and terahertz urban microcell propagation measurements and models,” _IEEE Communications Letters_ , vol. 25, no. 12, pp. 3755–3759, 2021.
  * [23] J. Huang, C.-X. Wang, H. Chang, J. Sun, and X. Gao, “Multi-frequency multi-scenario millimeter wave MIMO channel measurements and modeling for B5G wireless communication systems,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 9, pp. 2010–2025, 2020.
  * [24] Z. Cui, C. Briso-Rodriguez, K. Guan, Z. Zhong, and F. Quitin, “Multi-frequency air-to-ground channel measurements and analysis for UAV communication systems,” _IEEE access_ , vol. 8, pp. 110 565–110 574, 2020\.
  * [25] W. Khawaja, O. Ozdemir, and I. Guvenc, “UAV Air-to-Ground Channel Characterization for mmWave Systems,” in _Proc. IEEE VTC-Fall_ , 2017.
  * [26] A. Alkhateeb, “DeepMIMO: A generic deep learning dataset for millimeter wave and massive MIMO applications,” in _Proc. of Information Theory and Applications Workshop (ITA)_ , San Diego, CA, Feb 2019, pp. 1–8.
  * [27] “Remcom,” available on-line at https://www.remcom.com/, 2021-11-11.
  * [28] R. He, B. Ai, A. F. Molisch, G. L. Stuber, Q. Li, Z. Zhong, and J. Yu, “Clustering enabled wireless channel modeling using big data algorithms,” _IEEE Communications Magazine_ , vol. 56, no. 5, pp. 177–183, 2018.
  * [29] M. Shafi, A. F. Molisch, P. J. Smith, T. Haustein, P. Zhu, P. De Silva, F. Tufvesson, A. Benjebbour, and G. Wunder, “5G: A Tutorial Overview of Standards, Trials, Challenges, Deployment, and Practice,” _IEEE Journal on Selected Areas in Communications_ , vol. 35, no. 6, pp. 1201–1221, 2017\.
  * [30] M. Giordani, M. Polese, M. Mezzavilla, S. Rangan, and M. Zorzi, “Toward 6g networks: Use cases and technologies,” _IEEE Communications Magazine_ , vol. 58, no. 3, pp. 55–61, 2020.
  * [31] T. S. Rappaport, Y. Xing, O. Kanhere, S. Ju, A. Madanayake, S. Mandal, A. Alkhateeb, and G. C. Trichopoulos, “Wireless communications and applications above 100 GHz: Opportunities and challenges for 6G and beyond,” _IEEE access_ , vol. 7, pp. 78 729–78 757, 2019.
  * [32] P. J. Rousseeuw, “Silhouettes: a graphical aid to the interpretation and validation of cluster analysis,” _Journal of computational and applied mathematics_ , vol. 20, pp. 53–65, 1987.
  * [33] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved training of wasserstein gans,” _arXiv preprint arXiv:1704.00028_ , 2017.
  * [34] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in _International conference on machine learning_. PMLR, 2017, pp. 214–223.
  * [35] “mmWave channel modeling git hub repository,” available on-line at https://github.com/klmyyaqihu/mmwchanmod-GAN, 2021-12-14.