§ INTRODUCTION

§ BACKGROUND AND RELATED WORK

§ $\SUTRANETS$: SUB-SERIES AUTOREGRESSIVE NETWORKS

§ EXPERIMENTS

§ CONCLUSION

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Alexandrov, Benidis, Bohlke-Schneider,
Flunkert, Gasthaus, Januschowski, Maddix, Rangapuram, Salinas, Schulz,
Lorenzo, Türkmen, and WangAlexandrov et al2020]
authorpersonAlexander Alexandrov,
personKonstantinos Benidis, personMichael
Bohlke-Schneider, personValentin Flunkert, personJan
Gasthaus, personTim Januschowski, personDanielle C
Maddix, personSyama Rangapuram, personDavid Salinas,
personJasper Schulz, personStella Lorenzo,
personAli Caner Türkmen, and personYuyang
Wang. year2020.
GluonTS: Probabilistic and Neural Time Series
Modeling in Python.
journalJournal of Machine Learning Research
volume21, number116 (year2020),

[Anava and KuznetsovAnava and
authorpersonOren Anava and personVitaly
Kuznetsov. year2017.
titleWeb Traffic Time Series Forecasting.

[Athanasopoulos, Hyndman, Kourentzes, and
PetropoulosAthanasopoulos et al2017]
authorpersonGeorge Athanasopoulos,
personRob J Hyndman, personNikolaos Kourentzes, and
personFotios Petropoulos. year2017.
Forecasting with temporal hierarchies.
journalEuropean Journal of Operational Research
volume262, number1 (year2017),

[Baldi and SadowskiBaldi and
authorpersonPierre Baldi and personPeter J
Sadowski. year2013.
Understanding Dropout.
journalAdvances in neural information processing
systems  volume26 (year2013).

[Bender, Gebru, McMillan-Major, and
ShmitchellBender et al2021]
authorpersonEmily M Bender, personTimnit
Gebru, personAngelina McMillan-Major, and
personShmargaret Shmitchell. year2021.
On the Dangers of Stochastic Parrots: Can Language
Models Be Too Big?. In booktitleACM Conference on Fairness,
Accountability, and Transparency. pages610–623.

[Bengio, Vinyals, Jaitly, and ShazeerBengio
et al2015]
authorpersonSamy Bengio, personOriol
Vinyals, personNavdeep Jaitly, and personNoam
Shazeer. year2015.
Scheduled Sampling for Sequence Prediction with
Recurrent Neural Networks.
journalAdvances in Neural Information Processing
Systems  volume28 (year2015).

[Bengio, Ducharme, Vincent, and JanvinBengio
et al2003]
authorpersonYoshua Bengio, personRéjean
Ducharme, personPascal Vincent, and personChristian
Janvin. year2003.
A neural probabilistic language model.
journalJournal of Machine Learning Research
volume3 (year2003), pages1137–1155.

[Bergsma, Zeyl, Anaraki, and GuoBergsma
et al2022]
authorpersonShane Bergsma, personTim Zeyl,
personJavad Rahimipour Anaraki, and personLei Guo.
C2FAR: Coarse-to-Fine Autoregressive Networks for
Precise Probabilistic Forecasting. In booktitleAdvances in
Neural Information Processing Systems, Vol. volume35.

[Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal,
Neelakantan, Shyam, Sastry, Askell, et alBrown
et al2020]
authorpersonTom Brown, personBenjamin Mann,
personNick Ryder, personMelanie Subbiah,
personJared D Kaplan, personPrafulla Dhariwal,
personArvind Neelakantan, personPranav Shyam,
personGirish Sastry, personAmanda Askell,
et al year2020.
Language models are few-shot learners.
journalAdvances in Neural Information Processing
Systems  volume33 (year2020),

[Challu, Olivares, Oreshkin, Garza,
Mergenthaler-Canseco, and DubrawskiChallu et al2022]
authorpersonCristian Challu, personKin G
Olivares, personBoris N Oreshkin, personFederico
Garza, personMax Mergenthaler-Canseco, and
personArtur Dubrawski. year2022.
N-HiTS: Neural Hierarchical Interpolation for
Time Series Forecasting.
journalarXiv preprint arXiv:2201.12886

[Chang, Zhang, Han, Yu, Guo, Tan, Cui,
Witbrock, Hasegawa-Johnson, and HuangChang et al2017]
authorpersonShiyu Chang, personYang Zhang,
personWei Han, personMo Yu, personXiaoxiao
Guo, personWei Tan, personXiaodong Cui,
personMichael Witbrock, personMark A
Hasegawa-Johnson, and personThomas S Huang.
Dilated recurrent neural networks.
journalAdvances in Neural Information Processing
Systems  volume30 (year2017).

[Chen, Ma, and LinChen et al2021]
authorpersonZipeng Chen, personQianli Ma,
and personZhenxi Lin. year2021.
Time-Aware Multi-Scale RNNs for Time Series
Modeling.. In booktitleInternational Joint Conference On
Artificial Intelligence. pages2285–2291.

[Child, Gray, Radford, and SutskeverChild
et al2019]
authorpersonRewon Child, personScott Gray,
personAlec Radford, and personIlya Sutskever.
Generating long sequences with sparse
journalarXiv preprint arXiv:1904.10509

[Chow and LinChow and Lin1971]
authorpersonGregory C Chow and
personAn-loh Lin. year1971.
Best linear unbiased interpolation, distribution,
and extrapolation of time series by related series.
journalThe Review of Economics and Statistics
(year1971), pages372–375.

[Chung, Ahn, and BengioChung
et al2017]
authorpersonJunyoung Chung, personSungjin
Ahn, and personYoshua Bengio.
Hierarchical Multiscale Recurrent Neural Networks.
journalInternational Conference on Learning
Representations (year2017).

[Clark, Dyer, Lavie, and SmithClark
et al2011]
authorpersonJonathan H Clark, personChris
Dyer, personAlon Lavie, and personNoah A Smith.
Better Hypothesis Testing for Statistical Machine
Translation: Controlling for Optimizer Instability. In
booktitleAnnual Meeting of the Association for Computational
Linguistics: Human Language Technologies. pages176–181.

[Cortez, Bonde, Muzio, Russinovich, Fontoura,
and BianchiniCortez et al2017]
authorpersonEli Cortez, personAnand Bonde,
personAlexandre Muzio, personMark Russinovich,
personMarcus Fontoura, and personRicardo
Bianchini. year2017.
Resource Central: Understanding and Predicting
Workloads for Improved Resource Management in Large Cloud Platforms. In
booktitleSymposium on Operating Systems Principles.

[Didolkar, Gupta, Goyal, Gundavarapu, Lamb, Ke,
and BengioDidolkar et al2022]
authorpersonAniket Didolkar, personKshitij
Gupta, personAnirudh Goyal, personNitesh Bharadwaj
Gundavarapu, personAlex M Lamb, personNan Rosemary
Ke, and personYoshua Bengio. year2022.
Temporal Latent Bottleneck: Synthesis of Fast and
Slow Processing Mechanisms in Sequence Learning.
journalAdvances in Neural Information Processing
Systems  volume35 (year2022),

[Dua and GraffDua and Graff2017]
authorpersonDheeru Dua and personCasey
Graff. year2017.
titleUCI Machine Learning Repository.

[El Hihi and BengioEl Hihi and
authorpersonSalah El Hihi and
personYoshua Bengio. year1995.
Hierarchical Recurrent Neural Networks for
Long-Term Dependencies.
journalAdvances in Neural Information Processing
Systems  volume8 (year1995).

[Faloutsos, Flunkert, Gasthaus, Januschowski,
and WangFaloutsos et al2019]
authorpersonChristos Faloutsos,
personValentin Flunkert, personJan Gasthaus,
personTim Januschowski, and personYuyang Wang.
Forecasting Big Time Series: Theory and Practice.
In booktitleACM SIGKDD International Conference on Knowledge
Discovery & Data Mining. pages3209–3210.

[Fan, Zhang, Pan, Li, Zhang, Yuan, Wu, Wang,
Pei, and HuangFan et al2019]
authorpersonChenyou Fan, personYuze Zhang,
personYi Pan, personXiaoyue Li, personChi
Zhang, personRong Yuan, personDi Wu,
personWensheng Wang, personJian Pei, and
personHeng Huang. year2019.
Multi-Horizon Time Series Forecasting with Temporal
Attention Learning. In booktitleACM SIGKDD International
Conference on Knowledge Discovery & Data Mining.

[Gasthaus, Benidis, Wang, Rangapuram, Salinas,
Flunkert, and JanuschowskiGasthaus et al2019]
authorpersonJan Gasthaus, personKonstantinos
Benidis, personYuyang Wang, personSyama Sundar
Rangapuram, personDavid Salinas, personValentin
Flunkert, and personTim Januschowski.
Probabilistic Forecasting with Spline Quantile
Function RNNs. In booktitleInternational Conference on
Artificial Intelligence and Statistics. pages1901–1910.

[Gebru, Morgenstern, Vecchione, Vaughan,
Wallach, III, and CrawfordGebru et al2021]
authorpersonTimnit Gebru, personJamie
Morgenstern, personBriana Vecchione,
personJennifer Wortman Vaughan, personHanna Wallach,
personHal Daumé III, and personKate
Crawford. year2021.
Datasheets for Datasets.
journalCommun. ACM volume64,
number12 (year2021), pages86–92.

[Germain, Gregor, Murray, and
LarochelleGermain et al2015]
authorpersonMathieu Germain, personKarol
Gregor, personIain Murray, and personHugo
Larochelle. year2015.
MADE: Masked Autoencoder for Distribution
Estimation. In booktitleInternational Conference on Machine
Learning. pages881–889.

[Gneiting and RafteryGneiting and
authorpersonTilmann Gneiting and
personAdrian E Raftery. year2007.
Strictly Proper Scoring Rules, Prediction, and
journalJ. Amer. Statist. Assoc.
volume102, number477 (year2007),

[Gonen and GoldbergGonen and
authorpersonHila Gonen and personYoav
Goldberg. year2019.
Lipstick on a pig: Debiasing methods cover up
systematic gender biases in word embeddings but do not remove them.
journalarXiv preprint arXiv:1903.03862

[Goodfellow, Pouget-Abadie, Mirza, Xu,
Warde-Farley, Ozair, Courville, and BengioGoodfellow et al2014]
authorpersonIan Goodfellow, personJean
Pouget-Abadie, personMehdi Mirza, personBing Xu,
personDavid Warde-Farley, personSherjil Ozair,
personAaron Courville, and personYoshua Bengio.
Generative Adversarial Nets.
journalAdvances in Neural Information Processing
Systems (year2014), pages2672–2680.

[Gouttes, Rasul, Koren, Stephan, and
NaghibiGouttes et al2021]
authorpersonAdèle Gouttes, personKashif
Rasul, personMateusz Koren, personJohannes Stephan,
and personTofigh Naghibi. year2021.
Probabilistic Time Series Forecasting with Implicit
Quantile Networks.
journalarXiv preprint arXiv:2107.03743

authorpersonAlex Graves.
Generating Sequences with Recurrent Neural
journalarXiv preprint arXiv:1308.0850

[Graves, Mohamed, and HintonGraves
et al2013]
authorpersonAlex Graves, personAbdel-rahman
Mohamed, and personGeoffrey Hinton.
Speech Recognition with Deep Recurrent Neural
Networks. In booktitle2013 IEEE International Conference on
Acoustics, Speech and Signal Processing. pages6645–6649.

[Graves and SchmidhuberGraves and
authorpersonAlex Graves and
personJürgen Schmidhuber. year2008.
Offline handwriting recognition with
multidimensional recurrent neural networks.
journalAdvances in Neural Information Processing
Systems  volume21 (year2008).

[He, Zhang, Ren, and SunHe
et al2016]
authorpersonKaiming He, personXiangyu
Zhang, personShaoqing Ren, and personJian Sun.
Deep Residual Learning for Image Recognition. In
booktitleIEEE Conference on Computer Vision and Pattern
Recognition. pages770–778.

[Hewamalage, Ackermann, and
BergmeirHewamalage et al2023]
authorpersonHansika Hewamalage, personKlaus
Ackermann, and personChristoph Bergmeir.
Forecast evaluation for data scientists: common
pitfalls and best practices.
journalData Mining and Knowledge Discovery
volume37, number2 (year2023),

[Hochreiter and SchmidhuberHochreiter and
authorpersonSepp Hochreiter and
personJürgen Schmidhuber. year1997.
Long Short-Term Memory.
journalNeural computation volume9,
number8 (year1997), pages1735–1780.

[Hyndman and AthanasopoulosHyndman and
authorpersonRob J Hyndman and personGeorge
Athanasopoulos. year2018.
booktitleForecasting: Principles and Practice.

[Kalchbrenner, Elsen, Simonyan, Noury,
Casagrande, Lockhart, Stimberg, van den Oord, Dieleman, and
KavukcuogluKalchbrenner et al2018]
authorpersonNal Kalchbrenner, personErich
Elsen, personKaren Simonyan, personSeb Noury,
personNorman Casagrande, personEdward Lockhart,
personFlorian Stimberg, personAaron van den Oord,
personSander Dieleman, and personKoray
Kavukcuoglu. year2018.
Efficient Neural Audio Synthesis. In
booktitleInternational Conference on Machine Learning.

[Katharopoulos, Vyas, Pappas, and
FleuretKatharopoulos et al2020]
authorpersonAngelos Katharopoulos,
personApoorv Vyas, personNikolaos Pappas, and
personFrançois Fleuret. year2020.
Transformers are RNNs: Fast autoregressive
transformers with linear attention. In
booktitleInternational Conference on Machine Learning.

[Kingma and BaKingma and Ba2014]
authorpersonDiederik P Kingma and
personJimmy Ba. year2014.
Adam: A Method for Stochastic Optimization.
journalarXiv preprint arXiv:1412.6980

[Koutnik, Greff, Gomez, and
SchmidhuberKoutnik et al2014]
authorpersonJan Koutnik, personKlaus Greff,
personFaustino Gomez, and personJuergen
Schmidhuber. year2014.
A clockwork RNN. In
booktitleInternational conference on machine learning.

[Lai, Chang, Yang, and LiuLai
et al2018]
authorpersonGuokun Lai, personWei-Cheng
Chang, personYiming Yang, and personHanxiao Liu.
Modeling Long-and Short-Term Temporal Patterns with
Deep Neural Networks. In booktitleACM SIGIR Conference on
Research & Development in Information Retrieval. pages95–104.

[Lamb, Alias Parth Goyal, Zhang, Zhang,
Courville, and BengioLamb et al2016]
authorpersonAlex M Lamb, personAnirudh Goyal
Alias Parth Goyal, personYing Zhang, personSaizheng
Zhang, personAaron C Courville, and personYoshua
Bengio. year2016.
Professor forcing: A new algorithm for training
recurrent networks.
journalAdvances in Neural Information Processing
Systems  volume29 (year2016).

[Le, Jaitly, and HintonLe
et al2015]
authorpersonQuoc V Le, personNavdeep
Jaitly, and personGeoffrey E Hinton.
A Simple Way to Initialize Recurrent Networks of
Rectified Linear Units.
journalarXiv preprint arXiv:1504.00941

[LeCun, Cortes, and BurgesLeCun
et al1998]
authorpersonYann LeCun, personCorinna
Cortes, and personChristopher JC Burges.
titleThe MNIST database of handwritten digits.

[Li, Zhang, Zhou, Liu, Zhang, Xue, Xu, Fan,
Zhou, and LiLi et al2023]
authorpersonJi You Li, personJiachi Zhang,
personWenchao Zhou, personYuhang Liu,
personShuai Zhang, personZhuoming Xue,
personDing Xu, personHua Fan,
personFangyuan Zhou, and personFeifei Li.
Eigen: End-to-end Resource Optimization for
Large-Scale Databases on the Cloud.
journalProceedings of the VLDB Endowment
volume16, number12 (year2023),

[Li, Jin, Xuan, Zhou, Chen, Wang, and YanLi
et al2019]
authorpersonShiyang Li, personXiaoyong Jin,
personYao Xuan, personXiyou Zhou,
personWenhu Chen, personYu-Xiang Wang, and
personXifeng Yan. year2019.
Enhancing the Locality and Breaking the Memory
Bottleneck of Transformer on Time Series Forecasting.
journalAdvances in Neural Information Processing
Systems  volume32 (year2019).

[Li, Zhou, Xiao, He, Huang, and LiLi
et al2017]
authorpersonZimo Li, personYi Zhou,
personShuangjiu Xiao, personChong He,
personZeng Huang, and personHao Li.
Auto-conditioned recurrent networks for extended
complex human motion synthesis.
journalarXiv preprint arXiv:1707.05363

[Lin, Lin, Wu, Zhao, Mo, and ZhangLin
et al2023]
authorpersonShengsheng Lin, personWeiwei
Lin, personWentai Wu, personFeiyu Zhao,
personRuichao Mo, and personHaotong Zhang.
SegRNN: Segment Recurrent Neural Network for
Long-Term Time Series Forecasting.
journalarXiv preprint arXiv:2308.11200

[Liu, Yu, Liao, Li, Lin, Liu, and DustdarLiu
et al2021]
authorpersonShizhan Liu, personHang Yu,
personCong Liao, personJianguo Li,
personWeiyao Lin, personAlex X Liu, and
personSchahram Dustdar. year2021.
Pyraformer: Low-complexity pyramidal attention for
long-range time series modeling and forecasting. In
booktitleInternational Conference on Learning

[Lu, Liu, Fuh, Lian, Guo, Yang, Marelli, and
LiLu et al2021]
authorpersonXing Han Lu, personAihua Liu,
personShih-Chieh Fuh, personYi Lian,
personLiming Guo, personYi Yang,
personAriane Marelli, and personYue Li.
Recurrent disease progression networks for
modelling risk trajectory of heart failure.
journalPloS one volume16,
number1 (year2021).

[Madhusudhanan, Burchert, Duong-Trung, Born,
and Schmidt-ThiemeMadhusudhanan et al2021]
authorpersonKiran Madhusudhanan,
personJohannes Burchert, personNghia Duong-Trung,
personStefan Born, and personLars Schmidt-Thieme.
Yformer: U-net inspired transformer architecture
for far horizon time series forecasting.
journalarXiv preprint arXiv:2110.08255

[Melis, Dyer, and BlunsomMelis
et al2017]
authorpersonGábor Melis, personChris
Dyer, and personPhil Blunsom.
On the State of the Art of Evaluation in Neural
Language Models.
journalarXiv preprint arXiv:1707.05589

[Mitchell, Wu, Zaldivar, Barnes, Vasserman,
Hutchinson, Spitzer, Raji, and GebruMitchell et al2019]
authorpersonMargaret Mitchell, personSimone
Wu, personAndrew Zaldivar, personParker Barnes,
personLucy Vasserman, personBen Hutchinson,
personElena Spitzer, personInioluwa Deborah Raji,
and personTimnit Gebru. year2019.
Model Cards for Model Reporting. In
booktitleConference on Fairness, Accountability, and
Transparency. pages220–229.

[Mukherjee, Shankar, Ghosh, Tathawadekar,
Kompalli, Sarawagi, and ChaudhuryMukherjee et al2018]
authorpersonSrayanta Mukherjee,
personDevashish Shankar, personAtin Ghosh,
personNilam Tathawadekar, personPramod Kompalli,
personSunita Sarawagi, and personKrishnendu
Chaudhury. year2018.
AR-MDN: Associative and Recurrent Mixture Density
Networks for eRetail Demand Forecasting.
journalarXiv preprint arXiv:1803.03800

authorpersonKevin P Murphy.
booktitleMachine Learning: A Probabilistic
publisherMIT press.

[Nadeem, Bethke, and ReddyNadeem
et al2020]
authorpersonMoin Nadeem, personAnna Bethke,
and personSiva Reddy. year2020.
StereoSet: Measuring stereotypical bias in
pretrained language models.
journalarXiv preprint arXiv:2004.09456

[Nie, Nguyen, Sinthong, and KalagnanamNie
et al2022]
authorpersonYuqi Nie, personNam H Nguyen,
personPhanwadee Sinthong, and personJayant
Kalagnanam. year2022.
A time series is worth 64 words: Long-term
forecasting with transformers.
journalarXiv preprint arXiv:2211.14730

[O'Neill and BollegalaO'Neill and
authorpersonJames O'Neill and
personDanushka Bollegala. year2018.
Analysing Dropout and Compounding Errors in Neural
Language Models.
journalarXiv preprint arXiv:1811.00998

[Oreshkin, Carpov, Chapados, and
BengioOreshkin et al2019]
authorpersonBoris N Oreshkin, personDmitri
Carpov, personNicolas Chapados, and personYoshua
Bengio. year2019.
N-BEATS: Neural basis expansion analysis for
interpretable time series forecasting.
journalarXiv preprint arXiv:1905.10437

[Ouyang, Wu, Jiang, Almeida, Wainwright,
Mishkin, Zhang, Agarwal, Slama, Ray, et alOuyang
et al2022]
authorpersonLong Ouyang, personJeffrey Wu,
personXu Jiang, personDiogo Almeida,
personCarroll Wainwright, personPamela Mishkin,
personChong Zhang, personSandhini Agarwal,
personKatarina Slama, personAlex Ray,
et al year2022.
Training language models to follow instructions
with human feedback.
journalAdvances in Neural Information Processing
Systems  volume35 (year2022),

[Paszke, Gross, Massa, Lerer, Bradbury, Chanan,
Killeen, Lin, Gimelshein, Antiga, Desmaison, Köpf, Yang, DeVito,
Raison, Alykhan, Chilamkurthy, Steiner, Fang, Bai, and ChintalaPaszke
et al2019]
authorpersonAdam Paszke, personSam Gross,
personFrancisco Massa, personAdam Lerer,
personJames Bradbury, personGregory Chanan,
personTrevor Killeen, personZeming Lin,
personNatalia Gimelshein, personLuca Antiga,
personAlban Desmaison, personAndreas Köpf,
personEdward Yang, personZach DeVito,
personMartin Raison, personTejani Alykhan,
personSasank Chilamkurthy, personBenoit Steiner,
personLu Fang, personJunjie Bai, and
personSoumith Chintala. year2019.
PyTorch: An Imperative Style, High-Performance
Deep Learning Library.
journalAdvances in Neural Information Processing
Systems  volume32 (year2019).

[Rabanser, Januschowski, Flunkert, Salinas, and
GasthausRabanser et al2020]
authorpersonStephan Rabanser, personTim
Januschowski, personValentin Flunkert, personDavid
Salinas, and personJan Gasthaus.
The Effectiveness of Discretization in Forecasting:
An Empirical Study on Neural Time Series Models.
journalarXiv preprint arXiv:2005.10111

[Radford, Narasimhan, Salimans, and
SutskeverRadford et al2018]
authorpersonAlec Radford, personKarthik
Narasimhan, personTim Salimans, and personIlya
Sutskever. year2018.
Improving language understanding by generative

[Rangapuram, Werner, Benidis, Mercado,
Gasthaus, and JanuschowskiRangapuram et al2021]
authorpersonSyama Sundar Rangapuram,
personLucien D Werner, personKonstantinos Benidis,
personPedro Mercado, personJan Gasthaus, and
personTim Januschowski. year2021.
End-to-End Learning of Coherent Probabilistic
Forecasts for Hierarchical Time Series. In
booktitleInternational Conference on Machine Learning.

[Rasul, Seward, Schuster, and VollgrafRasul
et al2021]
authorpersonKashif Rasul, personCalvin
Seward, personIngmar Schuster, and personRoland
Vollgraf. year2021.
Autoregressive denoising diffusion models for
multivariate probabilistic time series forecasting. In
booktitleInternational Conference on Machine Learning.
PMLR, pages8857–8868.

[Rasul, Sheikh, Schuster, Bergmann, and
VollgrafRasul et al2020]
authorpersonKashif Rasul, personAbdul-Saboor
Sheikh, personIngmar Schuster, personUrs Bergmann,
and personRoland Vollgraf. year2020.
Multi-Variate Probabilistic Time Series Forecasting
via Conditioned Normalizing Flows.
journalarXiv preprint arXiv:2002.06103

[Rudinger, Naradowsky, Leonard, and
Van DurmeRudinger et al2018]
authorpersonRachel Rudinger, personJason
Naradowsky, personBrian Leonard, and personBenjamin
Van Durme. year2018.
Gender bias in coreference resolution.
journalarXiv preprint arXiv:1804.09301

[Salinas, Bohlke-Schneider, Callot, Medico, and
GasthausSalinas et al2019]
authorpersonDavid Salinas, personMichael
Bohlke-Schneider, personLaurent Callot, personRoberto
Medico, and personJan Gasthaus.
High-Dimensional Multivariate Forecasting with
Low-Rank Gaussian Copula Processes.
journalAdvances in Neural Information Processing
Systems  volume32 (year2019).

[Salinas, Flunkert, Gasthaus, and
JanuschowskiSalinas et al2020]
authorpersonDavid Salinas, personValentin
Flunkert, personJan Gasthaus, and personTim
Januschowski. year2020.
DeepAR: Probabilistic Forecasting with
Autoregressive Recurrent Networks.
journalInternational Journal of Forecasting
volume36, number3 (year2020),

[Sangiorgio and DercoleSangiorgio and
authorpersonMatteo Sangiorgio and
personFabio Dercole. year2020.
Robustness of LSTM neural networks for multi-step
forecasting of chaotic time series.
journalChaos, Solitons & Fractals
volume139 (year2020), pages110045.

[Sax and SteinerSax and Steiner2013]
authorpersonChristoph Sax and personPeter
Steiner. year2013.
Temporal disaggregation of time series.
journalThe R Journal volume5,
number2 (year2013), pages80–87.

authorpersonJürgen Schmidhuber.
Learning Complex, Extended Sequences Using the
Principle of History Compression.
journalNeural Computation volume4,
number2 (year1992), pages234–242.

authorpersonAmazon Web Services.
titleDeepAR+ Algorithm - Amazon Forecast.

Accessed: 2023-05-01.

[Shabani, Abdi, Meng, and SylvainShabani
et al2022]
authorpersonAmin Shabani, personAmir Abdi,
personLili Meng, and personTristan Sylvain.
Scaleformer: Iterative Multi-Scale Refining
Transformers for Time Series Forecasting.
journalarXiv preprint arXiv:2206.04038

[Sheng, Chang, Natarajan, and PengSheng
et al2019]
authorpersonEmily Sheng, personKai-Wei
Chang, personPremkumar Natarajan, and personNanyun
Peng. year2019.
The woman worked as a babysitter: On biases in
language generation.
journalarXiv preprint arXiv:1909.01326

authorpersonSlawek Smyl.
A hybrid method of exponential smoothing and
recurrent neural networks for time series forecasting.
journalInternational Journal of Forecasting
volume36, number1 (year2020),

[Sordoni, Bengio, Vahabi, Lioma, Grue Simonsen,
and NieSordoni et al2015]
authorpersonAlessandro Sordoni, personYoshua
Bengio, personHossein Vahabi, personChristina Lioma,
personJakob Grue Simonsen, and personJian-Yun Nie.
A Hierarchical Recurrent Encoder-Decoder for
Generative Context-Aware Query Suggestion. In
booktitleConference on Information and Knowledge
Management. pages553–562.

[Srivastava, Hinton, Krizhevsky, Sutskever, and
SalakhutdinovSrivastava et al2014]
authorpersonNitish Srivastava,
personGeoffrey Hinton, personAlex Krizhevsky,
personIlya Sutskever, and personRuslan
Salakhutdinov. year2014.
Dropout: A Simple Way to Prevent Neural Networks
from Overfitting.
journalJournal of Machine Learning Research
volume15, number1 (year2014),

authorpersonArthur Suilin.
titleKaggle Web Traffic Time Series Forecasting: How it

[Taieb and AtiyaTaieb and Atiya2015]
authorpersonSouhaib Ben Taieb and
personAmir F Atiya. year2015.
A Bias and Variance Analysis for Multistep-Ahead
Time Series Forecasting.
journalIEEE Transactions on Neural Networks and
Learning Systems volume27, number1
(year2015), pages62–76.

[Tang and MattesonTang and Matteson2021]
authorpersonBinh Tang and personDavid S
Matteson. year2021.
Probabilistic Transformer for Time Series
journalAdvances in Neural Information Processing
Systems  volume34 (year2021),

[Theis and BethgeTheis and Bethge2015]
authorpersonLucas Theis and personMatthias
Bethge. year2015.
Generative image modeling using spatial LSTMs.
journalAdvances in Neural Information Processing
Systems  volume28 (year2015).

[Trinh, Giupponi, and DiniTrinh
et al2018]
authorpersonHoang Duy Trinh, personLorenza
Giupponi, and personPaolo Dini.
Mobile traffic prediction from raw data using
LSTM networks. In booktitle2018 IEEE 29th Annual
International Symposium on Personal, Indoor and Mobile Radio Communications
(PIMRC). pages1827–1832.

[Uria, Côté, Gregor, Murray, and
LarochelleUria et al2016]
authorpersonBenigno Uria,
personMarc-Alexandre Côté, personKarol
Gregor, personIain Murray, and personHugo
Larochelle. year2016.
Neural Autoregressive Distribution Estimation.
journalJournal of Machine Learning Research
volume17, number1 (year2016),

[van den Oord, Kalchbrenner, and
Kavukcuogluvan den Oord et al2016]
authorpersonAaron van den Oord, personNal
Kalchbrenner, and personKoray Kavukcuoglu.
Pixel Recurrent Neural Networks. In
booktitleInternational Conference on Machine Learning.

[Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and PolosukhinVaswani et al2017]
authorpersonAshish Vaswani, personNoam
Shazeer, personNiki Parmar, personJakob Uszkoreit,
personLlion Jones, personAidan N Gomez,
personŁukasz Kaiser, and personIllia
Polosukhin. year2017.
Attention Is All You Need.
journalAdvances in Neural Information Processing
Systems  volume30 (year2017).

[Vezhnevets, Osindero, Schaul, Heess,
Jaderberg, Silver, and KavukcuogluVezhnevets et al2017]
authorpersonAlexander Sasha Vezhnevets,
personSimon Osindero, personTom Schaul,
personNicolas Heess, personMax Jaderberg,
personDavid Silver, and personKoray Kavukcuoglu.
Feudal networks for hierarchical reinforcement
learning. In booktitleInternational Conference on Machine
Learning. PMLR, pages3540–3549.

[Wen, Torkkola, Narayanaswamy, and MadekaWen
et al2017]
authorpersonRuofeng Wen, personKari
Torkkola, personBalakrishnan Narayanaswamy, and
personDhruv Madeka. year2017.
A Multi-Horizon Quantile Recurrent Forecaster.
journalarXiv preprint arXiv:1711.11053

authorpersonMonier Williams.
booktitleSanskrit-English Dictionary.
publisherOxford: Clarendon Press.

[Wu, Hu, Liu, Zhou, Wang, and LongWu
et al2022]
authorpersonHaixu Wu, personTengge Hu,
personYong Liu, personHang Zhou,
personJianmin Wang, and personMingsheng Long.
TimesNet: Temporal 2D-variation modeling for
general time series analysis.
journalarXiv preprint arXiv:2210.02186

[Wu, Xu, Wang, and LongWu
et al2021]
authorpersonHaixu Wu, personJiehui Xu,
personJianmin Wang, and personMingsheng Long.
Autoformer: Decomposition Transformers with
Auto-Correlation for Long-Term Series Forecasting.
journalAdvances in Neural Information Processing
Systems  volume34 (year2021).

[Wu, Xiao, Ding, Zhao, Wei, and HuangWu
et al2020]
authorpersonSifan Wu, personXi Xiao,
personQianggang Ding, personPeilin Zhao,
personYing Wei, and personJunzhou Huang.
Adversarial Sparse Transformer for Time Series
journalAdvances in Neural Information Processing
Systems  volume33 (year2020),

[Yu, Rao, and DhillonYu
et al2016]
authorpersonHsiang-Fu Yu, personNikhil Rao,
and personInderjit S Dhillon. year2016.
Temporal Regularized Matrix Factorization for
High-Dimensional Time Series Prediction.
journalAdvances in Neural Information Processing
Systems  volume29 (year2016).

[Zeng, Chen, Zhang, and XuZeng
et al2022]
authorpersonAiling Zeng, personMuxi Chen,
personLei Zhang, and personQiang Xu.
Are Transformers effective for time series
journalarXiv preprint arXiv:2205.13504

[Zhang, Wu, Che, Lin, Memisevic, Salakhutdinov,
and BengioZhang et al2016]
authorpersonSaizheng Zhang, personYuhuai
Wu, personTong Che, personZhouhan Lin,
personRoland Memisevic, personRuss R Salakhutdinov,
and personYoshua Bengio. year2016.
Architectural complexity measures of recurrent
neural networks.
journalAdvances in Neural Information Processing
Systems  volume29 (year2016).

[Zhou, Zhang, Peng, Zhang, Li, Xiong, and
ZhangZhou et al2021]
authorpersonHaoyi Zhou, personShanghang
Zhang, personJieqi Peng, personShuai Zhang,
personJianxin Li, personHui Xiong, and
personWancai Zhang. year2021.
Informer: Beyond Efficient Transformer for Long
Sequence Time-Series Forecasting. In booktitleAAAI
Conference on Artificial Intelligence. pages11106–11115.

[Zhou, Ma, Wen, Wang, Sun, and JinZhou
et al2022]
authorpersonTian Zhou, personZiqing Ma,
personQingsong Wen, personXue Wang,
personLiang Sun, and personRong Jin.
FEDformer: Frequency enhanced decomposed
transformer for long-term series forecasting. In
booktitleInternational Conference on Machine Learning.

[Zhu, Liu, Jin, Zhang, Chen, Zhou, and
ZhouZhu et al2021]
authorpersonZhibo Zhu, personZiqi Liu,
personGe Jin, personZhiqiang Zhang,
personLei Chen, personJun Zhou, and
personJianyong Zhou. year2021.
MixSeq: Connecting Macroscopic Time Series
Forecasting with Microscopic Time Series Data.
journalAdvances in Neural Information Processing
Systems  volume34 (year2021).


© 2020 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

Conference: IEEE Global Communications Conference 2020, Taipei, Taiwan , 7-11
December 2020

DOI: 10.1109/GLOBECOM42002.2020.9322432

URL: https://ieeexplore.ieee.org/document/9322432

# Recurrent Neural Network Assisted Transmitter Selection for Secrecy in
Cognitive Radio Network

Shalini Tripathi1, Chinmoy Kundu2, Octavia A. Dobre3, Ankur Bansal4, and Mark
F. Flanagan5<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>14Department
of EE, Indian Institute of Technology Jammu, India  25School of Electrical and
Electronic Engineering, University College Dublin, Ireland  3Engineering and
Applied Science, Memorial University, Canada

###### Abstract

In this paper, we apply the _long short-term memory_ (LSTM), an advanced
recurrent neural network based machine learning (ML) technique, to the problem
of transmitter selection (TS) for secrecy in an underlay small-cell cognitive
radio network with unreliable backhaul connections. The cognitive
communication scenario under consideration has a secondary small-cell network
that shares the same spectrum of the primary network with an agreement to
always maintain a desired outage probability constraint in the primary
network. Due to the interference from the secondary transmitter common to all
primary transmissions, the secrecy rates for the different transmitters are
correlated. LSTM exploits this correlation and matches the performance of the
conventional technique when the number of transmitters is small. As the number
grows, the performance degrades in the same manner as other ML techniques such
as support vector machine, $k$-nearest neighbors, naive Bayes, and deep neural
network. However, LSTM still significantly outperforms these techniques in
misclassification ratio and secrecy outage probability. It also reduces the
feedback overhead against conventional TS.

###### Index Terms:

Cognitive radio network, deep neural network, long short-term memory, physical
layer security, recurrent neural network, transmitter selection.

##  I Introduction

The pioneering work of Wyner in information-theoretic security has led to the
current popularity of wireless physical layer security [1]. This has shown
promising results in reducing complexity against traditional key-based
cryptographic techniques without requiring secret key sharing, management, and
complex algorithms. Enhancing security of a communication network by improving
selection diversity is a simple alternative against beamforming, jamming or
noise forwarding. This is extensively applied in transmit antenna selection
(TAS) in multiantenna systems, transmitter selection (TS) in multiple source
systems, and relay selection (RS) in cooperative systems [2, 3, 4]. Next-
generation wireless networks will be highly dense and heterogeneous due to the
increasing demand for high data rate applications. The combination of spectrum
sharing in cognitive radio (CR) and small-cell networks with wireless backhaul
is one of the probable solutions. Although wireless backhaul may be a cost-
effective alternative, it suffers from reliability issues. The security of
such a network is of paramount importance. A comprehensive analysis of secrecy
enhancement through TS in a cognitive small-cell network with uncertain
wireless backhaul can be found in [5, 6].

With the increasing application of machine learning (ML) in many different
domains, the application interest in the area of TAS problems in wireless
communications is also accelerating [7, 8, 9, 10]. The author in [7] first
proposed to solve the TAS problem as a multiclass-classification learning
problem using ML. Through the application of $k$-nearest neighbors ($k$-NN)
and support vector machine (SVM) algorithms, the author showed that the
computational complexity and feedback overhead can be reduced compared to the
traditional wireless techniques, while maintaining a reasonable performance
accuracy. In [8], the authors extended the idea of solving the TAS problem
through multiclass-classification ML algorithms in the multiple-input
multiple-output (MIMO) wiretap channel using SVM and naive Bayes (NB) schemes.
These three ML algorithms were also applied to TAS in an untrusted relay
networks in [9]. Recently, in [10], the multiclass-classification approach was
also extended to RS problems in dual-hop wireless networks using a decision-
tree-based ML scheme. We refer to these algorithms as “traditional” ML
algorithm as they require explicit programming to extract features.

Deep learning (DL), a subclass of ML, can also be used to predict and classify
from complex raw data without being explicitly programmed to extract features
as in ML [11]. DL can handle nonlinear problems and its performance improves
with the size of the data, which may not be the case for traditional ML models
[12]. The authors in [13] implemented a deep neural network (DNN) scheme for
TAS in an untrusted relay network. The authors showed that the DNN performs
better than traditional ML schemes and achieves almost the same secrecy rate
as the conventional scheme. In [14], the authors proposed a DNN-based approach
to implement the joint multicast beamforming and TAS problem to reduce the
computational complexity. An artificial neural network structure along with a
decision-tree-based approach was proposed for TAS in [15] to improve the error
performance in the presence of time-correlated channels and channel estimation
errors.

The drawback of DNN is that it has no memory and therefore cannot retain
information regarding previous computations. As a result, DNN does not work
well with dependencies in data set. Conventional recurrent neural networks
(RNNs) can overcome this challenge by having recurrent connections between
previous and current computations [12]. An RNN has short-term memory and
cannot handle long-term dependencies in the data set due to vanishing gradient
and exploding gradient problems [16]. Long short-term memory (LSTM), as an
advanced version of conventional RNN, can outperform them in cases where the
data set has long-term dependencies [16, 17]. Thus, LSTM is widely applied in
large-scale acoustic modeling, speech recognition, and text categorization
[18, 19]. In wireless communications, LSTM is applied for the channel state
information (CSI) prediction for 5G wireless communication in [20] and for
signal detection in multipath MIMO environment [21], where LSTM based
detection performed best. Despite its suitability in capturing long-term
dependencies in the data set, LSTM has not yet been considered for the
TAS/TS/RS problem through multiclass-classification.

In this paper, we propose LSTM as a tool for solving the TS problem to improve
the secrecy of a CR network with wireless backhaul. We assume that
interference exists from both primary to secondary network and vice versa. The
channel fading model is considered to be independent non-identically
distributed (INID) complex Gaussian, in contrast to the channel assumptions
considered in [5, 6]. Due to the common interference term from the primary
network to the secondary receiver, the secrecy rates of the individual
secondary transmitters are correlated. We propose LSTM for our TS problem to
exploit the long-term dependency generated in data due to this interference.
As the number of transmitters increases, the performance of all the learning
models, including LSTM, degrades compared to the conventional search. This has
not been pointed out by the earlier literature in ML multiclass-classification
problems. We will show that due to its ability to capture long-term data
dependency, LSTM has a significantly lower misclassification ratio than other
learning models and consequently superior secrecy outage probability (SOP).
LSTM can also reduce the feedback overhead by half compared to the
conventional scheme.

The rest of the paper is organised as follows. In Section II, the system model
is described. The SOP for the optimal selection is defined in Section III. The
training methodology for the ML based transmitter selection is described in
Section IV. Sections V and VI provide computational complexity and numerical
results, respectively. Finally, the paper is concluded in Section VII.

Notation: $\mathbb{P}[\cdot]$ is the probability of occurrence of an event.
$\mathbb{E}[\cdot]$ denotes expectation operation. The channel coefficient
between any two nodes $A$ and $B$ is denoted by $h_{AB}$ and the signal-to-
interference-plus-noise-ratio (SINR) at $B$ for the link $A$-$B$ is denoted as
$\Gamma_{AB}$.

##  II System Model

Figure 1: Cognitive radio network with unreliable backhaul.

The system model, depicted in Fig. 1, consists of a secondary small-cell
network and a primary cellular network where the former is sharing the
spectrum of the latter using underlay CR technology. In the secondary network,
a macro-cell base station (BS) provides wireless backhaul connections to $K$
small-cell transmitters which in turn serve a destination $D$. A passive
eavesdropper $E$ is present in the secondary network listening to the
communication from secondary transmitters. The primary network consists of a
transmitter $T$ and its receiver $R$. As both the primary and secondary
networks share the same spectrum, interference from the primary to secondary
network, as well as from the secondary to primary network, is considered. The
outage probability of the primary network is considered as the QoS constraint.
The channel coefficients of the links, $S_{k}$-$D$, $S_{k}$-$E$ and
$S_{k}$-$R$, for each $k$, and $T$-$R$, $T$-$D$, and $T$-$E$, have INID
complex Gaussian distribution with zero mean and variance
$1/\lambda_{s_{k}d}$, $1/\lambda_{s_{k}e}$, $1/\lambda_{s_{k}r}$, and
$1/\lambda_{tr}$, $1/\lambda_{td}$, and $1/\lambda_{te}$, respectively. Noise
at all receivers is considered as complex additive white Gaussian noise (AWGN)
with equal parameters of zero mean and variance $N_{0}$.

The SINR at the secondary destination or eavesdropper due to the interference
from the primary transmitter can be expressed as

$\displaystyle\Gamma_{SB}=\dfrac{P_{S}|h_{S_{k^{\ast}}B}|^{2}}{P_{T}|h_{TB}|^{2}+N_{0}}\,,$
(1)

where $P_{S}$ is the maximum transmit power allowed for the small-cell
transmitter satisfying the primary QoS constraint, $P_{T}$ is the primary
transmit power at $T$, $k^{\ast}$ signifies the optimal secondary transmitter,
and $B\in\\{D,E\\}$.

###  II-A Modelling Wireless Backhaul Uncertainty

The backhaul links between the BS and the small-cell transmitters are
unreliable due to the wireless propagation channel, and thus, have a non-zero
probability of failure. The backhaul uncertainty is modeled by INID Bernoulli
random variables $\mathbb{I}_{k}$ for $k=1,\ldots,K$, with success probability
$\mathbb{P}(\mathbb{I}_{k}=1)=\delta_{k}$ and failure probability
$\mathbb{P}(\mathbb{I}_{k}=0)=1-\delta_{k}$ $\forall k=1\ldots,K$. In contrast
to [6], the optimal transmitter is selected with the knowledge of active
backhaul links. This implies that the optimal transmitter lies within the set
of transmitters whose backhaul links are active at a certain point in time.

###  II-B Secondary Transmit Power Constraint

Due to the CR approach considered for simultaneous primary and secondary
transmission, the secondary transmission power is restricted to guarantee a
certain primary QoS. The primary constraint is taken to be the outage
probability and should be below a threshold level. The secondary transmit
power constraint can be obtained from the outage probability constraint of the
primary receiver as

$\displaystyle\mathbb{P}\left[{\Gamma_{TR}}<\Gamma_{0}\right]\leq\Phi,$ (2)

where $0<\Phi<1$ is the primary outage probability constraint,
$\Gamma_{0}=2^{R_{th}}-1$ is the SINR threshold for the primary outage, with
$\beta$ as the threshold rate of the primary outage, and $\Gamma_{TR}$ is
given as

$\displaystyle\Gamma_{TR}=\frac{P_{T}|h_{TR}|^{2}}{P_{S}|h_{S_{k^{*}}R}|^{2}+N_{0}}.$
(3)

As such, $P_{S}$ can be evaluated from (2) using the cumulative distribution
function of $\Gamma_{TR}$ and is given in [6].

##  III SOP for Optimal Selection

In this section we present the performance metric for the optimal TS. The
optimal transmitter is defined as the one which provides maximum instantaneous
secrecy rate. Considering the backhaul uncertainty, and assuming backhaul
activity knowledge is available before selection, the optimal transmitter is
selected as

$\displaystyle
k^{*}=\arg\max_{k}\\{{\mathbb{I}_{k}C^{k}_{s}}\\}=\arg\max_{k\in\mathcal{S}}\\{C^{k}_{s}\\},$
(4)

where $\mathcal{S}$ is the set of transmitters with active backhaul links, and
$C^{k}_{s}$ is the secrecy rate of the wiretap channel formed by the pair of
links $S_{k}$-$D$ and $S_{k}$-$E$, which is expressed as

$\displaystyle
C^{k}_{s}=\max\left\\{\log_{2}\left[\frac{1+\Gamma_{S_{k}D}}{1+\Gamma_{S_{k}E}}\right],0\right\\}.$
(5)

Equation (4) states that if backhaul knowledge is available, then optimal
selection can be performed within the set of transmitters with active
backhauls. To perform optimal TS using (4), global CSI is required. Both real
and imaginary parts of the complex channel coefficients are required to be fed
back to a central decision making unit.

To derive the performance of the optimal selection scheme, we will find the
SOP. This is defined as the probability that the maximum achievable secrecy
rate through selection is below a predefined threshold $R_{th}$. Including
backhaul reliability knowledge, the SOP can be evaluated as

$\displaystyle\mathcal{P}_{out}(R_{th})$
$\displaystyle=\sum\limits_{\mathcal{S}\subseteq\\{1,2,\ldots,K\\}}\mathbb{P}[\mathcal{S}]\mathbb{P}\left[\max_{k\in\mathcal{S}}\\{C_{s}^{k}\\}<R_{th}\right],$
(6)

where the probability that the active backhaul set is $\mathcal{S}$ is given
by

$\displaystyle\mathbb{P}[\mathcal{S}]=\prod_{i\in\mathcal{S}}\delta_{i}\prod_{j\notin{\mathcal{S}}}(1-\delta_{j}).$
(7)

In (7), ${\mathcal{S}}$ is the inactive backhaul set. Note that the evaluation
of (6) requires the computations of the summation which involves $2^{K}$
terms. Also, it is difficult to derive a closed-form expression of
$\mathbb{P}\left[\max_{k\in\mathcal{S}}\\{C_{s}^{k}\\}<R_{th}\right]$ in (6)
due to two reasons. The first one is that the links are not identically
distributed, and the second is that the values $C_{s}^{k}$ for different $k$
are correlated due to the involvement of the common terms $h_{TD}$ and
$h_{TE}$. Therefore, in this paper we focus on developing various ML based
techniques for optimal selection and use SOP as a performance measure.

##  IV ML Based Transmitter Selection

We utilize ML data classification property to design a TS mechanism which
maximizes the instantaneous secrecy capacity in (4). We shall construct a
training data set using the relevant channel gains that affect the
instantaneous secrecy capacity and will feed to an ML based classifier to
design a classification model. Once the model is developed, if we provide a
random sample input, the ML based classifier should predict the transmitter
that maximizes the instantaneous secrecy capacity. The process of data set
generation, labeling, and prediction are explained next.

###  IV-A Construction of the Training Data Sets

To construct the training data set, a three-step procedure is followed as
detailed below.

Data set generation: In this step, we generate the input variables for
training a learning model. From (4) and (5), we observe that TS depends on
$h_{S_{k}D}$, $h_{S_{k}E}$, $h_{TE}$, and $h_{TD}$, where $k\in\\{1,\ldots
K\\}$ for a given $\Phi$, $R_{th}$, and $P_{T}$. Therefore, we design $M$
training data set matrices of dimension $4\times K$, $\mathbf{D}^{m}$, where
$m\in\\{1,\ldots,M\\}$, such that its columns represent CSI data values
corresponding to each $k$, i.e.,
$[h_{S_{k}D},h_{S_{k}E},h_{TE},h_{TD}]^{\mathbb{T}}$ for $k\in{1,\cdots,K}$.
Here, $\mathbb{T}$ denotes the transpose operator. As traditional ML models
and the DNN model take a vector as input for the learning process, we design a
vector $\mathbf{d}^{m}$ of length $N=4K$ by arranging the columns of
$\mathbf{D}^{m}$ one after the other. $\mathbf{d}^{m}$ is called the feature
vector. As learning models act on real-valued data, it is necessary to
manipulate the complex CSI values to obtain real-valued data. In addition,
training data should be normalized to avoid significant learning bias. This
process is divided into the following steps:

  1. 1.

We find the absolute values of each complex CSI element to generate learning
compatible real-valued feature vector, $\mathbf{d}^{m}$,
$m\in\\{1,\cdots,M\\}$ from earlier $\mathbf{d}^{m}$. As learning models
require only absolute values of the complex channel coefficients, we can
reduce the feedback overhead by half compared to feeding back complex
coefficients.

  2. 2.

We normalize the real-valued feature vector $\mathbf{d}^{m}$ via

$\displaystyle
t_{i}^{m}=\frac{d_{i}^{m}-\mathbb{E}[\mathbf{d}^{m}]}{\max(\mathbf{d}^{m})-\min(\mathbf{d}^{m})},$
(8)

where $t_{i}^{m}$ and $d_{i}^{m}$ are the $i$th element of $\mathbf{t}^{m}$
and $\mathbf{d}^{m}$, respectively, $i\in\\{1,\ldots,4K\\}$, $\mathbf{t}^{m}$
is the normalized feature vector. From $\mathbf{t}^{m}$ we obtain the
normalized feature matrix $\mathbf{T}^{m}$ following the reverse process of
generating $\mathbf{D}^{m}$ to $\mathbf{d}^{m}$.

Key performance indicator (KPI) design: A KPI is intended for labelling
training samples. In this paper, maximizing the instantaneous secrecy capacity
of the system with backhaul uncertainty is the goal; hence, our KPI is the
term $(\mathbb{I}_{k}C_{s}^{k})$ in (4).

Labeling: We seek to find the maximum secrecy capacity among $K$ transmitters
considering backhaul uncertainty; therefore, we decide there should be
$\ell=(K+1)$ possible labels. The label should indicate the index of the
transmitter with maximum secrecy capacity only when its backhaul is also
active. The $(K+1)$th label is kept for the case when no active transmitters
are available due to backhaul inactivity. We calculate the $K$ KPIs using the
feature vector $\mathbf{d}^{m}$ and label each vector with one of the $(K+1)$
possible labels. A label vector is generated having $M$ labels for each
$\mathbf{d}^{m}$ where $m\in\\{1,\ldots,M\\}$.

###  IV-B Model Prediction

We train the traditional ML and DNN with $m$ data sets $\mathbf{t}^{m}$; in
contrast, LSTM is trained with $\mathbf{T}^{m}$ along with the corresponding
$M$ labels. Each column of $\mathbf{T}^{m}$ is correlated due to the common
interference terms $h_{TD}$ and $h_{TE}$. This introduces long-term dependency
in the data set if $K$ is large. The trained network is then used for label
prediction with the normalized test feature vector or matrix. The output of
the model provides the predicted label corresponding to the index of the
selected transmitter.

##  V Computational Complexity

In this section, the prediction complexity of each learning model is analyzed
as the training complexity can be trained offline. The complexities for SVM,
$k$-NN, NB, and DNN are presented in the Table I following [7, 8], and [9],
respectively. In the DNN scheme, $L_{1}=256$ and $L_{2}=128$ are the number of
neurons in the first and second hidden layers, respectively [9]. For the
standard LSTM network, the computational complexity of each time step (in our
case per transmitter due to our construction of the feature matrix,
$\mathbf{D}^{m}$) is $\mathcal{O}(W)$, where $\mathcal{O}(\cdot)$ is defined
as the “order of” symbol. Since the number of transmitters is $K$, the
complexity of our proposed LSTM model is $\mathcal{O}(KW)$. Here $W$
represents the total number of parameter computations assuming one memory cell
per memory block while ignoring the bias neuron [18]. It is defined as
$W=4n_{c}^{2}+4n_{i}n_{c}+n_{c}n_{o}+3n_{c}$, where $n_{c}$, $n_{i}$, and
$n_{o}$ denote the number of memory cells, the number of input units, and the
number of output units, respectively. As we have taken one LSTM layer with 100
hidden neurons, 4 input features, and $K+1$ output classes, $n_{c}=100$,
$n_{i}=4$, and $n_{o}=K+1$. The number of fixed input weights, recurrent
weights, and peephole weights in $W$ are $4$, $4$, and $3$, respectively [17].
In the conventional search, we find the optimal transmitter among all
candidates, directly using (4) with global CSI. The complexity of evaluation
of the values $C_{s}^{k}$ and their comparison to get the maximum are both
$\mathcal{O}(K)$, and therefore the complexity of this approach is
$\mathcal{O}(K)$. In the case where a large number of transmitters is
selected, the complexity of the conventional scheme increases significantly
due to combinatorial search, and can be written as
$\mathcal{O}(K+\binom{K}{n}\log\binom{K}{n})$, where $n$ is the number of
selected transmitters out of $K$.

TABLE I: Complexities of different TS schemes. Scheme | Complexity
---|---
LSTM | $\mathcal{O}(KW)$
DNN | $\mathcal{O}(NL_{1}+L_{1}L_{2}+L_{2}K)$
SVM | $\mathcal{O}(N^{2})$
$k$-NN | $\mathcal{O}(N)$
NB | $\mathcal{O}((K+1)N+K)$
Conventional | $\mathcal{O}(K)$

##  VI Results and Discussions

This section compares LSTM, DNN, SVM, $k$-NN, and NB schemes with the
conventional benchmark scheme. A training data set size of $M=10^{5}$ is
considered and the SOP is plotted by averaging the results from $M=10^{6}$
batches of test data. The parameters for the traditional ML (SVM, $k$-NN, and
NB) and DNN schemes are the same as those in [9] and [13], respectively. For
the LSTM parameters, activation functions for all gates are taken to be
sigmoid, whereas the hyperbolic tangent is used for the memory cell in
accordance with the standard LSTM architecture [18]. Further, the Adam
optimizer is implemented to find the optimum weights in the LSTM model [21].
We have implemented one LSTM layer with $100$ hidden neurons and set the
maximum number of epochs to $20$. The initial learning rate and minibatch size
are kept identical with that of the DNN.

For the physical layer system parameters, unless otherwise specified,
$R_{th}=0.5$, and $\Phi=0.1$. The channel parameters for the independent
identically distributed (IID) channel case are
$(1/\lambda_{tr},1/\lambda_{td},1/\lambda_{sd},1/\lambda_{sr},1/\lambda_{se},1/\lambda_{te})=(3,-6,3,-3,-3,6)$
dB, where $1/\lambda_{s_{k}d}=1/\lambda_{sd}$,
$1/\lambda_{s_{k}e}=1/\lambda_{se}$, $1/\lambda_{s_{k}r}=1/\lambda_{sr}$,
$\forall k$. In the INID channel case, the channel parameters are
$(1/\lambda_{tr},1/\lambda_{td},1/\lambda_{sr},1/\lambda_{te})=(3,-6,-3,6)$
dB, $1/\lambda_{s_{k}d}=(0,3,6,9)$ dB, $1/\lambda_{s_{k}e}=(-6,-3,0,3)$ dB,
$\forall k$, and $\delta=(0.90,0.92,0.94,0.96)$ when $K=4$, and
$1/\lambda_{s_{k}d}=(-12,\ldots,21)$ dB at a difference of 3 dB,
$1/\lambda_{s_{k}e}=(-21,\ldots,12)$ dB at a difference of 3 dB, $\forall k$,
and $\delta=(0.78,\ldots,1)$ at a difference of 0.2 when $K=12$.

(a)

(b)

(c)

Figure 2: Misclassification rate for TS at $\Gamma_{T}=8$ dB. Figure 3: SOP
versus $\Gamma_{T}$ performance of LSTM based model for different $K$ and
$\delta$ in IID channel case.

Figs. 2(a)-2(c) present the misclassification rate of the learning schemes for
$K$ transmitters at $\Gamma_{T}=8$ dB. Figs. 2(a) and 2(b) are obtained for
the IID channel case at $K=4$ and $12$, respectively, whereas Fig. 2(c) is for
the INID channel case with $K=12$. Specifically, $\Gamma_{T}=8$ dB is
considered as it provides minimum SOP at the chosen parameter values. In the
figure, each vertex of the polygon represents the misclassification for each
transmitter index. The corresponding misclassification rate is presented on
the straight line passing through the center of the polygon to the vertex. The
misclassification is considered when the learning model labels an input data
wrongly, i.e., $\ell\rightarrow\bar{\ell}$, where
$\forall\bar{\ell}\in\mathcal{L}$ such that $\ell\neq\bar{\ell}$ and
$\mathcal{L}$ is the set of all labels.

Comparing Figs. 2(a)-2(c), we can observe that the misclassification rate is
more or less the same in each class in the IID channel case, but this is not
true for the INID case. Due to the IID channel case, the number of CSI samples
belonging to each class are more or less same; however, that is not true for
the INID case. Hence, the misclassification rate is also different for INID.
However, it is clearly observed in the IID channel case that when the number
of transmitters increases from $K=4$ to $K=12$, the misclassification rate
increases. This leads to increased SOP performance degradation and will be
explained during the discussion of Fig. 3.

Note from Fig. 2(b) that when the number of transmitters is large, i.e.,
$K=12$, LSTM performs the best as a classifier (it results in the polygon with
the least area). We can also verify that in general, traditional ML based
schemes perform worse than DL based schemes. Among the traditional ML schemes,
the SVM model has achieved the best performance, $k$-NN performs moderately,
and NB is the worst. Since we have a finite number ($4K$) of training samples
of random channel gains with large dimensions, it is highly likely that
samples may contain some outliers. The SVM classifier has the capability of
handling such outliers and also can support high dimensional data sets, hence;
SVM performs best among the traditional ML schemes. However, the $k$-NN
classifier often suffers from high variations of classification when the
dimension of the data is large, thus explaining its moderate performance [7].
Due to limited samples, the probabilities calculated by the NB classifier are
often inaccurate, and hence, returns worst performance [9]. DNN performs
better than the traditional ML schemes because of its use of multiple hidden
layers, which helps to extract the features for classification with high
accuracy. LSTM has memory units which result in remembering the input
information and identifying long-term dependency in the input data set [16].
This property is successfully utilized in our proposed model, as the secrecy
rates of individual transmitters are related due to the common interference
term coming from the secondary transmitter. This data dependency cannot be
exploited fully during classification in the traditional ML models or in DNN;
therefore, LSTM outperforms the other learning schemes.

Fig. 3 shows the SOP versus $\Gamma_{T}$ performance for the LTSM scheme with
$K=4$, $K=8$, and $K=12$ for two different backhaul uncertainty values,
$\delta=0.8$ and $\delta=0.99$ in the IID channel case. We can see that when
$K=4$, the SOP obtained using the LSTM scheme matches with the SOP obtained
from the conventional search at a particular backhaul uncertainty. However, as
the number of transmitters increases, the LSTM scheme overestimate the SOP;
moreover, the gap between these two results increases. This observation
conforms with the misclassification diagram of $K=4$ and $K=12$ in Fig. 2(a)
and Fig. 2(b), respectively. This is reasonable because as the number of
possible classes increases, the probability of confusion between two classes
also increases during a learning based decision making process. As a result,
the SOP mismatch increases. In addition, we also observe for a particular $K$
that as the backhaul reliability decreases from $\delta=0.99$ to $0.8$, the
gap between SOPs of LSTM and conventional search decreases. This shows that at
a lower backhaul reliability, the misclassification rate decreases. At a low
backhaul reliability, fewer transmitters are active, which reduces the data
points in a particular feature vector and makes it sparse; as a consequence,
LSTM can classify feature vectors accurately and can minimize decision error.

Figure 4: SOP versus $\Gamma_{T}$ for different $K$ with $\delta=0.8$ in IID
channel case.

Fig. 4 compares SOP versus $\Gamma_{T}$ of the proposed learning schemes for
$K=4$ and $K=12$ in the IID channel case. Here, as in Fig. 3, we can see that
the SOP results of the learning schemes are closer to the conventional search
result when the number of transmitters is less, and the gaps between them
increase when the number of transmitters increases. This means that the
misclassification rate in learning models is in general lower when a smaller
number of transmitters (i.e., classes) exists, and that the misclassification
rate increases with an increase in the number of transmitters. As expected,
the LSTM technique performs the best, outperforming even the DNN scheme. This
is because the LSTM scheme can exploit the correlated structure of the feature
vectors better than the DNN scheme due to its internal recurrent structure. In
general, it is verified that the performance of DL schemes, LSTM and DNN, are
much better than the performance of the traditional ML techniques, SVM,
$k$-NN, and NB, for TS.

Figure 5: SOP versus $\Gamma_{T}$ for different $K$ and $\delta=0.8$ in INID
channel case.

The SOP results of Fig. 4 are replicated for the INID channel condition in
Fig. 5. The observations are broadly the same as for Fig. 4. However, it can
be observed that the NB technique performs worse in the INID channel
conditions.

##  VII Conclusion

In this work, we have applied LSTM, an advanced version of conventional RNN
based learning model, to a multiclass-classification problem of TS to improve
security in a small-cell CR network with unreliable backhaul.
Misclassification and secrecy outage performance of the model is compared with
three traditional ML based schemes, NB, $k$-NN, and SVM, and a DL based
scheme, DNN. We observe that LSTM matches the performance of the conventional
technique when the number of transmitters is low. Further, the performance of
all ML schemes degrades as the number of transmitters increases as compared to
the conventional scheme; however, LSTM has the best performance. We observe
that DL based schemes perform better than traditional ML based schemes as they
can handle nonlinear problems and large data set. We further notice that the
LSTM based model significantly outperforms the DNN based model as the number
of transmitters grows due to the memory based recurrent architecture. We
conclude that the LSTM based TS is the best suited when a large number of
classes with data dependency exist. It can also reduce feedback overhead
against the conventional TS scheme.

## Acknowledgement

This publication has emanated from research conducted with the financial
support of Science Foundation Ireland (SFI) under Grant Number 17/US/3445.

## References

  * [1] A. D. Wyner, “The wire-tap channel,” _Bell System Technical Journal_ , vol. 54, no. 8, pp. 1355–1387, Oct. 1975.
  * [2] N. Yang _et al._ , “Transmit antenna selection for security enhancement in MIMO wiretap channels,” _IEEE Trans. on Commun._ , vol. 61, no. 1, pp. 144–154, Jan. 2013.
  * [3] C. Kundu, T. M. N. Ngatched, and O. A. Dobre, “Relay selection to improve secrecy in cooperative threshold decode-and-forward relaying,” in _Proc. IEEE Global Communications Conference_ , Washington DC, USA, Dec. 2016, pp. 1–6.
  * [4] C. Kundu, S. Ghose, and R. Bose, “Secrecy Outage of Dual-hop Regenerative Multi-Relay System with Relay Selection,” _IEEE Trans. Wireless Commun._ , vol. 14, no. 8, pp. 4614–4625, Aug. 2015.
  * [5] V. Truong _et al._ , “Secure cognitive radio networks with source selection and unreliable backhaul connections,” _IET Commun._ , vol. 12, no. 15, pp. 1771–1777, Mar. 2018.
  * [6] J. Zhang _et al._ , “Secrecy performance of small-cell networks with transmitter selection and unreliable backhaul under spectrum sharing environment,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 11, pp. 10 895–10 908, Nov. 2019.
  * [7] J. Joung, “Machine learning-based antenna selection in wireless communications,” _IEEE Commun. Lett._ , vol. 20, no. 11, pp. 2241–2244, Nov. 2016.
  * [8] D. He _et al._ , “Transmit antenna selection in MIMO wiretap channels: A machine learning approach,” _IEEE Wireless Commun. Lett._ , vol. 7, no. 4, pp. 634–637, Aug. 2018.
  * [9] R. Yao _et al._ , “Machine learning-based antenna selection in untrusted relay networks,” in _Proc. International Conference on Artificial Intelligence and Big Data_ , Chengdu, China, Sep. 2019, pp. 323–328.
  * [10] X. Wang, “Decision-tree-based relay selection in dualhop wireless communications,” _IEEE Trans. Veh. Technol_ , vol. 68, no. 6, pp. 6212–6216, Apr. 2019.
  * [11] C. Zhang, P. Patras, and H. Haddadi, “Deep learning in mobile and wireless networking: A survey,” _IEEE Communications Surveys Tutorials_ , vol. 21, no. 3, pp. 2224–2287, third quarter, 2019.
  * [12] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_. MIT Press, 2016, http://www.deeplearningbook.org.
  * [13] R. Yao _et al._ , “Deep neural network assisted approach for antenna selection in untrusted relay networks,” _IEEE Wireless Commun. Lett_ , vol. 8, no. 6, pp. 1644–1647, Dec. 2019.
  * [14] M. S. Ibrahim _et al._ , “Learning-based antenna selection for multicasting,” in _Proc. IEEE International Workshop on Signal Processing Advances in Wireless Communications_ , Kalamata, Greece, Jun. 2018, pp. 1–5.
  * [15] S. Gecgel, C. Goztepe, and G. K. Kurt, “Transmit antenna selection for large-scale MIMO GSM with machine learning,” _IEEE Commun. Lett._ , vol. 9, no. 1, pp. 113–116, Jan. 2020.
  * [16] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, no. 8, pp. 1735–1780, Dec. 1997.
  * [17] K. Greff _et al._ , “LSTM: A search space odyssey,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 28, no. 10, pp. 2222–2232, Oct. 2017.
  * [18] H. Sak, A. W. Senior, and F. Beaufays, “Long short-term memory recurrent neural network architectures for large scale acoustic modeling,” in _Proc. Annual Conference of International Speech Communication Association_ , Singapore, Sep. 2014, pp. 338–342.
  * [19] A. Graves, N. Jaitly, and A. Mohamed, “Hybrid speech recognition with deep bidirectional LSTM,” in _Proc. IEEE Automatic Speech Recognition and Understanding Workshop_ , Olomouc, Czech Republic, Dec. 2013, pp. 273–278.
  * [20] C. Luo _et al._ , “Channel state information prediction for 5G wireless communications: A deep learning approach,” _IEEE Trans. Netw. Sci. Eng._ , vol. 7, no. 1, pp. 227–236, Mar. 2020.
  * [21] M. Baek _et al._ , “Implementation methodologies of deep learning-based signal detection for conventional MIMO transmitters,” _IEEE Trans. Broadcast._ , vol. 65, no. 3, pp. 636–642, Sep. 2019.