* Jenkinson et al. (2002) M. Jenkinson, P. Bannister, M. Brady, and S. Smith.  Improved optimization for the robust and accurate linear registration and motion correction of brain images.  _NeuroImage_ , 17:825 – 841, 2002.  doi: 10.1006/nimg.2002.1132.
  * Jenkinson et al. (2012) M. Jenkinson, C. F. Beckmann, T. E. Behrens, M. W. Woolrich, and S. M. Smith.  FSL.  _NeuroImage_ , 62(2):782 – 790, 2012.  doi: 10.1016/j.neuroimage.2011.09.015.
  * Jeurissen et al. (2014) B. Jeurissen, J.-D. Tournier, T. Dhollander, A. Connelly, and J. Sijbers.  Multi-tissue constrained spherical deconvolution for improved analysis of multi-shell diffusion MRI data.  _NeuroImage_ , 103:411–426, 2014.
  * Kim and Ye (2020) B.-H. Kim and J. C. Ye.  Understanding graph isomorphism network for rs-fMRI functional connectivity analysis.  _Frontiers in Neuroscience_ , 14:630, 2020.  doi: 10.3389/fnins.2020.00630.
  * Kingma and Ba (2014) D. Kingma and J. Ba.  Adam: A method for stochastic optimization.  2014\.
  * Ktena et al. (2018) S. I. Ktena, S. Parisot, E. Ferrante, M. Rajchl, M. C. H. Lee, B. Glocker, and D. Rueckert.  Metric learning with spectral graph convolutions on brain connectivity networks.  _NeuroImage_ , 169:431–442, 2018.
  * Lang et al. (2012) E. Lang, A. Tomé, I. Keck, J. Gorriz, and C. Puntonet.  Brain connectivity analysis: A short survey.  _Computational intelligence and neuroscience_ , 2012.  doi: 10.1155/2012/412512.
  * Li et al. (2019) X. Li, N. Dvornek, Y. Zhou, J. Zhuang, P. Ventola, and J. Duncan.  Graph neural network for interpreting task-fMRI biomarkers.  pages 485–493, 2019.
  * Li et al. (2018) Y. Li, R. Yu, C. Shahabi, and Y. Liu.  Diffusion convolutional recurrent neural network: Data-driven traffic forecasting, 2018.
  * Liang and Wang (2017) H. Liang and H. Wang.  Structure-Function Network Mapping and its Assessment via Persistent Homology.  _PLoS Computational Biology_ , 2017.
  * Liang et al. (2019) J. Liang, L. Jiang, L. Cao, Y. Kalantidis, L.-J. Li, and A. G. Hauptmann.  Focal visual-text attention for memex question answering.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41(8):1893–1908, 2019.  doi: 10.1109/TPAMI.2018.2890628.
  * Lim et al. (2019) S. Lim, F. Radicchi, M. P. van den Heuvel, and O. Sporns.  Discordant attributes of structural and functional brain connectivity in a two-layer multiplex network.  _Scientific Reports_ , 9, 12 2019.
  * Luetkepohl (2005) H. Luetkepohl.  _The New Introduction to Multiple Time Series Analysis_.  Springer, 2005.  doi: 10.1007/978-3-540-27752-1.
  * Mackinnon (1994) J. Mackinnon.  Approximate asymptotic distribution functions for unit-root and cointegration tests.  _Journal of Business and Economic Statistics_ , 12:167–76, 1994.  doi: 10.1080/07350015.1994.10510005.
  * Mele et al. (2019) G. Mele, C. Cavaliere, V. Alfano, M. Orsini, M. Salvatore, and M. Aiello.  Simultaneous eeg-fmri for functional neurological assessment.  _Frontiers in Neurology_ , 10, 08 2019.  doi: 10.3389/fneur.2019.00848.
  * Messé et al. (2014) A. Messé, D. Rudrauf, H. Benali, and G. Marrelec.  Relating Structure and Function in the Human Brain: Relative Contributions of Anatomy, Stationary Dynamics, and Non-stationarities.  _PLoS Computational Biology_ , 10(3), 2014.
  * Messé et al. (2015) A. Messé, M. T. Hütt, P. König, and C. C. Hilgetag.  A closer look at the apparent correlation of structural and functional connectivity in excitable neural networks.  _Scientific Reports_ , 5:7870, 2015.
  * Messé et al. (2015) A. Messé, D. Rudrauf, A. Giron, and G. Marrelec.  Predicting functional connectivity from structural connectivity via computational models using mri: An extensive comparison study.  _NeuroImage_ , 111, 05 2015.  doi: 10.1016/j.neuroimage.2015.02.001.
  * Mikolov et al. (2013) T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean.  Distributed representations of words and phrases and their compositionality.  _Advances in Neural Information Processing Systems_ , 26, 10 2013\.
  * Mill et al. (2017) R. D. Mill, A. Bagic, A. Bostan, W. Schneider, and M. W. Cole.  Empirical validation of directed functional connectivity.  _NeuroImage_ , 146:275–287, 2017.
  * Moeller et al. (2010) S. Moeller, E. Yacoub, C. A. Olman, E. Auerbach, J. Strupp, N. Y. Harel, and K. Ugurbil.  Multiband multislice ge-epi at 7 tesla, with 16-fold acceleration using partial parallel imaging with application to high spatial and temporal whole-brain fMRI.  _Magnetic resonance in medicine_ , 63 5:1144–53, 2010.
  * Olman et al. (2009) C. A. Olman, L. Davachi, and S. J. Inati.  Distortion and signal loss in medial temporal lobe.  _PLoS ONE_ , 4, 2009.
  * Panda et al. (2021) R. Panda, A. Thibaut, A. Lopez-Gonzalez, A. Escrichs, M. A. Bahri, A. Hillebrand, G. Deco, S. Laureys, O. Gosseries, J. Annen, and P. Tewarie.  Disruption in structural-functional network repertoire and time-resolved subcortical-frontoparietal connectivity in disorders of consciousness.  _bioRxiv_ , 2021.  doi: 10.1101/2021.12.10.472068.
  * Power et al. (2016) J. Power, M. Plitt, T. Laumann, and A. Martin.  Sources and implications of whole-brain fmri signals in humans.  _NeuroImage_ , 146, 10 2016.  doi: 10.1016/j.neuroimage.2016.09.038.
  * Prando et al. (2020) G. Prando, M. Zorzi, A. Bertoldo, M. Corbetta, M. Zorzi, and A. Chiuso.  Sparse dcm for whole-brain effective connectivity from resting-state fmri data.  _NeuroImage_ , 208:116367, 2020.  ISSN 1053-8119.
  * Ramsey et al. (2011) J. Ramsey, S. Hanson, and C. Glymour.  Multi-subject search correctly identifies causal connections and most causal directions in the dcm models of the smith et al. simulation study.  _NeuroImage_ , 58:838–48, 07 2011.  doi: 10.1016/j.neuroimage.2011.06.068.
  * Reid et al. (2019) A. Reid, D. Headley, R. Mill, R. sanchez romero, L. Uddin, D. Marinazzo, D. Lurie, P. Valdés-Sosa, S. Hanson, B. Biswal, V. Calhoun, R. Poldrack, and M. Cole.  Advancing functional connectivity research from association to causation.  _Nature Neuroscience_ , 22:1–10, 10 2019.  doi: 10.1038/s41593-019-0510-4.
  * Roebroeck et al. (2009) A. Roebroeck, E. Formisano, and R. Goebel.  The identification of interacting networks in the brain using fmri: Model selection, causality and deconvolution.  _NeuroImage_ , 58:296–302, 09 2009.  doi: 10.1016/j.neuroimage.2009.09.036.
  * Rosenthal et al. (2018) G. Rosenthal, F. Váša, A. Griffa, P. Hagmann, E. Amico, J. Goñi, G. Avidan, and O. Sporns.  Mapping higher-order relations between brain structure and function with embedded vector representations of connectomes.  _Nature Communications_ , 9, 12 2018.
  * Rumelhart et al. (1986) D. Rumelhart, G. E. Hinton, and R. J. Williams.  Learning representations by back-propagating errors.  _Nature_ , 323:533–536, 1986.
  * Salimi-Khorshidi et al. (2014) G. Salimi-Khorshidi, G. Douaud, C. F. Beckmann, M. F. Glasser, L. Griffanti, and S. M. Smith.  Automatic denoising of functional MRI data: Combining independent component analysis and hierarchical fusion of classifiers.  _NeuroImage_ , 90:449 – 468, 2014.  doi: 10.1016/j.neuroimage.2013.11.046.
  * Sarwar et al. (2021) T. Sarwar, Y. Tian, B. Yeo, K. Ramamohanarao, and A. Zalesky.  Structure-function coupling in the human connectome: A machine learning approach.  _NeuroImage_ , 226:117609, 02 2021.  doi: 10.1016/j.neuroimage.2020.117609.
  * Schnake et al. (2021) T. Schnake, O. Eberle, J. Lederer, S. Nakajima, K. T. Schutt, K.-R. Muller, and G. Montavon.  Higher-order explanations of graph neural networks via relevant walks.  _IEEE transactions on pattern analysis and machine intelligence_ , PP, 2021.
  * Seo et al. (2018) Y. Seo, M. Defferrard, P. Vandergheynst, and X. Bresson.  Structured sequence modeling with graph convolutional recurrent networks.  In L. Cheng, A. C. S. Leung, and S. Ozawa, editors, _Neural Information Processing_ , pages 362–373. Springer International Publishing, 2018\.
  * Seth et al. (2012) A. Seth, P. Chorley, and L. Barnett.  Granger causality analysis of fmri bold signals is invariant to hemodynamic convolution but not downsampling.  _NeuroImage_ , 65:540–55, 2012.  doi: 10.1016/j.neuroimage.2012.09.049.
  * Setsompop et al. (2012) K. Setsompop, B. Gagoski, J. R. Polimeni, T. Witzel, V. J. Wedeen, and L. L. Wald.  Blipped-controlled aliasing in parallel imaging for simultaneous multislice echo planar imaging with reduced g-factor penalty.  _Magnetic resonance in medicine_ , 67 5:1210–24, 2012.
  * Shinn et al. (2021) M. Shinn, A. Hu, L. Turner, S. Noble, S. Achard, A. Anticevic, D. Scheinost, R. T. Constable, D. Lee, E. T. Bullmore, and J. D. Murray.  Spatial and temporal autocorrelation weave human brain networks.  _bioRxiv_ , 2021.  doi: 10.1101/2021.06.01.446561.  URL https://www.biorxiv.org/content/early/2021/06/01/2021.06.01.446561.
  * Shuman et al. (2013) D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst.  The emerging field of signal processing on graphs: Ex-tending high-dimensional data analysis to networks and other irregular domains.  _IEEE Signal Processing Magazine_ , 30(3):83–98, 2013.
  * Singh et al. (2020) M. F. Singh, T. S. Braver, M. W. Cole, and S. Ching.  Estimation and validation of individualized dynamic brain models with resting state fmri.  _NeuroImage_ , 221:117046, 2020.  ISSN 1053-8119.  doi: https://doi.org/10.1016/j.neuroimage.2020.117046.
  * Smith et al. (1999) A. M. Smith, B. K. Lewis, U. E. Ruttimann, F. Q. Ye, T. M. Sinnwell, Y. Yang, J. H. Duyn, and J. A. Frank.  Investigation of low frequency drift in fmri signal.  _NeuroImage_ , 9:526–533, 1999.
  * Smith et al. (2012) R. Smith, J.-D. Tournier, F. Calamante, and A. Connelly.  Anatomically-constrained tractography: Improved diffusion mri streamlines tractography through effective use of anatomical information.  _NeuroImage_ , 62:1924–38, 2012.  doi: 10.1016/j.neuroimage.2012.06.005.
  * Smith et al. (2013a) R. Smith, J.-D. Tournier, F. Calamante, and A. Connelly.  Sift: Spherical-deconvolution informed filtering of tractograms.  _NeuroImage_ , 67:298–312, 2013a.  doi: 10.1016/j.neuroimage.2012.11.049.
  * Smith et al. (2011) S. M. Smith, K. L. Miller, G. Salimi-Khorshidi, M. Webster, C. F. Beckmann, T. E. Nichols, J. D. Ramsey, and M. W. Woolrich.  Network modelling methods for fmri.  _NeuroImage_ , 54(2):875–891, 2011.
  * Smith et al. (2013b) S. M. Smith, C. F. Beckmann, J. Andersson, E. J. Auerbach, J. Bijsterbosch, G. Douaud, E. Duff, D. A. Feinberg, L. Griffanti, M. P. Harms, M. Kelly, T. Laumann, K. L. Miller, S. Moeller, S. Petersen, J. Power, G. Salimi-Khorshidi, A. Z. Snyder, A. T. Vu, M. W. Woolrich, J. Xu, E. Yacoub, K. Uğurbil, D. C. V. Essen, and M. F. Glasser.  Resting-state fMRI in the human connectome project.  _NeuroImage_ , 80:144 – 168, 2013b.  doi: 10.1016/j.neuroimage.2013.05.039.
  * Sotiropoulos et al. (2013a) S. Sotiropoulos, S. Jbabdi, J. Xu, J. Andersson, S. Moeller, E. Auerbach, M. Glasser, M. Hernandez Fernandez, G. Sapiro, M. Jenkinson, D. Feinberg, E. Yacoub, C. Lenglet, V. DC, K. Ugurbil, and T. Behrens.  Advances in diffusion mri acquisition and processing in the human connectome project.  _NeuroImage_ , 80:125, 2013a.  doi: 10.1016/j.neuroimage.2013.05.057.
  * Sotiropoulos et al. (2013b) S. Sotiropoulos, S. Moeller, S. Jbabdi, J. Xu, J. Andersson, E. Auerbach, E. Yacoub, D. Feinberg, K. Setsompop, L. Wald, T. Behrens, K. Ugurbil, and C. Lenglet.  Effects of image reconstruction on fibre orientation mapping from multi-channel diffusion MRI: Reducing the noise floor using SENSE.  _Magnetic resonance in medicine : official journal of the Society of Magnetic Resonance in Medicine / Society of Magnetic Resonance in Medicine_ , (70), 2013b.  doi: 10.1002/mrm.24623.
  * Suarez et al. (2020) L. Suarez, R. Markello, R. Betzel, and B. Misic.  Linking structure and function in macroscale brain networks.  _Trends in Cognitive Sciences_ , 24, 02 2020.  doi: 10.1016/j.tics.2020.01.008.
  * Suarez et al. (2021) L. Suarez, B. Richards, G. Lajoie, and B. Misic.  Learning function from structure in neuromorphic networks.  _Nature Machine Intelligence_ , 3, 09 2021.  doi: 10.1038/s42256-021-00376-1.
  * Sutskever et al. (2014) I. Sutskever, O. Vinyals, and Q. V. Le.  Sequence to sequence learning with neural networks.  _CoRR_ , abs/1409.3215, 2014.  URL http://arxiv.org/abs/1409.3215.
  * Thomas et al. (2014) C. Thomas, F. Q. Ye, M. O. Irfanoglu, P. D. Modi, K. S. Saleem, D. A. Leopold, and C. Pierpaoli.  Anatomical accuracy of brain connections derived from diffusion MRI tractography is inherently limited.  _Proceedings of the National Academy of Sciences of the United States of America_ , 111 46:16574–9, 2014.
  * Tournier et al. (2004) J.-D. Tournier, F. Calamante, D. Gadian, and A. Connelly.  Direct estimation of the fiber orientation density function from diffusion-weighted mri data using spherical deconvolution.  _NeuroImage_ , 23:1176–85, 2004.  doi: 10.1016/j.neuroimage.2004.07.037.
  * Tournier et al. (2007) J.-D. Tournier, F. Calamante, and A. Connelly.  Robust determination of the fibre orientation distribution in diffusion mri: Non-negativity constrained super-resolved spherical deconvolution.  _NeuroImage_ , 35:1459–72, 2007.  doi: 10.1016/j.neuroimage.2007.02.016.
  * Tournier et al. (2019) J.-D. Tournier, R. Smith, D. Raffelt, R. Tabbara, T. Dhollander, M. Pietsch, D. Christiaens, B. Jeurissen, C.-H. Yeh, and A. Connelly.  MRtrix3: A fast, flexible and open software framework for medical image processing and visualisation.  _NeuroImage_ , 202, 2019.  doi: 10.1101/551739.
  * Uğurbil et al. (2013) K. Uğurbil, J. Xu, E. Auerbach, S. Moeller, A. Vu, J. Duarte-Carvajalino, C. Lenglet, X. Wu, S. Schmitter, P.-F. Van de Moortele, J. Strupp, G. Sapiro, F. De Martino, D. Wang, N. Harel, M. Garwood, L. Chen, D. Feinberg, S. Smith, and E. Yacoub.  Pushing spatial and temporal resolution for functional and diffusion MRI in the human connectome project.  _NeuroImage_ , 80, 2013.
  * van den Oord et al. (2016) A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu.  Wavenet: A generative model for raw audio.  In _Arxiv_ , 2016.  URL https://arxiv.org/abs/1609.03499.
  * Van Essen et al. (2013) D. Van Essen, S. Smith, D. Barch, T. Behrens, E. Yacoub, and K. Ugurbil.  The wu-minn human connectome project: an overview.  _NeuroImage_ , 80, 2013.
  * Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Vézquez-Rodríguez et al. (2020) B. Vézquez-Rodríguez, Z.-Q. Liu, P. Hagmann, and B. Misic.  Signal propagation via cortical hierarchies.  _Network Neuroscience_ , 4(4):1072–1090, 11 2020\.  ISSN 2472-1751.
  * Wang et al. (2014) H. E. Wang, C. G. Bénar, P. P. Quilichini, K. J. Friston, V. K. Jirsa, and C. Bernard.  A systematic framework for functional connectivity measures.  _Frontiers in Neuroscience_ , 8, 2014.
  * Wang et al. (2019) P. Wang, R. Kong, X. Kong, R. Liégeois, C. Orban, G. Deco, M. P. van den Heuvel, and B. T. Yeo.  Inversion of a large-scale circuit model reveals a cortical hierarchy in the dynamic resting human brain.  _Science Advances_ , 5(1):eaat7854, 2019.  doi: 10.1126/sciadv.aat7854.
  * Webb et al. (2013) J. T. Webb, M. A. Ferguson, J. A. Nielsen, and J. S. Anderson.  Bold granger causality reflects vascular anatomy.  _PLOS ONE_ , 8(12):null, 12 2013.
  * Wein et al. (2021a) S. Wein, G. Deco, A. Tomé, M. Goldhacker, W. Malloni, M. Greenlee, and E. Lang.  Brain connectivity studies on structure-function relationships: A short survey with an emphasis on machine learning.  _Computational Intelligence and Neuroscience_ , 2021:1–31, 05 2021a.  doi: 10.1155/2021/5573740.
  * Wein et al. (2021b) S. Wein, W. Malloni, A. Tomé, S. Frank, G.-I. Henze, S. Wüst, M. Greenlee, and E. Lang.  A graph neural network framework for causal inference in brain networks.  _Scientific Reports_ , 11, 04 2021b.  doi: 10.1038/s41598-021-87411-8.
  * Wen et al. (2013) X. Wen, G. Rangarajan, and M. Ding.  Is granger causality a viable technique for analyzing fmri data?  _PloS one_ , 8:e67428, 07 2013.  doi: 10.1371/journal.pone.0067428.
  * Werbos (1990) P. Werbos.  Backpropagation through time: what it does and how to do it.  _Proceedings of the IEEE_ , 78(10):1550–1560, 1990.
  * Wise et al. (2004) R. G. Wise, K. Ide, M. J. Poulin, and I. Tracey.  Resting fluctuations in arterial carbon dioxide induce significant low frequency variations in bold signal.  _NeuroImage_ , 21:1652–1664, 2004.
  * Wu et al. (2019) Z. Wu, S. Pan, G. Long, J. Jiang, and C. Zhang.  Graph wavenet for deep spatial-temporal graph modeling.  pages 1907–1913, 08 2019.  doi: 10.24963/ijcai.2019/264.
  * Wu et al. (2020) Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu.  A comprehensive survey on graph neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ , pages 1–21, 2020.
  * Xu et al. (2012) J. Xu, S. Moeller, J. Strupp, E. Auerbach, L. Chen, D. A. Feinberg, K. Ugurbil, and E. Yacoub.  Highly accelerated whole brain imaging using aligned-blipped-controlled-aliasing multiband EPI.  _Proceedings of the 20th Annual Meeting of ISMRM_ , page 2036, 2012\.
  * Xu et al. (2015) K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In _ICML_ , 2015.
  * Yan et al. (2021) T. Yan, T. Liu, J. Ai, Z. Shi, J. Zhang, G. Pei, and J. Wu.  Task-induced activation transmitted by structural connectivity is associated with behavioral performance.  _Brain Structure and Function_ , 226, 06 2021.  doi: 10.1007/s00429-021-02249-0.
  * Zeiler and Fergus (2013) M. Zeiler and R. Fergus.  Visualizing and understanding convolutional neural networks.  _ECCV 2014, Part I, LNCS 8689_ , 8689, 2013.
  * Zhang et al. (2012) H. Zhang, T. Schneider, C. Gandini Wheeler-Kingshott, and D. Alexander.  NODDI: Practical in vivo neurite orientation dispersion and density imaging of the human brain.  _NeuroImage_ , 61:1000–16, 2012.
  * Zheng et al. (2020) C. Zheng, X. Fan, C. Wang, and J. Qi.  Gman: A graph multi-attention network for traffic prediction.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34:1234–1241, 04 2020.  doi: 10.1609/aaai.v34i01.5477.
  * Zimmermann et al. (2019) J. Zimmermann, J. Griffiths, M. Schirner, P. Ritter, and A. R. McIntosh.  Subject-specificity of the correlation between large-scale structural and functional connectivity.  _Network Neuroscience_ , pages 1–35, 2019.
  * Zou et al. (2008) Q. Zou, C.-Z. Zhu, Y. Yang, X.-N. Zuo, X. Long, Q.-J. Cao, Y.-F. Wang, and Y.-F. Zang.  An improved approach to detection of amplitude of low-frequency fluctuation (alff) for resting-state fmri: Fractional alff.  _Journal of neuroscience methods_ , 172:137–41, 08 2008\.
  * Řehůřek and Sojka (2010) R. Řehůřek and P. Sojka.  Software framework for topic modelling with large corpora.  In _Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks_ , pages 45–50, 05 2010.  doi: 10.13140/2.1.2393.1847.

## Supplementary Information

### Supplement I

In this supplement the influence of the model hyperparameters for the
different neural network architectures is discussed. The hyperparameters are
chosen as described in the Model training section and held constant, while
only the hyperparameter of interest is varied in the following evaluations.
Figure S1 and S2 show that the performance of he DCRNN and GWN in general
still could slightly improve with a larger number of model parameters. Because
the computation time and memory requirements linearly grow with the number of
parameters, we chose the model hyperparameters as described in the Model
training section to yield a good trade-off between model performance and
computational requirements. Also the TAtt model in figure S3 shows some
improvement with a larger number of parameters, however the MAE is still
considerably higher compared to the RNN and WN based GNN architectures.

Figure S1: Here the influence of the hyperparameters on the prediction
accuracy of the DCRNN is depicted. In (A) the test error is shown in
dependence of the number of neurons in each layer, and in (B) the error in
dependence of the number of DCGRU layers.

Figure S2: In this figure the influence of the GWN hyperparameters on the
prediction accuracy is shown. In (A) the test error in dependence of number of
neurons (or feature maps) is illustrated. Here (B) shows the influence of the
number of DCC blocks used in the GWN architecture and (C) shows the impact of
the number of layers per DCC block.

Figure S3: Here the influence of hyperparameters on the TAtt accuracy is
illustrated. In (A) the test MAE in dependence of the number of neurons used
in each TAtt mechanism is shown. In (B) the influence of the number of TAtt
layers is depicted and (C) illustrates the impact of number of attention heads
incorporated.

### Supplement II

Table 1: List of ROIs involved in visual processing based on the multi-modal parcellation proposed by Glasser et al. (2016). The table shows the index of the region in the atlas for the right/left hemisphere including the name of the region. Index | Name
---|---
1/181 | V1
2/182 | MST
3/183 | V6
4/184 | V2
5/185 | V3
6/186 | V4
7/187 | V8
13/193 | V3A
16/196 | V7
19/199 | V3B
20/200 | LO1
21/201 | LO2
22/202 | PIT
23/203 | MT
152/332 | V6A
153/333 | VMV1
154/334 | VMV3
156/336 | V4t
158/338 | V3CD
159/339 | LO3
160/340 | VMV2
163/343 | VVC

### Supplement III

In this supplement the different spatial and temporal modeling approaches are
evaluated using the scale-free R-squared ($R^{2}$) measure and the similarity
of predicted connectivity states. The $R^{2}$ measure can be obtained with:

$R^{2}(\mathbf{x},\hat{\mathbf{x}})=1-\frac{\sum_{n,t}(x_{n}^{(t)}-\hat{x}_{n}^{(t)})^{2}}{\sum_{n,t}(x_{n}^{(t)}-\bar{x}_{n}^{(t)})^{2}}$
(23)

where $x_{n}^{(t)}$ represents the true BOLD signal in brain region $n$ at
timestep $t$. Correspondingly $\bar{x}_{n}^{(t)}$ denotes the average BOLD
signal across all $N$ regions and $T_{f}$ timepoints, and $\hat{x}_{n}^{(t)}$
indicates the predicted signal of the respective model. Using this scale-free
measure the evaluations of the section Comparison of GNN architectures are
replicated in figure S4. To test the significance also based on this measure,
the $R^{2}$ values were computed for the individual subjects, and by applying
a paired t-test the RNN and WN model both outperformed the TAtt model with a
$p$-value of $p\leq 0.0001$ (Cohen’s $d\gg 1$) based on this measure. Also the
impact of structural modeling showed to be significant with $p\leq 0.0001$
(Cohen’s $d>1$) for both STGNN models.

In addition, we evaluated the similarity of the predicted FC states to the
ground truth FC states. For this purpose we computed a FC matrix
$\mathbf{A}_{FC}\in\mathbb{R}^{N\times N}$ from the true ROI timecourses
$\mathbf{x}\in\mathbb{R}^{N\times T_{f}}$, and respectively the predicted FC
matrix $\mathbf{\hat{A}}_{FC}$ from predicted timecourses $\mathbf{\hat{x}}$
based on Pearson correlation. Then we sorted the elements below the diagonal
of the symmetric FC matrix $\mathbf{A}_{FC}$ into a vector
$\mathbf{v}\in\mathbb{R}^{\frac{1}{2}N(N-1)}$, and defined a correlation based
measure of FC similarity as:

$r(\mathbf{v},\mathbf{\hat{v}})=\frac{\sum_{c=1}^{\frac{1}{2}N(N-1)}(\mathbf{v}_{c}-\mathbf{\bar{v}}_{c})(\mathbf{\hat{v}}_{c}-\mathbf{\bar{\hat{v}}}_{c})}{\sigma_{\mathbf{v}}\sigma_{\mathbf{\hat{v}}}}$
(24)

with $\mathbf{\bar{v}}_{c}$ denoting the average and $\sigma_{\mathbf{v}}$ the
variance of FC values. Based on this metric we replicated the comparison of
spatial and temporal GNN approaches in figure S5. Also using this similarity
measure, the RNN and WN model outperformed the TAtt model with $p\leq 0.0001$
(Cohen’s $d\gg 1$) across subjects. The impact of structural modeling showed
to be significant with $p\leq 0.0001$ (Cohen’s $d>1$) across subjects for both
STGNN models. In addition the distributions of the test MAE across different
subjects, timepoints and brain ROIs are visualized for the STGNNs with and
without incorporating spatial modeling in figure S6.

Figure S4: Figure (A) shows a comparison of different modeling strategies for
temporal dynamics in the BOLD signal, comparing the $\mathbf{R}^{2}$ values of
the recurrent neural network (RNN), the WaveNet (WN) and the temporal
attention (TAtt) architecture. The errorbars represent the standard deviation
of the values across subjects. Figure (B), (C) and (D) show the prediction
accuracies of the DCRNN and GWN model in dependence of the walk order $K$. In
figure (B) the $\mathbf{R}^{2}$ values are shown when incorporating the SC as
an adjacency matrix $\mathbf{A}_{SC}$, figure (C) illustrates the test MAE
when employing CEs in an adjacency matrix $\mathbf{A}_{CE}$ to define spatial
relationships, and (D) displays the case when using a self-adaptive weight
matrix $\mathbf{A}_{Adap}$.

Figure S5: Figure (A) shows a comparison of different modeling strategies for
temporal dynamics in the BOLD signal, comparing the FC similarities of the
recurrent neural network (RNN), the WaveNet (WN) and the temporal attention
(TAtt) architecture. The errorbars represent the standard deviation of the
values across subjects. Figure (B), (C) and (D) show the prediction accuracies
of the DCRNN and GWN model in dependence of the walk order $K$. In figure (B)
the similarity values are shown when incorporating the SC as an adjacency
matrix $\mathbf{A}_{SC}$, figure (C) illustrates the test MAE when employing
CEs in an adjacency matrix $\mathbf{A}_{CE}$ to define spatial relationships,
and (D) displays the case when using a self-adaptive weight matrix
$\mathbf{A}_{Adap}$.

Figure S6: The distribution of the test MAE error across timepoins, subjects
and ROIs is illustrated for the STGNNs, with and without incorporating spatial
modeling. In (A) the prediction error in dependence of the forecasting horizon
is depicted. Further (B) shows the MAE across subjects and brain regions of
the RNN, visualized in boxplots on the left side. On the right side the MAEs
in dependence of the brain regions were projected onto the cortical surface.
The same analysis was carried out for the DCRNN, WN and GWN model in figure
(C), (D) and (E) respectively.

### Supplement IV

Figure S7: The figure shows a comparison of the model performances, based on
the $R^{2}$ measure as defined in equation 23. The errorbars represent the
standard deviation of the $R^{2}$ values across subjects. In (A) the model
accuracies when using a dataset of $50$ subjects are shown for the visual
network, the network within the right hemisphere and the whole brain network.
Figure (B) and (C) show the test performances in dependence of the network
size using the $25$ and $10$ subject dataset, respectively. Except for the
single network dataset, the improvements of accuracy of the DCRNN and GWN in
comparison to the VAR model became in all cases highly significant with $p\leq
0.0001$ (Cohen’s $d\gg 1$).

Figure S8: The figure shows a comparison of the model performances, based on
the similarity of predicted FC states, as defined in equation 24. The
errorbars represent the standard deviation of the values across subjects. In
(A) the model accuracies when using a dataset of $50$ subjects are shown for
the visual network, the network within the right hemisphere and the whole
brain network. Figure (B) and (C) show the test performances in dependence of
the network size using the $25$ and $10$ subject dataset, respectively. Except
for the single network dataset, the improvements of accuracy of the DCRNN and
GWN in comparison to the VAR model became in all cases highly significant with
$p\leq 0.0001$ (Cohen’s $d\gg 1$).

### Supplement V

Figure S9: In this figure the prediction accuracy of the different models is
presented when using a bandpass filter within the $0.01-0.1Hz$ frequency range
for the $25$ subject dataset and the brain network including the ROIs within
the right hemisphere (Glasser et al., 2016). In (A) the test MAE in dependence
of the forecasting horizon is shown, computed as an average across test
samples and brain regions. Figure (B), (C) and (D) show examples of
predictions generated by the VAR, GWN and DCRNN model respectively. The
examples in this figures were chosen to be representative for the whole test
set, by selecting only examples which errors maximally deviate by $0.02$ from
the corresponding average test MAE of the models.

### Supplement VI

Figure S10: This figure illustrates directed spatial relations learned by the
DCRNN model. Figure (A) shows the measures of influence
$\mathbf{I}(n^{\prime})$, derived from the DCRNN model when using the SC for
information propagation and figure (B) depicts the influence when
incorporating CEs for the information exchange. The values of the connectivity
measures were linearly mapped between $0$ and $100$ and the default scaling of
the color values provided by the connectome workbench (version 1.4.2) was
used, adjusting the colormap between the $2th$ and $98th$ percentile of the
values respectively. Figure S11: The figure illustrates the effect of a
perturbation in a STGNN model systematically induced in target regions V1 (A),
V2 (B), V3 (C), V4 (D), MT/V5 (E) and V6 (F). The connectivity values were
linearly mapped between $0$ and $100$ and the default scaling of the color
values provided by the connectome workbench (version 1.4.2) was used,
adjusting the colormap between the $2th$ and $98th$ percentile of the values
respectively.

Figure S12: In this figure the temporal average autocorrelation values of fMRI
timeseries in dependence of their lag order $j$ are shown. The values were
computed as the Pearson correlation between the original fMRI timeseries
$\mathbf{x}_{n}^{(t)}$ and its lagged values $\mathbf{x}_{n}^{(t-j)}$, and
then averaged across all subjects and brain regions. The errorbars represent
the standard deviations across subjects.

Figure S13: The FC connectivity of V1 to all other regions within the right
hemisphere is shown in (A). In (B) and (C) the GWN based directed connectivity
is illustrated when using the SC ($\mathbf{A}_{SC}$) and CE similarity
($\mathbf{A}_{CE}$) for the information propagation respectively.

### Supplement VII

Figure S14: The distribution of respiratory frequencies is shown in this
figure. For the 25 subjects resting-state fMRI dataset, physiological data was
available from 22 subjects, and the depicted spectrum was computed as the
average across the individual spectra of the 22 subjects.


Goals are Enough: Inducing AdHoc cooperation among unseen Multi-Agent systems in IMFs

Kaushik Dey and Satheesh K. Perepu
Ericsson Research (Artificial Intelligence)

Abir Das
Indian Institute of Technology, Kharagpur

Intent-based management will play a critical role in achieving customers' expectations in the next-generation mobile networks.
Traditional methods cannot perform efficient resource management since they tend to handle each expectation independently.
Existing approaches, e.g., based on multi-agent reinforcement learning (MARL) allocate resources in an efficient fashion when there are conflicting expectations on the network slice. However, in reality, systems are often far more complex to be addressed by a standalone MARL formulation. Often there exists a hierarchical structure of intent fulfillment where multiple pre-trained, self-interested agents may need to be further orchestrated by a supervisor or controller agent.
Such agents may arrive in the system adhoc, which then needs to be orchestrated along with other available agents. This is especially true for networks evolving with time as components can get added incrementally. Retraining the whole system every time is often infeasible given the associated time and cost.
Given the challenges, such adhoc coordination of pre-trained systems could be achieved through an intelligent supervisor agent which incentivizes pre-trained RL/MARL agents through sets of dynamic contracts (goals or bonuses) and encourages them to act as a cohesive unit towards fulfilling a global expectation. Some approaches use a rule-based supervisor agent and deploy the hierarchical constituent agents sequentially, based on human-coded rules.

In the current work, we propose a framework whereby pre-trained agents can be orchestrated in parallel leveraging an AI-based supervisor agent. For this, we propose to use Adhoc-Teaming approaches which assign optimal goals to the MARL agents and incentivize them to exhibit certain desired behaviours. Results on the network emulator show that the proposed approach results in faster and improved fulfilment of expectations when compared to rule-based approaches and even generalizes to changes in environments.

multi-agent reinforcement learning, adhoc-teaming approach, goal assignment

§ INTRODUCTION

Intent Management frameworks (IMF) form a key component in networks of the future to meet the ever-changing demands of customers [19, 31]. An intent can contain single/multiple expectations which are defined in the form of targets on a single/multiple KPI/s. For example, an intent could specify “70% of users using conversational video service should get Quality of experience $\geq$ 4". [19, 30]. In networks of the future, there could be multiple intents, in a single resource slice, each containing different expectations on multiple KPIs [2]. Constraints on available radio resources combined with varying demands from customers can lead to conflicting intents e.g., actions fulfilling certain expectations may affect the fulfilment of other intents on the slice.

In general, an intent-based framework may have multiple agents, each controlling one or more KPIs. It is necessary for such agents to coordinate in order to handle conflicts [1]. Existing approaches based on machine reasoning [14], machine learning [24], and reinforcement learning [21] try to address such conflicting intents. These agents being pre-trained may only work for specific scenarios. In the real world, the scenarios often change dynamically and hence may necessitate generalization of such pre-trained agents[13].

Using methods based on Multi-Agent Reinforcement Learning(MARL), we can train a predefined set of agents to collaborate towards a common goal.
Specifically, [21] considered the case where a MARL system (consisting of more than one RL agent) trained to control packet priority, coordinates with another set of MARL systems that modulates the Maximum Bit Rate (MBR). The coordination across both MARL systems is enforced by a rule-based controller which forces each MARL system to act sequentially for a predefined number of time steps. However, creating such human-driven rules can be a challenge as the number of intents increases.

Additionally for collaborative intent management frameworks in 6G, such MARL systems may arrive pre-trained without any information about each other. Given the pre-training, each MARL system would be self-interested and the supervisor agent will not have any control over their actions.
In such a situation, traditional MARL or hierarchical algorithms may often prove infeasible as it might be impractical or costly to either retrain the entire system from scratch or manually enforce expert-driven rules.
Hence in this paper, we look at a problem where sets of pre-trained, self-interested MARL agents need to collaborate together to achieve a cooperative behaviour aided by an AI-enabled controller, known as the supervisor agent.

Also, in [21] while the authors considered sequential execution of MARL agents which can increase the time taken for episode convergence, we propose a framework for simultaneous execution of MARL systems. To implement the same, we propose a method where modified AdHoc Teaming (AHT) techniques for Goal Engineering are integrated with MARL agents. The results presented in this work demonstrate, orchestrating the individual Goals are Enough for the supervisor to incentivize the pre-trained MARL agents to act in a cohesive manner towards a global intent. Our proposed method integrating the modified AHT and MARL in a single framework is henceforth referred to as AT-MARL.

AHT is a method of inducing cooperation between unseen agents on the fly [26, 18].
Here self-interested agents arrive in the system without prior knowledge about other agents.
The AHT framework involves training an agent, referred to as the learner/supervisor, to collaborate with an unfamiliar group of teammates without prior coordination.
The teammates possess relevant skills for the teamwork task, and the objective is to find a policy for the supervisor agent that will enable it to create dynamic sets of goals for the teammates. Such goals induce all the available agents/systems to perform a specific sequence of actions and thereby can create cooperative behaviour across the lower-level agents even though the supervisor cannot control the intrinsic actions of the agents directly.

The problem's inputs include the global task (list of intents with their KPI expectations), the dimension of action space for each agent, and the list of teammates. The output is represented by a policy that determines the supervisor's actions in any domain state, potentially including creating sub-goals that act as a short-term contract with trained agents.
Three key assumptions characterize the AHT problem:

* No prior coordination: The supervisor must coordinate with teammates without prior establishment of coordination mechanisms. It may have partial knowledge of teammates' attributes acquired from an expert.

* No control over teammates: The supervisor cannot change the environment or teammates' policies; it must adapt to the given conditions.

* Collaborative: All agents share a common objective, but individual teammates may have additional objectives that do not conflict with the team's common task.

In summary, the AHT problem involves training an agent to collaborate with unfamiliar teammates without prior coordination, while pursuing a common objective.

§ BACKGROUND

In this work, we consider the IMF as defined in [21].
In this framework three types of services are considered, namely Conversational Video (CV), Ultra Reliable Low Latency Communication (URLLC), and Massive IoT (mIoT) services each of which has distinct network traffic characteristics [29]. Especially, we are interested in controlling the KPIs, Quality of Experience (QoE) of CV service, and Packet Loss (PL) of both URLLC and mIoT services. To ensure tractability, we have selected Packet Priority and Maximum Bit Rate (MBR) as the two control parameters. Now, in a scenario where resources in the slice are limited, enhancing packet priority can enhance the Quality of Experience (QoE) for CV, but it might negatively impact packet loss for URLLC. Similarly, augmenting MBR for URLLC can improve packet loss and reduce latency for URLLC, but it may have adverse effects on the mIoT and CV service. In essence, the challenge lies in optimizing the fulfillment of various conflicting intents within the resource constraints of the network slice.

Authors in [21] used MARL-based methods to address the conflicting intents problem. In this approach, they have two MARL agents as in Figure <ref>, one for controlling priority (we refer to it as Priority MARL system) and another for controlling MBR (a.k.a MBR MARL system). Each system can influence both KPIs, PL (URLLC & mIoT) and QoE. To switch between these multiple MARL agents the authors proposed a rule-based supervisor that can switch between the respective MARL systems after a predefined interval of 5 time-steps.

[Block Diagram][Sample Results]Orchestrating multiple MARL systems through supervisor (Left Side) and corresponding results (Right Side) by using [21] framework which uses a rule-based supervisor. The white boxes show the existing solution. The grey and green boxes show additional intents and evolving controls and thereby the scalability need which necessitates the creation of complex and ever-changing rules. This motivates us to propose an AI-based supervisor agent that can scale  without human-coded rules

However, there are a few drawbacks to this approach:

* Rule-Based Control switching: The supervisor agent works on the basis of heuristics and rules which are designed by a human expert.
The expert needs to devise complex rules with the addition of new agents/new control parameters as shown in dotted grey/green boxes in Figure <ref>. Also, the rules may depend on various factors e.g. radio environment, which is humanely difficult to comprehend with an increase in scale and complexity.

* Execution for fixed timesteps: Since the rules are pre-configured, the execution for each system (5 time-steps in [21]) has to occur for a pre-determined time period and is not conditioned on the state of the environment. Ideally, this approach is not optimal as the optimal actions of MARL agents depend on the state of the environment prevailing at that point in time.

* Sequential execution: Sequential execution of each task, (either priority or MBR actions), increases the total time window for intent fulfilment. Parallel execution is not feasible in [21] since the supervisor cannot deal with the non-stationarity of simultaneous actions.

In order to address the drawbacks, we introduce AT-MARL which proposes and changes goals dynamically for each system of agents, and through a seamless interaction with multiple MARL systems, is able to orchestrate all the MARL systems to an optimal convergence with improved accuracy and precision. Our main contributions in the paper are

* AI-based Orchestration: An Autonomous Orchestration of independently trained systems (MARL systems here), which makes it scalable and doesn't need formulation of rules created by human experts.

* Event-based execution: The length of execution of episodes for each system can vary according to the global state making each sub-stage of the process more efficient

* Concurrent execution wherever needed: The execution process for every system is decided dynamically by the supervisor agent enabled by AHT. Hence two systems often execute in parallel reducing the overall convergence time for the realization of intents. The experiments shown below, demonstrate that our process is able to improve the end-to-end execution window by 30% and more importantly achieve a much more stable convergence eliminating oscillations in most cases. Eliminating oscillations around the desired KPI is important for mission-critical URLLC applications.

* Generalization with change in radio Environments: The rule-based supervisor is not able to adapt as the radio environment changes. Mobility patterns, weather, or the nature of data transmissions change the prevailing radio environment often. Our proposed method is able to generalize on changes in radio environments due to mobility and hence doesn't need any further retraining.

§.§ Related Work

Recently, there has been tremendous interest among researchers in training agents to perform complex tasks.
Since it is hard for a single agent to perform in a complex scenario, it is broken down into different sub-tasks and each agent is trained to perform one sub-task. Also, each of these sub-tasks requires different skills, and only one/more types of agents are trained to perform these sub-tasks. Hence to successfully complete the complex task, the participating agents should coordinate among themselves [9, 25, 27].
AHT is one of the efficient ways by which coordination can be learned among pre-trained agents where the agents are oblivious to each other's capabilities and actions till this point.

Research in AHT has been prevalent for a while and collaboration without pre-coordination has become more relevant as trained robots make their way into 5G-enabled Industry 4.0 scenarios.
AHT scenarios have been simulated in various environments and a variety of techniques have been explored to induce adhoc collaboration.
In Matrix Games, authors in [3] empirically evaluate different multi-agent learning algorithms in adhoc mixed teams while [11] introduce an optimal algorithm for cooperation with a Markovian teammate.

In the Predator-Prey environment, authors in [8] utilize Monte Carlo Tree Search with type-based reasoning using handcrafted types, while [23] extends AHT methods to work with teammates that can dynamically change their respective behaviours.
Authors in [20] assume only local observations are available for building the adhoc teamwork.
In the context of Large-scale Battlefields, the work in [4] develops type-based game theoretic reasoning.
[22] utilizes graph-based learning to handle a dynamic number of agents.

In the Robot-soccer environment, [9] proposes an approach for developing coordination with adhoc teams created impromptu, and [7] introduces a method for reusing policies learned from previous teammates.
Authors in [6] present algorithms for AHT based on previously met teammates, utilizing either policies or models.

[10] create a meta-strategy for solving adhoc teamwork in Hanabi using a diverse set of possible teammates.

In addition, [17] created an optimized metric for determining policy diversity in Hanabi self-play.

However, none of these scenarios deals with Adhoc teaming for multi-agent systems where two levels of decision-making may be present, one about coordinating individual actions between agents within each multi-agent system and another to decide synchronization of actions between multi-agent systems(proposed herein through AHT). The interaction of AHT and MARL-based frameworks is unique to our work and the proposed method(AT-MARL) may be leveraged to create a novel implementation for IMFs in 6G or future networks.

AT-MARL method for generating sub-goals for each agent. The agents' capabilities are introduced in the form of $\Gamma^i_t$  and merged with the current performance of the agents $(s_t^i,a_t^i,g_t^i)$ to obtain the embeddings $m_t^i$. The embeddings calculated for each agent are fused with global goals (given by the customers) to create future context $c_{t+1}$, which is used to train the goal policy $\pi_g$ to obtain the sub-goals for each agent in the next time-step.

Let us also discuss on complexities of AI-based coordination among pre-trained MARL systems.
Each MARL system, like the Priority MARL system or the MBR MARL system, arrives as a trained entity and has its own policy. Each system is trained to coordinate the actions within the system but is not trained to condition its actions based on the other system. Also, it may not be able to adjust to non-stationarity generated by another MARL system, which it may just discard as a noise in the environment. Notably, each MARL system does not have visibility about the actions of another system but can only see the global effect. In such a situation our challenge is to enable an intelligent coordinator/supervisor which can orchestrate the MARL systems autonomously. The supervisor should enable both Priority and MBR systems to act in parallel, thereby reducing the episode length in execution. Finally, the supervisor should ideally be able to adjust to changes in the radio environment and thereby be able to steer both systems to generalize their actions accordingly. Implementing such a method for orchestrating multiple MARL systems, which are pre-trained and thus have a mind of their own, is non-trivial. Additionally, the solution should be scalable, an aspect that has been a challenge with preset rule-based supervisor systems.

§ METHODOLOGY

As discussed in the earlier sections, the idea is to train the supervisor agent using AHT approaches. For this, we imbibe concepts mentioned across AHT in [26, 16, 5] and propose AT-MARL that binds the AHT to the pre-trained MARL framework of Priority and MBR systems discussed earlier.
The approach consists of training a goal policy (within the supervisor agent) that generates the sub-goals. To train the policy, two inputs are required (i) Individual agent's capabilities and (ii) Individual agent's current performance.

At first, we input the individual agent's capabilities in the form of a vector at every time instant $t$ [26].
The capability vector $\Gamma_{t}^i$ for agent $i$ at time $t$ represents the probability of agent $i$ achieving goal $g_p$.

\begin{align}
\Gamma_{t}^i = \begin{blockarray}{cccc}
g_{1} & g_{2} & \cdots & g_{P} \\
\begin{block}{[cccc]}
\rho_{t,1}^i & \rho_{t,2}^i & \cdots & \rho_{t,P}^i \\
\end{block}
\end{blockarray}
\end{align}

where $g_p, \; p = 1,\cdots, P$ is set of goals for the agent $i$ and $0\leq\rho_{t,p}^i\leq 1$ represents the probabilities for the agent $i$ achieving specific goal $p$ within time instant $t$. The capability vector $\Gamma_t^i$ is encoded using an encoder (fully connected network in our case) to project into low-dimensional space.

As mentioned earlier we also input the agent's current performance for better sub-goal assignment. For this, we create a state-action-goal tuple of the agent $(s_{t}^i, a_{t}^i, g_{t}^i)$, where $s_{t}^i$ is the state of the agent $i$ at time step $t$, $a_t^i$ is the action taken by the agent $i$ at time step $t$ and $g_t^i$ is the goal assigned for agent $i$ at time step $t$. We merge this tuple with the encoder output obtained earlier to create an embedding $m^i_t$.

For every agent, we use a separate encoder and merger and use it to create $N$ embeddings $m_t^i, \; i ={1,\cdots,N}$. Finally, we compute the embedding for all the agents and fuse them using the fusion layer. Also, we input the global goals (derived from intents) to the fusion layer since we would like the sub-goals generated to be conditional on the global goals.

The output of the fusion layer obtained by fusing the $N$ embeddings $m_{i,t}, \;\; i = 1,\cdots, N$ at time instant $t$ and global goals gives us the context representation for the future time step denoted as $c_{t+1}$, which can be used to train the goal policy $\pi_g$ to generate the sub-goals $g_{t+1}^i,\;i = 1,\cdots, N$ for next time step.
In this work, we use the RL-based actor-critic approach model to train the goal policy [28]. The block diagram of the proposed approach for sub-goal creation is shown in Figure <ref>.

The actor is chosen to be a 2-layer gated recurrent unit (GRU) network [12] and critic a $2$-layer fully connected network. Here we use fully connected layers to represent the encoder(2-layer), merger(1-layer), and fusion layers(3-layers).

Now before using the proposed method let us look at some baselines for comparison. This also helps to establish a lower bound and adds to the motivation for the AT-MARL.

[Without intermediate goals]
[With intermediate goals]
Parallel execution of the MARL agents with/without intermediate goals. These results are shown to demonstrate the problem which occurs without goal refinement when the MARL agents are executed in parallel.

The existing literature [21] for coordinating pre-trained MARL/RL systems uses a rule-based approach for orchestrating the Priority and MBR MARL systems.

Here we test situations where both MARL systems are executed simultaneously: In the first step, we introduce a simple supervisor agent which naively passes the same goals as is received from the intents.  Both the MARL systems execute simultaneously and are driven by the same goal. This results in sub-optimal performance and non-convergence which can be attributed to non-stationarity induced due to concurrent actions of both agents as shown in Figure <ref>.

Now in the proposed AT-MARL, we investigate three types of approaches (1) & (2): Sub-Goals generated at the service level and (3) Sub-Goals generated at the agent level

*  Goal generated at service level: Here we generate the goals at the service level i.e. at the KPI level using vanilla AHT framework. Since we plan to perform simultaneous execution of all the MARL systems, the same goal is passed to the respective agents in both MARL systems. At the start of the episode, we do not pass the final goal to the MARL systems but rather create intermediate sub-goals. These intermediate goals are expected to drive the trajectory in the right direction similar to the concept of auxiliary goals [15] in RL. However, given the same intermediate goals are passed to both MARL systems for each KPI, it results in significant oscillations around the target without any signs of convergence
(Figure <ref>). While this may look like a naive approach, this helps to analyze the challenges with respect to asynchronous and simultaneous execution.

*  Intuitive division of Goals: In the next step instead of sending the same generated sub-goals to both systems, we divide them in half and send them to respective agents. So if the AHT framework proposes an intermediate goal of 3 for QoE we divide it into equal parts and send it to respective Priority and MBR agents residing in two different MARL systems. The halving is a starting point based on our intuitive human logic of distributing a task equally among two participating cooperative systems. Although this is not optimal, however, we use this approach for a second baseline comparison.

* Goals generated at agent level: In this approach we improve on (2) whereby we generate unique intermediate goals at the agent level i.e. two different goals for each of the KPI, one in each of the MARL systems. This goal is conditioned on the global state of the system and the performance of the agents till that point.  Hence in the proposed approach,  we generate 6 unique goals for three different KPIs given there are two MARL systems in this experiment. We then explore the scalability of this approach by adding two additional intents to the system. In such a case we generate 10 goals, two for respective agents in both MARL agents. We refer to this version as the baseline AT-MARL method.

§ RESULTS AND DISCUSSIONS

Distribution of UE's across gNodeB's considered in the experiments for generalization.

Experiments are performed on a network emulator to demonstrate the efficacy of the proposed AHT-based approach. For effective comparison, the same network emulator is used as in [21]. The network emulator has the capability to generate different traffic scenarios from/to application layer to/from UE's through user-plane function (UPF) and gNodeB's. For every such link, we can configure the bandwidth to create different scenarios like enough resources or scarce resources for the UEs to achieve all the goals. However, since the main objective of the work is to demonstrate the usage of AHT approaches to train the supervisor agent we demonstrate only the enough resources scenario by configuring the airlink bandwidth to 10 MBPS.

In the network emulator, three types of services are simulated (i) CV, (ii) URLLC, and (iii) mIoT. The emulator has the capability of deploying multiple instances of URLLC and mIoT services. For our experiments, UE's are distributed uniformly across four gNodeB's as shown in Figure <ref>.

[Quality of Experience (CV)][Packet Loss (URLLC)][Packet Loss (mIoT)]
Comparison of approaches with uniformly distributed UEs measured by KPIs (a)QoE(CV), (b)PL(URLLC), (c)PL(mIoT).

We follow the procedure in [21] to train the priority and MBR MARL agents. For all three cases discussed in Section <ref>, we use these same pre-trained MARL agents.

Having established a baseline in Case 1 and 2 of Section <ref>, we used the AT-MARL method conditioned on Global and local observations for (3), i.e. Goals generated at agent level. This method generates individual sub-goals for each agent in the Priority and MBR MARL system. For example, we generate two unique QoE sub-goals, one for the RL agent modulating Priority for QoE (which is part of the Priority MARL system) and another for the RL agent modulating MBR for QoE intent(part of MBR MARL system). The sub-goals may again be revised after a few time steps by the supervisor based on the progress of the constituent agents. To the best of our knowledge, such an autonomous orchestration involving multiple pre-trained MARL systems and an AHT-based Supervisor agent has not been attempted before.

To quantify the performance we use a metric known as the integral absolute relative error (IAE) which is widely used in closed-loop control. Since IAE is subject to noise, we estimate it from the point the KPI first
reaches within 10% deviation from the target. The IAE is defined as

\begin{align}
IAE = \frac{1}{N} \displaystyle \sum_{k=1}^N \frac{|KPI[i]-T|}{T} \; \text{from the instant} \; KPI[i] \geq 0.9T
\end{align}

where $KPI[k]$ is the value of KPI measured at time instant $k$, $T$ is the target for the KPI and $N$ is the number of samples (counted from the instance the KPI value reaches $0.9T$ for the first time). The better the performance of the closed-loop control, the lower the IAE value.

The results are evaluated across three aspects:

* Convergence accuracy - measured by IAE and figures shown herein

* Scalability - Evaluated by increasing the number of intents to 5

* Generalization - Measured by training the AHT on a Uniform distribution of UEs and then testing on a Gaussian distribution. This simulates a change in the Radio environment due to changes in mobility patterns.

§.§ Uniform Distribution

The KPI level execution plot for the QoE, PL (URLLC), and PL (mIoT) are shown separately in Figures <ref>, <ref> and <ref> respectively. From the plots, it is evident that the AT-MARL method resulted in significantly improved performance than the rule-based approach. The oscillations around the expected value are reduced and the convergence is faster.  The IAE values obtained for different approaches for all KPIs are given in Table <ref>. From these values also, it is clear that the AT-MARL results in improved IAE values for every KPI, when compared to the rule-based approach by demonstrating accurate control and faster convergence. In terms of convergence time, we see that AT-MARL achieves convergence at an average of 14-time steps instead of 20-time steps in the rule-based approach (Fig <ref>). Hence the proposed approach improves the convergence time by about  30%.
Also, AT-MARL outperforms the goal-halving method by  21% as seen in IAE values.

Performance of the AT-MARL approach on multiple instances of 5 services to demonstrate the scalability

To demonstrate the scalability of the proposed approach we tested the proposed approach with five intents, QoE for CV, PL for two instances of URLLC service, and PL for two instances of mIoT service. The execution performance across KPIs using AT-MARL is shown in Figure <ref>. From the plot, it can be seen that the proposed approach results in accurate closed-loop control and thus is easily scalable. Hence as intents increase and correspondingly the complexity, the need for human-driven rules can be eliminated using AT-MARL. Although we limited experiments to 5 intents, the proposed approach can be easily scalable to many more number of intents.

[Quality of Experience (CV)][Packet Loss (URLLC)][Packet Loss (mIoT)]

Comparison of approaches with Gaussian distributed UE's across KPI's (a) QoE (CV), (b) PL (URLLC), (c) PL (mIoT).

IAE calculated on KPIs across approaches when UE's are distributed uniformly across gNodeB's

ApproachKPI     QoE (CV)       PL (URLLC)     PL (mIoT)
Rule-based                             0.51     0.88          1.87
Goal-Halving                           0.42     0.54          0.98
AT-MARL                    0.364     0.417         0.575

§.§ Gaussian Distribution

To demonstrate the generalization capabilities of AT-MARL, we tested the approach on the network emulator when the UE's are distributed as in Figure <ref> (Gaussian distribution).

Here the same agents from Sec <ref> are employed without further retraining i.e. the same Priority MARL, MBR MARL agent systems, and even the same M3RL agent. The KPI plot for the QoE, PL (URLLC), and PL (mIoT) are shown in Figures <ref>, <ref> and <ref> respectively. From the plots, it is evident that the AT-MARL achieves far better generalization when compared with the rule-based approach. Also, IAE values obtained for this case across different KPIs are given in Table <ref>. Hence, it can be concluded that the performance of AT-MARL demonstrates adequate generalization capabilities in the face of a changing radio environment.

Performance of updated AT-MARL with supervisor agent retrained on Gaussian distribution (Oracle*)

Performance of the agents with AT-MARL approach under continuous distribution change.

IAE calculated on KPIs compared across approaches when UE's spread in Gaussian distribution across gNodeBs. The first three options are test-time metrics for generalization

ApproachKPI     QoE (CV)       PL (URLLC)     PL (mIoT)
Rule-based                             0.64     2.31          4.74
Goal-Halving                           0.62     0.66          1.02
AT-MARL                    0.421     0.397         0.614
AT-MARL (with updated supervisor) ORACLE*     0.341     0.287     0.511

For purposes of benchmarking the generalization ability of AT-MARL with an upper bound, where we re-trained the supervisor agent on Gaussian distribution by keeping priority and MBR MARL agents fixed.
The execution performance of this approach is shown in Figure <ref>

and the IAE values obtained are shown in Table <ref>. Here it could be seen that AT-MARL's performance is closer to the best possible upper-bound case. Hence it can be concluded that the proposed method, AT-MARL, can generalize across radio environments without the need for retraining supervisor or lower-level MARL agents.

§.§ Real-time execution of the proposed approach

In the above two sub-sections, we have presented the results when the distribution of UE's across gNodeBs is either uniform or Gaussian. However, in a real-time scenario, the distribution change is continuous i.e. during the middle of one execution the distribution changes. To emulate the same, we performed the following experiment.

We initially trained the agents on uniform distribution. The KPI-level execution plot for this case is shown in Figure <ref>. In the middle of the execution, at $20^\text{th}$ time step, we changed the distribution to Gaussian (Figure <ref>). Now, because of this, we can see a small dip in KPIs. However, the supervisor agent redefined sub-goals such that performance is quickly restored.
Further at the $30^\text{th}$ time step, we change the distribution to Gamma (Figure <ref>). Here also it is evident that there is a very small dip in performance but the KPIs are promptly brought back to their respective targets as the supervisor agent realizes the same and reassigns the goals.

§ CONCLUSIONS

In this work, we introduce a method AT-MARL whereby multiple intents in a network can be fulfilled by autonomous orchestration of an AI-based Controller. The controller or supervisor learns to control more than one pre-trained group of RL agents that have not seen each other before and neither is allowed to communicate directly between groups. Results demonstrate that creating dynamic “Goals are Enough" to achieve such a complex control as it incentivizes sets of pre-trained MARL agents to act in cohesion.

The proposed method leverages AHT techniques on top of MARL framework. The synergy induced by our method across both the AHT and MARL agents is unique to our work.
The results demonstrate significantly improved convergence and also the scalability of the approach when compared with existing approaches. Most importantly the findings demonstrate the capability of the proposed approach to generalize to changes in the radio environment. Hence such a framework can be adequately leveraged for constructing a self-contained hierarchical IMF, which in turn will autonomously orchestrate multiple intents in the network.

Future work may look into the explainability of the AI controller such that the system becomes interpretable to a human expert and thereby more trustworthy for live deployments.

[1]
Khizar Abbas, Talha Ahmed Khan, Muhammad Afaq, and Wang-Cheol Song.
Network slice lifecycle management for 5g mobile networks: An
intent-based networking approach.
IEEE Access, 9:80128–80146, 2021.

[2]
Fred Aklamanu, Sabine Randriamasy, Eric Renault, Imran Latif, and Abdelkrim
Intent-based real-time 5g cloud service provisioning.
In 2018 IEEE Globecom Workshops (GC Wkshps), pages 1–6. IEEE,

[3]
Stefano V. Albrecht and Subramanian Ramamoorthy.
Comparative evaluation of mal algorithms in a diverse set of ad hoc
team problems.
In AAMAS '12, 2012.

[4]
Stefano V. Albrecht and Subramanian Ramamoorthy.
A game-theoretic model and best-response learning method for ad hoc
coordination in multiagent systems.
In Proceedings of the 2013 International Conference on
Autonomous Agents and Multi-Agent Systems, AAMAS '13, pages 1155–1156.
International Foundation for Autonomous Agents and Multiagent Systems, 2013.

[5]
Chris Baker, Rebecca Saxe, and Joshua Tenenbaum.
Action understanding as inverse planning.
Cognition, 113:329–349, 12 2009.

[6]
Samuel Barrett, Avi Rosenfeld, Sarit Kraus, and Peter Stone.
Making friends on the fly: Cooperating with new teammates.
Artificial Intelligence, 242:132–171, 2017.

[7]
Samuel Barrett and Peter Stone.
Cooperating with unknown teammates in robot soccer.
In AAAI Workshop on Multiagent Interaction without Prior
Coordination, page 6, 2014.

[8]
Samuel Barrett, Peter Stone, and Sarit Kraus.
Empirical evaluation of ad hoc teamwork in the pursuit domain.
In AAMAS '11, volume 2, pages 567–574, 2011.

[9]
Michael Bowling and Peter Mccracken.
Coordination and adaptation in impromptu teams.
In AAAI Conference on Artificial Intelligence, 2005.

[10]
Rodrigo Canaan, Xianbo Gao, Julian Togelius, Andy Nealen, and Stefan Menzel.
Generating and adapting to diverse ad-hoc cooperation agents in
arXiv preprint arXiv:2004.13710, 2020.

[11]
Subhro Chakraborty and Peter Stone.
Cooperating with a markovian ad hoc teammate.
In Proceedings of the 12th International Conference on
Autonomous Agents and Multiagent Systems, volume 1 of AAMAS '13, pages
1085–1092. International Foundation for Autonomous Agents and Multiagent
Systems, 2013.

[12]
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio.
On the properties of neural machine translation: Encoder-decoder
arXiv preprint arXiv:1409.1259, 2014.

[13]
Kaushik Dey, Satheesh K. Perepu, Pallab Dasgupta, and Abir Das.
Domain adaptation of reinforcement learning agents based on network
service proximity.
In 2023 IEEE 9th International Conference on Network
Softwarization (NetSoft), pages 152–160, 2023.

[14]
ETSI GS ZSM 009-1.
Zero-touch network and service management (zsm): Closed-loop
automation; enablers group specification.

[15]
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z
Leibo, David Silver, and Koray Kavukcuoglu.
Reinforcement learning with unsupervised auxiliary tasks.
arXiv preprint arXiv:1611.05397, 2016.

[16]
Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman.
Deep successor reinforcement learning.
arXiv preprint arXiv:1606.02396, 2016.

[17]
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster.
Trajectory diversity for zero-shot coordination.
In Proceedings of the 38th International Conference on Machine
Learning, pages 7204–7213, 2021.

[18]
Jean P. Martins, Igor Almeida, Ricardo Souza, and Silvia Lins.
Policy distillation for real-time inference in fronthaul congestion
IEEE Access, 9:154471–154483, 2021.

[19]
Jorg Niemoller, Leo Mokrushin, Swarup Mohalik, Vlachou Konchylaki, and
Cognitive processes for adaptive intent-based networking.
Technical report, Ericsson Technology Review, Kista, Sweden, 2020.

[20]
Georgios Papoudakis, Filippos Christianos, and Stefano V. Albrecht.
Local information agent modelling in partially-observable
arXiv preprint arXiv:2006.09447, 2021.

[21]
Satheesh K. Perepu, Jean P. Martins, Ricardo Souza S, and Kaushik Dey.
Intent-based multi-agent reinforcement learning for service assurance
in cellular networks.
In GLOBECOM 2022 - 2022 IEEE Global Communications Conference,
pages 2879–2884, 2022.

[22]
Arrasy Rahman, Niklas H"opner, Filippos Christianos, and Stefano V. Albrecht.
Towards open ad hoc teamwork using graph-based policy learning.
In International Conference on Machine Learning, volume 139.
PMLR, 2021.

[23]
Manish Ravula, Shani Alkoby, and Peter Stone.
Ad hoc teamwork with behavior switching agents.
In International Joint Conference on Artificial Intelligence,
pages 550–556, 2019.

[24]
Diego Fernando Preciado Rojas and Andreas Mitschele-Thiel.
Machine learning-based son function conflict resolution.
In 2019 IEEE Symposium on Computers and Communications (ISCC),
pages 1–6, 2019.

[25]
Michael Rovatsos and Marco Wolf.
Towards social complexity reduction in multiagent learning: the adhoc
In Collaborative Learning Agents. Papers from 2002 AAAI Spring
Symposium on, number SS-02-02 in AAAI Technical Report, pages 90–97. AAAI
Press, 2002.

[26]
Tianmin Shu and Yuandong Tian.
M$^3$rl: Mind-aware multi-agent management reinforcement learning.
In International Conference on Learning Representations, 2019.

[27]
Peter Stone, Gal A. Kaminka, Sarit Kraus, and Jeffrey S. Rosenschein.
Ad hoc autonomous agent teams: Collaboration without
In Proceedings of the Twenty-Fourth AAAI Conference on
Artificial Intelligence, AAAI'10, page 1504–1509. AAAI Press, 2010.

[28]
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.
Policy gradient methods for reinforcement learning with function
Advances in neural information processing systems, 12, 1999.

[29]
Tarik Taleb, Konstantinos Samdanis, Badr Mada, Hannu Flinck, Sunny Dutta, and
Dario Sabella.
On multi-access edge computing: A survey of the emerging 5g network
edge cloud architecture and orchestration.
IEEE Communications Surveys & Tutorials, 19(3):1657–1681,

[30]
Luis Velasco, Marco Signorelli, Oscar González De Dios, Chrysa Papagianni,
Roberto Bifulco, Juan Jose Vegas Olmos, Simon Pryor, Gino Carrozzo, Julius
Schulz-Zander, Mehdi Bennis, et al.
End-to-end intent-based networking.
IEEE communications Magazine, 59(10):106–112, 2021.

[31]
Engin Zeydan and Yekta Turk.
Recent advances in intent-based networking: A survey.
In 2020 IEEE 91st Vehicular Technology Conference
(VTC2020-Spring), pages 1–5. IEEE, 2020.