§ ACKNOWLEDGEMENT

We thank Martin Jaggi for his comments and support.
We also thank anonymous reviewers and Soumajit Majumder<EMAIL_ADDRESS>for their constructive and helpful reviews.
This work was supported in part by the National Key R&D Program of China (Project No. 2022ZD0115100), the Research Center for Industries of the Future (RCIF) at Westlake University, and Westlake Education Foundation.

[Achituve et al., 2021]
Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya.
Personalized federated learning with gaussian processes.
Advances in Neural Information Processing Systems, 34, 2021.

[Amosy et al., 2021]
Ohad Amosy, Gal Eyal, and Gal Chechik.
Inference-time personalized federated learning.
arXiv preprint arXiv:2111.08356, 2021.

[Arivazhagan et al., 2019]
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav
Federated learning with personalization layers.
arXiv preprint arXiv:1912.00818, 2019.

[Arjovsky et al., 2019]
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.
Invariant risk minimization.
arXiv preprint arXiv:1907.02893, 2019.

[Ba et al., 2016]
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
Layer normalization.
arXiv preprint arXiv:1607.06450, 2016.

[Caldas et al., 2018]
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub
Konečnỳ, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar.
Leaf: A benchmark for federated settings.
arXiv preprint arXiv:1812.01097, 2018.

[Chen et al., 2018]
Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He.
Federated meta-learning with fast convergence and efficient
arXiv preprint arXiv:1802.07876, 2018.

[Chen & Chao, 2022]
Hong-You Chen and Wei-Lun Chao.
On bridging generic and personalized federated learning for image
In International Conference on Learning Representations, 2022.
URL <https://openreview.net/forum?id=I1hQbx10Kxn>.

[Chen et al., 2020]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.
A simple framework for contrastive learning of visual
In International conference on machine learning, pp.  1597–1607. PMLR, 2020a.

[Chen et al., 2020]
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.
Improved baselines with momentum contrastive learning.
arXiv preprint arXiv:2003.04297, 2020b.

[Chrabaszcz et al., 2017]
Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter.
A downsampled variant of imagenet as an alternative to the cifar
arXiv preprint arXiv:1707.08819, 2017.

[Coates et al., 2011]
Adam Coates, Andrew Ng, and Honglak Lee.
An analysis of single-layer networks in unsupervised feature
In Proceedings of the fourteenth international conference on
artificial intelligence and statistics, pp.  215–223. JMLR Workshop and
Conference Proceedings, 2011.

[Collins et al., 2021]
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai.
Exploiting shared representations for personalized federated
In International Conference on Machine Learning, pp.  2089–2099. PMLR, 2021.

[Deng et al., 2009]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In 2009 IEEE conference on computer vision and pattern
recognition, pp.  248–255. Ieee, 2009.

[Deng et al., 2020]
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi.
Adaptive personalized federated learning.
arXiv preprint arXiv:2003.13461, 2020a.

[Deng et al., 2020]
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi.
Distributionally robust federated averaging.
Advances in Neural Information Processing Systems,
33:0 15111–15122, 2020b.

[Diao et al., 2021]
Enmao Diao, Jie Ding, and Vahid Tarokh.
Heterofl: Computation and communication efficient federated learning
for heterogeneous clients.
In International Conference on Learning Representations, 2021.
URL <https://openreview.net/forum?id=TNkPBBYFkXg>.

[Du et al., 2021]
Wei Du, Depeng Xu, Xintao Wu, and Hanghang Tong.
Fairness-aware agnostic federated learning.
In Proceedings of the 2021 SIAM International Conference on
Data Mining (SDM), pp.  181–189. SIAM, 2021.

[Duchi et al., 2011]
John Duchi, Elad Hazan, and Yoram Singer.
Adaptive subgradient methods for online learning and stochastic
Journal of machine learning research, 120 (7), 2011.

[Fallah et al., 2020]
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar.
Personalized federated learning with theoretical guarantees: A
model-agnostic meta-learning approach.
Advances in Neural Information Processing Systems,
33:0 3557–3568, 2020.

[Ganin et al., 2016]
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo
Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.
Domain-adversarial training of neural networks.
The journal of machine learning research, 170
(1):0 2096–2030, 2016.

[Gasanov et al., 2021]
Elnur Gasanov, Ahmed Khaled, Samuel Horváth, and Peter Richtárik.
Flix: A simple and communication-efficient alternative to local
methods in federated learning.
arXiv preprint arXiv:2111.11556, 2021.

[Ghosh et al., 2020]
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran.
An efficient framework for clustered federated learning.
Advances in Neural Information Processing Systems,
33:0 19586–19597, 2020.

[Grandvalet & Bengio, 2004]
Yves Grandvalet and Yoshua Bengio.
Semi-supervised learning by entropy minimization.
Advances in neural information processing systems, 17, 2004.

[Gretton et al., 2012]
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and
Alexander Smola.
A kernel two-sample test.
The Journal of Machine Learning Research, 130
(1):0 723–773, 2012.

[Gulrajani & Lopez-Paz, 2020]
Ishaan Gulrajani and David Lopez-Paz.
In search of lost domain generalization.
arXiv preprint arXiv:2007.01434, 2020.

[Guo et al., 2021]
Yongxin Guo, Tao Lin, and Xiaoying Tang.
Towards federated learning on time-evolving heterogeneous data.
arXiv preprint arXiv:2112.13246, 2021.

[Haddadpour et al., 2021]
Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi.
Federated learning with compression: Unified analysis and sharp
In International Conference on Artificial Intelligence and
Statistics, pp.  2350–2358. PMLR, 2021.

[Hand, 2006]
David J Hand.
Classifier technology and the illusion of progress.
Statistical science, 210 (1):0 1–14, 2006.

[Hanzely & Richtárik, 2020]
Filip Hanzely and Peter Richtárik.
Federated learning of a mixture of global and local models.
arXiv preprint arXiv:2002.05516, 2020.

[Hassani et al., 2021]
Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and
Humphrey Shi.
Escaping the big data paradigm with compact transformers.
arXiv preprint arXiv:2104.05704, 2021.

[He et al., 2020]
Chaoyang He, Songze Li, Jinhyun So, Mi Zhang, Hongyi Wang, Xiaoyang Wang,
Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, et al.
Fedml: A research library and benchmark for federated machine
arXiv preprint arXiv:2007.13518, 2020a.

[He et al., 2016]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp.  770–778, 2016.

[He et al., 2020]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.
Momentum contrast for unsupervised visual representation learning.
In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition, pp.  9729–9738, 2020b.

[Hendrycks & Dietterich, 2019]
Dan Hendrycks and Thomas Dietterich.
Benchmarking neural network robustness to common corruptions and
arXiv preprint arXiv:1903.12261, 2019a.

[Hendrycks & Dietterich, 2019]
Dan Hendrycks and Thomas Dietterich.
Benchmarking neural network robustness to common corruptions and
Proceedings of the International Conference on Learning
Representations, 2019b.

[Hendrycks & Dietterich, 2018]
Dan Hendrycks and Thomas G Dietterich.
Benchmarking neural network robustness to common corruptions and
surface variations.
arXiv preprint arXiv:1807.01697, 2018.

[Hendrycks et al., 2021]
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan
Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.
The many faces of robustness: A critical analysis of
out-of-distribution generalization.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pp.  8340–8349, 2021a.

[Hendrycks et al., 2021]
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.
Natural adversarial examples.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  15262–15271, 2021b.

[Hsieh et al., 2020]
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons.
The non-iid data quagmire of decentralized machine learning.
In International Conference on Machine Learning, pp.  4387–4398. PMLR, 2020.

[Hsu et al., 2019]
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown.
Measuring the effects of non-identical data distribution for
federated visual classification.
arXiv preprint arXiv:1909.06335, 2019.

[Huang et al., 2021]
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei,
and Yong Zhang.
Personalized cross-silo federated learning on non-iid data.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 35, pp.  7865–7873, 2021.

[Ioffe & Szegedy, 2015]
Sergey Ioffe and Christian Szegedy.
Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
In International conference on machine learning, pp.  448–456. PMLR, 2015.

[Iwasawa & Matsuo, 2021]
Yusuke Iwasawa and Yutaka Matsuo.
Test-time classifier adjustment module for model-agnostic domain
Advances in Neural Information Processing Systems, 34, 2021.

[Jiang et al., 2019]
Yihan Jiang, Jakub Konečnỳ, Keith Rush, and Sreeram Kannan.
Improving federated learning personalization via model agnostic meta
arXiv preprint arXiv:1909.12488, 2019.

[Kairouz et al., 2021]
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi
Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham
Cormode, Rachel Cummings, et al.
Advances and open problems in federated learning.
Foundations and Trends® in Machine Learning,
140 (1–2):0 1–210, 2021.

[Karimireddy et al., 2020]
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian
Stich, and Ananda Theertha Suresh.
Scaffold: Stochastic controlled averaging for federated learning.
In International Conference on Machine Learning, pp.  5132–5143. PMLR, 2020.

[Karimireddy et al., 2021]
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J
Reddi, Sebastian U Stich, and Ananda Theertha Suresh.
Mime: Mimicking centralized stochastic algorithms in federated
In Advances in Neural Information Processing Systems, 2021.

[Kingma & Ba, 2014]
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980, 2014.

[Koh et al., 2021]
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang,
Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips,
Irena Gao, et al.
Wilds: A benchmark of in-the-wild distribution shifts.
In International Conference on Machine Learning, pp.  5637–5664. PMLR, 2021.

[Krizhevsky et al., 2009]
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.

[Kumar et al., 2022]
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy
Fine-tuning can distort pretrained features and underperform
In International Conference on Learning Representations, 2022.
URL <https://openreview.net/forum?id=UYneFzXSJWh>.

[Lai et al., 2022]
Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha
Madhyastha, and Mosharaf Chowdhury.
Fedscale: Benchmarking model and system performance of federated
learning at scale.
In International Conference on Machine Learning, pp.  11814–11827. PMLR, 2022.

[LeCun et al., 1998]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.
Gradient-based learning applied to document recognition.
Proceedings of the IEEE, 860 (11):0
2278–2324, 1998.

[Lee et al., 2018]
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.
A simple unified framework for detecting out-of-distribution samples
and adversarial attacks.
Advances in neural information processing systems, 31, 2018.

[Li et al., 2018]
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.
Learning to generalize: Meta-learning for domain generalization.
In Thirty-Second AAAI Conference on Artificial Intelligence,

[Li et al., 2020]
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith.
Federated learning: Challenges, methods, and future directions.
IEEE Signal Processing Magazine, 370 (3):0
50–60, 2020a.

[Li et al., 2020]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and
Virginia Smith.
Federated optimization in heterogeneous networks.
Proceedings of Machine Learning and Systems, 2:0
429–450, 2020b.

[Li et al., 2020]
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith.
Fair resource allocation in federated learning.
In International Conference on Learning Representations,
URL <https://openreview.net/forum?id=ByexElSYDr>.

[Li et al., 2021]
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith.
Tilted empirical risk minimization.
In International Conference on Learning Representations,
URL <https://openreview.net/forum?id=K5YasWXZT3O>.

[Li et al., 2021]
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith.
Ditto: Fair and robust federated learning through personalization.
In International Conference on Machine Learning, pp.  6357–6368. PMLR, 2021b.

[Li et al., 2021]
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou.
FedBN: Federated learning on non-IID features via local batch
In International Conference on Learning Representations,
URL <https://openreview.net/forum?id=6YEQUn0QICG>.

[Lin et al., 2020]
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi.
Ensemble distillation for robust model fusion in federated learning.
In Advances in Neural Information Processing Systems,

[Lin et al., 2020]
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi.
Don't use large mini-batches, use local sgd.
In International Conference on Learning Representations,
URL <https://openreview.net/forum?id=B1eyO1BFPr>.

[Liu et al., 2021]
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor
Mordan, and Alexandre Alahi.
TTT++: When does self-supervised test-time training fail or thrive?
Advances in Neural Information Processing Systems, 34, 2021.

[Mansour et al., 2020]
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh.
Three approaches for personalization with applications to federated
arXiv preprint arXiv:2002.10619, 2020.

[Marfoq et al., 2022]
Othmane Marfoq, Giovanni Neglia, Richard Vidal, and Laetitia Kameni.
Personalized federated learning through local memorization.
In International Conference on Machine Learning, pp.  15070–15092. PMLR, 2022.

[McMahan et al., 2017]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera
y Arcas.
Communication-efficient learning of deep networks from decentralized
In Artificial Intelligence and Statistics, pp.  1273–1282.
PMLR, 2017.

[Miller et al., 2021]
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh,
Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt.
Accuracy on the line: On the strong correlation between
out-of-distribution and in-distribution generalization.
In International Conference on Machine Learning, pp.  7721–7735. PMLR, 2021.

[Mitra et al., 2021]
Aritra Mitra, Rayana Jaafar, George J Pappas, and Hamed Hassani.
Linear convergence in federated learning: Tackling client
heterogeneity and sparse gradients.
Advances in Neural Information Processing Systems,
34:0 14606–14619, 2021.

[Mohri et al., 2019]
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh.
Agnostic federated learning.
In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.),
Proceedings of the 36th International Conference on Machine Learning,
ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of
Proceedings of Machine Learning Research, pp.  4615–4625. PMLR,
URL <http://proceedings.mlr.press/v97/mohri19a.html>.

[Quiñonero-Candela et al., 2008]
Joaquin Quiñonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D
Dataset shift in machine learning.
Mit Press, 2008.

[Radford et al., 2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
Learning transferable visual models from natural language
In International Conference on Machine Learning, pp.  8748–8763. PMLR, 2021.

[Recht et al., 2018]
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.
Do cifar-10 classifiers generalize to cifar-10?
arXiv preprint arXiv:1806.00451, 2018.

[Recht et al., 2019]
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.
Do imagenet classifiers generalize to imagenet?
In International Conference on Machine Learning, pp.  5389–5400. PMLR, 2019.

[Reddi et al., 2021]
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush,
Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan.
Adaptive federated optimization.
In International Conference on Learning Representations, 2021.
URL <https://openreview.net/forum?id=LkFG3lB13U5>.

[Reisizadeh et al., 2020]
Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, and Ali Jadbabaie.
Robust federated learning: The case of affine distribution shifts.
arXiv preprint arXiv:2006.08907, 2020.

[Ren et al., 2020]
Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and
Hongsheng Li.
Balanced meta-softmax for long-tailed visual recognition.
arXiv preprint arXiv:2007.10740, 2020.

[Ren et al., 2021]
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and
Balaji Lakshminarayanan.
A simple fix to mahalanobis distance for improving near-ood
arXiv preprint arXiv:2106.09022, 2021.

[Ryffel et al., 2018]
Theo Ryffel, Andrew Trask, Morten Dahl, Bobby Wagner, Jason Mancuso, Daniel
Rueckert, and Jonathan Passerat-Palmbach.
A generic framework for privacy preserving deep learning.
arXiv preprint arXiv:1811.04017, 2018.

[Sagawa et al., 2019]
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang.
Distributionally robust neural networks for group shifts: On the
importance of regularization for worst-case generalization.
arXiv preprint arXiv:1911.08731, 2019.

[Sagawa et al., 2021]
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick
Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, et al.
Extending the wilds benchmark for unsupervised adaptation.
arXiv preprint arXiv:2112.05090, 2021.

[Sattler et al., 2020]
Felix Sattler, Klaus-Robert Müller, and Wojciech Samek.
Clustered federated learning: Model-agnostic distributed multitask
optimization under privacy constraints.
IEEE Transactions on Neural Networks and Learning Systems,

[Shamsian et al., 2021]
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik.
Personalized federated learning using hypernetworks.
In International Conference on Machine Learning, pp.  9489–9502. PMLR, 2021.

[Shannon, 2001]
Claude Elwood Shannon.
A mathematical theory of communication.
ACM SIGMOBILE mobile computing and communications review,
50 (1):0 3–55, 2001.

[Shen et al., 2021]
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng
Towards out-of-distribution generalization: A survey.
arXiv preprint arXiv:2108.13624, 2021.

[Shi et al., 2021]
Yuxin Shi, Han Yu, and Cyril Leung.
A survey of fairness-aware federated learning.
arXiv preprint arXiv:2111.01872, 2021.

[Singhal et al., 2021]
Karan Singhal, Hakim Sidahmed, Zachary Garrett, Shanshan Wu, John Rush, and
Sushant Prakash.
Federated reconstruction: Partially local federated learning.
Advances in Neural Information Processing Systems, 34, 2021.

[Smith et al., 2017]
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar.
Federated multi-task learning.
arXiv preprint arXiv:1705.10467, 2017.

[Sun et al., 2020]
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.
Test-time training with self-supervision for generalization under
distribution shifts.
In International Conference on Machine Learning, pp.  9229–9248. PMLR, 2020.

[T Dinh et al., 2020]
Canh T Dinh, Nguyen Tran, and Josh Nguyen.
Personalized federated learning with moreau envelopes.
Advances in Neural Information Processing Systems,
33:0 21394–21405, 2020.

[Tan et al., 2022]
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang.
Towards personalized federated learning.
IEEE Transactions on Neural Networks and Learning Systems,

[Tan et al., 2021]
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi
Fedproto: Federated prototype learning over heterogeneous devices.
arXiv preprint arXiv:2105.00243, 2021.

[Taori et al., 2020]
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht,
and Ludwig Schmidt.
Measuring robustness to natural distribution shifts in image
Advances in Neural Information Processing Systems,
33:0 18583–18599, 2020.

[Tenison et al., 2022]
Irene Tenison, Sai Aravind Sreeramadas, Vaikkunth Mugunthan, Edouard Oyallon,
and Irina Rish.
Gradient masked averaging for federated learning.
arXiv preprint arXiv:2201.11986, 2022.

[Vapnik, 1998]
Vladimir N. Vapnik.
Statistical Learning Theory.
Wiley-Interscience, 1998.

[Wang et al., 2020]
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell.
Tent: Fully test-time adaptation by entropy minimization.
arXiv preprint arXiv:2006.10726, 2020a.

[Wang et al., 2020]
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor.
Tackling the objective inconsistency problem in heterogeneous
federated optimization.
arXiv preprint arXiv:2007.07481, 2020b.

[Wang et al., 2020]
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat.
Slowmo: Improving communication-efficient distributed sgd with slow
In International Conference on Learning Representations,
URL <https://openreview.net/forum?id=SkxJ8REYPH>.

[Wang et al., 2019]
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise
Beaufays, and Daniel Ramage.
Federated evaluation of on-device personalization.
arXiv preprint arXiv:1910.10252, 2019.

[Wang et al., 2022]
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai.
Continual test-time domain adaptation.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  7201–7211, 2022.

[Wang et al., 2020]
Yufei Wang, Haoliang Li, and Alex C Kot.
Heterogeneous domain generalization via domain mixup.
In ICASSP 2020-2020 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pp.  3622–3626. IEEE,

[Wortsman et al., 2021]
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh
Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt.
Robust fine-tuning of zero-shot models.
arXiv preprint arXiv:2109.01903, 2021.

[Wu et al., 2022]
Bingzhe Wu, Zhipeng Liang, Yuxuan Han, Yatao Bian, Peilin Zhao, and Junzhou
Drflm: Distributionally robust federated learning with inter-client
noise via local mixup.
arXiv preprint arXiv:2204.07742, 2022a.

[Wu et al., 2022]
Shanshan Wu, Tian Li, Zachary Charles, Yu Xiao, Ziyu Liu, Zheng Xu, and
Virginia Smith.
Motley: Benchmarking heterogeneity and personalization in federated
arXiv preprint arXiv:2206.09262, 2022b.

[Wu & He, 2018]
Yuxin Wu and Kaiming He.
Group normalization.
In Proceedings of the European conference on computer vision
(ECCV), pp.  3–19, 2018.

[Yan et al., 2020]
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren.
Improve unsupervised domain adaptation with mixup training.
arXiv preprint arXiv:2001.00677, 2020.

[Yu et al., 2020]
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov.
Salvaging federated learning by local adaptation.
arXiv preprint arXiv:2002.04758, 2020.

[Yuchen Lin et al., 2022]
Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Christophe
Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr.
Fednlp: Benchmarking federated learning methods for natural language
processing tasks.
Findings of NAACL, 2022.

[Yurochkin et al., 2019]
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia
Hoang, and Yasaman Khazaeni.
Bayesian nonparametric federated learning of neural networks.
In International Conference on Machine Learning, pp.  7252–7261. PMLR, 2019.

[Zhang et al., 2021]
Marvin Zhang, Sergey Levine, and Chelsea Finn.
Memo: Test time robustness via adaptation and augmentation.
arXiv preprint arXiv:2110.09506, 2021a.

[Zhang et al., 2021]
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez.
Personalized federated learning with first order model optimization.
In International Conference on Learning Representations,
URL <https://openreview.net/forum?id=ehJqJQk9cw>.

[Zhang et al., 2021]
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng.
Test-agnostic long-tailed recognition by test-time aggregating
diverse experts with self-supervision.
arXiv preprint arXiv:2107.09249, 2021c.

[Zhu et al., 2022]
Chen Zhu, Zheng Xu, Mingqing Chen, Jakub Konečný, Andrew Hard, and
Tom Goldstein.
Diurnal or nocturnal? federated learning of multi-branch networks
from periodically shifting distributions.
In International Conference on Learning Representations, 2022.
URL <https://openreview.net/forum?id=E4EE_ohFGz>.


# Learning Music Representations with wav2vec 2.0

###### Abstract

Learning music representations that are general-purpose offers the flexibility
to finetune several downstream tasks using smaller datasets. The wav2vec 2.0
speech representation model showed promising results in many downstream speech
tasks, but has been less effective when adapted to music. In this paper, we
evaluate whether pre-training wav2vec 2.0 directly on music data can be a
better solution instead of finetuning the speech model. We illustrate that
when pre-training on music data, the discrete latent representations are able
to encode the semantic meaning of musical concepts such as pitch and
instrument. Our results show that finetuning wav2vec 2.0 pre-trained on music
data allows us to achieve promising results on music classification tasks that
are competitive with prior work on audio representations. In addition, the
results are superior to the pre-trained model on speech embeddings,
demonstrating that wav2vec 2.0 pre-trained on music data can be a promising
music representation model.

Index Terms—  music representations, self-supervision, pre-training

##  1 Introduction

Learning feature representations with deep architectures has shown remarkable
success over hand-crafted features in Music Information Retrieval (MIR) [1].
Approaches such as transfer learning from music auto-tagging [2, 3, 4] allows
someone to pre-train neural networks using large datasets and extracting
features for downstream MIR tasks such as instrument classification or genre
recognition. In this way, downstream MIR tasks can be solved using smaller
annotated datasets, which is desired since labeling is costly and difficult to
achieve. One issue with auto-tagging models is that they require very large
annotated datasets that are still difficult to obtain. To overcome the need
for large annotated datasets, new music representation techniques have emerged
that do not directly use waveform-related labels emerged. For example, pre-
training from language models [5] or using noisy language descriptors of the
musical content [6].

A different approach that is based on using proxy tasks to learn
representations is self-supervised learning (SSL), where information from
input data is extracted to provide labels. This is advantageous since labels
can be generated automatically without requiring human intervention. Some SSL
models have been proposed for music representations showing competitive
performance in several downstream MIR tasks [7, 8, 9, 10]. Beyond music
representation learning, SSL models have grown popularity for speech
representations and downstream tasks such as speaker identification, automatic
speech recognition, phoneme recognition, and speech translation [11]. Examples
of speech SSL models include wav2vec 2.0 [12] which is a contrastive learning-
based approach where the model learns to distinguish a target sample
(positive) from distractors (negative). The original model was pre-trained on
the LibriSpeech dataset [13] and its success is highlighted by the ability to
retain high performance even when dedicated datasets for downstream tasks are
very small, e.g., 10 minutes only for speech recognition [12] or 1000
observations for non-intrusive speech quality assessment [14].

The wav2vec 2.0 SSL model has been extensively evaluated for speech tasks.
However, its adaptation to music tasks (such as pitch classification or
instrument classification) has been explored so far without success, as shown
in two studies. In the NeurIPS challenge HEAR [15], wav2vec 2.0 embeddings are
extracted from the model pre-trained on the LibriSpeech dataset and are used
as input features without finetuning. Their performance is relatively low in
music tasks, even if wav2vec 2.0 speech embeddings can still represent some
musical concepts to some degree, such as pitch [15]. Wang et al. [16] have
also evaluated wav2vec 2.0 outside of the speech domain. In this case, the
authors found that wav2vec 2.0 did not perform well when pre-trained on
AudioSet [17], possibly due to the limitation of the masked prediction
objective of learning from a dataset more complex than LibriSpeech [16].

An approach that is still unexplored is pre-training wav2vec 2.0 on music data
only. The transferability of deep networks becomes more challenging when the
source and the target tasks have different domains [18] and it has been shown
that wav2vec 2.0 might be sensitive to a domain shift. For example, pre-
training wav2vec 2.0 with cross-lingual datasets improves performance of ASR
systems [19] and finetuning with non-English languages shows a performance
drop for speech quality assessment [14].

In this paper, we study whether the domain shift between the pre-trained model
and the downstream tasks observed can cause this performance drop in music
tasks as reported in the studies above. We explore further the capacity of
wav2vec 2.0 features in non-speech tasks asking the following questions:

  1. 1.

Does wav2vec 2.0 pre-trained on music encode meaningful music representations,
i.e. related to musical concepts such as pitch or instruments?

  2. 2.

Is it possible to obtain competitive performance on MIR tasks when finetuning
wav2vec 2.0 pre-trained on music?

  3. 3.

Can we establish if wav2vec 2.0 is a potential candidate model for music tasks
other than speech?

The paper is structured as follows. In Section 2 we illustrate how we pre-
train wav2vec 2.0 with music data. Section 3 is dedicated to the analysis of
the features learned by wav2vec 2.0. We show whether the information encoded
in the codebooks is related to music labels and we compare the encoded
representations in the continuous layers of wav2vec 2.0 pre-trained on music
with the information encoded in the original speech model. Section 4 shows the
results of finetuning wav2vec 2.0 pre-trained on music on two MIR tasks:
instrument classification and pitch classification111In this paper, we use the
term pitch classification since the NSynth dataset is made of isolated note
segments. This is different from the more common term ”pitch detection” where
note segments are not isolated.. In Section 5 we discuss whether wav2vec 2.0
pre-trained on music provides promising potential for broader downstream MIR
tasks.

##  2 Method

###  2.1 Pre-Trained Model

The wav2vec 2.0 model can be summarized in the following blocks:

  1. 1.

A feature encoder $f:\mathbb{X}\mapsto\mathbb{Z}$ that converts input audio
chunks of 20 milliseconds $X$ into a sequence of latent speech representations
$\mathbb{Z}=\\{z_{1},z_{2},...,z_{T}\\}$ for $T$ timesteps. The encoder
consists of 7 1D convolutional layers, each with 512 filters.

  2. 2.

A context network $g:\mathbb{Z}\mapsto\mathbb{C}$ based on the Transformer
architecture [20] that builds context representations for each audio segment
that capture the entire audio sequence
$\mathbb{C}=\\{c_{1},c_{2},...,c_{T}\\}$

  3. 3.

A quantization module that transforms encoder output representations into
discrete speech representations $\mathbb{C}\mapsto\mathbb{Q}$. The discrete
latent features are learned with product quantization and are needed to create
targets for the loss function, but they are not used as input for the context
network. A vector that concatenates an entry from each of the 2 codebooks is
linearly transformed to get the quantized representations
$\mathbb{Q}=\\{q_{1},q_{2},...,q_{T}\\}$. The Gumbel-Softmax is used to choose
the codebook entries in a differentiable way.

  4. 4.

A contrastive loss function is used to learn how to identify the true
quantized speech representation from 100 quantized negative samples that are
uniformly sampled. Given an audio chunk at time step $t$, the model compares
the cosine similarity between the Transformer output at time step $t$ and the
quantized speech representation in the same step $t$ against the similarity
with negative distractors. The high similarity with the negative samples is
penalized by contrastive loss. The latent speech representation at the time
step $t$ created by the feature encoder is masked before being fed to the
Transformer-based context network. Negative samples are sampled from other
masked time steps of the same utterance.

We use the BASE model configuration [12], which consists of 12 Transformer
blocks and produces $768$-dimensional feature vectors.

To learn music representations, we pre-trained wav2vec 2.0 on the MusicNet
dataset [21]. MusicNet consists of $\approx 34$ hours of audio across 330
classical music recordings provided as raw waveforms, covering 11 musical
instruments. To pre-train wav2vec 2.0 we use the fairseq toolkit [22]. The
data is split into overlapped segments of 20 seconds, whose length is
recommended in the fairseq repository instructions. To increase the dataset
size we take overlapped segments with hop size equal to 10 seconds collecting
$\approx 65$ hours of audio in total. The dataset that we used to pre-train
represents only $\approx 7\%$ of the LibriSpeech dataset size which is the one
used to pre-train wav2vec 2.0 for speech [12]. However, we will show that this
is sufficient to address whether wav2vec 2.0 learns meaningful music
representations. The model was trained for 1790 epochs and it took 7 days on
the NVIDIA A100 64GB GPU.

###  2.2 Finetuning

Evaluation of downstream tasks is performed on the NSynth dataset [23] using
the original train, validation, and test splits. The NSynth dataset includes
305,979 samples of 4 seconds. Two tasks are evaluated on this dataset, pitch
classification 1 and instrument classification. Pitch labels on the isolated
note recordings are provided as MIDI numbers. Instrument labels represent the
instrument family and include the following 11 instruments: bass, brass,
flute, guitar, keyboard, mallet, organ, reed, string, synth_lead, vocal.
Notice that synth_lead is only present in the training set, which makes
validation and test splits made of 10 instrument classes.

The output of the last Transformer block is a matrix of size $(n\times l)$
where $n$ is the number of time frames and $l$ is the size of the feature
vector equal to $768$. To remove the time dimension, we simply average across
time, obtaining an $l$-dimensional vector at the output. The latter is
connected to a linear layer that consists of the number of output neurons
equal to the number of classes of the task: 112 neurons for pitch
classification and 11 neurons for instrument classification.

The pre-trained wav2vec 2.0 model with music data is used in 3 different
configurations for the downstream tasks: 1) finetuning (FT1) the entire
network, 2) finetuning (FT2) the context network (Transformer) while keeping
the feature encoder frozen, 3) Freezing both feature and context networks and
doing a simple feature extraction (FE) which consists of training only the
output linear layer. Finetuning on models FT1 and FT2 is performed using the
Adam optimizer with a learning rate of 0.00001 for the pre-trained part and
0.0001 for the output layer. The FE model is trained using the Adam optimizer
with a learning rate of 0.001 and only the weights of the output linear layer
are optimized. In all 3 configurations, training is stopped if the average
loss in the validation set did not decrease for 10 epochs. The cross-entropy
loss is used for classification.

##  3 Feature Analysis

Our first research question in Section 1 asked whether wav2vec 2.0 learns
meaningful representations when pre-trained on music data. We first explored
whether the learned discrete latent representations used in the loss function
encode a semantic meaning related to musical concepts. The discrete
representations are an important step in wav2vec 2.0 since learning a finite
set of discrete audio units encourages the model not to learn all the
variations in the data when minimizing the contrastive loss. We use the NSynth
dataset to compute the co-occurrence between both pitch and instrument family
labels and the discrete latent features produced by wav2vec 2.0 pre-trained on
MusicNet without finetuning.

Fig. 1: Co-occurrence between the discrete latent representations and
instrument family labels on the NSynth test set.

Figure 1 and Figure 2 show that discrete latent representations specialize in
both instrument and pitch classes, respectively. Many latents co-occur with
bass, which is the most frequent class in the NSynth test. The discrete latent
representations share a similar pattern as the wav2vec 2.0 speech model, where
the encoded semantic meaning of the codebooks has been shown to be represented
by phonemes [12].

A deeper insight in the analysis of the wav2vec 2.0 features can be obtained
by analyzing the Transformer layers. Given a masked latent representation
$z_{t}$, the objective of the model is to learn a context representation
$c_{t}$ in order to correctly guess the quantized representation $q_{t}$ among
the negative samples. For this reason, it should be expected that the final
layers of the Transformer should have higher similarity with the Transformer
input. This behaviour should be observed regardless of the input signal type
(speech or music). To confirm whether the Transformer layers evolve in the
pre-trained model as expected, we follow the same approach of Pasad et al.
[24] where they observed this phenomenon occurring in the wav2vec 2.0 pre-
trained on speech. We computed the canonical correlation analysis (CCA)
between each Transformer layer and the output of the feature encoder. Given a
matrix $W\in\mathbb{R}^{n\times k}$ and $Y\in\mathbb{R}^{n\times j}$ with
$k<j$, CCA finds two basis such that when the matrices are projected onto the
basis their correlation is the highest. More specifically, the CCA is
calculated as follows:

$\rho_{i}=\max_{u^{i}_{w},u^{i}_{y}}corr(Wu^{i}_{w},Yu^{i}_{y}),$ (1)

$CCA(W,Y)=\frac{\sum\limits_{i=1}^{k}\rho_{i}}{k}$ (2)

where $\rho_{i}$ represents the $i$-th canonical correlation coefficient,
$u^{i}_{w}$ and $u^{i}_{y}$ are the vectors found by CCA that maximize the
canonical weights, and the final CCA is obtained with the average. In our
analysis, the matrices are represented by the feature vectors at each
timestep. We use a variant projection weighted canonical correlation analysis
(PWCCA) [25] that is less sensitive to perturbation since it uses a weighted
mean to assign a higher weight to the correlation coefficients that have more
importance.

Fig. 2: Co-occurrence between the discrete latent representations and pitch
classes on the NSynth test set. Fig. 3: Evolution of the Transformer layers of
the pre-trained model and the finetuned models using PWCCA between each layer
and the output of the feature encoder (CNN).

The PWCCA is calculated using the FT1 approach where the feature encoder is
frozen, and by using frames extracted from the MusicNet dataset. Due to the
high computational effort, we take 4 seconds in the middle of each MusicNet
observation using half of the dataset size. Figure 3 shows that the pre-
trained model attempts to reconstruct the input features (i.e. the output of
the feature encoder) while the similarity between the final layers of the
finetuned models tend to be lower than the pre-trained model. This confirms
that the evolution of the Transformer layers with respect to the feature
encoder output is the same in both the speech model (Pasad et al. [24]) and
the music model (this study), which is aligned with the objective of wav2vec
2.0.

##  4 Downstream Tasks

The second research question in Section 1 asked whether finetuning wav2vec 2.0
pre-trained on music data shows competitive performance in downstream MIR
tasks. The performance of wav2vec 2.0 pre-trained on music data is evaluated
on pitch and instrument classification using the NSynth test set. The model is
compared with previous works as shown in Table LABEL:tab:w2v_res.

For pitch classification we use CREPE [26] which is the best pitch classifier
in the HEAR challenge [15], SF NFNet-50 which is the best model in a
comparison of audio representations reported by Wang et al. [16], and features
extracted from wav2vec 2.0 pre-trained on LibriSpeech which is fundamental to
understanding the differences with wav2vec 2.0 pre-trained on music. All the
models, except for CREPE, are designed to learn general purpose audio
representations.

For instrument classification, we consider MuLaP [6] which learns music
representations by using weak supervision from noisy language descriptors of
the musical content, the work of Favory et al. [27] that we call contextual
tag embeddings (CTE) where the learned audio representations are aligned to
music tags, SF NFNet-50 which is also the best model for instrument
classification in the same study from Wang et al. [16], and the feature
extracted from wav2vec 2.0 pre-trained on LibriSpeech.

It should be noted that the models reported in Table LABEL:tab:w2v_res have
some differences that do not allow for direct comparisons such as pre-training
datasets, supervision strategies, hyper-parameters, and strategies to use the
learned features. However, the choice of the above prior work models helps us
to contextualize the results obtained with wav2vec 2.0 pre-trained on music.

The results in Table LABEL:tab:w2v_res show that pre-training wav2vec 2.0 on
music shows comparable results with prior work and improvement over wav2vec
2.0 pre-trained on speech. For pitch classification, finetuning the entire
wav2vec 2.0 pre-trained on music achieves the best results together with CREPE
and it is the best model among the ones trained to learn general-purpose audio
representations. Also, extracting features from the music model shows an 11%
increase over the speech embeddings, indicating that pre-training wav2vec 2.0
on music is the contributing factor to the observed performance improvement.
For instrument classification, wav2vec 2.0 pre-trained on music is the second
best model after SF NFNet-50 and it shows better results than MuLaP and CTE.
We also observe that feature extraction of wav2vec 2.0 from the music model
has significant improvement over the speech model, which confirms the positive
contribution of the music data used in the pre-training phase. An important
aspect to consider is that we pre-trained wav2vec 2.0 on a relatively smaller
dataset which still shows competitive results with the other approaches that
are pre-trained on larger datasets.

Table 1: Performance evaluation using accuracy (%) on NSynth test set. The
results of the baseline model are taken from [15]+,[16]++[6]+++

Pitch Instrument Pre-training Data wav2vec 2.0 Music FE 76.0 64.0 Music,
$\approx 65$ hours wav2vec 2.0 Music FT1 82.0 70.0 Music, $\approx 65$ hours
wav2vec 2.0 Music FT2 90.0 75.0 Music, $\approx 65$ hours CREPE [26]+ 90.0 SF
NFNet-50 [16]++ 88.0 78.2 Audio, $\approx 5800$ hours wav2vec 2.0 Speech FE
[12]+ 65.0 Speech,$\approx 960$ hours wav2vec 2.0 Speech FE [12]++ 35.0 40.2
Speech,$\approx 960$ hours MuLaP [6]+++ 71.7 CTE [27]+++ 70.0 Music, $\approx
562$ hours

##  5 Discussion & Conclusions

In this paper, our aim was to study the potential of wav2vec 2.0 on learning
meaningful representations from music data. We pre-trained wav2vec 2.0 on
music data and evaluated the model on pitch and instrument classification. We
demonstrated that wav2vec 2.0 encodes semantic meaning related to musical
concepts in the discrete latent representations and that the Transformer layer
behaviour is the same of the speech model. We showed that finetuning wav2vec
2.0 pre-trained on music has significant improvement over the original model
pre-trained on speech and other audio-representations models. We posed the
question: is wav2vec 2.0 pre-trained on music a potential model for learning
general-purpose music representations? Our results and analysis support
further application of wav2vec 2.0 with music pre-training for broader
downstream MIR tasks.

Specifically, we propose to extend these findings by performing a direct
comparison with the other models and addressing the following: (i) we pre-
trained the model using a small dataset which was sufficient for the questions
addressed in this paper but not for general-purpose audio representations that
require pre-training with much larger datasets; (ii) the evaluation of the MIR
task was conducted using a monophonic dataset (NSynth) so the generalization
for downstream polyphonic tasks should be explored; (iii) a broader mix of
genres in the pre-trained dataset should be explored as MusicNet includes
Western classical music and also non-Western music; (iv) the model
hyperparameters were not adjusted or optimized and may be better suited to
speech than to music; (iv) the model has been trained for a fixed number of
epochs due to GPU capacity constraints but more training epochs can be used
e.g. by monitoring the contrastive loss with a validation set.

## References

  * [1] Eric J Humphrey, Juan P Bello, and Yann LeCun,  “Feature learning and deep architectures: New directions for music informatics,”  Journal of Intelligent Information Systems, vol. 41, no. 3, pp. 461–481, 2013.
  * [2] Keunwoo Choi, György Fazekas, Mark Sandler, and Kyunghyun Cho,  “Transfer learning for music classification and regression tasks,”  in Proc. of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017. International Society for Music Information Retrieval, 2017, pp. 141–149.
  * [3] Aäron Van Den Oord, Sander Dieleman, and Benjamin Schrauwen,  “Transfer learning by supervised pre-training for audio-based music classification,”  in Proc. of the 15th International Society for Music Information Retrieval Conference, ISMIR 2014, 2014.
  * [4] Jordi Pons and Xavier Serra,  “musicnn: pre-trained convolutional neural networks for music audio tagging,”  in Late-breaking/demo session in 20th International Society for Music Information Retrieval Conference (LBD-ISMIR2019), 2019.
  * [5] Rodrigo Castellon, Chris Donahue, and Percy Liang,  “Codified audio language modeling learns useful representations for music information retrieval,”  Proc. of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, 2021.
  * [6] Ilaria Manco, Emmanouil Benetos, Elio Quinton, and György Fazekas,  “Learning music audio representations via weak language supervision,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 456–460.
  * [7] Ho-Hsiang Wu, Chieh-Chi Kao, Qingming Tang, Ming Sun, Brian McFee, Juan Pablo Bello, and Chao Wang,  “Multi-task self-supervised pre-training for music classification,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 556–560.
  * [8] Hongyuan Zhu, Ye Niu, Di Fu, and Hao Wang,  “Musicbert: A self-supervised learning of music representation,”  in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 3955–3963.
  * [9] Andrew N Carr, Quentin Berthet, Mathieu Blondel, Olivier Teboul, and Neil Zeghidour,  “Self-supervised learning of audio representations from permutations with differentiable ranking,”  IEEE Signal Processing Letters, vol. 28, pp. 708–712, 2021.
  * [10] Janne Spijkervet and John Ashley Burgoyne,  “Contrastive learning of musical representations,”  Proc. of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, 2021.
  * [11] Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, et al.,  “Self-supervised speech representation learning: A review,”  arXiv preprint arXiv:2205.10643, 2022.
  * [12] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in Neural Information Processing Systems, vol. 33, pp. 12449–12460, 2020.
  * [13] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an asr corpus based on public domain audio books,”  in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210.
  * [14] Helard Becerra, Alessandro Ragano, and Andrew Hines,  “Exploring the influence of fine-tuning data on wav2vec 2.0 model for blind speech quality prediction,”  Proc. Interspeech, pp. 4088–4092, 2022.
  * [15] Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W Schuller, Christian J Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, et al.,  “Hear: Holistic evaluation of audio representations,”  in NeurIPS 2021 Competitions and Demonstrations Track. PMLR, 2022, pp. 125–145.
  * [16] Luyu Wang, Pauline Luc, Yan Wu, Adria Recasens, Lucas Smaira, Andrew Brock, Andrew Jaegle, Jean-Baptiste Alayrac, Sander Dieleman, Joao Carreira, et al.,  “Towards learning universal audio representations,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 4593–4597.
  * [17] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter,  “Audio set: An ontology and human-labeled dataset for audio events,”  in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 776–780.
  * [18] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson,  “How transferable are features in deep neural networks?,”  Advances in neural information processing systems, vol. 27, 2014\.
  * [19] Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli,  “Unsupervised cross-lingual representation learning for speech recognition,”  Ninth International Conference on Learning Representations (ICLR), 2021.
  * [20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [21] J. Thickstun, Z. Harchaoui, and S.M. Kakade,  “Learning features of music from scratch,”  in 5th International Conference on Learning Representations, ICLR 2017, 2017.
  * [22] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli,  “fairseq: A fast, extensible toolkit for sequence modeling,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), 2019, pp. 48–53.
  * [23] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan,  “Neural audio synthesis of musical notes with wavenet autoencoders,”  in International Conference on Machine Learning. PMLR, 2017, pp. 1068–1077.
  * [24] Ankita Pasad, Ju-Chieh Chou, and Karen Livescu,  “Layer-wise analysis of a self-supervised speech representation model,”  in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 914–921.
  * [25] Ari Morcos, Maithra Raghu, and Samy Bengio,  “Insights on representational similarity in neural networks with canonical correlation,”  Advances in Neural Information Processing Systems, vol. 31, 2018\.
  * [26] Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello,  “Crepe: A convolutional representation for pitch estimation,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 161–165.
  * [27] Xavier Favory, Konstantinos Drossos, Tuomas Virtanen, and Xavier Serra,  “Learning contextual tag embeddings for cross-modal alignment of audio and tags,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 596–600.


11institutetext: Graduate School of Informatics, Nagoya University, Nagoya,
Japan. 22institutetext: Center for Advanced Intelligence Project, RIKEN,
Suita, Japan. 33institutetext: PRESTO, Japan Science and Technology Agency,
Saitama, Japan.

# Action valuation of on- and off-ball soccer players
based on multi-agent deep reinforcement learning

Hiroshi Nakahara 11 Kazushi Tsutsui 11 Kazuya Takeda 11 Keisuke Fujii 112233

###### Abstract

Analysis of invasive sports such as soccer is challenging because the game
situation changes continuously in time and space, and multiple agents
individually recognize the game situation and make decisions. Previous studies
using deep reinforcement learning have often considered teams as a single
agent and valued the teams and players who hold the ball in each discrete
event. Then it was challenging to value the actions of multiple players,
including players far from the ball, in a spatiotemporally continuous state
space. In this paper, we propose a method of valuing possible actions for on-
and off-ball soccer players in a single holistic framework based on multi-
agent deep reinforcement learning. We consider a discrete action space in a
continuous state space that mimics that of Google research football and
leverages supervised learning for actions in reinforcement learning. In the
experiment, we analyzed the relationships with conventional indicators, season
goals, and game ratings by experts, and showed the effectiveness of the
proposed method. Our approach can assess how multiple players move
continuously throughout the game, which is difficult to be discretized or
labeled but vital for teamwork, scouting, and fan engagement.

###### Keywords:

multi-agent reinforcement learning sports football

##  1 Introduction

With advanced data analysis in sports, tactical planning, player evaluation,
and coaching methods based on data have recently become available. The data
analytics of dynamic movements in team sports such as soccer is considered
challenging because game situations are continuous in time and space, and
multiple agents individually recognize the game situations and make decisions.
In particular, each action’s value has been quantified based on the strength
of its association with scores for on-ball players (i.e., with a ball). In
previous work using supervised learning, machine learning models were used to
compute an action’s value by predicting whether scores or other events occur
or not in the following actions [3, 40, 41, 43]. In these frameworks, it would
be difficult to consider possible (i.e., counterfactual) actions as time goes
back from a goal or other events. To value on-ball actions in terms of
obtaining rewards (e.g., goals), there have been studies using reinforcement
learning (RL) [25, 24, 33, 31]. These works typically consider teams as a
single agent and valuate an on-ball player or a team in irregularly occurring
events (e.g., passes and shots). Considering the nature of soccer, it should
be modeled as a continuous space, and valuate actions without event labels
including on- and off-ball plays.

Although most studies focus on the evaluation of on-ball players, players can
indirectly contribute to scoring even when they are off-ball (i.e., without a
ball) which is a large part of their playing time (e.g., approximately 87 min
of 90 min [8]). A previous work [37] estimated the value of the state from the
positional information of the ball and players based on a rule-based model
called Off-Ball Scoring Opportunities (OBSO). However, valuing other attacking
players who do not receive the ball and reflecting the valuation of several
possible actions in the state value is challenging. Another study [39]
quantified every off-ball player’s impact on scores in terms of the difference
between predicted and real player movements. The method quantitatively values
only a single player’s contribution once through their predicted movement
trajectories. Thus it is challenging to calculate the contributions of
multiple players at each time stamp using a comprehensive learning-based
framework.

In this paper, we propose a valuation method of on- and off-ball soccer
players in a single holistic framework based on multi-agent deep RL.
Specifically, we consider a discrete action space in a continuous state space
that mimics that of the Google research football (GFootball) [19] and leverage
supervised learning for actions in RL. Based on a deep RL model with a
discrete action space, we estimate the possible action value of multiple
players in real games, including those far from the ball. The proposed network
estimates state-action values (i.e., Q-values) based on the game states (e.g.,
player and ball locations) and actions (e.g., shot and pass). For the off-ball
action, we define the directions of movement as actions. For player
valuations, our method can compute the overall contribution of each player
during the attack by aggregating the Q-values in the RL model. The proposed
method enables us to comprehensively value on- and off-ball actions, which
makes it possible to compare the contributions of the two types of actions,
and thus provides important information for understanding the characteristics
of players. Assessing the movements of all players for teammates is important
for building teamwork, assessment of players’ salaries, recruitment, and
scouting.

In summary, our main contributions were as follows. (1) We propose a valuation
method of on- and off-ball soccer players in a single holistic framework based
on multi-agent deep RL. (2) We consider a discrete action space in a
continuous state space that mimics GFootball and leverages supervised learning
for actions in RL. Furthermore, since the proposed method can counterfactually
compute possible action values that were not chosen in the actual game, it can
be also used for valuating counterfactual actions. (3) In the experiment, we
analyzed the relationships with conventional indicators, season goals, and
game ratings by experts, and showed the effectiveness of the proposed method.
Our approach can assess how multiple players move continuously throughout the
game, which is difficult to be discretized or labeled but vital for teamwork,
scouting, and fan engagement. The remainder of this paper is structured as
follows. First, we describe our methods in Section 2 and present experimental
results in Section 3. Next, we overview the related works in Section 4 and
conclude this paper in Section 5.

##  2 Method

In this section, we describe our problem setting and propose our RL framework.
Then we describe the dataset and preprocessing, and valuation framework.

###  2.1 Problem setting

In this study, we aim to value players by computing state-action values (i.e.,
Q-values) using an RL framework. For simplicity, here we consider independent
multi-agent RL and then omit the agent index.

The RL model in this study consists of three components: state $s$, action
$a$, and reward $r$. At the time $t$, when an action $a_{t}$ is chosen in a
state $s_{t}$, a reward $r_{t+1}$ is given. In on-policy RL, the agent learns
a policy $\pi$ that can maximize $\sum_{t=1}^{T}\gamma^{t}r_{t}$, where
$\gamma\in[0,1]$ is the discount factor and $T$ is the time horizon
(hereafter, we consider the case of $\gamma=1$ [24] with no discount for
simplicity).

For the state $s_{t}$, we used 2-dimensional position coordinates and
velocities data of the players ($22$) and the ball ($23\times 2\times 2$
dimensions). Inspired by 19 actions defined in GFootball [19], we selected 14
actions of the attacking players: movement actions defined as $8$ different
movement directions ($8$ directions in $45$ degree increments), idle, starting
and stopping a sprint, the release of the movement direction, passing, and
shooting. The passing and shooting actions were defined by the labels (event
data) in the dataset described later. Movement directions were computed based
on the player’s velocity direction. Other, idle, starting and stopping a
sprint, and release of the movement direction were computed based on the
stopping (0.1 m/s) and sprint thresholds (24 km/h). Regarding on-ball players’
actions, the dataset included other behavior labels (e.g., dribbling and
trapping), which were not used in this study for simplicity.

For the reward $r$, we added the following three values at the last time $T$
of the sequence of attacks. (1) Goal: $1$ if a series of attacks end in a
score, and $0$ otherwise, (2) Expected score with no goal: the EPV (expected
possession value) [9], which is an indicator of the probability of scoring
based on the ball position coordinates ($x,y$) at time $T$, (3) Conceding:
$-1$ if a goal is scored by an opponent’s attack immediately after a sequence
of attacks, and $0$ otherwise. For the rewards at $t$ other than time $T$
($t<T$), we used $0$ in this study.

Figure 1: Our deep RL model with action supervision.

###  2.2 Deep reinforcement learning with action supervision

Next, we explain our RL algorithm with action supervision. We compute state-
action values (Q-value) for the valuation of plays, which has been used in
previous sports studies [25, 24, 7, 31]. In this framework, the optimal state-
action value function $Q^{*}(s_{t},a_{t})$ is determined by solving the
Bellman equation as follows:

$\displaystyle Q^{*}(s_{t},a_{t})$ $\displaystyle=$ $\displaystyle
E_{s_{t+1},a_{t+1}}[r_{t+1}+Q(s_{t+1},a_{t+1})|s_{t},a_{t}]$ (1)
$\displaystyle=$
$\displaystyle\sum_{r_{t+1}}P(r_{t+1}|s_{t},a_{t})r_{t+1}+\sum_{\begin{subarray}{c}s_{t+1}\\\
a_{t+1}\end{subarray}}P(s_{t+1},a_{t+1}|s_{t},a_{t})Q(s_{t+1},a_{t+1}),$ (2)

where $Q(s_{t},a_{t})$ refers to the Q-value of taking action $a$ in state
$s$. In other words, the Q-value of taking action $a_{t}$ in state $s_{t}$ is
defined as the sum of the expected reward when the sequence of attacks ends at
time $t$ and the expected Q-value at time $t+1$.

To train the RL model, following by [25, 24], we adopt a temporal difference
(TD) prediction method called SARSA (state-action-reward-state-action), which
is a model-free and on-policy RL algorithm, with recurrent neural network
(RNN). As the input sequences, we used a matrix of $T$ frames of state (92
dim.) and the one-hot action for each agent (14 dim.) as input. A conceptual
diagram of the architecture is shown in Fig. 1. We used GRUs (gated recurrent
units) as RNNs. The middle layer was a single layer with $64$-dimensional
neurons and the activation function was ReLU. In the output layer, we output
the Q-values for each player for each action (14 dim.). To estimate Q values
based on Eq. (2), we compute the following TD loss:

$\mathcal{L}_{TD}=\sum_{t\in
T}(r_{t+1}+Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t}))^{2}.$ (3)

In addition, we introduce a supervised loss for actions because the above
approach might lead to insufficient learning of all possible actions and it
would be necessary to restrict the state spaces from the limited data. Based
on the discussion in [16, 13], we propose a simple action supervision loss
represented by the cross-entropy of softmax values of the Q-function such that

$\mathcal{L}_{AS}=-\sum_{t\in
T}\mathbf{a}_{t}\cdot\log\left(\mathrm{softmax}(\mathbf{Q}_{s_{t}})\right),$
(4)

where $\mathbf{a}_{t}\in\\{0,1\\}^{|A|}$ (i.e., one-hot vector of actions),
$|A|$ is the size of action space,
$\mathbf{Q}_{s_{t}}=[Q(s_{t},a_{t}=1),...,Q(s_{t},a=|A|)]$, and the $\log$
applies element-wise. This loss aims to maximize the Q-function values for the
action of the data. Note that we assume that the action of the actual players
is better than random selection. This is an inductive bias for efficient
learning to estimate Q-values for all possible actions but the main loss is
$\mathcal{L}_{TD}$ and the weight of $\mathcal{L}_{AS}$ should be much smaller
than $\mathcal{L}_{TD}$.

We also add an $L_{1}$ regularization loss applied to the weights and biases
of the network to help prevent over-fitting on the relatively small
demonstration dataset. The total loss in our training is a combination of
three losses:

$\mathcal{L}_{total}=\mathcal{L}_{TD}+{\lambda_{1}}\mathcal{L}_{AS}+{\lambda_{2}}\mathcal{L}_{L_{1}}.$
(5)

We set $\lambda_{1}=0.01$ and $\lambda_{2}=0.1$, which control the weighting
among the losses. We trained the models using the Adam optimizer [18] with
default parameters.

###  2.3 Dataset and preprocessing

In this study, we used 54 games data in the Meiji J1 League (a professional
soccer league in Japan) 2019 season including all 34 games data of Yokohama F
Marinos to perform specific player-level evaluations in limited data. Note
that currently the tracking data for all players and timesteps were not
publicly shared in such amounts. The dataset includes event data (i.e., labels
of actions, e.g., passing and shooting, recorded at 30 Hz and the simultaneous
xy coordinates of the ball) and tracking data (i.e., xy coordinates of all
players recorded at 25 Hz) provided by Data Stadium Inc. The season goals for
each player in each match were collected from [17]. The ratings by experts in
each match [36] were also used, which was scored in 0.5-point increments with
a maximum of 10 points.

As a preprocessing step, we first split a sequence of a game (usually about
$90$ minutes) into possessions (i.e., sequences of attacks) from the beginning
of the possession (ball recovery) to the end of the possession (ball loss or
goal) as an input of RL models. The minimum number of frames was $50$, and the
maximum number of frames was $T=300$ (equivalent to $30$ seconds). The
tracking data were down-sampled to 10 Hz based on [11, 39]. We analyzed 10
attacking players (without a goalkeeper), i.e., constructed 10 RL agent
models.

###  2.4 Validation of the model and valuation of players

In the experiment, we first validated the RL model and then analyzed the
valuations of professional soccer players. To train the model, we used attack
sequences other than those of Yokohama F. Marinos ($1669$ and $186$ sequences)
as the training and validation data. To value the players, we used attack
sequences of Yokohama F. Marinos ($1236$ series) as the test data. For
simplicity, only attacks within the attacking third were used. We used attack
sequences other than those of Yokohama F. Marinos ($1669$ series) as the
training data, and attack sequences of Yokohama F. Marinos ($1236$ series) as
the test data for valuating the actions.

Figure 2: Example of estimated Q-values. (a) shows an example attack. Blue,
red, and black indicate the attacker, defender, and the ball, respectively
(note that the ball is over a defender). (b) and (c) show the Q-value of
player A for each action using the RL model with and without action
supervision, respectively. Note that $i,p,sh,sp,rd,ss$ correspond to idle,
pass, shot, sprint, decelerate, and sprint end, respectively, and arrows
correspond to the direction of movement.

##  3 Results

In this section, we validated the RL model, and then analyzed the valuations
of professional soccer players.

###  3.1 Validation of our RL model

Here we compared our RL models with and without action supervision (i.e.,
$\lambda_{1}=0$ and other conditions were the same). We quantitatively
compared the TD loss in RL $\mathcal{L}_{TD}$ and action supervision loss
$\mathcal{L}_{AS}$. The $\mathcal{L}_{TD}$ with supervision ($0.0034\pm
0.0001$; the mean and standard deviation of test samples) was smaller than
that without supervision ($0.0063\pm 0.0004$), but $\mathcal{L}_{AS}$ with
supervision ($3.9550\pm 0.0005$) was slightly larger than that without
supervision ($3.9407\pm 0.0005$). Considering the fact that the reward scale
is $[-1,1]$, the learning of Q-values was considered to be within the
acceptable range for both models. It should be noted that we can
quantitatively compare the optimization results (i.e., losses) but can only
qualitatively investigate the effectiveness of the model in terms of modeling
of soccer agents. In particular, the action supervision may require careful
interpretation, because much less supervision (e.g., $\lambda_{1}\ll 0.01$)
would lead to insufficient learning of counterfactual action values, whereas
much more supervision (e.g., $\lambda_{1}\gg 0.01$) may overfit to the actual
actions and would not consider counterfactual actions. Then we carried out a
qualitative analysis as described below.

Next, an example of the Q-value output is shown. Fig. 2a shows the coordinates
of the player and the ball in the frame to be valuated. In this case, player A
in the actual game passed the ball to player B, who was considered to have
more space for a shot. Figs. 2b and c show the Q-value of player A for each
action using the RL model with and without action supervision, respectively.
As shown in Fig. 2b, our model with action supervision indicates that the
Q-value of a pass was higher than those of others, suggesting that passing may
produce a more favorable result rather than other actions (e.g., a shot) in
this case. In contrast, in Fig. 2c, our model without action supervision shows
more distributed Q-values closer to zero and small value of the pass action.
Quantitatively, the average on-ball Q-values with supervision ($0.0137\pm
2.597\times 10^{-4}$) were larger than those without supervision ($0.0008\pm
4.431\times 10^{-5}$), and the average off-ball Q-values with supervision
($0.0008\pm 0.0035$) were also larger than those without supervision
($-7.123\pm 0.0018$). In particular, our approach with supervision emphasized
the on-ball valuation, which was similar to the conventional valuation of the
players, but in some cases, a fairer valuation including off-ball situations
may be required. Although which models were best in a practical sense cannot
be determined from the data, for validation of the model output, we mainly
show the results of the model with action supervision based on the above
verification.

Figure 3: The relationship between (a) average Q-values computed by the
proposed method and season goals and (b) average Q-values of on- and off-ball
states.

###  3.2 Comparison with conventional indicators and player valuation

Since there is no true value for the Q-value, the performance of the model
cannot be directly validated. Here we show the usefulness of the proposed
indicators by investigating the relationship with the existing indicators.
Specifically, we explain the relationships between the average Q-values
computed by the proposed method and four indicators: season goals, ratings by
experts, OBSO [37], and creating off-ball scoring opportunity (C-OBSO) [39].
OBSO was used to estimate the value of the state based on the rule-based
model, which basically values attacking players who will receive the ball.
C-OBSO quantifies the off-ball player’s impact on scores created by the
difference between predicted and real player movements. Spearman’s rank
correlation coefficient (hereafter denoted as $\rho$) was used as the
correlation coefficient, and the players to be valuated were limited to those
who played at least $10$ games. Since the sample size was small ($N=14$) in
the correlation analysis, the $\rho$ value was used as an effect size for
evaluation (criteria are based on [15]), rather than the $p$-value.

First, the relationship between the season goals and the average Q-values
obtained by the proposed method is shown in Fig. 3a. There was a high negative
correlation between them ($\rho=-0.761$) whereas many players had 0 or 1 total
goal. In the season, four players (Nakagawa, Marcos, Thiago, and Kida) were
selected as the best 11 players award in the league. Among these four players,
the proposed method showed higher values for a defender (Thiago), midfielder
(Kida), and forward players (Nakagawa and Marcos) in this order. Considering
the high negative correlation, our indicator tended to value defensive players
who provided many passes, because Thiago ranked 6th in terms of the players
with the most passes in the league in spite of being a defender. Similarly,
Hatanaka, who was a defender ranked 1st in our indicator, also ranked 2nd in
terms of most passes in the league. The results suggest that their ability
indirectly contributing to the goal with many passes and other off-ball
movements can be reflected in our indicator, rather than goals.

Next, we compared the contributions of on- and off-ball plays. In Fig. 3,
there was a moderate positive correlation between average on- and off-ball
Q-values ($\rho=0.618$), suggesting that our indicator may tend to value
similar play style between on- and off-ball situations. Because of a similar
tendency, in the following analysis, we also used the Q-values including on-
and off-ball cases.

Figure 4: The relationship between average Q-values by the proposed method and
(a) the average rating by the experts, (b) OBSO [37], and (c) C-OBSO [39].

Next, the relation between the average rating by the experts and the average
Q-value is shown in Fig. 4a. There was a low negative correlation between the
two indicators ($\rho=-0.218$). This result may be related to the tendency
that our indicator tended to value defense players who provided many passes,
and there was a significant correlation between the goals and this subjective
rating in a previous work [39].

Finally, we compared our results with existing off-ball indicators such as
OBSO [37] and C-OBSO [39]. Figure 4b shows a low negative correlation with
average OBSO ($\rho=-0.305$). Since OBSO values attacking players who will
receive the ball, the tendency was similar to the correlation with the season
goal. OBSO can also value the players’ all time stamps, thus the offensive
players can be valued separately based on both OBSO and our Q-values. For
example, regarding Miyoshi (a midfielder) and Theerathon (a defender), who had
three season goals, Miyoshi’s performance can be highly valued in terms of
off-ball movement to receive the ball (i.e., OBSO), and Theerathon’s can be
highly valued in terms of passing and other off-ball movements (i.e., our
method).

Figure 4c shows no correlation with average C-OBSO ($\rho=0.182,p>0.05$).
Compared with C-OBSO [39], which can value forward players such as Edigar and
Nakagawa more than our Q-values, our Q-values tended to value midfielders and
defenders such as Kida (0 season goals) and Hirose (1 season goal) indirectly
contributing to the goals. From these results, we clarified the properties of
our Q-values compared with those of conventional indicators.

##  4 Related work

In team sports tactical behaviors, agents’ behavioral process bears
resemblance to the RL framework (e.g., [10]). Due to various challenges in
modeling the entire framework from data, two approaches can be adopted:
estimating related variables and functions from data as a sub-problem (i.e.,
the inverse approach) and constructing a model (e.g., an RL model) to generate
data in a virtual environment (i.e., the forward approach such as [19, 35]).
In this paper, we have concentrated on the inverse approach but also
considered a forward model.

Numerous approaches have been employed to quantitatively assess the actions of
attacking players in terms of scoring, such as using expected scores derived
from tracking data [26, 4, 33, 34, 1], and action data such as dribbling and
passing [3, 6, 43]. Some researchers have valuated passes [29, 2, 5], while
others have assessed actions to receive a ball by attributing a value to the
location with the highest expected score [37, 23] and using a rule-based
approach [14]. Defensive behaviors have also been valued using data-driven
methods [32, 40]. However, such approaches would have difficulties in in
considering possible (i.e., counterfactual) actions as time goes back from a
goal or other events.

From the RL perspective, numerous studies have focused on inverse approaches.
To value on-ball actions, several studies have estimated Q-function or other
policy functions [25, 24, 33, 31]. However, they often consider teams as a
single agent and did not valuate off-ball players in all time steps (without
events). In terms of inverse RL, research on estimating reward functions has
also been conducted [27, 30]. To estimate policy functions, researchers have
sometimes performed trajectory prediction through imitation learning [21, 20,
38, 11] and behavioral modeling [44, 42, 22, 12], aiming to mimic (rather than
optimize) a policy using neural networks. Our approach considers the RL model
overall rather than as a sub-problem using hand-crafted reward functions, and
estimates multiple players’ Q-functions for simultaneously valuating on- and
off-ball players even when no event occurs.

##  5 Conclusion

In this paper, we proposed a comprehensive evaluation method for soccer
players in attacking scenes using the framework of deep RL with action
supervision. In the experiment, we analyzed the relationship with conventional
indicators, season goals, and game ratings by experts, and showed the
effectiveness of the proposed method. Our approach can assess how multiple
players move continuously throughout the game, which is difficult to be
discretized or labeled but vital for teamwork, scouting, and fan engagement. A
possible future direction is to improve RL models for a more valid valuation
of players. The proposed RL model and Q-value computation can be further
improved by more sophisticated rewards per action (e.g., [31]) and a forward
RL simulation e.g., ([28]).

## Acknowledgments

This work was supported by JSPS KAKENHI (Grant Numbers 20H04075 and 21H05300)
and JST Presto (Grant Number JPMJPR20CA).

## References

  * [1] Bransen, L., Van Haaren, J.: Measuring football players’ on-the-ball contributions from passes during games. In: International workshop on machine learning and data mining for sports analytics. pp. 3–15. Springer (2018)
  * [2] Brooks, J., Kerr, M., Guttag, J.: Developing a data-driven player ranking in soccer using predictive model weights. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 49–55 (2016)
  * [3] Decroos, T., Bransen, L., Van Haaren, J., Davis, J.: Actions speak louder than goals: Valuing player actions in soccer. In: KDD. pp. 1851–1861 (2019)
  * [4] Decroos, T., Dzyuba, V., Van Haaren, J., Davis, J.: Predicting soccer highlights from spatio-temporal match event streams. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 31 (2017)
  * [5] Dick, U., Link, D., Brefeld, U.: Who can receive the pass? a computational model for quantifying availability in soccer. Data Mining and Knowledge Discovery 36(3), 987–1014 (2022)
  * [6] Dick, U., Tavakol, M., Brefeld, U.: Rating player actions in soccer. Frontiers in Sports and Active Living p. 174 (2021)
  * [7] Ding, N., Takeda, K., Fujii, K.: Deep reinforcement learning in a racket sport for player evaluation with technical and tactical contexts. IEEE Access 10, 54764–54772 (2022)
  * [8] Fernández, J., Bornn, L.: Wide open spaces: A statistical technique for measuring space creation in professional soccer. In: 12th MIT Sloan Sports Analytics Conference (2018)
  * [9] Fernández, J., Bornn, L., Cervone, D.: Decomposing the immeasurable sport: A deep learning expected possession value framework for soccer. In: 13th MIT Sloan Sports Analytics Conference (2019)
  * [10] Fujii, K.: Data-driven analysis for understanding team sports behaviors. Journal of Robotics and Mechatronics 33(3), 505–514 (2021)
  * [11] Fujii, K., Takeishi, N., Kawahara, Y., Takeda, K.: Policy learning with partial observation and mechanical constraints for multi-person modeling. arXiv preprint arXiv:2007.03155 (2020)
  * [12] Fujii, K., Takeuchi, K., Kuribayashi, A., Takeishi, N., Kawahara, Y., Takeda, K.: Estimating counterfactual treatment outcomes over time in complex multi-agent scenarios. arXiv preprint arXiv:2206.01900 (2022)
  * [13] Fujii, K., Tsutsui, K., Scott, A., Nakahara, H., Takeishi, N., Kawahara, Y.: Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations. arXiv preprint arXiv:2305.13030 (2023)
  * [14] Fujii, K., Yoshihara, Y., Matsumoto, Y., Tose, K., Takeuchi, H., Isobe, M., Mizuta, H., Maniwa, D., Okamura, T., Murai, T., et al.: Cognition and interpersonal coordination of patients with schizophrenia who have sports habits. PLoS One 15(11), e0241863 (2020)
  * [15] Guilford, J.P.: Fundamental statistics in psychology and education. McGraw-Hill (1950)
  * [16] Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al.: Deep q-learning from demonstrations. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference. pp. 3223–3230 (2018)
  * [17] JLEAGUE: Jleague.jp 2019 data (2019), https://www.jleague.jp/stats/2019/goal.html
  * [18] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (2015)
  * [19] Kurach, K., Raichuk, A., Stańczyk, P., Zając, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michalski, M., Bousquet, O., et al.: Google research football: A novel reinforcement learning environment. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 4501–4510 (2020)
  * [20] Le, H.M., Carr, P., Yue, Y., Lucey, P.: Data-driven ghosting using deep imitation learning. In: Proceedings of MIT Sloan Sports Analytics Conference (2017)
  * [21] Le, H.M., Yue, Y., Carr, P., Lucey, P.: Coordinated multi-agent imitation learning. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. pp. 1995–2003. JMLR. org (2017)
  * [22] Li, L., Yao, J., Wenliang, L., He, T., Xiao, T., Yan, J., Wipf, D., Zhang, Z.: Grin: Generative relation and intention network for multi-agent trajectory prediction. Advances in Neural Information Processing Systems 34, 27107–27118 (2021)
  * [23] Link, D., Lang, S., Seidenschwarz, P.: Real time quantification of dangerousity in football using spatiotemporal tracking data. PloS one 11(12), e0168768 (2016)
  * [24] Liu, G., Luo, Y., Schulte, O., Kharrat, T.: Deep soccer analytics: learning an action-value function for evaluating soccer players. Data Mining and Knowledge Discovery 34(5), 1531–1559 (2020)
  * [25] Liu, G., Schulte, O.: Deep reinforcement learning in ice hockey for context-aware player evaluation. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence. pp. 3442–3448 (2018)
  * [26] Lucey, P., Bialkowski, A., Monfort, M., Carr, P., Matthews, I.: quality vs quantity: Improved shot prediction in soccer using strategic features from spatiotemporal data. In: Proceedings of MIT Sloan Sports Analytics Conference. pp. 1–9 (2014)
  * [27] Luo, Y., Schulte, O., Poupart, P.: Inverse reinforcement learning for team sports: Valuing actions and players. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20. pp. 3356–3363. International Joint Conferences on Artificial Intelligence Organization (7 2020)
  * [28] Mendes-Neves, T., Mendes-Moreira, J., Rossetti, R.J.: A data-driven simulator for assessing decision-making in soccer. In: Progress in Artificial Intelligence: 20th EPIA Conference on Artificial Intelligence, EPIA 2021, Virtual Event, September 7–9, 2021, Proceedings. pp. 687–698. Springer (2021)
  * [29] Power, P., Ruiz, H., Wei, X., Lucey, P.: Not all passes are created equal: Objectively measuring the risk and reward of passes in soccer from tracking data. In: KDD. pp. 1605–1613 (2017)
  * [30] Rahimian, P., Toka, L.: Inferring the strategy of offensive and defensive play in soccer with inverse reinforcement learning. In: Machine Learning and Data Mining for Sports Analytics: 8th International Workshop, MLSA 2021, Virtual Event, September 13, 2021, Revised Selected Papers. pp. 26–38. Springer (2022)
  * [31] Rahimian, P., Van Haaren, J., Abzhanova, T., Toka, L.: Beyond action valuation: A deep reinforcement learning framework for optimizing player decisions in soccer. In: 16th Annual MIT Sloan Sports Analytics Conference. Boston, MA, USA: MIT. p. 25 (2022)
  * [32] Robberechts, P.: Valuing the art of pressing. In: Proceedings of the StatsBomb Innovation In Football Conference. pp. 1–11. StatsBomb (2019)
  * [33] Routley, K., Schulte, O.: A markov game model for valuing player actions in ice hockey. In: Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence. p. 782–791. UAI’15, AUAI Press, Arlington, Virginia, USA (2015)
  * [34] Schulte, O., Khademi, M., Gholami, S., Zhao, Z., Javan, M., Desaulniers, P.: A markov game model for valuing actions, locations, and team performance in ice hockey. Data Mining and Knowledge Discovery 31(6), 1735–1757 (2017)
  * [35] Scott, A., Fujii, K., Onishi, M.: How does AI play football? An analysis of RL and real-world football strategies. In: 14th International Conference on Agents and Artificial Intelligence (ICAART’ 22). vol. 1, pp. 42–52 (2022)
  * [36] Soccer-digest: Soccer digest web j1 rating (2019), https://www.soccerdigestweb.com
  * [37] Spearman, W.: Beyond expected goals. In: Proceedings of the 12th MIT sloan sports analytics conference. pp. 1–17 (2018)
  * [38] Teranishi, M., Fujii, K., Takeda, K.: Trajectory prediction with imitation learning reflecting defensive evaluation in team sports. In: 2020 IEEE 9th Global Conference on Consumer Electronics (GCCE). pp. 124–125. IEEE (2020)
  * [39] Teranishi, M., Tsutsui, K., Takeda, K., Fujii, K.: Evaluation of creating scoring opportunities for teammates in soccer via trajectory prediction. In: International Workshop on Machine Learning and Data Mining for Sports Analytics. Springer (2022)
  * [40] Toda, K., Teranishi, M., Kushiro, K., Fujii, K.: Evaluation of soccer team defense based on prediction models of ball recovery and being attacked. PLoS One 17(1), e0263051 (2022)
  * [41] Umemoto, R., Tsutsui, K., Fujii, K.: Location analysis of players in uefa euro 2020 and 2022 using generalized valuation of defense by estimating probabilities. arXiv preprint arXiv:2212.00021 (2022)
  * [42] Yeh, R.A., Schwing, A.G., Huang, J., Murphy, K.: Diverse generation for multi-agent sports games. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [43] Yeung, C.C.K., Sit, T., Fujii, K.: Transformer-based neural marked spatio temporal point process model for football match events analysis. arXiv preprint arXiv:2302.09276 (2022)
  * [44] Zhan, E., Zheng, S., Yue, Y., Sha, L., Lucey, P.: Generating multi-agent trajectories using programmatic weak supervision. In: International Conference on Learning Representations (2019)