11institutetext:  Laboratoire Lagrange, Université Côte d’Azur, Observatoire
de la Côte d’Azur, CNRS, Blvd de l’Observatoire, CS 34229, 06304 Nice cedex 4,
France 22institutetext: LLR, CNRS, École Polytechnique, Institut Polytechnique
de Paris 33institutetext: Université Paris Cité, CNRS, Astroparticule et
Cosmologie, F-75013 Paris, France 44institutetext: Laboratoire d’Annecy de
Physique des Particules, Université Savoie Mont Blanc, CNRS/IN2P3, F-74941
Annecy, France 55institutetext: Department of Astronomy, University of Geneva,
ch. d’Ecogia 16, CH-1290 Versoix, Switzerland 66institutetext: School of
Physics and Astronomy, Cardiff University, Queenâs Buildings, The Parade,
Cardiff CF24 3AA, UK 77institutetext: Université Paris-Saclay, Université
Paris Cité, CEA, CNRS, AIM, F-91191, Gif-sur-Yvette, France (2022)
88institutetext: European Space Astronomy Centre (ESA/ESAC), Operations
Department, Villanueva de la Canãda, Madrid, Spain 99institutetext:
Laboratoire de Physique Subatomique et de Cosmologie, Université Grenoble
Alpes, CNRS/IN2P3, 53, avenue des Martyrs, Grenoble, France 1010institutetext:
Aix Marseille Université, CNRS, LAM (Laboratoire d’Astrophysique de Marseille)
UMR 7326, 13388, Marseille, France 1111institutetext: Institut Néel, CNRS and
Université Grenoble Alpes, France 1212institutetext: Institut de
RadioAstronomie Millimétrique (IRAM), Grenoble, France 1313institutetext: HH
Wills Physics Laboratory, University of Bristol, Tyndall Avenue, Bristol, BS8
1TL, UK 1414institutetext: Istituto Nazionale di Astrofisica (INAF) -
Osservatorio di Astrofisica e Scienza dello Spazio (OAS), via Gobetti 93/3,
I-40127 Bologna, Italy 1515institutetext: Dipartimento di Fisica, Sapienza
Università di Roma, Piazzale Aldo Moro 5, I-00185 Roma, Italy
1616institutetext: Univ. Grenoble Alpes, CNRS, IPAG, F-38000 Grenoble, France
1717institutetext: INAF - IASF Milan, via A. Corti 12, I-20133 Milano, Italy
1818institutetext: Department of Physics and Astronomy, University of Sussex,
Brighton BN1 9QH, UK 1919institutetext: Centro de Astrobiología (CSIC-INTA),
Torrejón de Ardoz, 28850 Madrid, Spain 2020institutetext: Department of Space,
Earth and Environment, Chalmers University of Technology, Onsala Space
Observatory, SE-439 92 Onsala, Sweden 2121institutetext: High Energy Physics
Division, Argonne National Laboratory, 9700 South Cass Avenue, Lemont, IL
60439, USA 2222institutetext: Institute for Astronomy & Astrophysics, Space
Applications & Remote Sensing, National Observatory of Athens, GR-15236 Palaia
Penteli,Greece 2323institutetext: Institut de RadioAstronomie Millimétrique
(IRAM), Granada, Spain 2424institutetext: LERMA, Observatoire de Paris, PSL
Research University, CNRS, Sorbonne UniversitÃs, UPMC Univ. Paris 06, 75014
Paris, France 2525institutetext: School of Earth and Space Exploration and
Department of Physics, Arizona State University, Tempe, AZ 85287
2626institutetext: Argelander Institut für Astronomie, Universität Bonn, Auf
dem Huegel 71, DE-53121 Bonn, Germany 2727institutetext: European Southern
Observatory, Alonso de Cordova 3107, Vitacura, 19001 Casilla, Santiago 19,
Chile 2828institutetext: INAF-Osservatorio Astronomico di Cagliari, Via della
Scienza 5, 09047 Selargius, Italy 2929institutetext: LPENS, Ecole Normale
Supérieure, 24 rue Lhomond, 75005, Paris (FR) 3030institutetext: Department of
Physics and Astronomy, University of Pennsylvania, 209 South 33rd Street,
Philadelphia, PA 19104, USA 3131institutetext: Institut d’Astrophysique de
Paris, Sorbonne Universités, UPMC Univ. Paris 06, CNRS UMR 7095, 75014 Paris,
France 3232institutetext: Univ. Lyon, Univ. Claude Bernard Lyon 1, CNRS/IN2P3,
IP2I Lyon, 69622 Villeurbanne, France 3333institutetext: INFN, Sezione di
Bologna, viale Berti Pichat 6/2, 40127 Bologna, Italy 3434institutetext:
Institut d’Astrophysique Spatiale (IAS), CNRS and Université Paris Sud, Orsay,
France

# NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relation

R. Adam NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relationNIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation
<EMAIL_ADDRESS>M. Ricci NIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3
low-mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relationNIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relationNIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation D.
Eckert NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation P. Ade
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation H. Ajeddig
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation B. Altieri
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation P. André
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation E. Artis
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation H. Aussel
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation A. Beelen
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation C. Benoist
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation A. Benoît
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation S. Berta
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation L. Bing
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation M.
Birkinshaw DeceasedNIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation O. Bourrion NIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3
low-mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation D. Boutigny NIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3
low-mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation M. Bremer NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation M. Calvo NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation A. Cappi NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relationNIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation A.
Catalano NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation M.
De Petris NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation
F.-X. Désert NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation S.
Doyle NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation E. F. C.
Driessen NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation L.
Faccioli NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation C.
Ferrari NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation F.
Gastaldello NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation P.
Giles NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation A. Gomez
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation J. Goupy
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation O. Hahn
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation C. Hanser
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation C.
Horellou NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation F.
Kéruzoré NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation E.
Koulouridis NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relationNIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation C.
Kramer NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation B.
Ladjelate NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation G.
Lagache NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation S.
Leclercq NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation
J.-F. Lestrade NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation
J.F. Macías-Pérez NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation S. Madden NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation B. Maughan NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation S. Maurogordato NIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3
low-mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation A. Maury NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation P. Mauskopf NIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3
low-mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relationNIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation A.
Monfardini NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation M.
Muñoz-Echeverría NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation F. Pacaud NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation L. Perotto NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation M. Pierre NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation G. Pisano NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation E. Pompei NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation N. Ponthieu NIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3
low-mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation V. Revéret NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation A. Rigby NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relation A. Ritacco NIKA2 observations of 3 low-mass galaxy clusters at $z\sim
1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-
mass galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$
relationNIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$:
pressure profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass
galaxy clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation C.
Romero NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation H. Roussel
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation F. Ruppin
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation M. Sereno
NIKA2 observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure
profile and $Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy
clusters at $z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relationNIKA2
observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure profile and
$Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation K. Schuster NIKA2
observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure profile and
$Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation A. Sievers NIKA2
observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure profile and
$Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation G. Tintoré Vidal NIKA2
observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure profile and
$Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation C. Tucker NIKA2
observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure profile and
$Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation R. Zylka NIKA2
observations of 3 low-mass galaxy clusters at $z\sim 1$: pressure profile and
$Y_{\rm SZ}-M$ relationNIKA2 observations of 3 low-mass galaxy clusters at
$z\sim 1$: pressure profile and $Y_{\rm SZ}-M$ relation

###### Abstract

Three galaxy clusters selected from the XXL X-ray survey at high redshift and
low mass ($z\sim 1$ and $M_{500}\sim 1-2\times 10^{14}$ M⊙) were observed with
NIKA2 to image their Sunyaev-Zel’dovich effect (SZ) signal. They all present
an SZ morphology, together with the comparison with X-ray and optical data,
that indicates dynamical activity related to merging events. Despite their
disturbed intracluster medium, their high redshifts, and their low masses, the
three clusters follow remarkably well the pressure profile and the SZ flux-
mass relation expected from standard evolution. This suggests that the physics
that drives cluster formation is already in place at $z\sim 1$ down to
$M_{500}\sim 10^{14}$ M⊙.

##  1 Introduction

Galaxy clusters are important astrophysical objects that provide insights into
the formation and evolution of cosmic structures. Understanding the properties
of their intracluster medium (ICM) is crucial for both cosmology and
astrophysics. The thermal pressure profile of the ICM provides valuable
information about the matter distribution within galaxy clusters. It reflects
how the gas is compressed within the cluster’s gravitational potential well.
Additionally, the $Y_{\rm SZ}-M$ scaling relation is an important tool for
measuring cluster masses, as it relates to the Sunyaev-Zel’dovich (SZ, see
Sunyaev1972, ) flux ($Y_{\rm SZ}$, which directly measures the thermal energy
of the ICM) to the total cluster mass, $M$ (see Mroczkowski2019, , for a
review).

While the thermal pressure profile and the $Y_{\rm SZ}-M$ scaling relation
have been extensively studied up to intermediate redshifts (e.g., Arnaud2010,
; PlanckV2013, ), there is a lack of detailed observations in the low-mass,
high-redshift regime. In this proceeding, we bridge this observational gap by
presenting detailed SZ observations of three low-mass ($M_{500}\sim 2\times
10^{14}$ M⊙) galaxy clusters at $z\sim 1$ selected from the XXL X-ray survey.
These clusters, namely XLSSC 072, XLSSC 100, and XLSSC 102 were imaged using
the NIKA2 millimeter camera on the IRAM 30-meter telescope at 150 GHz and 260
GHz (Adam2018a, ).

By combining the SZ observations with X-ray and optical data, we investigate
the dynamic state of the clusters and derive their thermal pressure profiles.
We also extract the SZ fluxes and compare them with expectations based on the
standard evolution of the ICM properties calibrated on nearby massive systems.
This analysis allows us to explore any deviations from the expected evolution
in the ICM properties for low-mass clusters at high redshifts, where
astrophysical processes are expected to have a more significant impact
(Fakhouri2010, ; Pop2022, ).

This proceeding is organized as follows: Section 2 presents the cluster
selection and the observations, Section 3 discusses the data analysis used to
address the cluster dynamical state, extract the pressure profile and the
$Y_{\rm SZ}-M$ relation, in Section 4, we summarize our main findings and
conclusions. See adam2023xxl  for the full results. See also (Ricci2020, ) for
the detailed analysis of the first cluster XLSSC 102.

##  2 Target selection and observations

We selected our target clusters from the XXL survey (Pierre2016, ). The XXL
survey’s selection function enabled the identification of clusters at low mass
and high redshift (Pacaud2016, ). We specifically focused on the securely
detected XXL clusters (C1) from the northern region (XXL-N), observable from
the IRAM 30m telescope. Optical detections using galaxy overdensity were used
for independent confirmation, with robust spectroscopic redshift estimates
(Adami2018, ). We selected three clusters, XLSSC 072, XLSSC 100, and XLSSC
102, at a redshift $z\sim 1$ and $M_{500}\sim 1-2\times 10^{14}$ M⊙, for NIKA2
observations.

The observations were conducted between January 2018 and February 2020. XLSSC
072, XLSSC 100, and XLSSC 102 were observed for 10, 10, and 6.6 hours,
respectively. The data reduction followed the procedure described in Adam2015
and Adam2016 . The data analysis included estimating the astrophysical signal
filtering induced by the data reduction and deriving the noise statistical
properties through power spectrum analysis and Monte Carlo simulations.
Overall, the observational details and data reduction process for all three
clusters were consistent with the characteristics of the NIKA2 instrument
(Perotto2020, ).

After smoothing the NIKA2 data to an effective resolution of 27 arcsec, XLSSC
072, XLSSC 100, and XLSSC 102, are detected with a peak signal-to-noise ratio
(S/N) of -9.7, -9.2, and -6.9, respectively. Radio and submillimeter point
sources were removed from the SZ images prior to further analysis.

##  3 Analysis and results

###  3.1 Dynamical state

The SZ data were combined with X-ray (XXL survey Pierre2016 ) and optical
images (CFHTLS Gwyn2012  and HSC Aihara2022 ) and analyzed to trace the ICM
thermal pressure, the ICM thermal density, and the collisionless galaxy
population, respectively. Comparing these tracers provides valuable insights
into the clusters’ dynamical properties.

The multiwavelength analysis shows that all three clusters exhibit deviations
from spherical symmetry, indicating the presence of disturbances in the gas
and galaxy distribution, likely due to merging events. The SZ and X-ray
signals agree well on large scales but may differ on smaller scales,
suggesting local compressions caused by merging. The surface brightness
distributions of both SZ and X-ray signals are relatively flat, indicating
dynamically disturbed systems without prominent X-ray peaks associated with
relaxed clusters.

###  3.2 Mass measurement

The masses were estimated using two main methods.

  1. 1.

Combining the X-ray derived density profile (Eckert2011, ; Eckert2020, ) and
the SZ-derived pressure profile (see below) via the hydrostatic equilibrium
assumption,

$M_{\rm HSE}(r)=-\frac{r^{2}}{\mu_{\rm gas}m_{\rm p}n_{\rm
e}(r)G}\frac{dP_{e}(r)}{dr}.$ (1)

This provides a direct measurement but is more sensitive to systematics
(hydrostatic mass bias, clumping, etc).

  2. 2.

Using the scaling relation between $Y_{\rm X,500}=M_{\rm gas}(R_{500})\ k_{\rm
B}T$ and the mass from Arnaud2010 ,

$E(z)^{-2/3}\left(\frac{Y_{{\rm X},500}}{2\times 10^{14}{\rm M}_{\odot}{\rm
keV}}\right)=10^{0.376}\times\left(\frac{M_{{\rm HSE},500}}{6\times
10^{14}{\rm M}_{\odot}}\right)^{1.78}.$ (2)

The gas mass $M_{\rm gas}$ and temperature $k_{\rm B}T$ were extracted from
X-ray data. This provides a robust mass proxy but relies on local calibration.

We obtain direct hydrostatic masses of $M_{\rm HSE,500}=1.93_{-0.30}^{+0.38}$,
$2.55_{-0.73}^{+1.46}$ and $1.12_{-0.20}^{+0.22}$ M⊙ for XLSSC 072, XLSSC 100
and XLSSC 102, respectively. The masses based on the $Y_{\rm X}$ proxy are
$M_{Y_{\rm X},500}=1.98_{-0.17}^{+0.31}$, $2.13_{-0.33}^{+0.49}$, and
$1.88_{-0.19}^{+0.28}$ M⊙ for the same clusters. They agree well within error
bars, although a $2\sigma$ tension is observed for XLSSC 102 (see discussion
hereafter).

###  3.3 Pressure profile

The pressure profile of the three clusters was measured using three different
methodologies.

  1. 1.

Forward modeling of the NIKA2 data using a gNFW pressure profile (Nagai2007,
), given by

$P_{e}(r)=\frac{P_{0}}{\left(\frac{r}{r_{p}}\right)^{c}\left(1+\left(\frac{r}{r_{p}}\right)^{a}\right)^{\frac{b-c}{a}}}.$
(3)

  2. 2.

Non-parametric fitting of a binned pressure profile. We defined the pressure
at five radii, logarithmically spaced from 50 kpc to 1 Mpc.

  3. 3.

Forward modeling of the NIKA2 data using an NFW (Navarro1996, ) hydrostatic
mass profile combined with the density profile derived from X-ray data.
According to equation 1, the pressure is modeled as

$P_{e}(r)=P_{e}(r_{0})+\int_{r}^{r_{0}}\frac{\mu_{\rm gas}m_{\rm
p}Gn_{e}(r^{\prime})M_{\rm HSE}(r^{\prime})}{{r^{\prime}}^{2}}dr^{\prime},$
(4)

Assuming standard evolution and given our mass estimates, the pressure
profiles align well with the dynamical state analysis. Alternatively,
considering the prior knowledge of the cluster dynamical states, the data
agree favorably with the pressure profile calibrated on low-redshift clusters
(Arnaud2010, ) and scaled to low mass and high redshift using standard
evolution (see Figure 1). All the pressure profiles recovered with the various
methodologies exhibit excellent agreement within the uncertainties at all
radii. Most of the uncertainty is due to the difficulty in having precise and
robust mass measurements at these redshifts and mass scales.

Figure 1: NIKA2 derived pressure profile and comparison with expectations from
the universal pressure profile (UPP) from Arnaud2010 , as well as
morphologically disturbed (MD) and cool-core systems (CC), given $Y_{\rm
X}$-based masses. Figure extracted from (adam2023xxl, ).

###  3.4 Scaling relation

Our targets, XLSSC072, XLSSC100, and XLSSC 102, lie at the low-mass end of the
Planck calibration sample (PlanckXX2014, ) but have higher redshifts (around
$z\sim 1$) compared to the Planck clusters (around $z\sim 0.2$). When using
the $Y_{\rm X}-M$ relation for mass estimation, they follow the scaling
relation remarkably well. When direct hydrostatic equilibrium mass
measurements are used, XLSSC102 deviates from the scaling relation by
approximately $2\sigma$. However, this deviation may be attributed to
systematic uncertainties in mass measurement due to the complex morphology and
dynamical state of this cluster or a significant hydrostatic mass bias, as
discussed in detail in Ricci2020 . The other clusters agree very well with the
relation.

##  4 Conclusions

Our study aimed to investigate the SZ structure and properties of the ICM in
three low-mass galaxy clusters at a redshift $z\sim 1$. By utilizing the NIKA2
camera, we obtained valuable resolved SZ data for these clusters, representing
some of the first detailed observations of such low-mass systems at high
redshifts.

Our analysis revealed evidence of ongoing merging activity in all three
clusters, as indicated by their disturbed morphologies and deviations from
compact spherically symmetric distributions. The pressure profiles of these
clusters, rescaled according to standard evolution in mass and redshift, were
found to be in agreement with those of local dynamically disturbed systems.
Despite their perturbed ICM, low masses, and high redshifts, we did not
observe any significant deviations in the $Y_{\rm SZ}-M$ scaling relation for
our target clusters. While the sample size is not sufficient to draw
statistical conclusions, these findings provide an initial indication that the
scaling relations are stable down to masses of $M_{500}\sim 2\times 10^{14}$
M⊙ and redshifts of $z\sim 1$. This may indicate that the cluster formation
physics is already in place down to these masses and up to these redshifts

It is important to note that the comparison of the pressure profiles and
scaling relation to those of local samples is limited by uncertainties in the
mass estimates. This highlights the challenges in accurately determining and
robustly estimating the mass in this relatively unexplored regime.

## References

  * (1) R.A. Sunyaev, Y.B. Zeldovich, Comments on Astrophysics and Space Physics 4, 173 (1972)
  * (2) T. Mroczkowski, D. Nagai, K. Basu, J. Chluba, J. Sayers, R. Adam, E. Churazov, A. Crites, L. Di Mascolo, D. Eckert et al., Space Sci. Rev. 215, 17 (2019)
  * (3) M. Arnaud, G.W. Pratt, R. Piffaretti, H. Böhringer, J.H. Croston, E. Pointecouteau, A&A 517, A92 (2010)
  * (4) Planck Collaboration, P.A.R. Ade, N. Aghanim, M. Arnaud, M. Ashdown, F. Atrio-Barandela, J. Aumont, C. Baccigalupi, A. Balbi, A.J. Banday et al., A&A 550, A131 (2013)
  * (5) R. Adam, A. Adane, P.A.R. Ade, P. André, A. Andrianasolo, H. Aussel, A. Beelen, A. Benoît, A. Bideaud, N. Billot et al., A&A 609, A115 (2018)
  * (6) O. Fakhouri, C.P. Ma, M. Boylan-Kolchin, MNRAS 406, 2267 (2010)
  * (7) A.R. Pop, L. Hernquist, D. Nagai, R. Kannan, R. Weinberger, V. Springel, M. Vogelsberger, D. Nelson, R. Pakmor, A. Pillepich et al., arXiv e-prints arXiv:2205.11528 (2022)
  * (8) R. Adam, M. Ricci, D. Eckert, P. Ade, H. Ajeddig, B. Altieri, P. AndrÃ, E. Artis, H. Aussel, A. Beelen et al., A&A (accepted), arXiv e-prints arXiv:2310.05819 (2023)
  * (9) M. Ricci, R. Adam, D. Eckert, P. Ade, P. André, A. Andrianasolo, B. Altieri, H. Aussel, A. Beelen, C. Benoist et al., A&A 642, A126 (2020)
  * (10) M. Pierre, F. Pacaud, C. Adami, S. Alis, B. Altieri, N. Baran, C. Benoist, M. Birkinshaw, A. Bongiorno, M.N. Bremer et al., A&A 592, A1 (2016)
  * (11) F. Pacaud, N. Clerc, P.A. Giles, C. Adami, T. Sadibekova, M. Pierre, B.J. Maughan, M. Lieu, J.P. Le Fèvre, S. Alis et al., A&A 592, A2 (2016)
  * (12) C. Adami, P. Giles, E. Koulouridis, F. Pacaud, C.A. Caretta, M. Pierre, D. Eckert, M.E. Ramos-Ceja, F. Gastaldello, S. Fotopoulou et al., A&A 620, A5 (2018)
  * (13) R. Adam, B. Comis, J.F. Macías-Pérez, A. Adane, P. Ade, P. André, A. Beelen, B. Belier, A. Benoît, A. Bideaud et al., A&A 576, A12 (2015)
  * (14) R. Adam, B. Comis, I. Bartalucci, A. Adane, P. Ade, P. André, M. Arnaud, A. Beelen, B. Belier, A. Benoît et al., A&A 586, A122 (2016)
  * (15) L. Perotto, N. Ponthieu, J.F. Macías-Pérez, R. Adam, P. Ade, P. André, A. Andrianasolo, H. Aussel, A. Beelen, A. Benoît et al., A&A 637, A71 (2020)
  * (16) S.D.J. Gwyn, AJ 143, 38 (2012)
  * (17) H. Aihara, Y. AlSayyad, M. Ando, R. Armstrong, J. Bosch, E. Egami, H. Furusawa, J. Furusawa, S. Harasawa, Y. Harikane et al., PASJ 74, 247 (2022)
  * (18) D. Eckert, S. Molendi, S. Paltani, A&A 526, A79 (2011)
  * (19) D. Eckert, A. Finoguenov, V. Ghirardini, S. Grandis, F. Kaefer, J. Sanders, M. Ramos-Ceja, The Open Journal of Astrophysics 3, 12 (2020)
  * (20) D. Nagai, A.V. Kravtsov, A. Vikhlinin, ApJ 668, 1 (2007)
  * (21) J.F. Navarro, C.S. Frenk, S.D.M. White, ApJ 462, 563 (1996)
  * (22) Planck Collaboration, P.A.R. Ade, N. Aghanim, C. Armitage-Caplan, M. Arnaud, M. Ashdown, F. Atrio-Barandela, J. Aumont, C. Baccigalupi, A.J. Banday et al., A&A 571, A20 (2014)


Recent studies show that transformer-based architectures emulate gradient descent during a forward pass, contributing to in-context learning capabilities— an ability where the model adapts to new tasks based on a sequence of prompt examples without being explicitly trained or fine tuned to do so. This work investigates the generalization properties of a single step of gradient descent in the context of linear regression with well-specified models. A random design setting is considered and analytical expressions are derived for the statistical properties and bounds of generalization error in a non-asymptotic (finite sample) setting. These expressions are notable for avoiding arbitrary constants, and thus offer robust quantitative information and scaling relationships. These results are contrasted with those from classical least squares regression (for which analogous finite sample bounds are also derived), shedding light on systematic and noise components, as well as optimal step sizes. Additionally,  identities involving high-order products of Gaussian random matrices are presented as a byproduct of the analysis.

Gradient descent, In-context learning,  Regression, Generalization, Statistical bounds

62J05, 68T10

§ INTRODUCTION

Significant developments in large language models have led to much research  on enhancing and understanding  learning processes of Transformers [23, 14]. Among other characteristics, Transformer models have been shown to mimic the mechanisms of gradient descent during their forward pass [24]. This behavior has been related to in-context learning abilities of Transformers [8]. In-context examples typically consist of input-output pairs that are directly related to a specific task. This mechanism allows the Transformer to process a new query input by leveraging the prompt examples to generate the corresponding predictive output effectively. The connections between in-context learning and gradient descent have been widely studied over the past two years. von Oswald et al. [24] show that training Transformers on auto-regressive tasks mimics gradient-based meta-learning. In other words, Transformers learn in-context by emulating gradient descent, becoming meta-optimizers and excelling in regression tasks. Dai et al. [6] hypothesize that attention values act as meta-gradients, enabling implicit fine-tuning for in-context learning.

While the present work is strongly motivated by the above connections and hypotheses, the primary focus of this paper is on the foundational aspects of gradient descent method. We consider a case of in-context linear regression with a well-specified model, and aim to  investigate the extent to which a single step of gradient descent can generalize across examples from a noisy distribution. The focus on the single step is because of the relevance to in-context learning as described above.

The problem setting is as follows. Assume we have a model $y = W_0 x$, where for simplicity we assume $x \in \reals^n$ corresponds to realizations from the standard normal distribution. $W_0 \in \reals^{m \times n}$ is the existing weight matrix and $y \in \reals^m$ is the output.  As in the in-context setting, assume that we get N i.i.d. data pairs $\{x_1,y_1\},...\{x_N,y_N\}$ where $p(y_i|x_i) = \mathcal{N}(y_i;W_1 x_i,  \sigma^2 I)$.  We are interested in the properties of one-shot gradient descent, for an testing input $x_{N+1} \triangleq \hat{x}$. Particularly, we would like to know the statistical properties of the prediction $\hat{y}$ and bounds on the generalization error.  We will compare these results to least squares regression (on the given data pairs). A random design setting will be concerned (i.e. the in-context data available to us is drawn randomly).

Indeed,  properties of gradient descent have been studied for a long time, with much of the work devoted to convergence properties (e.g.  [22, 13]). Optimality of stochastic gradient descent has also been addressed from various perspectives (e.g.  [21]). Ref. [19] presents  upper bounds on the generalization error that depend on local statistics of the stochastic gradients using  information-theoretic constructs.
We seek results that are non-asymptotic (i.e. by considering finite samples), not containing arbitrary constants, and not requiring bounded covariates. In fact, it can be argued that such results are hard to come by even in conventional linear least-squares regression. For instance,  Gyorfi et al. [10] and Catoni's [5] error bounds for least squares regression are well-crafted, but contain an arbitrary constant. Many publications contain terms such as $O(\cdot)$, thus rendering an asymptotic error estimate. Audibert & Catoni [1] require
$N >> n \log n$ and Hsu et al. [12] require $N >> n$.  Classical PAC-Bayesian bounds [16] require bounded loss functions or additional parameters beyond the data [9].  The author is careful to emphasize that the   above works are rigorous, and focused on a more general - and thus more practically relevant - setting than the present one, and that a bound can be useful even under the above conditions.

The outline and main contributions of the paper are as follows: The expected generalization error of in-context gradient descent (for linear regression with well-specified models) is derived in Section <ref>. Comparisons are made with classical least squares regression and a breakdown of the systematic and noise components and an expression is provided for the optimal step size .  In sections <ref> and  <ref>, probabilistic bounds are derived for gradient descent and least squares regression. Section <ref> explores connections to existing work. As a byproduct of this work, several identities were derived involving high order products of Gaussian random matrices, and provided in the Appendix.

§ EXPECTED GENERALIZATION ERROR

As mentioned in the introduction, the in-context learning setting is equated to one step of gradient descent over a `prompt' of $N$ i.i.d. data-pairs from a new task. Gradient descent  yields an output with the new weights, i.e. $\hat{y} = (W_0 + \Delta W)\hat{x}$.

Given $X \triangleq [x_1 \ \ x_2  \ \ ...\ \ x_N] \in \reals^{n \times N}$, which is a i.i.d standard normal random matrix and $Y \triangleq [y_1 \ \ y_2 \ \ .. \ \ y_N] \in \reals^{m \times N}$ with $y_i = W_1 x_i + \sigma^2 z_i$ and $z_i \in \reals^m$ is an i.i.d. sample from the standard normal distribution. For a step size $\eta$, the expected mean squared error for a step of gradient descent is
$\mathbb{E}[\ell] =||W_1 - W_0 ||_F^2  \left((1-\eta)^2+ \eta^2 \frac{n+1}{N}\right) + \sigma^2 \left(m +  \eta^2 \frac{n}{N} \right)  $

For a mean squared training loss, it is easy to see that the update to the weights after one step of gradient descent is

\begin{align*}
\Delta W &= - \frac{\eta}{N} \sum_i (W_0 x_i - y_i) x_i^T  \triangleq - \frac{\eta}{N} (W_0 X - Y) X^T.
\end{align*}

Given $ y_i = W_1 x_i + \sigma z_i$,

\begin{align*}
\Delta W &= - \frac{\eta}{N} (W_0 X - W_1 X { - \sigma Z}) X^T \\
&= \frac{\eta}{N}(W_1-W_0) X X^T { + \sigma \frac{\eta}{N}Z  X^T},
\end{align*}

where $Z \triangleq [z_1 \ \ z_2 \ \ .. \ \ z_N] \in \reals^{m \times N}$. Given $X,Z$, it can be shown that the distribution of the weight update is
$$\Delta W \sim \mathcal{W}\left(\eta(W_1-W_0),\frac{\eta^2}{N}(n+1)(W_1-W_0)^T(W_1-W_0)\right),$$

where $\mathcal{W}$ is the Wishart distribution.

We are interested in determining the statistical properties of the generalization error for the above random design case. We will begin with the expected mean squared error, which for a
given test location $\hat{x}$ is
$\ell  \triangleq \norm{W_1 \hat{x} + \sigma \hat{z} - (W_0 + \Delta W) \hat{x}}^2.$ Substituting $\Delta W$,

\begin{align*}
\ell &=  \norm{\left( W_1 \hat{x} - W_0 - \frac{\eta}{N}(W_1-W_0) X X^T \right)\hat{x}    { - \sigma \frac{\eta}{N}Z  X^T \hat{x}+ \sigma \hat{z}}}^2 \\
\mathbb{E}[\ell]&= \mathbb{E}[ \norm{\left((W_1-W_0)(I- \frac{\eta}{N} X X^T \right)\hat{x} }^2] {+ \sigma^2 \mathbb{E}[\hat{z}^T \hat{z}]   + \sigma^2 \frac{\eta^2}{N^2} \mathbb{E}\left[ \hat{x} X Z^T Z X^T \hat{x}\right]}\\
&= \mathbb{E}[ \norm{ A(I- a Q) \hat{x} }^2]  { + \sigma^2 m + \sigma^2 a^2 N n} \ \ \textrm{where} \ \ a \triangleq \frac{\eta}{N}, Q \triangleq X X^T, A \triangleq W_1-W_0 \\
\mathbb{E}_{\hat{x}}[\ell] &= \tr [(I- a Q)A^T A(I- a Q)] { + \sigma^2 m + \sigma^2 a^2 N n} \\
&= \tr [A^T A - a Q A^T A - A^T A a Q +a^2 Q A^T A Q] { + \sigma^2 m + \sigma^2 a^2 N n} \\
\mathbb{E}[\ell]&= \tr [A^T A](1 - a N - a N + a^2((n+1)N+N^2)) {+ \sigma^2 m + \sigma^2 a^2 N n} \\
&= \tr [A^T A](1 - 2 a N + a^2((n+1)N+N^2)) { + \sigma^2 m + \sigma^2 a^2 N n}.
\end{align*}

Consider $m=1$  for simplicity. The above yields

\begin{equation}
\label{eq:ExpectedLoss}
\mathbb{E}[\ell]=  ||W_1 - W_0 ||_2^2  \left((1-\eta)^2+ \eta^2 \frac{n+1}{N}\right) { + \sigma^2 \left(1 +  \eta^2 \frac{n}{N} \right)}.
\end{equation}

Note that standard least squares regression yields (see Section <ref>)

\mathbb{E}[\ell] = ||W_1-W_0||_2^2 \left( 1 - \frac{N}{n} \right) { + \sigma^2 \left( 1 + \frac{N}{n-N-1} \right)} \ \ \textrm{for}  \ \  2 \leq N \leq n-1

\mathbb{E}[\ell] =  { \sigma^2 \left( 1 + \frac{n}{N-n-1} \right)} \ \ \textrm{for}  \ \  n+1 \leq N \le \infty.

In the under-parametrized regime (i.e. $N < n$), the systematic error in Least squares (or more precisely, Least Norm) regression drops linearly, whereas the error due to noise grows. The blow up near $N = n$ and the double descent behavior is attributable to noise amplification, and is - by now - well studied [2, 3]. For $N \ge n$, there is no systematic error as is well known.

Figure  <ref> shows for $n=40, m=1, \eta=1$, empirical evaluations (for each N, we used 500 random designs and 500 test evaluations) compared to the above analytical expressions. $W_1$ was sampled from the standard normal distribution and normalized such that $\norm{W_1} =1$ and $\sigma^2$ was varied to yield different signal-to-noise ratios.
The breakdown of the different components of the testing error is also shown for $\norm{W_1}^2 = \sigma^2$.

Expected Generalization Error for $n=40; m=1; \eta=1$. Top : $\norm{W_1}^2/\sigma^2=1$; Bottom left : $\norm{W_1}^2/\sigma^2=2$; Bottom right: $\norm{W_1}^2/\sigma^2=0.5$

$n=40; m=1, \norm{W_1}^2/\sigma^2=1$

Examining Equation <ref>, we can determine the optimal step size
$$\eta_{opt} = \frac{N}{N+n+1 + \frac{\sigma^2}{||W_1||^2}n}$$
for gradient descent. In realistic scenarios, we do not  know the signal to noise ratio, and thus a practical guide would be

\eta_{opt} = \frac{N}{N+n+1} = 1 - \frac{n+1}{N+n+1},

which for the noiseless case yields

$$\mathbb{E}[\ell] = ||W_1||^2 \left(1- \frac{N}{N+n+1}\right).$$
This is shown to greatly reduce the generalization error as shown in Fig <ref>.

$n=40; m=1, \sigma=0$

§ BOUNDS

In the previous section, we examined the expected error in in-context regression. In this section, we derive probabilistic bounds for the generalization error.

As in the prior setting, assume $X \triangleq $ $[x_1 \ \  ...\ \ x_N]$ $ \in \reals^{n \times N}$ as an i.i.d standard normal random matrix and $Y \triangleq [y_1 \ \ y_2 \ \ .. \ \ y_N] \in \reals^{1 \times N}$ with $y_i = w_1^T x_i + \sigma^2 z_i$ and $z_i \in \reals^m$ is an i.i.d. sample from the standard normal distribution.  With at least a probability of $1-\delta$,the generalization error of gradient descent (with a step size of 1) is bounded by

\begin{equation}
\ell \leq \left[\frac{||w_1-w_0||_2^2(n+1)  + \sigma^2 (n+N)}{N}  + \sqrt{\frac{V}{\delta}} \right],
\end{equation}

\begin{align*}
V & \triangleq  \sigma^4\left(2  + \frac{12 n+6 n^2}{N^3} + \frac{6 n + 2 n^2}{N^2} + \frac{4 n}{N}\right)\\
&+\sigma^2 \norm{w_1-w_0}^2\left(\frac{72  +  60 n  + 12 n^2}{N^3} + \frac{12 + 16 n + 4 n^2}{N^2} + \frac{4 + 4 n}{N}\right) \\
&+\norm{w_1-w_0}^4 \left( \frac{90  + 48 n + 6 n^2}{N^3} + \frac{20  + 10 n + 2 n^2}{N^2} \right)
\end{align*}

Note: For the noiseless case, therefore with at least a probability of $1-\delta$, we can guarantee that the generalization error is bounded by

\begin{equation}
\ell \leq \left[\frac{n+1}{N}+ \sqrt{\frac{ \left[ \frac{90+48 n+6 n^2}{N^3} + \frac{20+ 10 n +2 n^2}{N^2}\right]}{\delta}} \right] \norm{w_1-w_0}^2.
\end{equation}

For large n, we have

\begin{equation}
\ell \leq \frac{n}{N}\left(1+ \sqrt{\frac{2}{\delta}} \right) \norm{w_1-w_0}^2.
\end{equation}

Figure <ref> verifies the empirical CDF (generated using $8 \times 10^6$ samples) for a case with signal to noise ratio of 1.

Empirical CDF and theoretical bounds for $n=40; m=1$. Left : N=8; Middle: N=20; Right: N=60

We will use the Chebyshev concentration inequality, which requires the variance. We first pursue the noise free case.  Following the setup of Proof 1, for the noiseless case

\ell^2_{sys} \triangleq\hat{x}^T(I-a Q)A^T A(I-aQ)\hat{x} \hat{x}^T(I-a Q)A^T A(I-aQ)\hat{x}. $$

$a \triangleq \frac{\eta}{N}, Q \triangleq X X^T, A \triangleq W_1-W_0$ .

\begin{align*}
\mathbb{E}_{\hat{x}}[\ell^2_{sys}] &= 2 \tr[A(I- a Q)(I- a Q)A^T A(I- a Q)(I- a Q)A^T] + (\tr[A(I- a Q)(I- a Q)A^T])^2\\
&= 2 \tr[B(I- a Q)(I- a Q)B(I- a Q)(I- a Q)] + (\tr[B(I- a Q)(I- a Q)])^2,
\end{align*}

where  $B \triangleq A^T A$.  We have used the following identity:

If $X \sim \mathcal{N}(0,I)$ then
$\mathbb{E}_X[x^T C^T C x x^T C^T C x] = 2\tr(CC^TCC^T)+ \tr[C C^T]^2.$ The manipulations below require expectations involving permutations of products of 8th order Gaussians. See Appendix.

\begin{align*}
\mathbb{E}[\ell^2_{sys}]&=  2\mathbb{E} \tr[B(I-2aQ+a^2QQ)B(I-2aQ+a^2QQ)]\\
&+ \mathbb{E}[\Tr[B(I-aQ)(I-aQ)]\Tr[B(I-aQ)(I-aQ)]]\\
%&=  2\mathbb{E} \tr[B(I-2aQ+a^2QQ)B(I-2aQ+a^2QQ)]+  \mathbb{E}[\Tr[B-2aBQ+a^2 BQQ]\Tr[B-2aBQ+a^2 BQQ]] \\
%&=  2\mathbb{E} \tr[(B-2aBQ+a^2BQQ)(B-2aBQ+a^2BQQ)]+   \mathbb{E}[\Tr[B-2aBQ+a^2 BQQ]\Tr[B-2aBQ+a^2 BQQ]]   \\
%&=  2\mathbb{E}\tr[BB-2aBBQ+a^2BBQQ-2aBQ(B-2aBQ+a^2BQQ)+a^2BQQ(B-2aBQ+a^2BQQ)]\\
%&+ \mathbb{E}[\tr[B]^2-4a\tr[B]\tr[BQ]+2a^2\tr[B]\tr[BQQ]+4a^2\tr[BQ]\tr[BQ]-4a^3\tr[BQ]\tr[BQQ]+a^4\tr[BQQ]\tr[BQQ]]\\
%&=  2\mathbb{E}\tr[BB-2aBBQ+a^2BBQQ-2aBQB+4a^2BQBQ-2a^3BQBQQ+a^2BQQB-2a^3BQQBQ+a^4BQQBQQ)]\\
&=  2\mathbb{E}\tr[BB-4aBBQ+2a^2BBQQ+4a^2BQBQ-4a^3BQBQQ\\
&+a^4BQQBQQ]+ \mathbb{E}[\tr[B]^2-4a\tr[B]\tr[BQ]+2a^2\tr[B]\tr[BQQ]+4a^2\tr[BQ]\tr[BQ]\\
&=  2\tr[B^2](1-4aN+2a^2((n+1)N+N^2))+2\mathbb{E}\tr[4a^2BQBQ-4a^3BQBQQ\\
&+a^4BQQBQQ]+ \tr[B]^2(1-4aN+2a^2(1+n+N)) + \mathbb{E}[4a^2\tr[BQ]\tr[BQ\\
\end{align*}

\begin{align*}
\mathbb{E}[\ell^2_{sys}]&=  2\tr[B^2](1-4aN+2a^2((n+1)N+N^2))\\
&+8 a^2 N((N+1)\tr[B^2]+\tr[B]^2)\\
&-8a^3 N ((4 + n + (3 + n) N + N^2) \tr[B^2] + (2 + n + 2 N)\tr[B]^2)\\
&+2 a^4 N((20 + n(11 + n) + 21 N + n (7 + n) N + 2 (3 + n) N^2 + N^3) \tr[B^2] \\
&+ (10 + 5 n + n^2 + 5 (2 + n) N + 4 N^2) \tr[B]^2)\\
&+ 4a^2 N (N \tr[B]^2+2 \tr[B^2]) \\
&-4a^3 N(2(2 + n + 2 N) \tr[B^2] + (2 + N(1 + n + N))\tr[B]^2)\\
&+a^4 N(2(10 + 5n + n^2 + 5(2 + n)N + 4N^2)*\tr[B^2] + (n^2 N + 2 n (3 + N + N^2) \\
&+ (1 + N)(10 + N + N^2)) \tr[B]^2)\\
&=(1 + a N (-4 +
a (2 (5 + n + 3 N) +
a (2 + N) (3 + n + N) (-4 + a (5 + n + N)))))\\
& (2 \tr[B^2] + \tr[B]^2).
\end{align*}

The varjance $Var(\ell_{sys}) = \mathbb{E}[\ell^2_{sys}] - \mathbb{E}^2[\ell_{sys}]$ was verified using $10^{10}$ samples for several combinations of $N$ and $n$. Nevertheless the above formula is unwieldy. Examining the case where $m=1,w_0=0$ and $\eta=1$: $\mathbb{E}[\ell_{sys}] = \frac{n+1}{N}\norm{w_1-w_0}^2$ and
$Var[\ell] = \left[ \frac{90+48 n+6 n^2}{N^3} + \frac{20+ 10 n +2 n^2}{N^2}\right]\norm{w_1-w_0}^4$.

Chebyshev's inequality states that for any random variable $X$, and $\epsilon \in \reals^+$,

$$ P( |x-\mathbb{E}[X]| \geq \epsilon)  \leq  \frac{Var(X)}{\epsilon^2}. $$

\begin{align*}
P( |\ell_{sys}-\frac{n+1}{N}\norm{w_1-w_0}^2| \leq \epsilon)   &\geq 1- \frac{1}{\epsilon^2} \left[ \frac{90+48 n+6 n^2}{N^3} + \frac{20+ 10 n +2 n^2}{N^2}\right]\norm{w_1-w_0}^4 \\
\end{align*}

Define $\delta \triangleq \frac{Var[\ell]}{\epsilon^2} $. Therefore with at least a probability of $1-\delta$.

\begin{equation}
\ell_{sys} \leq \left[\frac{n+1}{N}+ \sqrt{\frac{ \left[ \frac{90+48 n+6 n^2}{N^3} + \frac{20+ 10 n +2 n^2}{N^2}\right]}{\delta}} \right] \norm{w_1-w_0}^2.
\end{equation}

Now we consider the noise terms.
Define $D \triangleq A(I-aQ)  - a \sigma Z X^T$

\begin{align*}
\ell^2 &=  (D \hat{x} + \sigma \hat{z})^T (D \hat{x} + \sigma \hat{z}) (D \hat{x} + \sigma \hat{z})^T (D \hat{x} + \sigma \hat{z}) \\
\mathbb{E}_{\hat{x}}[\ell^2] &=2 \tr[D D^T D D^T] + 4 \sigma^2 \hat{z}^T D D^T \hat{z} + (\tr[D D^T]+\sigma^2 \hat{z}^T \hat{z})^2 \\
&=2 \tr[D D^TD D^T] + 4 \sigma^2 \hat{z}^T D D^T \hat{z} + \tr[D D^T]^2+\sigma^4 \hat{z}^T \hat{z} \hat{z}^T \hat{z} + 2 \sigma^2 \hat{z}^T \hat{z} \tr[D D^T]\\
\mathbb{E}_{\hat{z} \hat{x}}[\ell^2] &=2 \tr[D D^T D D^T] + 4 \sigma^2  \tr[D D^T] + \tr[D D^T]^2+(2m+m^2) \sigma^4 + 2 \sigma^2 m \tr[D D^T].
%&=2 \tr[D D^T D D^T] + (4 \sigma^2 + 2 \sigma^2 m)  \tr[D D^T] + \tr[D D^T]^2+(2m+m^2) \sigma^4 \\
\end{align*}

Let's switch to $m=1$ for tractability
$  c^T  \triangleq A(I-aQ) ; C^T \triangleq  - a \sigma X^T$ and $z = Z^T$. The expressions below require new identities that are presented in the Appendix.

\begin{align*}
\mathbb{E}_{\hat{z} \hat{x}}[\ell^2] &= 3 (Cz+c)^T(Cz+c)(Cz+c)^T(Cz+c)+ 6 \sigma^2  (Cz+c)^T(Cz+c) + 3 \sigma^4 \\
\mathbb{E}_{z \hat{z} \hat{x}}[\ell^2] &= 6 \Tr[C C^T C C^T] + 12 c^T C C^T c + 3 (\Tr[CC^T]  + c^T c)^2 + 6 \sigma^2 (\tr[C C^T]+c^T c) + 3 \sigma^4 \\
%&=6 \Tr[C C^T C C^T] + 12 c^T C C^T c + 3 \Tr[CC^T]\Tr[C C^T]  + 3 c^T c c^T c + 6 \Tr[C C^T]c^T c + 6 \sigma^2 (\tr[C C^T]+c^T c) + 3 \sigma^4 \\
%&= 6 a^4 \sigma^4\Tr[X X^T X X^T] + 12a^2 \sigma^2 A(I-aQ) X X^T (I-aQ)A^T +3 a^4 \sigma^4 \Tr[XX^T]  \Tr[XX^T]{\color{blue}+ \mathbb{E}[\ell_{sys}^2]}\\
%&+6a^2\sigma^2 \Tr[X X^T]A(I-aQ)(I-aQ)A^T + 6 \sigma^4 a^2 \Tr[X X^T] + 6 \sigma^2 A(I-aQ)(I-aQ)A^T + 3 \sigma^4\\
%&= 6 a^4 \sigma^4\Tr[Q Q] + 12a^2 \sigma^2 A(I-aQ) Q(I-aQ)A^T +3 a^4 \sigma^4 \Tr[Q] \Tr[Q]{\color{blue}+3 A(I-aQ)(I-aQ)A^TA(I-aQ)(I-aQ)A^T}\\
%&+6a^2\sigma^2 \Tr[Q]A(I-aQ)(I-aQ)A^T + 6 \sigma^4 a^2 \Tr[Q] + 6 \sigma^2 A(I-aQ)(I-aQ)A^T + 3 \sigma^4\\
%&= 6 a^4 \sigma^4\Tr[Q Q] + 12a^2 \sigma^2 \Tr[B(I-aQ) Q(I-aQ)]  +3 a^4 \sigma^4 \Tr[Q]\Tr[Q]{\color{blue}+ \mathbb{E}[\ell_{sys}^2]}\\
%&+6a^2\sigma^2 \Tr[Q] A(I-aQ)(I-aQ)A^T + 6 \sigma^4 a^2 \Tr[Q] + 6 \sigma^2 \tr[B (I-aQ)(I-aQ)] + 3 \sigma^4\\
%&= 6 a^4 \sigma^4\Tr[Q Q] + 12a^2 \sigma^2 \Tr[(B-aBQ) (Q-aQQ)] +3 a^4 \sigma^4 \Tr[Q]\Tr[Q]{\color{blue}+ \mathbb{E}[\ell_{sys}^2]}\\
%&+6a^2\sigma^2 \Tr[Q](A A^T -2aAQA^T+a^2 AQQA^T)+ 6 \sigma^4 a^2 \Tr[Q] + 6 \sigma^2 \tr[(B-aBQ)(I-aQ)] + 3 \sigma^4\\
&= 6 a^4 \sigma^4\Tr[Q Q] + 12a^2 \sigma^2 \Tr[BQ-2aBQQ +a^2BQQQ] +3 a^4 \sigma^4 \Tr[Q]\Tr[Q]{\color{blue}+ \mathbb{E}[\ell_{sys}^2]}\\
&+6a^2\sigma^2 \Tr[Q] (A A^T -2aAQA^T+a^2 AQQA^T) + 6 \sigma^4 a^2 \Tr[Q] \\
&+ 6 \sigma^2 \tr[B-2aBQ+a^2 BQQ] + 3 \sigma^4\\
\mathbb{E}[\ell^2]&= 6 a^4 \sigma^4 Nn (N+n+1) + 12a^2 \sigma^2 \Tr[B](N-2aN(N+n+1)\\
& +a^2N (4 + n^2 + 3 n (1 + N) + N (3 + N))) +3 a^4 \sigma^4 N (2n +N n^2)\\
&+6a^2\sigma^2 \tr[B] (Nn  -2aN (2+Nn)+a^2 N(1+n+N)(4+nN)) \\
&+ 6 \sigma^4 a^2 nN + 6 \sigma^2 \tr[B](1-2aN+a^2 N(N+n+1)) + 3 \sigma^4  {\color{blue}+ \mathbb{E}[\ell_{sys}^2]}
\end{align*}

For $\eta = 1$, we get

\begin{align*}\mathbb{E}[\ell^2]&= \sigma^4 \left( 3 + \frac{12 n+ 6 n^2 }{N^3} + \frac{6 n + 3 n^2}{N^2} + \frac{6 n}{N}\right) \\
& + \tr[B]  \sigma^2\left(  \frac{72   + 60 n  +12 n^2 }{N^3} + \frac{12  + 18 n +6 n^2 }{N^2} + \frac{6 n}{N} + \frac{6}{N} \right)  {\color{blue}+ \mathbb{E}[\ell_{sys}^2]}.
\end{align*}

\begin{align*}
Var[\ell] &=\sigma^4\left(2  + \frac{12 n+6 n^2}{N^3} + \frac{6 n + 2 n^2}{N^2} + \frac{4 n}{N}\right)\\
&+\sigma^2 \tr[B]\left(\frac{72  +  60 n  + 12 n^2}{N^3} + \frac{12 + 16 n + 4 n^2}{N^2} + \frac{4 + 4 n}{N}\right) \\
&+\tr[B]^2 \left( \frac{90  + 48 n + 6 n^2}{N^3} + \frac{20  + 10 n + 2 n^2}{N^2} \right).
\end{align*}

§ RESULTS FOR LEAST NORM AND LEAST SQUARES REGRESSION

Given $X \triangleq [x_1 \ \ x_2  \ \ ...\ \ x_N] \in \reals^{n \times N}$, which is a i.i.d standard normal random matrix and $Y \triangleq [y_1 \ \ y_2 \ \ .. \ \ y_N] \in \reals^{1 \times N}$ with $y_i = w_1^T x_i + \sigma^2 z_i$ and $z_i \in \reals^m$ is an i.i.d. sample from the standard normal distribution. Then, the generalization error in Least norm (centered on $W_0$) and Least Squares regression has the following properties

\begin{align*}
\mathbb{E}[\ell] &=\norm{W_1 - W_0}^2 \left(1-\frac{N}{n} \right) { + \sigma^2 \left(1 + \frac{N}{n-N-1} \right)} \ \  ; \ \ 1 \leq N \leq n-1 \\
\mathbb{E}[\ell] &= \sigma^2 \left(1 + \frac{n}{N-n-1} \right) \ \  ; \ \ n+1 \leq N  \\
\mathbb{E}[\ell^2] &= \sigma^4 \frac{3 (N-1) (N-3)}{(N-n-1) (N-n-3)} \ \ ; N  > n+3 \\
\mathbb{E}[\ell^2] &=\sigma^4 \frac{3 (n-1) (n-3)}{(n-N-1) (n-N-3)} + 6 \sigma^2 \norm{W_1 - W_0}^2 \frac{(n-1)(n-N)}{n(n-N-1)} \\
& + 3 \norm{W_1 - W_0}^4  \frac{(n-N)(n-N+2)}{n(n+2)}  \ \ ; \ \ n \ge N+3.
\end{align*}

Note: A version of the first two equalities can be found in Belkin et al. [3] and are verified in Figure <ref>. The last two equalities above are novel to the knowledge of the author, and are verified in Figure <ref>. Hastie et al. [11] have derived formulae  for variance under slightly more general conditions, but these results are asymptotic in nature. Very recently, Zhou et al. [26] derive a related expression, but they do not assume $\hat{x}$ and $\hat{y}$ are random as in the present context [They have cited an  unpublished, early version of this work [7]].  Belkin et al. [3] derive statistical bounds that are  related.
These results can be employed with concentration inequalities as in the previous section to construct bounds for the generalization error.  Figure  <ref> shows such a comparison for the $n=40$ example considered in previously.

$\mathbb{E}[\ell^2]$ for $n=40; m=1; \eta=1$. Top : $\norm{W_1}^2/\sigma^2=1$; Bottom left : $\norm{W_1}^2/\sigma^2=2$; Bottom right: $\norm{W_1}^2/\sigma^2=0.5$

Analytical bounds for $n=40; m=1$. Left : N=60;  Right: N=80

Over-parametrized case

Let us consider the over-parametrized case first:

\begin{align*}
\ell &= ||(W_1 - W)\hat{x} {+ \sigma \hat{z} }||_2^2 = ||  \sigma \hat{z}  - \sigma Z X^+ \hat{x}  ||_2^2\\
\mathbb{E}[\ell]&= \mathbb{E}[(\sigma \hat{z}  - \sigma Z X^+ \hat{x} )^T(\sigma \hat{z}  - \sigma Z X^+ \hat{x} )] \\&=  \mathbb{E}[\sigma^2 \hat{z}^T \hat{z}]   +\mathbb{E}[\sigma^2 \hat{x}^T X^{+T} Z^T Z  X^+ \hat{x}] \\
&=  \sigma^2    + \sigma^2 \mathbb{E}_{X Z}[ \tr[X^{+T} Z^T Z  X^+]] \\
&=  \sigma^2    + \sigma^2 \mathbb{E}_{X}[ \tr[X^{+T}   X^+]] \\
&=  \sigma^2    + \sigma^2 \mathbb{E}_{X}[ \tr[(X X^T)^{-1}]] \\
&= \sigma^2 \left(1 + \frac{n}{N-n-1} \right).
\end{align*}

\begin{align*}
\ell^2 &= (\sigma \hat{z}  - \sigma Z X^+ \hat{x} )^T(\sigma \hat{z}  - \sigma Z X^+ \hat{x} )(\sigma \hat{z}  - \sigma Z X^+ \hat{x} )^T(\sigma \hat{z}  - \sigma Z X^+ \hat{x} )\\
\mathbb{E}_{\hat{x}}[\ell^2] &= \sigma^4 ( 2 \tr[Z X^+ X^{+T} Z^T Z X^+ X^{+T} Z^T] + 4  \hat{z}^T Z X^+ X^{+T} Z^T \hat{z} + \tr[Z X^+ X^{+T} Z^T ]^2 + \hat{z}^T \hat{z} \hat{z}^T \hat{z}\\
& + 2  \hat{z}^T \hat{z} \tr[Z X^+ X^{+T} Z^T ])\\
\mathbb{E}_{\hat{z}\hat{x}}[\ell^2]&=  \sigma^4 ( 2 \tr[z^T (X X^T)^{-1} z z^T (X X^T)^{-1} z] + 4  \tr[ z^T (X X^T)^{-1} z] + \tr[z^T (X X^T)^{-1} z ]^2 + 3 \\
&+ 2  \tr[z^T (X X^T)^{-1} z ])\\
\mathbb{E}_{z\hat{z}\hat{x}}[\ell^2]&=  \sigma^4 ( 6 \tr[(X X^T)^{-2}] + 3 \tr[X X^T)^{-1}]^2 + 6  \tr[(X X^T)^{-1}] + 3)\\
&\sim \sigma^4( 6 \tr(W_n^{-1}(I,N)W_n^{-1}(I,N)) + 3 \tr(W_n^{-1}(I,N))\tr(W_n^{-1}(I,N))+ 6 \tr( W_n^{-1}(I,N))  +3),
\end{align*}

where $W_m^{-1}$ represents the Inverse Wishart distribution. The expectation of the last expression can be compactly reduced using standard Wishart distribution identities, except the term involving the trace of the product of inverse Wishart matrices, which we obtain from Pielaszkiewicz & Holgersson [20] (page 8). With this identity,

\begin{align*}
\mathbb{E}_X \mathbb{E}_{\xh} \mathbb{E}_{z} \mathbb{E}_{\hat z} \ell^2&= 6 \sigma^4 \frac{(N-1)n}{(N-n-3)(N-n-1)(N-n)} +3 \sigma^4 \frac{n(n(N-n-2)+2)}{(N-n-3)(N-n-1)(N-n)}\\
& + 6 \sigma^4 \frac{n}{N-n-1}  + 3 \sigma^4 \\
&=\sigma^4 \frac{3 (N-1) (N-3)}{(N-n-1) (N-n-3)},
\end{align*}

as long as $N>n+3$.

Under-parametrized case

Next, we consider the under-parameterized case for which the Lagrangian \( \mathcal{L} \) is defined as:

\[
\mathcal{L}(W, \Lambda) = \text{tr}((W - W_0)(W - W_0)^T) + \text{tr}(\Lambda^T (W X - Y))
\]

where \( \Lambda \) is a  matrix of Lagrange multipliers.

\[
\frac{\partial \mathcal{L}}{\partial W} = 2(W - W_0) + \Lambda X^T = 0  \implies W = W_0 - \frac{1}{2} \Lambda X^T
\]

Substituting \( W \) into the constraint:

\[
(W_0 - \frac{1}{2} \Lambda X^T) X = Y \implies W_0 X - \frac{1}{2} \Lambda X^T X = Y
\implies
\Lambda = -2 (X^T X)^{-1}(W_0 X - Y)
\]

Substituting back for \( W_1 \):

\[
W = W_0 - \frac{1}{2} (-2 (X^T X)^{-1}(W_0 X - Y)) X^T = W_0 + (Y - W_0 X) (X^T X)^{-1} X^T
\]

It is easily shown that first order optimality yields $W=W_0 + (Y - W_0 X) (X^T X)^{-1} X^T$. In the present setup, $Y = W_1 X {+ \sigma Z}$. Therefore
W = W_0(I-P) + W_1 P { + \sigma Z X^+} ,
where $X^+ \triangleq (X^T X)^{-1} X^T$ and  $ P  \triangleq  X (X^T X)^{-1} X^T$.

For a test location $\hat{x}$,

$$ \ell =  ||(W_1 - W)\hat{x} {+ \sigma \hat{z}}||_2^2 = || (W_1-W_0)(I-P)  \hat{x} {+ \sigma \hat{z} - \sigma Z X^+ \hat{x}}||_2^2$$

Taking the expectation of the first term (wrt) $\hat{x}$, we have  $  \tr[(I-P)(W_1-W_0)^T(W_1-W_0)(I-P)] = \tr[(I-P)(W_1-W_0)^T(W_1-W_0)]$
Then taking the expectation of the first term wrt to X, we have

\begin{align*}
\tr[(W_1-W_0)^T(W_1-W_0)\mathbb{E}[I-P]] = \tr[(W_1-W_0)^T(W_1-W_0)]\left(1-\frac{N}{n}\right).
\end{align*}

The contribution of the second term to $\mathbb{E}[\ell]$ is $\sigma^2$.

The third term is $
\mathbb{E}_{\hat x}[\hat{x}^T X^{+T} Z^T Z X^+ \hat{x}] = \Tr[X^{+T} Z^T Z X^+]$, thus
$\mathbb{E}_Z[\Tr[X^{+T} Z^T Z X^+]] = \Tr[X^{+T} X^+] = \Tr[(X^T X)^{-1}]$. Finally,

$$\mathbb{E}_X[\tr[(X^T X)^{-1}]] = \frac{N}{n-N-1}.$$

$$\mathbb{E}[\ell] = ||W_1 - W_0||^2 \left(1-\frac{N}{n} \right) { + \sigma^2 \left(1 + \frac{N}{n-N-1} \right)}. $$

Now we consider the second moment. Define $C  \triangleq  -  \sigma  X^{+T}; c\triangleq (I-P) b; z \triangleq Z^T; b \triangleq (W_1-W_0)^T.$

\begin{align*}
\mathbb{E}_{z \hat{z} \hat{x}}[\ell^2] &= 6 \Tr[C C^T C C^T] + 12 c^T C C^T c + 3 (\Tr[CC^T]  + c^T c)^2 + 6 \sigma^2 (\tr[C C^T]+c^T c) + 3 \sigma^4 \\
&= 6 \sigma^4 \Tr[X^{+T} X^{+} X^{+T} X^{+}] + 12 \sigma^2 b^T (I-P)  X^{+T} X^{+} (I-P)b + 3 \sigma^4 \Tr[X^{+T}X^{+}]^2  \\
&+ 3 b^T (I-P)b b^T (I-P)b  \\
&+ 6 \sigma^2 \tr[ X^{+T}  X^{+}]b^T(I-P)b + 6 \sigma^4 \tr[X^{+T} X^{+}]+6 \sigma^2 b^T(I-P)b+ 3 \sigma^4 \\
&= 6 \sigma^4 \Tr[(X^T X)^{-2}] + 3 \sigma^4 \Tr[(X^T X)^{-1}]^2  + 3 b^T (I-P)b b^T (I-P)b  \\
&+ 6 \sigma^2 \tr[ (X^T X)^{-1}]b^T(I-P)b + 6 \sigma^4 \tr[(X^T X)^{-1}]+6 \sigma^2 b^T(I-P)b+ 3 \sigma^4 \\
\mathbb{E}[\ell^2] &=6 \sigma^4 \frac{(n-1)N}{(n-N-3)(n-N-1)(n-N)} \\&+ 3 \sigma^4 \frac{N(N(n-N-2)+2)}{(n-N-3)(n-N-1)(n-N)}  + 3 \norm{b}^4  \frac{(n-N)(n-N+2)}{n(n+2)}  \\
&+ 6 \sigma^2 \frac{N}{n-N-1}\norm{b}^2 \left(1-\frac{N}{n}\right) + 6 \sigma^4\frac{N}{n-N-1}+6 \sigma^2 \norm{b}^2 \left(1-\frac{N}{n}\right)+ 3 \sigma^4\\
&=\sigma^4 \frac{3 (n-1) (n-3)}{(n-N-1) (n-N-3)} + 6 \sigma^2 \frac{(n-1)(n-N)}{n(n-N-1)}\norm{b}^2  + 3 \norm{b}^4  \frac{(n-N)(n-N+2)}{n(n+2)}.
\end{align*}

In the derivation above, we have used the following

* Noting that $I-P$ will have $n-N$ unity eigenvalues (rest are zero). We will write $(I-P) = \sum_{i=1}^{n-N} v_i v_i^T$, where $v_i$ are its eigenvectors.

\begin{align*}
b^T(I- P)b b^T(I- P)b&= b^T \sum_{i=1}^{n-N} (v_i v_i^T) b b^T \sum_{j=1}^{n-N} (v_j v_j^T) b\\
&=(n-N)b^T  v_1 v_1^T b b^T  v_1 v_1^T b +  (n-N)(n-N-1)b^T  v_1 v_1^T b b^T  v_2 v_2^T b\\
&= (n-N) (b^T  v_1)^4  +  (n-N)(n-N-1) (b^T  v_1)^2 (b^T  v_2)^2\\
\mathbb{E}[b^T(I- P)b b^T(I- P)b]&= \norm{b}^4 (n-N) \left( \frac{3}{n(n+2)}  +  (n-N-1) \frac{1}{n(n+2)}  \right)\\
&= \norm{b}^4  \frac{(n-N)(n-N+2)}{n(n+2)}.
\end{align*}

The penultimate step uses identities 29 and 30 in Section <ref>.

* $X^+ P = X^+$ and hence $X^+(I-P) = 0.$

* Write $X = U \Sigma V^T$

\begin{align*}
\tr[ (X^T X)^{-1}]b^T(I-P)b & = \left(\tr[(V \Sigma U^T U \Sigma V^T )^{-1}]  \right) \left(b^T (I-U\Sigma V^T (V \Sigma U^T U \Sigma V^T)^{-1} V \Sigma U^T)b\right)\\
&= \left(\sum_{i=1}^{N} \frac{1} {\sigma_i^2} \right) \left( b^T(I-UU^T)b\right).
\end{align*}

Since the elements of $X$ are i.i.d Gaussian random variables, the singular vectors $U$ and $V$
are uniformly distributed on the unit sphere. Because of the rotational invariance of the singular vectors, the columns of
$U$ and $V$  do not depend on the particular values of the singular values $\sigma$, and thus the two bracketed terms above are independent of each other. Thus

\begin{align*}
\mathbb{E}[\tr[ (X^T X)^{-1}]b^T(I-P)b] & = \mathbb{E}[\tr[ (X^T X)^{-1}]] \mathbb{E}[b^T(I-P)b] \\
&= \frac{N}{n-N-1}  \norm{b}^2 \left(1-\frac{N}{n} \right).
\end{align*}

Derivations Thus Far (expressions for $m=1$, $W_0 = 0$)

Method          $\mathbb{E}[\ell]$       $\mathbb{E}[\ell^2]$ w/o noise      $\mathbb{E}[\ell^2]$
$GD$                    $  ||W_1||^2  \frac{n+1}{N}   + \sigma^2 \left(1 +  \frac{n}{N} \right)$         $  \frac{90+48 n+6 n^2}{N^3} + \frac{21+12 n+3 n^2}{N^2} $     $ TBD $
$LS(N<n-1)$          $  ||W_1||^2 \left( 1 - \frac{N}{n} \right)  + \sigma^2 \left( 1 + \frac{N}{n-N-1} \right) $         $ TBD $      $ TBD $
$LS(N > n+3)$     $   \sigma^2 \left( 1 + \frac{n}{N-n-1} \right)  $        $  0 $     $ \sigma^4 \frac{3 (N-1) (N-3)}{(N-n-1) (N-n-3)} $

Figure <ref> shows the different components of $\mathbb{E}[\ell^2]$ for a sample case, comparing the systematic (terms involving $\norm{W_1-W_0}^4$), pure noise (terms involving $\sigma^4$) and 'interactions' (terms involving $\norm{W_1-W_0}^2 \sigma^2$). Even for the un-optimized gradient descent (i.e. $\eta=1$), the implicit regularization appears to be highly beneficial.

$\mathbb{E}[\ell^2]$ for $n=40; m=1$ . Break down shows pure noise, interactive and systematic components for Least Squares and one-shot gradient descent (with $\eta=1$).

§ CONNECTIONS TO EXISTING  WORK & CONCLUSIONS

This study investigated the performance of gradient descent in a in-context linear regression setting, providing qualitative insights and quantitative characterization of the statistical properties of the generalization error. The derived generalization error bounds were contrasted with those from least squares regression, demonstrating that a single step of gradient descent can offer a comparable performance in certain contexts, especially in high noise settings.  An expression for the optimal step size was derived. The  analysis of systematic and noise components offered a comprehensive view of the factors contributing to generalization error.

The fact that gradient descent can provide reasonable results with a single step has implications for reducing computational complexity, especially in one-shot scenarios and resource-constrained environments. Additionally, generalization error bounds were extended to least squares and least norm regression. This work uncovered new identities involving high-order products of Gaussian random matrices, which may have broader applications in regression tasks and beyond. All of the derived results are verified using empirical computations on a large number of samples. It is  intriguing that in much of the literature, probabilistic bounds are often not verified using numerical experiments (indeed, there are clear exceptions, for instance [11, 17, 26]).

Our study, while restricted in scope to well-specified models, addresses some key gaps in the literature: Despite extensive research on convergence and optimality in gradient-based methods, finite sample, non-asymptotic results that do not rely on arbitrary constants are rare in the literature, and in the case of probabilistic bounds, this appears to be the case for linear regression as well. Closely related to our work – though focused more on Transformers directly – Mahankali et al. [15] show that  linear self-attention layers can emulate one step of gradient descent on least-squares regression tasks, and exhibit optimal in-context learning capabilities in synthetic linear regression scenarios. Zhang et al [25] show that transformers emulate gradient descent by achieving global minimums through suitable initialization, enabling competitive prediction errors on new tasks, while being sensitive to covariate shifts. These works are focused on optimality and do not consider bounds.

Existing literature on gradient descent and both linear and non-linear regression is extensive and rigorous, and might prompt questions about the value of a simpler setting of the  present work.  Nevertheless,  clean, finite sample results without arbitrary constants can yield much insight into the behavior of more complex architectures and regression tasks. As a comparable example,  Belkin et al. [3] also consider well-specified linear regression and explain the so-called double-descent phenomenon which challenges classical notions of the bias-variance trade-off. This finding has been reproduced in more complex problems  involving deep neural networks (e.g.  [18]), and thus has led to profound implications for the design and understanding of learning algorithms.

Overall, this study underscores the potential for a single step of gradient descent to generalize effectively in in-context learning scenarios. Future research could extend these findings to more complex regression tasks, including non-linear and incomplete parametrizations, i.e. considering model form errors. Additionally, the implications of these results can be explored on transformer-based architectures and other machine learning algorithms on practical applications.

§ APPENDIX

§.§ Identities

Some of the derivations presented in the manuscript were extremely lengthy. To aid further work, we present a compact set of identities below.  A few of these can be easily derived using the excellent Matrix reference manual [4]. Many of the below expressions, however, require many hours of manipulations. Proofs are also provided for a few of the more complex identities.

All the expressions assume that $x_i \sim \mathcal{N}(0,I_n)$  ;  $Q\triangleq \sum_{i=1}^N x_i x_i^T$ ; $B \in \reals^{n \times n}$ ; $b \in \reals^n$

* $ \mathbb{E}[x x^T  x x^T ] = (2+n) I$

* $\mathbb{E}[\tr[BQ]] = N \tr[B]$

* $ \mathbb{E}[x x^T  x x^T x x^T] = (8+6n+n^2) I$

* $ \mathbb{E}[x x^T B  x x^T ] =  B + B^T + \tr[B] I$

* $ \mathbb{E}[x^T B  x x^T B x] =  \tr[B (B+B^T)] + \tr[B]^2$

* $\mathbb{E}[\tr[Q]\tr[Q]] = N(2n+N n^2)$

* $\mathbb{E}[\tr[QQ]] = N n (N+n+1)$

The following 3 identities [20] assume invertibility of $X X^T$ which is almost surely guaranteed under the present settings for $N>n$. Switch n and N for analogous identities involving the inverse of $X^T X$.

* $\tr[(XX^T)^{-1}] = \frac{n}{N-n-1}$

* $\tr[(XX^T)^{-2}] = \frac{(N-1)n}{(N-n-3)(N-n-1)(N-n)}$

* $\tr[(XX^T)^{-1}]^2 = \frac{n(n(N-n-2)+2)}{(N-n-3)(N-n-1)(N-n)}$

* $\mathbb{E}[x^T x b^T x x^T x x^T b] = b^T b (n^2+6n+8)$

* $\mathbb{E}[(Bx + b)^T(Bx + b) (Bx + b)^T(Bx + b)] = 2\tr(BB^TBB^T) + 4b^TBB^Tb + (tr(BB^T) + b^T b)^2.$

* $\mathbb{E}[\tr[Q]b^TQQb]] = b^T b N (1 + n + N) (4 + n N)$

* $\mathbb{E}[\tr[Q]b^TQb]] = b^T b N (2 + n N) $

* $\mathbb{E}[QQQ] = N (4 + n^2 + 3 n (1 + N) + N (3 + N)) I $

* $\mathbb{E}[\tr[BQQ]] = \tr[B] N(N+n+1)$

* $\mathbb{E}[\tr[Q]b^TQb] = N(2+n N)b^T b$

* $\mathbb{E}[\tr[Q]b^TQ Q b] = N (1+n+N) (4+nN)b^Tb$

* $ \mathbb{E}[x^T B  x b^T x x^T] =  b^T(B+B^T+\tr[B]I)$

The following assume that B is symmetric

* $ \mathbb{E}[x^T B  x x^T B x x^T x] =  (n+4) \tr[B]^2+ (8+2n) \tr[B^2]$

* $ \mathbb{E}[x^T B  x x^T x x^T B x x^T x] =  (n^2+24+10 n) \tr[B]^2+ (2n^2+20n+48) \tr[B^2]$

* $\mathbb{E}[\tr[BQBQ]] = N ((N+1)\tr[B^2]+\tr[B]^2)$

* $\mathbb{E}[\tr[BQBQQ]] = N((4 + n + (3 + n)N + N^2)\tr[B^2] + (2 + n + 2 N) \tr[B]^2)$

* $\mathbb{E}[\tr[BQQBQQ]] = N((20 + n(11 + n) + 21 N + n(7 + n)N + 2(3 + n)N^2 + N^3)\tr[B^2] + (10 + 5n + n^2 + 5(2 + n)N + 4 N^2) \tr[B]^2)$

* $\mathbb{E}[\tr[BQ]\tr[BQ]] = (N \tr[B]^2+2 \tr[B^2])N$

* $\mathbb{E}[\tr[BQ]\tr[BQQ]] = N(2(2 + n + 2N)\tr[B^2] + (2 + N(1 + n + N))\tr[B]^2)$

* $\mathbb{E}[\tr[BQQ]\tr[BQQ]] =N(2(10 + 5n + n^2 + 5(2 + n)N + 4N^2)\tr[B^2] + (n^2 N + 2n(3 + N + N^2) + (1 + N)(10 + N + N^2))\tr[B]^2)$

The following assume that $q_i $ and $q_j$ are n-dimensional orthonormal vectors uniformly distributed on the surface of a sphere.

* $\mathbb{E}[(b^T q_i)^2] = \norm{b}^2 \frac{1}{n}$

* $\mathbb{E}[(b^T q_i)^4] = \norm{b}^4 \frac{3}{n(n+2)}$

* $\mathbb{E}[(b^T q_i)^2 (b^T q_j)^2] = \norm{b}^4 \frac{1}{n(n+2)}$

§.§ Some Proofs

§.§.§ $\mathbb{E}[\tr[BQBQQ]] $

\begin{align*}
\mathbb{E}[BQBQQ] &= \mathbb{E}\left[B \left(\sum_{i=1}^N Q_i \right)B \left(\sum_{j=1}^N Q_j \right) \left(\sum_{k=1}^N Q_k \right)\right]\\
&= N \mathbb{E}\left[B Q_1 B \left(\sum_{j=1}^N Q_j \right) \left(\sum_{k=1}^N Q_k \right)\right] \\
&= N \left( \mathbb{E}\left[B Q_1 B Q_1 Q_1  + (N-1) B Q_1 B Q_2 Q_2 + 2 (N-1)  B Q_1 B Q_1 Q_2 + (N-1)(N-2) B Q_1 B Q_2 Q_3 \right] \right) \\
\tr[\mathbb{E}[BQBQQ] ] &= N \left((n+4) \tr[B]^2+ (8+2n) \tr[B^2]  + (N-1) \tr[B^2] (n+2) + 2 (N-1)  (2 \tr[B^2] + \tr[B]^2) + (N-1)(N-2) \tr[B^2] \right) \\
&=N((4 + n + (3 + n)N + N^2)\tr[B^2] + (2 + n + 2 N) \tr[B]^2)
\end{align*}

§.§.§ $\mathbb{E}[\tr[BQQBQQ]] $

\begin{align*}
\mathbb{E}[BQQBQQ]] &= \mathbb{E}\left[B \left(\sum_{i=1}^N Q_i \right) \left(\sum_{j=1}^N Q_j \right)B \left(\sum_{k=1}^N Q_k \right) \left(\sum_{l=1}^N Q_l \right)\right]\\
&= N \mathbb{E}\left[B \left(\sum_{i=1}^N Q_i \right) \left(\sum_{j=1}^N Q_j \right)B \left(\sum_{k=1}^N Q_k \right)  Q_1 \right]
\end{align*}

This requires quartic products of outer products of Gaussians. To track the permutations, we take N=4 and represent k slices of the i,j interactions. The rows of the slices represent $Q_i$ and columns represent $Q_j$. The color coding distinguishes the eight unique interactions.

red!25 I     III     III     III

III     blue!25II     green!25IV    green!25IV

III     green!25IV     blue!25 II     green!25IV

III     green!25IV     green!25IV     blue!25II

III    	black!25  V     	red!55 VI     	red!55 VI

orange!50 	VII     III    	red!55 VI     	red!55 VI

red!55	VI    	red!55 VI    green!25 IV     yellow!50VIII

red!55 	VI     	red!55 VI     yellow!50VIII     green!25IV

III    	red!55  VI     	black!25 V     	red!55 VI

red!55 	VI     green!25 IV    	red!55 VI     	yellow!50 VIII

orange!50	VII    	red!55 VI     III     red!55VI

red!55 	VI     	yellow!50 VIII     red!55VI     green!25IV

III      	red!55 VI     	red!55 VI     		black!25 V

red!55 	VI     green!25 IV    		yellow!50 VIII     	red!55 VI

red!55 	VI   	yellow!50 VIII     green!25 IV      red!55VI

orange!50	VII    	red!55 	VI      red!55VI      III

\begin{align*}
\tr[\mathbb{E}[BQQBQQ]] &= N \tr\left[\mathbb{E}\left[B \left(\sum_{i=1}^N Q_i \right) \left(\sum_{j=1}^N Q_j \right)B \left(\sum_{k=1}^N Q_k \right) Q_1 \right]\right]\\
&= N \tr[ \mathbb{E}\left[B Q_1 Q_1 B Q_1  Q_1 \right]] \\
&+N (N-1)  \tr[\mathbb{E}\left[B Q_2 Q_2 B Q_1  Q_1 \right]] \\
&+4 N (N-1)  \tr[\mathbb{E}\left[B Q_1 Q_2 B Q_1  Q_1 \right]] \\
&+2 N (N-1) (N-2)  \tr[\mathbb{E}\left[B Q_2 Q_3 B Q_1  Q_1 \right]] \\
&+N (N-1)  \tr[\mathbb{E}\left[B Q_1 Q_2 B Q_2  Q_1 \right]] \\
&+4 N (N-1)(N-2) \tr[\mathbb{E}\left[B Q_1 Q_3 B Q_2  Q_1 \right]] \\
&+N (N-1) \tr[\mathbb{E}\left[B Q_2 Q_1 B Q_2  Q_1 \right]] \\
&+N (N-1)(N-2)(N-3) \tr[\mathbb{E}\left[B Q_4 Q_3 B Q_2  Q_1 \right]] \\
&= N ((n^2+24+10 n) \tr[B]^2+ (2n^2+20n+48) \tr[B^2])\\
&+N (N-1)  \tr[B^2](2+n)^2 \\
&+4 N (N-1)  ((n+4) \tr[B]^2+ (8+2n) \tr[B^2])\\
&+2 N (N-1) (N-2) \tr[B^2](2+n) \\
&+N (N-1)  (4 \tr[B^2]+\tr[B]^2(4+n)) \\
&+4 N (N-1)(N-2)(2 \tr[B^2] + \tr[B]^2) \\
&+N (N-1) (\tr[B^2] (6 + n) + 2 \tr[B]^2 )\\
&+N (N-1)(N-2)(N-3) \tr[B^2].
\end{align*}

§.§.§ $\mathbb{E}[\tr[BQQ]\tr[BQ]]$

\begin{align*}
&=N \mathbb{E}[(x_1^T BQ x_1+x_2^T B Q x_2+...+x_N^T BQ x_N)x_1^T B x_1] \\
&= N \mathbb{E}[(x_1^T B (x_1 x_1^T+x_2 x_2^T+...+x_N x_N^T) x_1 x_1^T B x_1]\\
&+ N \mathbb{E}[(x_2^T B (x_1 x_1^T+x_2 x_2^T+...+x_N x_N^T) x_2 x_1^T B x_1]\\
&+ .... \\
&+ N \mathbb{E}[(x_N^T B (x_1 x_1^T+x_2 x_2^T+...+x_N x_N^T) x_N x_1^T B x_1]\\
&= N \mathbb{E}[(x_1^T B (x_1 x_1^T+x_2 x_2^T+...+x_N x_N^T) x_1 x_1^T B x_1]\\
&+ N (N-1) \mathbb{E}[(x_2^T B (x_1 x_1^T+x_2 x_2^T+...+x_N x_N^T) x_2 x_1^T B x_1]\\
&= N \mathbb{E}[(x_1^T B (x_1 x_1^T+x_2 x_2^T+...+x_N x_N^T) x_1 x_1^T B x_1]\\
&+ N (N-1) \mathbb{E}[x_2^T B x_1 x_1^T x_2 x_1^T B x_1] \\
&+ N (N-1) \mathbb{E}[x_2^T B x_2 x_2^T x_2 x_1^T B x_1] \\
&+ N (N-1) (N-2) \mathbb{E}[x_2^T B x_3 x_3^T x_2 x_1^T B x_1] \\
&=N ((n+4) \tr[B]^2+ (8+2n) \tr[B^2] + (N-1)(2\tr[B^2]+\tr[B]^2)) \\
&+ N (N-1) (2 \tr[B^2]+\tr[B]^2)\\
&+ N (N-1) (2+n)\tr[B]^2\\
&+ N (N-1)(N-2) \tr[B]^2
\end{align*}

§.§.§ $\mathbb{E}[\tr[BQQ]\tr[BQQ]]$

This involves terms of the form
\mathbb{E}[x_j^T B x_i x_i^T x_j x_1^T B x_k x_k^T x_1]

Term I

\begin{align*}
\mathbb{E}[x_1^T B x_1 x_1^T x_1 x_1^T B x_1 x_1^T x_1] = (n^2+24+10 n) \tr[B]^2+ (2n^2+20n+48) \tr[B^2]
\end{align*}

Term II

\begin{align*}
\mathbb{E}[x_2^T B x_2 x_2^T x_2 x_1^T B x_1 x_1^T x_1] = \tr[B]^2(n+2)^2
\end{align*}

Term III

\begin{align*}
\mathbb{E}[x_2^T B x_1 x_1^T x_2 x_1^T B x_1 x_1^T x_1] = \mathbb{E}[  x_1^T B x_1 x_1^T B x_1 x_1^T x_1]  = (n+4) \tr[B]^2+ (8+2n) \tr[B^2]
\end{align*}

Term IV

\begin{align*}
\mathbb{E}[x_3^T B x_2 x_2^T x_3 x_1^T B x_1 x_1^T x_1] = \mathbb{E}[x_2^T B x_2  x_1^T B x_1 x_1^T x_1] = \tr[B]^2(2+n)
\end{align*}

Term V

\begin{align*}
\mathbb{E}[x_2^T B x_1 x_1^T x_2 x_1^T B x_2 x_2^T x_1]& =\mathbb{E}[x_2^T C x_2 a^T x_2 x_2^T b] = \mathbb{E}[a^T (C+C^T)b + a^T \tr[C] b] \\
&= \mathbb{E}[x_1^T B (B x_1 x_1^T + x_1 x_1^T B)x_1 + x_1^T B \tr[B x_1 x_1^T]x_1]\\
&= \mathbb{E}[x_1^T B^2  x_1 x_1^T x_1 + 2 x_1^T B x_1  x_1^T B x_1]\\
&= \tr[B^2](2+n) + 4 \tr[B^2] + 2 \tr[B]^2
\end{align*}

Term VI

\begin{align*}
\mathbb{E}[x_3^T B x_1 x_1^T x_3 x_1^T B x_2 x_2^T x_1] = \mathbb{E}[x_3^T B x_1 x_1^T x_3 x_1^T B  x_1]  = \mathbb{E}[x_1^T B x_1 x_1^T B  x_1] = 2 \tr[B^2] + \tr[B]^2
\end{align*}

Term VII

\begin{align*}
\mathbb{E}[x_1^T B x_2 x_2^T x_1 x_1^T B x_2 x_2^T x_1]& = \mathbb{E}[\tr[B x_2 x_2^T (B x_2 x_2^T+ x_2 x_2^T B)] + \tr[B x_2 x_2^T]^2 ] \\
& = \mathbb{E}[\tr[B x_2 x_2^T (B x_2 x_2^T+ x_2 x_2^T B)] + \tr[B x_2 x_2^T]^2 ] \\
& = 2 \mathbb{E}[x_2^T B x_2 x_2^T B x_2 ] + \mathbb{E}[x_2^T x_2 x_2^T B^2 x_2] \\
& = 4 \tr[B^2] + 2 \tr[B]^2 + (2+n)\tr[B^2] \\
\end{align*}

Term VIII

\begin{align*}
\mathbb{E}[x_4^T B x_3 x_3^T x_4 x_1^T B x_2 x_2^T x_1] &= \mathbb{E}[x_4^T B  x_4 x_1^T B x_1]  = \tr[B]^2
\end{align*}

§.§.§ $\mathbb{E}[\tr[Q]b^TQQb]] $

\begin{align*}
&= N \mathbb{E}\left[ x_1^T x_1 b^T \left(\sum_{j=1}^N Q_j \right) \left(\sum_{k=1}^N Q_k \right)b\right] \\
&= N \left( \mathbb{E}\left[x_1^T x_1 b^T x_1 x_1^T x_1 x_1^T b  + (N-1) x_1^T x_1 b^T x_2 x_2^T x_2 x_2^T b + 2 (N-1)  x_1^T x_1 b^T x_1 x_1^T x_2 x_2^T b + (N-1)(N-2) x_1^T x_1 b^T x_2 x_2^T x_3 x_3^T b \right] \right) \\
&= N b^T b \left( (n^2+6n+8)+ (N-1)(n(2+n)) + 2 (N-1)(2+n)   + (N-1)(N-2)n  \right) \\
&= b^T b N (1 + n + N) (4 + n N)
\end{align*}

[1]
J.-Y. Audibert and O. Catoni, Linear regression through pac-bayesian
truncation, arXiv preprint arXiv:1010.0072,  (2010).

[2]
M. Belkin, D. Hsu, S. Ma, and S. Mandal, Reconciling modern
machine-learning practice and the classical bias–variance trade-off,
Proceedings of the National Academy of Sciences, 116 (2019),
pp. 15849–15854.

[3]
M. Belkin, D. Hsu, and J. Xu, Two models of double descent for weak
features, SIAM Journal on Mathematics of Data Science, 2 (2020),
pp. 1167–1180.

[4]
M. Brookes, The matrix reference manual,
http://www.ee.imperial.ac.uk/hp/staff/dmb/matrix/intro.html,  (2020).

[5]
O. Catoni, Statistical learning theory and stochastic optimization:
Ecole d'Eté de Probabilités de Saint-Flour, XXXI-2001, vol. 1851,
Springer Science & Business Media, 2004.

[6]
D. Dai, Y. Sun, L. Dong, Y. Hao, S. Ma, Z. Sui, and F. Wei, Why can
gpt learn in-context? language models implicitly perform gradient descent as
meta-optimizers, arXiv preprint arXiv:2212.10559,  (2022).

[7]
K. Duraisamy, Basic'generalization error bounds for least squares
regression with well-specified models, arXiv preprint arXiv:2109.09647,

[8]
S. Garg, D. Tsipras, P. S. Liang, and G. Valiant, What can
transformers learn in-context? a case study of simple function classes,
Advances in Neural Information Processing Systems, 35 (2022),
pp. 30583–30598.

[9]
P. Germain, F. Bach, A. Lacoste, and S. Lacoste-Julien, Pac-bayesian
theory meets bayesian inference, arXiv preprint arXiv:1605.08636,  (2016).

[10]
L. Györfi, M. Kohler, A. Krzyzak, H. Walk, et al., A
distribution-free theory of nonparametric regression, vol. 1, Springer,

[11]
T. Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani, Surprises
in high-dimensional ridgeless least squares interpolation, Annals of
statistics, 50 (2022), p. 949.

[12]
D. Hsu, S. M. Kakade, and T. Zhang, Random design analysis of ridge
regression, in Conference on learning theory, JMLR Workshop and Conference
Proceedings, 2012.

[13]
T. Hu, Q. Wu, and D.-X. Zhou, Convergence of gradient descent for
minimum error entropy principle in linear regression, IEEE Transactions on
Signal Processing, 64 (2016), pp. 6571–6579.

[14]
S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, Transformers in vision: A survey, ACM computing surveys (CSUR), 54 (2022),
pp. 1–41.

[15]
A. Mahankali, T. B. Hashimoto, and T. Ma, One step of gradient
descent is provably the optimal in-context learner with one layer of linear
self-attention, arXiv preprint arXiv:2307.03576,  (2023).

[16]
D. A. McAllester, Pac-bayesian model averaging, in Proceedings of
the twelfth annual conference on Computational learning theory, 1999,
pp. 164–170.

[17]
S. Mei and A. Montanari, The generalization error of random features
regression: Precise asymptotics and the double descent curve, Communications
on Pure and Applied Mathematics,  (2019).

[18]
P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever,
Deep double descent: Where bigger models and more data hurt, Journal of
Statistical Mechanics: Theory and Experiment, 2021 (2021), p. 124003.

[19]
G. Neu, G. K. Dziugaite, M. Haghifam, and D. M. Roy, Information-theoretic generalization bounds for stochastic gradient descent,
in Conference on Learning Theory, PMLR, 2021, pp. 3526–3545.

[20]
J. Pielaszkiewicz and T. Holgersson, Mixtures of traces of wishart
and inverse wishart matrices, Communications in Statistics-Theory and
Methods,  (2019), pp. 1–17.

[21]
A. Rakhlin, O. Shamir, and K. Sridharan, Making gradient descent
optimal for strongly convex stochastic optimization, arXiv preprint
arXiv:1109.5647,  (2011).

[22]
A. Shapiro and Y. Wardi, Convergence analysis of gradient descent
stochastic algorithms, Journal of optimization theory and applications, 91
(1996), pp. 439–454.

[23]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
Ł. Kaiser, and I. Polosukhin, Attention is all you need, Advances
in neural information processing systems, 30 (2017).

[24]
J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev,
A. Zhmoginov, and M. Vladymyrov, Transformers learn in-context by
gradient descent, in International Conference on Machine Learning, PMLR,
2023, pp. 35151–35174.

[25]
R. Zhang, S. Frei, and P. L. Bartlett, Trained transformers learn
linear models in-context, arXiv preprint arXiv:2306.09927,  (2023).

[26]
L. Zhou, F. Koehler, D. J. Sutherland, and N. Srebro, Optimistic
rates: A unifying theory for interpolation learning and regularization in
linear regression, ACM/JMS Journal of Data Science, 1 (2024), pp. 1–51.