# Higher order concentration on Stiefel and Grassmann manifolds

Friedrich Götze  Friedrich Götze, Faculty of Mathematics, Bielefeld
University, Germany<EMAIL_ADDRESS>and  Holger Sambale  Holger
Sambale, Faculty of Mathematics, Ruhr University Bochum, Germany
<EMAIL_ADDRESS>
(Date: August 28, 2024)

###### Abstract.

We prove higher order concentration bounds for functions on Stiefel and
Grassmann manifolds equipped with the uniform distribution. This partially
extends previous work for functions on the unit sphere. Technically, our
results are based on logarithmic Sobolev techniques for the uniform measures
on the manifolds. Applications include Hanson–Wright type inequalities for
Stiefel manifolds and concentration bounds for certain distance functions
between subspaces of $\mathbb{R}^{n}$.

###### Key words and phrases:

Concentration of measure phenomenon, Stiefel manifold, Grassmann manifold,
logarithmic Sobolev inequality, Hanson-Wright inequality

###### 1991 Mathematics Subject Classification:

Primary 60E15, Secondary 28C10, 58C35

This research was supported by the German Research Foundation via CRC 1283.

##  1\. Introduction

In recent years, functions on the Stiefel manifold (i. e., the collection of
all $d$-tuples of orthonormal vectors in $\mathbb{R}^{n}$) and on the closely
related Grassmann manifold (the collection of all $d$-dimensional linear
subspaces of $\mathbb{R}^{n}$) have attracted increasing attention from
various fields of research. A central reason is that they admit a wealth of
applications in various directions like data analysis, subspace estimation,
computer vision or statistical learning. In addition, they have also been
studied from more theoretical points of view, in particular in convex geometry
but quite recently also in large deviations theory. For a selection of various
results, see e. g. [TVC08, HL08, WLT11, ZZJH18, LLY20] (data analysis) and
[LPT06, GKZ21, KP21] (including large deviation principles).

A frequent question which naturally arises in many applications is how to
control the fluctuations of some (Stiefel or Grassmann) functional around a
typical value like its expectation, i. e., one asks for suitable concentration
of measure results. Often times, the functional is of Lipschitz-type, which in
classical situations (e. g., functions of independent sub-Gaussian random
variables or in presence of a log-Sobolev inequality) leads to sub-Gaussian
tail bounds. For a brief review especially adapted to Stiefel and Grassmann
manifolds, cf. Section 1.2 below. However, this approach fails if the
functionals under consideration are no longer Lipschitz. Still, one may hope
for useful concentration bounds if they are Lipschitz of a “higher order”,
which typically means that their higher order derivatives are absolutely
bounded. A classical example for order two are quadratic forms. In this
situation, various questions of interest include exponential moment bounds,
refined tail bounds with various levels of decay and centering around non-
deterministic quantities which may correspond to a suitable decomposition.

In this note, we provide higher order concentration results for Stiefel and
Grassmann manifolds equipped with the uniform distribution. Higher order
concentration has been studied in various settings in the past two decades, e.
g. in [La06] (polynomials in independent Gaussian variables) [AW15] (measures
satisfying certain Sobolev-type inequalities) as well as [KV00, SS12, AL12,
KL15, GSS21a, GSS21b]. In this paper, we especially continue the line of
research begun in [BCG17, GSS19], where second and higher order concentration
results for the unit sphere were established (noting that the unit sphere can
be understood as a Stiefel manifold).

###  1.1. Grassmann and Stiefel manifolds

For any natural numbers $d\leq n$, the Stiefel manifold $W_{n,d}$ is the set
of all $d$-tupels of orthonormal vectors in $\mathbb{R}^{n}$. Clearly,
$W_{n,d}$ may be written as

$W_{n,d}=\\{A\in\mathbb{R}^{n\times d}\colon A^{T}A=I_{d}\\},$

where $I_{d}\in\mathbb{R}^{d\times d}$ denotes the identity matrix. This is
the representation we shall use throughout this note. $W_{n,d}$ is manifold of
dimension $\mathrm{dim}(W_{n,d})=nd-d(d+1)/2$. Obviously, $W_{n,1}=S^{n-1}$
(the unit sphere), $W_{n,n}=O(n)$ (the orthogonal group), and $W_{n,n-1}$ can
be identified with the special orthogonal group $SO(n)=\\{O\in
O(n)\colon\mathrm{det}(O)=1\\}$.

We may equip $W_{n,d}$ with the subspace topology and distances inherited from
$\mathbb{R}^{n\times d}$, including the scalar product on $\mathbb{R}^{n\times
d}$ and the induced (Hilbert–Schmidt) norm

$\langle A,B\rangle:=\mathrm{tr}(A^{T}B),\qquad\lvert A\rvert^{2}\equiv\lVert
A\rVert_{\mathrm{HS}}^{2}=\mathrm{tr}(A^{T}A)=\sum_{i,j}A_{ij}^{2}.$

In particular, $W_{n,d}$ is a compact topological space. The product of the
orthogonal groups $O(n)\times O(d)$ acts transitively on $W_{n,d}$ by the two-
sided multiplication $O_{n}\times O_{d}\mapsto O_{n}AO_{d}$, turning $W_{n,d}$
into a homogeneous space. Hence, it may be equipped with a unique invariant
(Haar) probability measure $\mu_{n,d}$ in the sense that if $A\sim\mu_{n,d}$
(i. e., $A$ has distribution $\mu_{n,d}$), $O_{n}AO_{d}\sim\mu_{n,d}$ for any
$O_{n}\in O(n)$, $O_{d}\in O(d)$. We call $\mu_{n,d}$ the uniform distribution
on $W_{n,d}$. If $G=(G_{ij})_{i,j=1}^{d,n}$ is an $n\times d$ random matrix
whose entries are i.i.d. standard normal, then
$G(G^{T}G)^{-1/2}\sim\mu_{n,d}$, see [KPT20, Lemma 3.1].

There are several ways of introducing the closely related Grassmann manifold,
i. e. the set of all $d$-dimensional subspaces of $\mathbb{R}^{n}$. For an
overview, cf. e. g. [BZA20]. For instance, identifying a subspace with its
basis (which is unique up to orthogonal transformations), we may understand
the Grassmann manifold as the quotient $W_{n,d}/O(d)$, identifying any two
$X,X^{\prime}\in W_{n,d}$ such that $X^{\prime}=XO$ for some $O\in O(d)$.
However, the calculus of (higher order) derivatives on these equivalence
classes turns out to be troublesome, and therefore, we choose a different
approach.

For any $A\in W_{n,d}$, $\pi_{\mathrm{St}}(A):=AA^{T}\equiv P_{A}$ is a
projection matrix of rank $d$ (more precisely, the projection onto the
subspace a basis of which is given by the columns of $A$), and for any
$O_{d}\in O(d)$, we have $P_{AO_{d}}=P_{A}$. Therefore, identifying elements
of the Grassmannian with projection matrices, we may define

$G_{n,d}:=\\{P\in\mathbb{R}^{n\times n}_{\mathrm{sym}}\colon P^{2}=P,\
\mathrm{rank}(P)=d\\},$

where $\mathbb{R}^{n\times n}_{\mathrm{sym}}$ denotes the space of the
symmetric $n\times n$ matrices. $G_{n,d}$ is a manifold of dimension
$\mathrm{dim}(G_{n,d})=d(n-d)$. If $d=1$, we get back the half-sphere (where
$\theta$ and $-\theta$ are identified), which can also be regarded as the
projective space $\mathbb{R}P^{n-1}$. Depending on the situation, we will
either regard $G_{n,d}$ as a submanifold of $\mathbb{R}^{n\times
n}_{\mathrm{sym}}$ or $\mathbb{R}^{n\times n}$ (the latter is often more
convenient when taking derivatives), equipping it with the inherited topology
and distances similarly to the case of the Stiefel manifold (in particular,
$G_{n,d}$ is compact).

Many of the properties of Stiefel manifolds can be extended to Grassmann
manifolds. Especially, the group action $O(n)\ni O_{n}\mapsto P_{O_{n}A}$
turns $G_{n,d}$ into a symmetric (not just homogeneous) space. We denote the
uniform distribution on $G_{n,d}$ by $\nu_{n,d}$. Clearly, $\nu_{n,d}$ is the
pushforward of $\mu_{n,d}$ under the map $\pi_{\mathrm{St}}$. If
$G=(G_{ij})_{i,j=1}^{d,n}$ is an $n\times d$ random matrix whose entries are
i.i.d. standard normal, it follows from the discussion above that
$G(G^{T}G)^{-1}G^{T}\sim\nu_{n,d}$.

###  1.2. Concentration results

If $f\colon W_{n,d}\to\mathbb{R}$ is $L$-Lipschitz with mean $\mu_{n,d}(f)$
with respect to $\mu_{n,d}$, a standard concentration result (cf. e. g. [Le01,
p. 27]) yields

(1.1) $\mu_{n,d}(f-\mu_{n,d}(f)\geq t)\leq\exp(-(n-1)t^{2}/(8L^{2})).$

By switching to the pushforward, the same result also holds for
$(G_{n,d},\nu_{n,d})$ with the constant $8$ replaced by $16$ (cf. Section 6
for details, in particular Lemma 6.1).

The aim of this note is to extend results of type (1.1) to higher orders,
involving $\mathcal{C}^{k}$ functions and derivatives up to order $k$. Here, a
$\mathcal{C}^{k}$ function on $W_{n,d}$ (likewise, $G_{n,d}$) may be
understood as a function which admits an extension to some open neighborhood
of the manifold which is $\mathcal{C}^{k}$-smooth. Let us first fix some
notation. If $A=(a_{i_{1}\ldots i_{k}})\in\mathbb{R}^{m^{k}}$ is any matrix or
tensor of order $k$, we write

$\displaystyle\lVert A\rVert_{\mathrm{HS}}$
$\displaystyle:=\Big{(}\sum_{i_{1},\ldots,i_{k}}a_{i_{1}\ldots
i_{k}}^{2}\Big{)}^{1/2},$ $\displaystyle\lVert A\rVert_{\mathrm{op}}$
$\displaystyle:=\sup\Big{\\{}\sum_{i_{1},\ldots,i_{k}}a_{i_{1}\ldots
i_{k}}x^{(1)}_{i_{1}}\cdots x^{(k)}_{i_{k}}\colon x^{(j)}\in S^{m-1}\ \forall
j\Big{\\}}$

for the respective Hilbert–Schmidt and operator type norms. For any function
$g\colon W_{n,d}\to\mathbb{R}$ and any $p\geq 1$, we denote by $\lVert
g\rVert_{p}$ the $L^{p}$ norm of $g$ with respect to $\mu_{n,d}$. If $g$ is
matrix-valued (e. g., some tensor of derivatives), we use the short-hand
notation

$\lVert g\rVert_{\mathrm{HS},p}:=\lVert\lVert
g\rVert_{\mathrm{HS}}\rVert_{p}=\Big{(}\int_{W_{n,d}}\lVert
g\rVert_{\mathrm{HS}}^{p}d\mu_{n,d}\Big{)}^{1/p}$

and similarly $\lVert g\rVert_{\mathrm{op},p}$.

In Section 3, we will introduce a notion of differentiability for functions on
$W_{n,d}$. In particular, using the intrinsic (Stiefel) gradient
$\nabla_{W}f\in\mathbb{R}^{n\times d}$ and assuming $f\colon
W_{n,d}\to\mathbb{R}$ to be $\mathcal{C}^{1}$-smooth, we may reformulate (1.1)
as

(1.2) $\mu_{n,d}(f-\mu_{n,d}(f)\geq
t)\leq\exp\Big{(}-\frac{(n-1)t^{2}}{8\lVert\nabla_{W}f\rVert_{\mathrm{HS},\infty}^{2}}\Big{)}.$

The same result holds for $\mathcal{C}^{1}$-smooth functions on
$(G_{n,d},\nu_{n,d})$, involving the Grassmann gradient
$\nabla_{G}f\in\mathbb{R}^{n\times n}_{\mathrm{sym}}$ and with $8$ replaced by
$16$.

Our first theorem complements (1.2) by a second order concentration bound for
functions on Stiefel manifolds of any order $d\leq n-1$ in the spirit of
[BCG17] (cf. also [SS21], where this approach to second order concentration
has been generalized and adapted to a wealth of different situations). In
addition to the intrinsic gradient, this also involves an intrinsic Hessian
$f^{\prime\prime}_{W}\in\mathbb{R}^{nd\times nd}$, a notion to be made precise
in Section 3 as well.

######  Theorem 1.1.

Let $f\colon W_{n,d}\to\mathbb{R}$ be a $\mathcal{C}^{2}$-smooth function such
that $\mu_{n,d}(f)=0$.

  1. (1)

Assuming $\lVert\nabla_{W}f\rVert_{\mathrm{HS},2}\leq 2/\sqrt{n-2}$ and
$\lVert f^{\prime\prime}_{W}\rVert_{\mathrm{op},\infty}\leq 1$, we have

$\int_{W_{n,d}}\exp\Big{(}\frac{n-2}{32e}|f|\Big{)}d\mu_{n,d}\leq 2.$

  2. (2)

For $C=16e^{2}/\log(2)$ and any $t\geq 0$,

$\mu_{n,d}(|f-\mu_{n,d}(f)|\geq t)\leq
2\exp\Big{(}-\frac{n-2}{C}\min\Big{(}\frac{t^{2}}{\lVert\nabla_{W}f\rVert_{\mathrm{HS},2}^{2}},\frac{t}{\lVert
f_{W}^{\prime\prime}\rVert_{\mathrm{op},\infty}}\Big{)}\Big{)}.$

  3. (3)

If $\mu_{n,d}(\nabla_{W}f)=0$, the bounds in (1) and (2) continue to hold with
$\lVert\nabla_{W}f\rVert_{\mathrm{HS},2}$ replaced by $\sqrt{8/(n-2-8d)}\lVert
f_{W}^{\prime\prime}\rVert_{\mathrm{HS},2}$.

Note that in (3), the integral $\mu_{n,d}(\nabla_{W}f)$ has to be understood
componentwise. In fact, the condition in (3) can also be modified to $\lVert
f_{W}^{\prime\prime}\rVert_{\mathrm{HS},2}\leq b$ similarly as in [BCG17,
Theorem 1.1] (in particular, arriving at $b$-dependent bounds). We skip the
details. With only minor modifications, these results also hold for functions
on Grassmann manifolds:

######  Theorem 1.2.

Let $f\colon G_{n,d}\to\mathbb{R}$ be a $\mathcal{C}^{2}$-smooth function such
that $\nu_{n,d}(f)=0$.

  1. (1)

Assuming $\lVert\nabla_{G}f\rVert_{\mathrm{HS},2}\leq\sqrt{8/(n-2)}$ and
$\lVert f^{\prime\prime}_{G}\rVert_{\mathrm{op},\infty}\leq 1$, we have

$\int_{G_{n,d}}\exp\Big{(}\frac{n-2}{64e}|f|\Big{)}d\mu_{n,d}\leq 2.$

  2. (2)

For $C=32e^{2}/\log(2)$ and any $t\geq 0$,

$\nu_{n,d}(|f-\mu_{n,d}(f)|\geq t)\leq
2\exp\Big{(}-\frac{n-2}{C}\min\Big{(}\frac{t^{2}}{\lVert\nabla_{G}f\rVert_{\mathrm{HS},2}^{2}},\frac{t}{\lVert
f_{G}^{\prime\prime}\rVert_{\mathrm{op},\infty}}\Big{)}\Big{)}.$

  3. (3)

If $\nu_{n,d}(\nabla_{G}f)=0$, the bounds in (1) and (2) continue to hold with
$\lVert\nabla_{G}f\rVert_{\mathrm{HS},2}$ replaced by
$\sqrt{16/(n-2-16d)}\lVert f_{G}^{\prime\prime}\rVert_{\mathrm{HS},2}$.

In general, once we have a function $f$ which is defined and smooth in a
neighborhood of $W_{n,d}$ (which may be achieved by choosing a suitable
extension of $f$), we may also formulate concentration of measure results
involving the usual (Euclidean) derivatives only. The same holds for functions
defined in a neighborhood of $G_{n,d}$ in $\mathbb{R}^{n\times
n}_{\mathrm{sym}}$ or $\mathbb{R}^{n\times n}$.

As shown in Sections 3 and 5, we always have
$|\nabla_{W}f|,|\nabla_{G}f|\leq|\nabla f|$. In particular, results depending
on Euclidean derivatives are somewhat less accurate than bounds involving
intrinsic derivatives. On the other hand, the underlying calculus is typically
much less involved, especially as the order of the derivatives increases, and
in many situations, the usual derivatives may already be sufficient for
meaningful concentration bounds. Here, we have the following result, which
complements [BGS19].

######  Theorem 1.3.

Let $f$ be a real-valued $\mathcal{C}^{k}$-smooth function defined in some
neighborhood of $W_{n,d}$ such that $\mu_{n,d}(f)=0$.

  1. (1)

Assuming $\lVert f^{(\ell)}\rVert_{\mathrm{op},2}\leq(4/(n-2))^{(k-\ell)/2}$
for any $\ell=1,\ldots,k-1$ and $\lVert
f^{(k)}\rVert_{\mathrm{op},\infty}\linebreak[2]\leq 1$, we have

$\int_{W_{n,d}}\exp\Big{(}\frac{n-2}{32e}|f|^{2/k}\Big{)}d\mu_{n,d}\leq 2.$

  2. (2)

For $C=4e^{2}/\log(2)$ and any $t\geq 0$,

$\mu_{n,d}(|f|\geq t)\leq
2\exp\Big{(}-\frac{n-2}{Ck^{2}}\min\Big{(}\min_{\ell=1,\ldots,k-1}\frac{t^{2/\ell}}{\lVert
f^{(\ell)}\rVert_{\mathrm{op},2}^{2/\ell}},\frac{t^{2/k}}{\lVert
f^{(k)}\rVert_{\mathrm{op},\infty}^{2/k}}\Big{)}\Big{)}.$

The Grassmann version of this theorem reads as follows:

######  Theorem 1.4.

Let $f$ be a real-valued $\mathcal{C}^{k}$-smooth function defined in some
neighborhood of $G_{n,d}$ such that $\nu_{n,d}(f)=0$.

  1. (1)

Assuming $\lVert f^{(\ell)}\rVert_{\mathrm{op},2}\leq(8/(n-2))^{(k-\ell)/2}$
for any $\ell=1,\ldots,k-1$ and $\lVert
f^{(k)}\rVert_{\mathrm{op},\infty}\linebreak[2]\leq 1$, we have

$\int_{G_{n,d}}\exp\Big{(}\frac{n-2}{64e}|f|^{2/k}\Big{)}d\nu_{n,d}\leq 2.$

  2. (2)

For $C=8e^{2}/\log(2)$ and any $t\geq 0$,

$\nu_{n,d}(|f|\geq t)\leq
2\exp\Big{(}-\frac{n-2}{Ck^{2}}\min\Big{(}\min_{\ell=1,\ldots,k-1}\frac{t^{2/\ell}}{\lVert
f^{(\ell)}\rVert_{\mathrm{op},2}^{2/\ell}},\frac{t^{2/k}}{\lVert
f^{(k)}\rVert_{\mathrm{op},\infty}^{2/k}}\Big{)}\Big{)}.$

Examples where these results are applied to concrete situations and are
briefly compared to related bounds known from the literature are given in
Section 7.

###  1.3. Overview

In view of the parametrizations of the manifolds via matrices, we first
provide a brief review on some basic facts about derivatives in matrix
arguments and related topics in Section 2. First and second order intrinsic
derivatives for functions on Stiefel manifolds are introduced and discussed in
Sections 3 and 4, including a number of inequalities relating first and second
order. These results are adapted to Grassmann manifolds in Section 5. The
proofs of our main results are given in Section 6. Applications to polynomial
chaos (in particular of order $2$), including concentration of the distance to
a given subspace, are provided in Section 7.

##  2\. Basic facts about matrix calculus

The aim of this section is to fix some notation and to collect some basic
facts about matrix calculus mostly for the sake of reference. Most of the
results are elementary and will be stated without proofs.

If $A=(A_{ij})$ is an $n\times m$ matrix (i. e., $A\in\mathbb{R}^{n\times
m}$), we may vectorize it by setting

$\mathrm{vec}(A):=(A_{11},\ldots,A_{n1},A_{12},\ldots,A_{n2},\ldots,A_{1m},\ldots,A_{nm})^{T},$

i. e. $\mathrm{vec}(A)$ is the vector in $\mathbb{R}^{nm}$ with $m$ $n$-blocks
corresponding to the columns of $A$. Occasionally, we will also need the
inverse operation of $\mathrm{vec}$, which we denote by $\mathrm{mat}$. For
$B\in\mathbb{R}^{nd\times nd}$ and $U,V\in\mathbb{R}^{n\times d}$, we
introduce the short-hand notation

(2.1) $BV:=B\mathrm{vec}(V),\qquad\langle BV,U\rangle:=\langle
B\mathrm{vec}(V),\mathrm{vec}(U)\rangle.$

There is an $nm\times nm$ permutation matrix $K_{n,m}$ such that for any
$A\in\mathbb{R}^{n\times m}$,

(2.2) $\mathrm{vec}(A^{T})=K_{n,m}\mathrm{vec}(A).$

$K_{n,m}$ is uniquely defined by (2.2) and is called the $(n,m)$ commutation
matrix. Note that $K_{n,m}^{T}=K_{m,n}$.

Vectorization is closely related to the Kronecker product $A\otimes B$ for any
$A\in\mathbb{R}^{n\times m}$ and any $B\in\mathbb{R}^{p\times q}$. The
following lemma lists some of its elementary properties.

######  Lemma 2.1.

  1. (1)

The Kronecker product is bilinear and associative.

  2. (2)

We have $(A\otimes B)^{T}=A^{T}\otimes B^{T}$.

  3. (3)

$A\otimes B$ is invertible whenever $A$ and $B$ are invertible, and in this
case, $(A\otimes B)^{-1}=A^{-1}\otimes B^{-1}$.

  4. (4)

For any $A\in\mathbb{R}^{n\times m},B\in\mathbb{R}^{p\times
q},C\in\mathbb{R}^{m\times r},D\in\mathbb{R}^{q\times s}$, we have $(A\otimes
B)(C\otimes D)=(AC)\otimes(BD)$.

  5. (5)

For any $A\in\mathbb{R}^{n\times m},B\in\mathbb{R}^{m\times p}$, we have
$\mathrm{vec}(AB)=(I_{p}\otimes A)\mathrm{vec}(B)=(B^{T}\otimes
I_{n})\mathrm{vec}(A)$.

  6. (6)

For any $A\in\mathbb{R}^{n\times m},B\in\mathbb{R}^{m\times
p},C\in\mathbb{R}^{p\times q}$, we have $\mathrm{vec}(ABC)=(C^{T}\otimes
A)\mathrm{vec}(B)=(I_{q}\otimes AB)\mathrm{vec}(C)=(C^{T}B^{T}\otimes
I_{n})\mathrm{vec}(A)$.

  7. (7)

For any $A\in\mathbb{R}^{n\times m}$ and any $B\in\mathbb{R}^{p\times q}$, we
have $K_{p,n}(A\otimes B)K_{m,q}=B\otimes A$.

If $f\colon\mathbb{R}^{n\times m}\to\mathbb{R}^{p\times q}$ is any
differentiable (possibly matrix-valued) function of $X\in\mathbb{R}^{n\times
m}$, we define

$Df(X):=D\mathrm{vec}(f(X)):=\frac{df(X)}{dX}:=\frac{d\mathrm{vec}(f(X))}{d\mathrm{vec}(X)},$

which is a $pq\times mn$ matrix. Let us recall the usual differentiation rules
for derivatives in matrix arguments together with the derivatives of several
standard types of functions.

######  Lemma 2.2.

Let $X\in\mathbb{R}^{n\times m}$, $A\in\mathbb{R}^{p\times n}$ and
$B\in\mathbb{R}^{m\times q}$.

  1. (1)

Let $f(X)\in\mathbb{R}^{p\times q}$, $g(X)\in\mathbb{R}^{q\times k}$ be
differentiable functions of $X$. Then, the _product rule_ holds:

$D(f(X)g(X))=(g(X)^{T}\otimes I_{p})Df(X)+(I_{k}\otimes f(X))Dg(X).$

  2. (2)

For differentiable functions $f$ on $\mathbb{R}^{q\times k}$ and
$g(X)\in\mathbb{R}^{q\times k}$, the usual _chain rule_ holds:

$D[f(g(X))]=Df(g(X))\cdot Dg(X).$

  3. (3)

We have $\frac{dX^{T}}{dX}=K_{n,m}$.

  4. (4)

We have $\frac{d(AXB)}{dX}=B^{T}\otimes A$.

  5. (5)

We have $\frac{d\mathrm{tr}(AX)}{dX}=\mathrm{vec}(A^{T})^{T}$.

##  3\. Derivatives on Stiefel manifolds

To formally introduce intrinsic derivatives of first and second order on
Stiefel manifolds, we follow and extend the spherical case as discussed in
[BCG17]. If $f$ is any real-valued locally Lipschitz function on some metric
space $(M,d)$ (with no isolated points), we may always define the generalized
modulus of the gradient of $f$ in $x\in M$

(3.1) $|\nabla^{*}f(x)|=\limsup_{x^{\prime}\rightarrow
x}\frac{|f(x)-f(x^{\prime})|}{d(x,x^{\prime})}.$

In particular, we may use (3.1) for functions $f(X)$ on the Stiefel manifold
$W_{n,d}$ together with the Euclidean (Hilbert–Schmidt) metric
$d(X,X^{\prime})=|X-X^{\prime}|$. Here we will always write
$|\nabla^{*}f(X)|\equiv|\nabla_{W}f(X)|$, cf. the discussion after (3.2)
below. Note we could also use the (point-dependent) canonical instead of the
Euclidean metric, which leads to different notions of differentiability, cf.
e. g. [EAS98].

To introduce a notion of differentiability of a function $f$ on $G_{n,d}$
which is consistent with (3.1), recall that the tangent space in $A\in
W_{n,d}$ is given by

$T_{A}:=\\{N\in\mathbb{R}^{n\times
d}:A^{T}N+N^{T}A=0\\}=\\{N\in\mathbb{R}^{n\times d}:A\circ N=0\\},$

where for any $M,N\in\mathbb{R}^{n\times d}$ (or $M,N\in\mathbb{R}^{d\times
d}$ as we shall also need later on), $M\circ N$ denotes the “symmetric
product”

$M\circ N:=\frac{1}{2}(M^{T}N+N^{T}M),$

which is a symmetric $d\times d$ matrix (for $d=1$, this reduces to the
Euclidean scalar product). Hence, $T_{A}$ is the set of all the matrices
$N\in\mathbb{R}^{n\times d}$ such that $A^{T}N$ is antisymmetric. Now, a
function $f\colon W_{n,d}\to\mathbb{R}$ is differentiable at $A\in W_{n,d}$ if
it admits a Taylor expansion

(3.2) $f(A^{\prime})=f(A)+\langle
M,A^{\prime}-A\rangle+o\big{(}|A^{\prime}-A|\big{)}\quad{\rm as}\ \
A^{\prime}\rightarrow A,\ \ A^{\prime}\in W_{n,d}$

with some $M\in{\mathbb{R}}^{n\times d}$. The unique $M_{0}$ of smallest
(Euclidean) length among all such $M$ is called the intrinsic derivative or
gradient of $f$ at $A$ and is denoted $\nabla_{W}f(A)$. The length of
$\nabla_{W}f(A)$ agrees with (3.1).

Once $M\in\mathbb{R}^{n\times d}$ satisfies (3.2), any matrix $M-B$ satisfies
(3.2) as well if

$\langle B,A^{\prime}-A\rangle=o(|A^{\prime}-A|)$

as $A^{\prime}\to A$, $A,A^{\prime}\in W_{n,d}$. The latter is equivalent to
$\langle B,N\rangle=0$ for all $N\in T_{A}$, i. e. $B\in T_{A}^{\perp}$.
Therefore, the minimization problems translates into

$\lVert M-B\rVert\to\mathrm{min}\qquad\text{over all $B\in T_{A}^{\perp}$,}$

which is solved uniquely for the orthogonal projection $M$ onto $T_{A}$.

Often, it is convenient to consider functions $f$ which are defined and smooth
in an open neighborhood of $W_{n,d}$, which may be achieved by choosing a
suitable extension of the function under consideration. In this case, we may
take the Euclidean derivative $Df(A)\in\mathbb{R}^{1\times nd}$ of $f$ in
$A\in W_{n,d}$ (cf. the previous section) and set $\nabla
f(A):=\mathrm{mat}(Df(A)^{T})\in\mathbb{R}^{n\times d}$ (the usual gradient
rewritten as a matrix). Clearly, $\nabla f(A)$ satisfies (3.2), and we may
project it onto $T_{A}$. This way, we get back $\nabla_{W}f(A)$.

######  Proposition 3.1.

Let $f$ be defined and $\mathcal{C}^{1}$-smooth in a neighborhood of
$W_{n,d}$. Then, the intrinsic first derivative of $f$ at $A\in W_{n,d}$ is
given by the projection onto $T_{A}$

(3.3) $\nabla_{W}f(A)=\nabla f(A)-A(A\circ\nabla f(A)).$

In particular, $|\nabla_{W}f(A)|\leq|\nabla f(A)|$ for any $A\in W_{n,d}$.

Defining the projection $\pi_{A}\colon\mathbb{R}^{n\times d}\to T_{A}$ by

(3.4) $\pi_{A}M:=M-A(A\circ M),$

Proposition 3.1 states that $\nabla_{W}f(A)=\pi_{A}\nabla f(A)$. In
particular, by the contractivity of orthogonal projections, this shows that
indeed, $|\nabla_{W}f(A)|\leq|\nabla f(A)|$.

Next, we introduce second order (intrinsic) derivatives (Hessians). For any
$C^{2}$-smooth function $f\colon\mathbb{R}^{n\times d}\to\mathbb{R}$ at a
given point $A\in W_{n,d}$, consider the Taylor expansion up to the quadratic
term (using the notation (2.1))

(3.5)
$f(A^{\prime})=f(A)+\langle\nabla_{W}f(A),A^{\prime}-A\rangle+\frac{1}{2}\langle
B(A^{\prime}-A),A^{\prime}-A\rangle+o\big{(}|A^{\prime}-A|^{2}\big{)}$

as $A^{\prime}\rightarrow A$, $A^{\prime}\in W_{n,d}$, where
$B\in\mathbb{R}^{nd\times nd}$ is some matrix. The collection of all $B$’s
satisfying (3.5) represents an affine subspace of $\mathbb{R}^{nd\times nd}$.
Therefore, among all of them, there exists a unique matrix of smallest
Hilbert–Schmidt norm. It is called the (intrinsic) second derivative of $f$ at
the point $X$ and will be denoted $f_{W}^{\prime\prime}(A)$.

If $f$ is $C^{2}$-smooth function on some open neighborhood of $W_{n,d}$, by
the usual (Euclidean) Taylor expansion, (3.5) holds with

(3.6) $B=f^{\prime\prime}(A)-(A\circ\nabla f(A))\otimes I_{n},$

where $f^{\prime\prime}(A)$ is the $nd\times nd$ matrix of the usual
(Euclidean) second order derivatives of $f$ at $A$. Indeed, plugging (3.3) and
(3.6) into (3.5), this follows from

$\displaystyle\qquad\langle A(A\circ\nabla
f(A)),A^{\prime}-A\rangle+\frac{1}{2}\langle((A\circ\nabla f(A))\otimes
I_{n})(A^{\prime}-A),A^{\prime}-A\rangle$
$\displaystyle=\frac{1}{2}\langle(A^{\prime}+A)(A\circ\nabla
f(A)),A^{\prime}-A\rangle=\frac{1}{2}\mathrm{tr}((A\circ\nabla f(A))(A^{\prime
T}+A^{T})(A^{\prime}-A))$ $\displaystyle=\frac{1}{2}\mathrm{tr}((A\circ\nabla
f(A))(A^{\prime T}A-A^{T}A^{\prime}))=0,$

where in the first identity we have used that by (2.1) and Lemma 2.1 (5),

$((A\circ\nabla f(A))\otimes
I_{n})(A^{\prime}-A)=\mathrm{vec}((A^{\prime}-A)(A\circ\nabla f(A))),$

the third step follows from $A,A^{\prime}\in W_{n,d}$ and in the last one we
use that the trace of a product of a symmetric and an antisymmetric matrix is
$0$.

Given $C\in\mathbb{R}^{nd\times nd}$, the matrix $B-C$ satisfies (3.5) if and
only if

$\langle
C(A^{\prime}-A),A^{\prime}-A\rangle=o\big{(}|A^{\prime}-A|^{2}\big{)}$

for $A^{\prime}\to A$, $A^{\prime}\in W_{n,d}$. Similarly to the first order
case, this is equivalent to $\langle CX,X\rangle=0$ for all $X\in T_{A}$. This
condition defines a linear subspace $L$ of $\mathbb{R}^{nd\times nd}$, and the
minimization problem translates into

$\lVert B-C\rVert\to\mathrm{min}\qquad\text{over all $C\in L$,}$

which is solved uniquely for the orthogonal projection of $B$ onto the linear
space $L^{\perp}$ of all matrices orthogonal to $L$. Since $B$ is symmetric,
we may restrict ourselves to symmetric matrices. As a result, we obtain the
following description.

######  Proposition 3.2.

The intrinsic second derivative of $f$ at each $A\in W_{n,d}$ is the symmetric
matrix which is given by the orthogonal projection

$f_{W}^{\prime\prime}(A)=P_{L^{\perp}}B,\qquad
B=f^{\prime\prime}(A)-(A\circ\nabla f(A))\otimes I_{n},$

to the orthogonal complement of the linear subspace $L=L_{A}$ of all symmetric
matrices $C$ in $\mathbb{R}^{nd\times nd}$ such that $\langle CX,X\rangle=0$
for all $X\in T_{A}$. Equivalently,

$f_{W}^{\prime\prime}(A)=P_{A}BP_{A},$

where $P_{A}=AA^{T}$ and $P_{A}BP_{A}$ is the $nd\times nd$ matrix which is
defined by the relation

(3.7)
$(P_{A}BP_{A})\mathrm{vec}(V)=\mathrm{vec}(P_{A}\mathrm{mat}[B\mathrm{vec}(P_{A}V)])$

for any $V\in\mathbb{R}^{n\times d}$.

In particular, for all $A\in W_{n,d}$ and $V\in\mathbb{R}^{n\times d}$,
$f_{W}^{\prime\prime}(A)V\in T_{A}$, $f^{\prime\prime}_{W}(A)A=0$, and hence
$\langle f_{W}^{\prime\prime}(A)V,A\rangle=0$ for all $V$. Furthermore, we
have the contraction property

$\|f_{W}^{\prime\prime}(A)\|_{\mathrm{HS}}\,\leq\,\|f^{\prime\prime}(A)-(A\circ\nabla
f(A))\otimes I_{n}\|_{{\rm HS}},$

which also holds for the operator norm.

In the notation of (2.1), it is not hard to show that for any
$V\in\mathbb{R}^{n\times d}$

$f^{\prime\prime}(X)V=\nabla\langle\nabla f(X),V\rangle$

for the usual Euclidean derivatives. The analogue for intrinsic derivatives
reads as follows:

######  Proposition 3.3.

Given a $C^{2}$-smooth function $f$ on $W_{n,d}$, for all $A\in W_{n,d}$ and
$V\in\mathbb{R}^{n\times d}$, we have

$f^{\prime\prime}_{W}(A)V=\nabla_{W}\langle\nabla_{W}f(A),V\rangle+P_{A}(\nabla_{W}f(A)(A\circ
V)).$

Here, the left-hand side has to be read as
$f^{\prime\prime}(W)\mathrm{vec}(V)$ and the right-hand side has to be
understood as vectorized.

In particular, if $V\in T_{A}$, then $P_{A}V=V$ and $A\circ V=0$, so that we
obtain

$f^{\prime\prime}_{W}(A)V=\nabla_{W}\langle\nabla_{W}f(A),V\rangle.$

###### Proof.

Consider the function $\psi_{V}(A):=\langle\nabla_{W}f(A),V\rangle$. In view
of Proposition 3.1, any smooth extension of $f$ to a neighborhood of $W_{n,d}$
also yields a smooth extension of $\psi_{V}$ which is given by

$\displaystyle\psi_{V}(X)$ $\displaystyle=\langle\nabla f(X),V\rangle-\langle
X(X\circ\nabla f(X)),V\rangle$ $\displaystyle=\langle\nabla
f(X),V\rangle-\mathrm{tr}(V^{T}X(X\circ\nabla f(X))).$

Let us calculate $\nabla\psi_{V}(X)$. First, we have

$\mathrm{vec}(\nabla\langle\nabla
f(X),V\rangle)=f^{\prime\prime}(X)\mathrm{vec}(V).$

Next, we consider

$\displaystyle\frac{d}{dX}[V^{T}X(X\circ\nabla f(X))]$
$\displaystyle=((X\circ\nabla f(X))\otimes I_{d})\frac{d(V^{T}X)}{dX}$
$\displaystyle\quad+(I_{d}\otimes(V^{T}X))\frac{d(X\circ\nabla f(X))}{dX},$

where we have applied Lemma 2.2 (1). By Lemma 2.2 (4),

$\frac{d(V^{T}X)}{dX}=I_{d}\otimes V^{T},$

so that

$((X\circ\nabla f(X))\otimes I_{d})\frac{d(V^{T}X)}{dX}=(X\circ\nabla
f(X))\otimes V^{T}.$

Moreover, using Lemma 2.2 (1&3), we have

$\frac{d(X^{T}\nabla f(X))}{dX}=(\nabla f(X)^{T}\otimes
I_{d})K_{n,d}+(I_{d}\otimes X^{T})f^{\prime\prime}(X)$

as well as

$\frac{d(\nabla f(X)^{T}X)}{dX}=(X^{T}\otimes
I_{d})K_{n,d}f^{\prime\prime}(X)+I_{d}\otimes\nabla f(X)^{T},$

and hence, using Lemma 2.1 (7), it follows that

$\frac{d(X\circ\nabla
f(X))}{dX}=\frac{1}{2}(I_{d^{2}}+K_{d,d})((I_{d}\otimes\nabla
f(X)^{T})+(I_{d}\otimes X^{T})f^{\prime\prime}(X)).$

Putting everything together, it follows by Lemma 2.2 (2&5) and (switching from
$d/(dX)$ to $\mathrm{vec}(\nabla)$, hence transposing) Lemma 2.1 (2) that

$\displaystyle\mathrm{vec}(\nabla\psi_{V}(X))$
$\displaystyle=f^{\prime\prime}(X)\mathrm{vec}(V)-((X\circ\nabla f(X))\otimes
V)\mathrm{vec}(I_{d})$ $\displaystyle\quad-\frac{1}{2}((I_{d}\otimes\nabla
f(X))+f^{\prime\prime}(X)(I_{d}\otimes
X))(I_{d^{2}}+K_{d,d})(I_{d}\otimes(X^{T}V))\mathrm{vec}(I_{d}).$

Using Lemma 2.1 (6), is possible to simplify this impression further. First,
note that

$((X\circ\nabla f(X))\otimes V)\mathrm{vec}(I_{d})=\mathrm{vec}(V(X\circ\nabla
f(X))).$

Moreover, we have

$(I_{d}\otimes(X^{T}V))\mathrm{vec}(I_{d})=\mathrm{vec}(X^{T}V)$

and by (2.2), it follows that

$K_{d,d}\mathrm{vec}(X^{T}V)=\mathrm{vec}(V^{T}X),$

so that

$\frac{1}{2}(I_{d^{2}}+K_{d,d})(I_{d}\otimes(X^{T}V))\mathrm{vec}(I_{d})=\mathrm{vec}(X\circ
V).$

To continue,

$(I_{d}\otimes\nabla f(X))\mathrm{vec}(X\circ V)=\mathrm{vec}(\nabla
f(X)(X\circ V))$

as well as

$(I_{d}\otimes X)\mathrm{vec}(X\circ V)=\mathrm{vec}(X(X\circ V)).$

Putting everything together, we thus obtain

$\displaystyle\mathrm{vec}(\nabla\psi_{V}(X))$
$\displaystyle=f^{\prime\prime}(X)\mathrm{vec}(V)-\mathrm{vec}(V(X\circ\nabla
f(X)))$ $\displaystyle\quad-\mathrm{vec}(\nabla f(X)(X\circ
V))-f^{\prime\prime}(X)\mathrm{vec}(X(X\circ V)).$

Restricting this from $X\in\mathbb{R}^{n\times d}$ to $A\in W_{n,d}$ and
recalling the projection $P_{A}$, we therefore have

$\mathrm{vec}(\nabla\psi_{V}(A))=f^{\prime\prime}(A)\mathrm{vec}(P_{A}V)-\mathrm{vec}(V(A\circ\nabla
f(A))+\nabla f(A)(A\circ V)).$

In terms of intrinsic derivatives (and now using the short-hand notation
introduced in Proposition 3.2), we obtain

$\nabla_{W}\psi_{V}(A)=P_{A}f^{\prime\prime}(A)P_{A}V-P_{A}[V(A\circ\nabla
f(A))]-P_{A}[\nabla f(A)(A\circ V)].$

Recall the matrix $B=f^{\prime\prime}(A)-(A\circ\nabla f(A))\otimes I_{n}$
from (3.6). By Proposition 2.1 (5),

$\tilde{B}\mathrm{vec}(V)=((A\circ\nabla f(A))\otimes
I_{n})\mathrm{vec}(V)=\mathrm{vec}(V(A\circ\nabla f(A))),$

so that if we write $B=f^{\prime\prime}(A)-\tilde{B}$, we obtain

$\displaystyle\nabla_{W}\psi_{V}(A)$
$\displaystyle=P_{A}f^{\prime\prime}(A)P_{A}V-P_{A}\tilde{B}V-P_{A}[\nabla
f(A)(A\circ V)]$
$\displaystyle=P_{A}f_{W}^{\prime\prime}(A)P_{A}V-P_{A}\tilde{B}(V-P_{A}V)-P_{A}[\nabla
f(A)(A\circ V)]$

in view of Proposition 3.2. As above, we note that

$\tilde{B}V=(V-P_{A}V)(A\circ\nabla f(A)),$

so that altogether, we arrive at

$\displaystyle f^{\prime\prime}_{W}(A)V$
$\displaystyle=\nabla_{W}\langle\nabla_{W}f(A),V\rangle+P_{A}[(V-P_{A}V)(A\circ\nabla
f(A))]$ $\displaystyle\quad+P_{A}[\nabla f(A)(A\circ V)].$

To finish the proof, it remains to note that

$\displaystyle\quad P_{A}[(V-P_{A}V)(A\circ\nabla f(A))]+P_{A}[\nabla
f(A)(A\circ V)]-P_{A}[\nabla_{W}f(A)(A\circ V)]$ $\displaystyle=P_{A}[A(A\circ
V)(A\circ\nabla f(A))]+P_{A}[A(A\circ\nabla f(A))(A\circ V)]$
$\displaystyle=P_{A}[A((A\circ V)\circ(A\circ\nabla f(A)))]=0,$

where the last step follows by an easy calculation using $A^{T}A=I_{d}$. ∎

##  4\. Second order modulus of gradient

Recall that the generalized second order modulus of the gradient on the
Stiefel manifold $W_{n,d}$ is defined by

$\displaystyle|\nabla^{(2)}f(A)|$ $\displaystyle=|\nabla_{W}|\nabla_{W}f(A)||$
$\displaystyle=\limsup_{A^{\prime}\to
A}\frac{\big{|}|\nabla_{W}f(A)|-|\nabla_{W}f(A^{\prime})|\big{|}}{|A-A^{\prime}|}.$

Typically, explicitly calculating $|\nabla^{(2)}f(A)|$ is not easy, however,
and therefore, motivated by the Euclidean or spherical calculus, we may hope
for an estimate of the form $|\nabla^{(2)}f(A)|\leq\lVert
f^{\prime\prime}_{W}(A)\rVert_{\mathrm{op}}$. Indeed, we have the following
result.

######  Proposition 4.1.

For any $\mathcal{C}^{2}$-smooth function $f$ on $W_{n,d}$, $|\nabla_{W}f|$
has finite Lipschitz semi-norm, and for any $A\in W_{n,d}$,

$|\nabla^{(2)}f(A)|=|\nabla_{W}f(A)|^{-1}|f^{\prime\prime}_{W}(A)\nabla
f_{W}(A)|,$

where the right-hand side has to be understood as $\lVert
f^{\prime\prime}_{W}(A)\rVert_{\mathrm{op}}$ if $|\nabla_{W}f(A)|=0$. In
particular, we always have

$|\nabla^{(2)}f(A)|\leq\lVert f^{\prime\prime}_{W}(A)\rVert_{\mathrm{op}}.$

###### Proof.

Let us first prove that the function $|\nabla_{W}f(A)|$ has finite Lipschitz
semi-norm. Since the first two intrinsic derivatives of $f$ are continuous and
therefore bounded on the compact manifold $W_{n,d}$, we obtain from
Proposition 3.3 that

$|\nabla_{W}\langle\nabla_{W}f(A),V\rangle|\leq C$

for any $V\in\mathbb{R}^{n\times d}$ such that $|V|\equiv\lVert
V\rVert_{\mathrm{HS}}=1$, where $C$ is some constant independent of $A$ and
$V$. Hence, the function $A\mapsto\langle\nabla_{W}f(A),V\rangle$ has
Lipschitz semi-norm bounded by $C$, i. e.

$|\langle\nabla_{W}f(A^{\prime}),V\rangle-\langle\nabla_{W}f(A),V\rangle|\leq
Cd(A,A^{\prime})$

for all $A,A^{\prime}\in W_{n,d}$. Therefore, taking the supremum over all $V$
and applying the triangle inequality yields

$||\nabla_{W}f(A^{\prime})|-|\nabla_{W}f(A)||\leq|\nabla_{W}f(A^{\prime})-\nabla_{W}f(A)|\leq
Cd(A^{\prime},A),$

which had to be proven.

To show the identity for the second order modulus of the gradient, fix $A\in
W_{n,d}$. By the definition of the intrinsic gradient and Proposition 3.3, we
have

$\langle\nabla_{W}f(A^{\prime}),V\rangle=\langle\nabla_{W}f(A),V\rangle+\langle\tilde{V},A^{\prime}-A\rangle+o(|A^{\prime}-A|),$

where

$\tilde{V}=f^{\prime\prime}_{W}(A)V-P_{A}(\nabla_{W}f(A)(A\circ V)).$

Moreover, a closer analysis (using the integral form of the Taylor formula and
the compactness of $W_{n,d}$, which implies that every continuous function is
already uniformly continuous) yields that the remainder term in the Taylor
expansion can be bounded independently of $V\in\mathbb{R}^{n\times d}$ such
that $|V|=1$, i. e.

$\sup_{|V|=1}|\langle\nabla_{W}f(A^{\prime}),V\rangle-\langle\nabla_{W}f(A),V\rangle-\langle\tilde{V},A^{\prime}-A\rangle|\leq\varepsilon(|A-A^{\prime}|),$

where $\varepsilon(t)$ is some function which satisfies $\varepsilon(t)\to 0$
as $t\to 0$.

The next step is to rewrite the Taylor formula as

(4.1)
$\langle\nabla_{W}f(A^{\prime}),V\rangle=\langle\nabla_{W}f(A)+L,V\rangle+o(|A^{\prime}-A|).$

To this end, we first choose a suitable $\tilde{L}$ such that
$\langle\tilde{V},A^{\prime}-A\rangle=\langle\tilde{L},V\rangle$. Obviously,

$\langle f_{W}^{\prime\prime}(A)V,A^{\prime}-A\rangle=\langle
f^{\prime\prime}_{W}(A)(A^{\prime}-A),V\rangle.$

Moreover,

$\displaystyle\quad P_{A}(\nabla_{W}f(A)(A\circ V))$
$\displaystyle=\nabla_{W}f(A)(A\circ V)-A(A\circ(\nabla_{W}f(A)(A\circ V)))$
$\displaystyle=\frac{1}{2}(\nabla_{W}f(A)A^{T}V+\nabla_{W}f(A)V^{T}A)-\frac{1}{4}(AA^{T}\nabla_{W}f(A)A^{T}V$
$\displaystyle\qquad+AA^{T}\nabla_{W}f(A)V^{T}A+AA^{T}V\nabla_{W}f(A)^{T}A+AV^{T}A\nabla_{W}f(A)^{T}A).$

Recall that $\langle U,V\rangle=\mathrm{tr}(U^{T}V)$ and that the trace is
invariant under cyclic permutations. Therefore, we may easily verify the
general rules

(4.2) $\langle UVW,X\rangle=\langle U^{T}XW^{T},V\rangle,\qquad\langle
UV^{T}W,X\rangle=\langle WX^{T}U,V\rangle,$

where $V,X\in\mathbb{R}^{n\times d}$ and $U\in\mathbb{R}^{n\times n}$,
$W\in\mathbb{R}^{d\times d}$ in the first identity or
$U,W\in\mathbb{R}^{n\times d}$ in the second one. We now apply this inequality
to all six terms appearing in $P_{A}(\nabla_{W}f(A)(A\circ V))$ with $V$ as
above and $X=A^{\prime}-A$. For instance, for the first term this yields

$\langle\nabla_{W}f(A)A^{T}V,A^{\prime}-A\rangle=\langle
A\nabla_{W}f(A)^{T}(A^{\prime}-A),V\rangle.$

Proceeding similarly, we arrive at

$\langle\nabla_{W}f(A)(A\circ V),A^{\prime}-A\rangle=\langle
A(\nabla_{W}f(A)\circ(A^{\prime}-A)),V\rangle$

as well as

$\langle A(A\circ(\nabla_{W}f(A)(A\circ V))),A^{\prime}-A\rangle=\langle
A((A^{T}\nabla_{W}f(A))\circ(A\circ(A^{\prime}-A))),V\rangle.$

Altogether, we obtain

$\tilde{L}=f_{W}^{\prime\prime}(A)(A^{\prime}-A)-A[\nabla_{W}f(A)\circ(A^{\prime}-A)-(A^{T}\nabla_{W}f(A))\circ(A\circ(A^{\prime}-A))].$

Now we define

$L:=f_{W}^{\prime\prime}(A)(A^{\prime}-A)-A(\nabla_{W}f(A)\circ(A^{\prime}-A))$

To see that (4.1) holds, it remains to show that
$\tilde{L}-L=o(|A^{\prime}-A|)$ as $A^{\prime}\to A$, $A,A^{\prime}\in
W_{n,d}$. To this end, note that by an easy calculation,

$A\circ(A^{\prime}-A)=\frac{1}{2}(A^{\prime}-A)^{T}(A-A^{\prime})=o(|A^{\prime}-A|).$

From here, the claim immediately follows by compactness arguments.

Now we take an absolute value on both sides of the Taylor formula (4.1) and
take the supremum over all $V$ such that $\mathrm{vec}(V)\in S^{nd-1}$. This
leads to

$|\nabla_{W}f(A^{\prime})|=|\nabla_{W}f(A)+L|+o(|A^{\prime}-A|).$

Next, we write

$|\nabla_{W}f(A)+L|^{2}=|\nabla_{W}f(A)|^{2}+2\langle\nabla_{W}f(A),L\rangle+|L|^{2}.$

Noting that

$\langle\nabla_{W}f(A),A(A\circ\nabla
f(A))\rangle=\mathrm{tr}(\nabla_{W}f(A)^{T}A(A\circ\nabla f(A)))=0$

since $\nabla_{W}f(A)^{T}A$ is antisymmetric (as $\nabla_{W}f(A)\in T_{A}$)
and $A\circ\nabla f(A)$ is symmetric, we obtain

$\langle\nabla_{W}f(A),L\rangle=\langle\nabla_{W}f(A),f_{W}^{\prime\prime}(A)(A^{\prime}-A)\rangle=\langle
U,A^{\prime}-A\rangle,$

where

$U:=f_{W}^{\prime\prime}(A)\nabla_{W}f(A)\equiv
f_{W}^{\prime\prime}(A)\mathrm{vec}(\nabla_{W}f(A)).$

Since $|L|^{2}=O(|A^{\prime}-A|^{2})$, we obtain

$|\nabla_{W}f(A)+L|^{2}=|\nabla_{W}f(A)|^{2}+2\langle
U,A^{\prime}-A\rangle+o(|A^{\prime}-A|).$

If $|\nabla_{W}f(A)|>0$, it therefore follows that

$|\nabla_{W}f(A)+L|=|\nabla_{W}f(A)|+|\nabla_{W}f(A)|^{-1}\langle
U,A^{\prime}-A\rangle+o(|A^{\prime}-A|).$

Hence,

$|\nabla_{W}f(A^{\prime})|-|\nabla_{W}f(A)|=|\nabla_{W}f(A)|^{-1}\langle
U,A^{\prime}-A\rangle+o(|A^{\prime}-A|)$

and thus

$\displaystyle\limsup_{A^{\prime}\to
A}\frac{\big{|}|\nabla_{W}f(A^{\prime})|-|\nabla_{W}f(A)|\big{|}}{|A^{\prime}-A|}$
$\displaystyle=|\nabla_{W}f(A)|^{-1}\limsup_{A^{\prime}\to
A}\frac{\big{|}\langle U,A^{\prime}-A\rangle\big{|}}{|A^{\prime}-A|}$
$\displaystyle=|\nabla_{W}f(A)|^{-1}|\nabla_{W}\psi_{U}(A)|,$

where $\psi_{U}(A):=\langle U,A\rangle$. As noted after Proposition 3.2, $U\in
T_{A}$, so that $\nabla_{W}\psi_{U}(A)=U$. Thus, we arrive at

$|\nabla^{(2)}_{W}(A)|=|\nabla_{W}f(A)|^{-1}|f_{W}^{\prime\prime}(A)\nabla_{W}f(A)|$

if $|\nabla_{W}f(A)|>0$.

It remains to consider the case where $|\nabla_{W}f(A)|=0$. Here,
$L=f_{W}^{\prime\prime}(A)(A^{\prime}-A)$, and the Taylor formula reads

$|\nabla_{W}f(A^{\prime})|=|L|+o(|A^{\prime}-A|).$

It follows that

$\displaystyle|\nabla^{(2)}_{W}f(A)|$ $\displaystyle=\limsup_{A^{\prime}\to
A}\frac{|\nabla_{W}f(A^{\prime})|}{|A^{\prime}-A|}=\limsup_{A^{\prime}\to
A}\frac{|f_{W}^{\prime\prime}(A)(A^{\prime}-A)|}{|A^{\prime}-A|}$
$\displaystyle=\limsup_{V\to 0,V\in
T_{A}^{\perp}}\frac{|f_{W}^{\prime\prime}(A)V|}{|V|}=\lVert
f_{W}^{\prime\prime}(A)\rVert_{\mathrm{op}},$

which finishes the proof. ∎

##  5\. Derivatives on Grassmann manifolds

Let us adapt the results of the previous two sections to functions on
Grassmann manifolds. Much of what follows relies on similar arguments as in
the Stiefel case, and for this reason we will often only sketch the arguments.
To introduce a notion of differentiability on $G_{n,d}$, first recall that the
tangent space in $P\in G_{n,d}$ is given by

$T_{P}:=\\{S\in\mathbb{R}^{n\times n}_{\mathrm{sym}}\colon
S=SP+PS\\}\equiv\\{S\in\mathbb{R}^{n\times n}_{\mathrm{sym}}\colon
S=[[S,P],P]\\},$

where for any $M,N\in\mathbb{R}^{n\times n}$, $[M,N]=MN-NM$ denotes the matrix
commutator. A function $f\colon G_{n,d}\to\mathbb{R}$ is differentiable at
$X\in G_{n,d}$ if it admits a Taylor expansion

(5.1) $f(P^{\prime})=f(P)+\langle
M,P^{\prime}-P\rangle+o\big{(}|X^{\prime}-X|\big{)}\quad\mathrm{as}\ \
P^{\prime}\rightarrow P,\ \ P^{\prime}\in G_{n,d}$

with some $M\in\mathbb{R}^{n\times n}_{\mathrm{sym}}$. Among all such $M$,
there exists a unique $M_{0}$ of smallest (Euclidean) length, called the
intrinsic (first) derivative or gradient of $f$ at $P$ and denoted
$\nabla_{G}f(P)$. The length of $\nabla_{G}f(P)$ agrees with the generalized
modulus of the gradient (3.1) applied to the Grassmann manifold. In passing,
note that unlike in case of the Stiefel manifold, for Grassmann manifolds the
Euclidean metric and the canonical metric lead to the same notion of
differentiablity. As in case of the Stiefel manifold, the minimization problem
(5.1) translates into

$\lVert M-B\rVert\to\mathrm{min}\qquad\text{over all $B\in T_{P}^{\perp}$,}$

which is solved uniquely for the orthogonal projection $M$ onto $T_{P}$.

If we consider functions which are defined and smooth in an open neighborhood
of $G_{n,d}$ in the ambient space, e. g. $\mathbb{R}^{n\times
n}_{\mathrm{sym}}$, we may take the Euclidean gradient $\nabla
f(P)=\mathrm{mat}(Df(P)^{T})$ (cf. Section 2) and project it onto $T_{P}$,
which gives back $\nabla_{G}f(P)$. As the projection
$\pi_{P}\colon\mathbb{R}^{n\times n}_{\mathrm{sym}}\to T_{P}$ is given by

(5.2) $\pi_{P}M:=[P,[P,M]]=PM+MP-2PMP,$

we immediately arrive at the following analogue of Proposition 3.1.

######  Proposition 5.1.

Let $f$ be defined and $\mathcal{C}^{1}$-smooth in some open neighborhood of
$G_{n,d}$ in $\mathbb{R}^{n\times n}_{\mathrm{sym}}$. Then, the intrinsic
first derivative of $f$ at $P\in G_{n,d}$ is given by the projection onto
$T_{P}$

$\nabla_{G}f(P)=\pi_{P}\nabla f(P)=[P,[P,\nabla f(P)]].$

In particular, $|\nabla_{G}f(P)|\leq|\nabla f(P)|$ for any $P\in G_{n,d}$.

In fact, sometimes yet a further embedding might be convenient, so that we
regard $G_{n,d}$ as a submanifold of $\mathbb{R}^{n\times n}$ and take the
Euclidean derivatives of some extension of $f$ to an open neighborhood in
$\mathbb{R}^{n\times n}$. However, in this case, we may project $\nabla f(P)$
onto the tangent space of $\mathbb{R}^{n\times n}_{\mathrm{sym}}$ (which
equals $\mathbb{R}^{n\times n}_{\mathrm{sym}}$) by applying the projection
$\pi_{\mathrm{sym}}\colon\mathbb{R}^{n\times n}\to\mathbb{R}^{n\times
n}_{\mathrm{sym}}$ given by $\pi_{\mathrm{sym}}(M):=(M+M^{T})/2$. Then, we may
proceed as in Proposition 5.1 for $\pi_{\mathrm{sym}}(\nabla f(P))$.

For any $C^{2}$-smooth function $f$ on $G_{n,d}$ at a given point $P\in
W_{n,d}$, the intrinsic second order derivative is the matrix
$B\in\mathbb{R}^{n^{2}\times n^{2}}$ of smallest Hilbert–Schmidt norm which
satisfies

(5.3)
$f(P^{\prime})=f(P)+\langle\nabla_{G}f(P),P^{\prime}-P\rangle+\frac{1}{2}\langle
B(P^{\prime}-P),P^{\prime}-P\rangle+o\big{(}|P^{\prime}-P|^{2}\big{)}$

as $P^{\prime}\rightarrow P$, $P^{\prime}\in G_{n,d}$. It will be denoted
$f_{G}^{\prime\prime}(X)$. Considering functions $f$ which are defined and
smooth in some open neighborhood of $G_{n,d}$ in $\mathbb{R}^{n\times
n}_{\mathrm{sym}}$, it is possible to express the intrinsic second order
derivative in terms of the Euclidean derivatives of $f$. Instead of providing
the details, we refer to [HHT07, Theorem 2.4] where corresponding calculations
have been done. Adapting them to our framework yields the following analogue
of Proposition 3.2.

######  Proposition 5.2.

The intrinsic second derivative of $f$ at each $P\in G_{n,d}$ is given by the
relation

$f_{G}^{\prime\prime}(P)V=\pi_{P}f^{\prime\prime}(P)\pi_{P}V-[P,[\nabla
f(P),\pi_{P}V]]$

for any $V\in\mathbb{R}^{n\times n}_{\mathrm{sym}}$. Here, the expression on
the right hand side has to be understood as

$\mathrm{vec}(\pi_{P}\mathrm{mat}[f^{\prime\prime}(P)\mathrm{vec}(\pi_{P}V)])-\mathrm{vec}([P,[\nabla
f(P),\pi_{P}V]]).$

Note that $[P,[\nabla f(P),\pi_{P}V]]\in T_{P}$. In particular, for all $P\in
G_{n,d}$ and $V\in\mathbb{R}^{n\times n}_{\mathrm{sym}}$,
$f_{G}^{\prime\prime}(P)V\in T_{P}$, $f^{\prime\prime}_{G}(P)P=0$, and hence
$\langle f_{G}^{\prime\prime}(P)V,P\rangle=0$ for all $V\in\mathbb{R}^{n\times
n}_{\mathrm{sym}}$. Furthermore, we have the contraction property

$\|f_{G}^{\prime\prime}(P)\|_{\mathrm{HS}}\leq\|f^{\prime\prime}(P)-[P,[\nabla
f(P),\pi_{P}V]]\|_{{\rm HS}},$

which also holds for the operator norm. Similarly to the first order case,
sometimes it is necessary to extend $f$ to a smooth function on some
neighborhood in $\mathbb{R}^{n\times n}$. In this case, we may take the usual
Euclidean Hessian $f^{\prime\prime}(P)$, from which we get back the Hessian on
$\mathbb{R}^{n\times n}_{\mathrm{sym}}$ by considering
$\pi_{\mathrm{sym}}f^{\prime\prime}(P)\pi_{\mathrm{sym}}$.

Moreover, we have the following analogue of Proposition 3.3.

######  Proposition 5.3.

Given a $C^{2}$-smooth function $f$ on $G_{n,d}$, for all $P\in G_{n,d}$ and
$V\in\mathbb{R}^{n\times n}_{\mathrm{sym}}$, we have

$f_{G}^{\prime\prime}(P)V=\nabla_{G}\psi_{V}(P)-[P,[\nabla_{G}f(P),V]].$

Here, the left-hand side has to be read as
$f^{\prime\prime}(W)\mathrm{vec}(V)$ and the right-hand side has to be
understood as vectorized.

In particular, if $V\in T_{P}$ it follows from [HHT07, Lemma 2.2] that

$f^{\prime\prime}_{G}(P)V=\nabla_{G}\langle\nabla_{G}f(P),V\rangle.$

Finally, we also have an analogue of Proposition 4.1.

######  Proposition 5.4.

For any $\mathcal{C}^{2}$-smooth function $f$ on $G_{n,d}$, $|\nabla_{G}f|$
has finite Lipschitz semi-norm, and for any $P\in G_{n,d}$,

$|\nabla^{(2)}_{G}f(P)|=|\nabla_{G}f(P)|^{-1}|f^{\prime\prime}_{G}(P)\nabla
f_{G}(P)|,$

where the right-hand side has to be understood as $\lVert
f^{\prime\prime}_{G}(P)\rVert_{\mathrm{op}}$ if $|\nabla_{G}f(P)|=0$. In
particular, we always have

$|\nabla^{(2)}_{G}f(P)|\leq\lVert
f^{\prime\prime}_{G}(P)\rVert_{\mathrm{op}}.$

As the proofs of Proposition 5.3 and Proposition 5.4 are mostly an adaption of
the arguments known from the Stiefel case, we defer them to the appendix.

##  6\. Proofs

To prepare the proofs of our main results, we briefly revisit the
representations of the Stiefel and Grassmann manifolds we use in this paper.
In particular, we shall discuss the Lipschitz properties of the map
$\pi_{\mathrm{St}}\colon W_{n,d}\to G_{n,d}$, $A\mapsto AA^{T}\equiv P_{A}$.
Clearly, $\pi_{\mathrm{St}}$ is $2\sqrt{d}$-Lipschitz as

$\lVert AA^{T}-A^{\prime}A^{\prime T}\rVert_{\mathrm{HS}}\leq\lVert
A(A^{T}-A^{\prime T})\rVert_{\mathrm{HS}}+\lVert(A-A^{\prime})A^{\prime
T}\rVert_{\mathrm{HS}}\leq 2\sqrt{d}\lVert A-A^{\prime}\rVert_{\mathrm{HS}}$

since $\lVert A\rVert_{\mathrm{HS}},\lVert
A^{\prime}\rVert_{\mathrm{HS}}=\sqrt{d}$. However, this Lipschitz constant is
too weak for our purposes. To avoid the dependency on $d$, more subtle
arguments are needed.

Recall that if $M_{1},M_{2}\subset\mathbb{R}^{n}$ are two subspaces of
dimension $d$, the principal angles $\theta_{j}$, $j=1,\ldots,d$, between
$M_{1}$ and $M_{2}$ are recursively defined by

$\cos\theta_{j}=\max_{u\in M_{1}}\max_{v\in M_{2}}\lvert\langle
u,v\rangle\rvert=\langle u_{j},v_{j}\rangle$

subject to the constraints

$\langle u_{i},u\rangle=0,\quad\langle v_{i},v\rangle=0,\quad
i=1,2,\ldots,j-1.$

The vectors $\\{u_{1},\ldots,u_{d}\\}$, $\\{v_{1},\ldots,v_{d}\\}$ are called
the principal vectors of the pair $(M_{1},M_{2})$. The concept of principal
angles goes back to Jordan. Here we mainly follow the survey article [Ga08].

Note that the principal angles are uniquely defined and satisfy
$0\leq\theta_{1}\leq\ldots\leq\theta_{d}\leq\pi/2$, while the principal
vectors are not unique. However, they form orthonormal $d$-frames and can thus
be interpreted as elements $U,V\in W_{n,d}$. Moreover, we have $\langle
u_{i},v_{j}\rangle=\delta_{ij}\cos\theta_{i}$. By [Af57], the respective
orthogonal projections $P_{U}=\pi_{\mathrm{St}}(U)$ and
$P_{V}=\pi_{\mathrm{St}}(V)$ satisfy

$P_{U}P_{V}u_{j}=(\cos^{2}\theta_{j})u_{j},\quad
P_{V}P_{U}v_{j}=(\cos^{2}\theta_{j})v_{j},\quad j=1,\ldots,d,$

i. e., the non-zero eigenvalues of the matrices $P_{U}P_{V}$ and $P_{V}P_{U}$
are $\cos^{2}\theta_{j}$, $j=1,\ldots,d$. This relation extends to all $A,B\in
W_{n,d}$ such that $A=UO$ and $B=VO^{\prime}$ for some $O,O^{\prime}\in O(d)$
(since $P_{A}=P_{U}$ and $P_{B}=P_{V}$). Note also that by definition of the
principal angles, for any $A,A^{\prime}\in W_{n,d}$

$\max_{O\in O(d)}\langle A,AO^{\prime}\rangle=\langle
A,A^{\prime\prime}\rangle=\mathrm{tr}(A^{T}A^{\prime\prime})=\sum_{j=1}^{d}\cos\theta_{j},$

where $\theta_{j}$ are the principal angles between the two subspaces induced
by $A$ and $A^{\prime}$ and $A^{\prime\prime}$ is the Stiefel matrix
maximizing the term on the left-hand side.

######  Lemma 6.1.

The map $\pi_{\mathrm{St}}\colon W_{n,d}\to G_{n,d}$, $A\mapsto AA^{T}=P_{A}$
is $\sqrt{2}$-Lipschitz.

###### Proof.

Using principle angles and the notation introduced above, we have

$\displaystyle\lVert P_{A}-P_{A^{\prime}}\rVert_{\mathrm{HS}}^{2}$
$\displaystyle=2d-2\mathrm{tr}(P_{A}P_{A^{\prime}})=2d-2\sum_{j=1}^{d}\cos^{2}\theta_{j}=2\sum_{j=1}^{d}(1-\cos^{2}\theta_{j})$
$\displaystyle\leq
2\sum_{j=1}^{d}2(1-\cos\theta_{j})=2(2d-2\mathrm{tr}(A^{T}A^{\prime\prime}))$
$\displaystyle\leq
2(2d-2\mathrm{tr}(A^{T}A^{\prime}))=2||A-A^{\prime}||_{\mathrm{HS}}^{2},$

where in the first inequality we used that $(1-x^{2})/(1-x)\leq 2$ for any
$x\in[0,1]$. ∎

One may wonder whether the map $P_{A}$ might even be $1$-Lipschitz. However,
simple examples are sufficient to show that this cannot be true (e. g.,
consider $d=1$ and the vectors $A=(1,0,\ldots,0)^{T}$ and
$A^{\prime}=(1/\sqrt{n},\ldots,1/\sqrt{n})^{T}$).

The core of our arguments is a logarithmic Sobolev inequality for Stiefel and
Grassmann manifolds. Even if we are not aware of a source where log-Sobolev
inequalities for Stiefel and Grassmann manifolds are rigorously formulated,
they may easily be derived by a simple projection argument. We emphasize that
these inequalities are designed for the representations of $W_{n,d}$ and
$G_{n,d}$ we use all over this paper.

######  Proposition 6.2.

  1. (1)

For any $d<n$, $W_{n,d}$ satisfies a logarithmic Sobolev inequality with
constant $4/(n-2)$, i. e. for any $f\colon W_{n,d}\to\mathbb{R}$ sufficiently
smooth,

$\mathrm{Ent}_{\mu_{n,d}}(f^{2})\leq\frac{8}{n-2}\int_{W_{n,d}}|\nabla_{W}f|^{2}d\mu_{n,d}.$

  2. (2)

For any $d<n$, $G_{n,d}$ satisfies a logarithmic Sobolev inequality with
constant $8/(n-2)$, i. e. for any $f\colon G_{n,d}\to\mathbb{R}$ sufficiently
smooth,

$\mathrm{Ent}_{\nu_{n,d}}(f^{2})\leq\frac{16}{n-2}\int_{G_{n,d}}|\nabla_{G}f|^{2}d\nu_{n,d}.$

Let us briefly mention that for $d=1$ (i. e. the sphere), the optimal Sobolev
constant is known to be $1/(n-1)$ as shown in [MW82]. In particular, even if
Proposition 6.2 is not fully accurate, the behavior of the Sobolev constant
for large values of $n$ agrees with the optimal result in case of the sphere.
Moreover, note that if $d=n$, $W_{n,n}=O(n)$ has two connected components,
which in particular implies that a log-Sobolev inequality cannot hold.

###### Proof of Proposition 6.2.

First recall that if $d<n$, we have

$W_{n,d}\cong SO(n)/SO(n-d).$

Indeed, identifying any matrix in $SO(n)$ with its first $n-1$ columns
$e_{1},\ldots,e_{n-1}$, this follows readily using the projection map
$\varphi\colon SO(n)\to W_{n,d}$ which is given by
$\varphi(e_{1},\ldots,e_{n-1}):=(e_{1},\ldots,e_{d})$. By [Me19, Theorem
5.16], the special orthogonal group $SO(n)$ equipped with the uniform
probability measure and Hilbert–Schmidt metric satisfies a log-Sobolev
inequality with constant $4/(n-2)$. Noting that the map $\varphi$ is
$1$-Lipschitz, (1) therefore follows immediately.

To see (2), it remains to note that $(G_{n,d},\nu_{n,d})$ is the the
pushforward of $(W_{n,d},\mu_{n,d})$ under $\pi_{\mathrm{St}}$, which is
$\sqrt{2}$-Lipschitz according to Lemma 6.1 (so that the Sobolev constant is
doubled). ∎

In the sequel, we will give the proofs of the concentration bound for function
on Stiefel manifolds, i. e., Theorem 1.1 and Theorem 1.3. As already shown in
[AS94], logarithmic Sobolev inequalities imply certain $L^{p}$ norm
inequalities. In the situation under consideration, for any locally Lipschitz
function $g\colon W_{n,d}\to\mathbb{R}$ and for any $p\geq 2$,

(6.1) $\lVert g\rVert_{p}^{2}\leq\lVert
g\rVert_{2}^{2}+4(n-2)^{-1}(p-2)\lVert\nabla_{W}g\rVert_{p}^{2}.$

Using Proposition 3.1, we moreover have

(6.2) $\lVert g\rVert_{p}^{2}\leq\lVert
g\rVert_{2}^{2}+4(n-2)^{-1}(p-2)\lVert\nabla g\rVert_{p}^{2}.$

Let us also recall that since $\mu_{n,d}$ satisfies a log-Sobolev inequality,
it also satisfies a Poincaré inequality with the same constant, i. e.

(6.3)
$\mathrm{Var}_{\mu_{n,d}}(f)\equiv\int_{W_{n,d}}(f-\mu_{n,d}(f))^{2}d\mu_{n,d}\leq\frac{4}{n-2}\int_{W_{n,d}}|\nabla_{W}f|^{2}d\mu_{n,d}$

for all $f\colon W_{n,d}\to\mathbb{R}$ sufficiently smooth.

We now first prove Theorem 1.3. Note that its proof follow the lines of the
proofs established in [BGS19, GSS21b].

###### Proof of Theorem 1.3.

Applying (6.2) to $g=\lVert f^{(\ell)}\rVert_{\mathrm{op}}$ and recalling that
for Euclidean derivatives, $|\nabla\lVert
f^{(\ell)}\rVert_{\mathrm{op}}|\leq\lVert f^{(\ell+1)}\rVert_{\mathrm{op}}$
(cf. [BGS19, Lemma 4.1]), we obtain that

$\lVert f^{(\ell)}\rVert_{\mathrm{op},p}^{2}\leq\lVert
f^{(\ell)}\rVert_{\mathrm{op},2}^{2}+4(n-2)^{-1}(p-2)\lVert\nabla
f^{(\ell+1)}\rVert_{\mathrm{op},p}^{2}$

for any $\ell=1,\ldots,k-1$, which yields

$\displaystyle\lVert f\rVert_{p}^{2}$ $\displaystyle\leq\lVert
f\rVert_{2}^{2}+\sum_{\ell=1}^{k-1}\Big{(}\frac{4(p-2)}{n-2}\Big{)}^{\ell}\lVert
f^{(\ell)}\rVert_{\mathrm{op},2}^{2}+\Big{(}\frac{4(p-2)}{n-2}\Big{)}^{k}\lVert
f^{(k)}\rVert_{\mathrm{op},\infty}^{2}$ $\displaystyle\leq\frac{4}{n-2}\lVert
f^{(1)}\rVert_{2}^{2}+\sum_{\ell=1}^{k-1}\Big{(}\frac{4(p-2)}{n-2}\Big{)}^{\ell}\lVert
f^{(\ell)}\rVert_{\mathrm{op},2}^{2}+\Big{(}\frac{4(p-2)}{n-2}\Big{)}^{k}\lVert
f^{(k)}\rVert_{\mathrm{op},\infty}^{2}$ (6.4)
$\displaystyle\leq\sum_{\ell=1}^{k-1}\Big{(}\frac{4(p-1)}{n-2}\Big{)}^{\ell}\lVert
f^{(\ell)}\rVert_{\mathrm{op},2}^{2}+\Big{(}\frac{4(p-1)}{n-2}\Big{)}^{k}\lVert
f^{(k)}\rVert_{\mathrm{op},\infty}^{2}$

for any $p\geq 2$, where the second step follows by Poincaré inequality.

To see (1), plugging in the assumptions we arrive at

$\lVert
f\rVert_{p}^{2}\leq\Big{(}\frac{4}{n-2}\Big{)}^{k}\sum_{\ell=1}^{k}p^{\ell}\leq\frac{1}{1-p^{-1}}\Big{(}\frac{4p}{n-2}\Big{)}^{k}\leq\Big{(}\frac{8p}{n-2}\Big{)}^{k}$

and therefore $\lVert f\rVert_{p}\leq(8p/(n-2))^{k/2}$ for any $p\geq 2$. If
$p\leq 2$, we may estimate $\lVert f\rVert_{p}\leq\lVert
f\rVert_{2}\leq(16/(n-2))^{k/2}$. In particular, we obtain that for any $m\geq
1$,

$\lVert|f|^{2/k}\rVert_{m}=\lVert f\rVert_{2m/k}^{2/k}\leq\gamma m$

with $\gamma=16/(n-2)$. In particular, by an easy calculation (cf. [BGS19, Eq.
(2.17)], this yields that $\int_{W_{n,d}}\exp(c^{\prime}|f|)d\mu_{n,d}\leq 2$
for $c^{\prime}=1/(2\gamma e)$, i. e. (1).

To show (2), taking roots in (6) yields

$\lVert
f\rVert_{p}\leq\sum_{\ell=1}^{k-1}\Big{(}\frac{4(p-1)}{n-2}\Big{)}^{\ell/2}\lVert
f^{(\ell)}\rVert_{\mathrm{op},2}+\Big{(}\frac{4(p-1)}{n-2}\Big{)}^{k/2}\lVert
f^{(k)}\rVert_{\mathrm{op},\infty}.$

Now, [GSS21b, Proposition 4] yields

$\mu_{n,d}(|f-\mu_{n,d}(f)|\geq t)\leq
2\exp\Big{(}-\frac{1}{C}\min\Big{(}\min_{\ell=1,\ldots,k-1}\frac{t^{2/\ell}}{\lVert
f^{(\ell)}\rVert_{\mathrm{op},2}^{2/\ell}},\frac{t^{2/k}}{\lVert
f^{(k)}\rVert_{\mathrm{op},\infty}^{2/k}}\Big{)}\Big{)},$

where we may choose

$C=\frac{4e^{2}k^{2}}{\log(2)(n-2)},$

which completes the proof. ∎

For the proof of Theorem 1.1, we moreover need the following lemma which
relates the $L^{2}$ norms of the derivatives of first and second order under a
centering assumption.

######  Lemma 6.3.

Let $f\colon W_{n,d}\to\mathbb{R}$ be a $\mathcal{C}^{2}$-smooth function
which has centered first order intrinsic derivatives, i. e.
$\mu_{n,d}(\nabla_{W}f)=0$. Then,

$\int_{W_{n,d}}|\nabla_{W}f|^{2}d\mu_{n,d}\leq\frac{8}{n-2-8d}\int_{W_{n,d}}\lVert
f^{\prime\prime}_{W}\rVert_{\mathrm{HS}}^{2}d\mu_{n,d}.$

###### Proof.

Write

$|\nabla_{W}f(A)|^{2}=\sum_{i,j}\langle\nabla_{W}f(A),e_{ij}\rangle^{2},$

where $i\leq n$, $j\leq d$ and $e_{ij}$ is the “indicator matrix” whose
$(i,j)$-th entry equals $1$ while all other entries are zero. By assumption,
the functions $A\mapsto\langle\nabla_{W}f(A),e_{ij}\rangle$ are centered.
Therefore, applying the Poincaré inequality (6.3) yields

$\frac{n-2}{4}\int_{W_{n,d}}\langle\nabla_{W}f(A),e_{ij}\rangle^{2}d\mu_{n,d}\leq\int_{W_{n,d}}|\nabla_{W}(\nabla_{W}f(A),e_{ij}\rangle)|^{2}d\mu_{n,d}.$

By Proposition 3.3, we have

$\nabla_{W}(\nabla_{W}f(A),e_{ij}\rangle)=f^{\prime\prime}_{W}(A)e_{ij}-P_{A}(\nabla_{W}f(A)(A\circ
e_{ij})),$

so that we may estimate

$\displaystyle\quad\int_{W_{n,d}}|\nabla_{W}(\nabla_{W}f(A),e_{ij}\rangle)|^{2}d\mu_{n,d}$
$\displaystyle\leq
2\int_{W_{n,d}}|f^{\prime\prime}_{W}(A)e_{ij}|^{2}d\mu_{n,d}+2\int_{W_{n,d}}|P_{A}(\nabla_{W}f(A)(A\circ
e_{ij}))|^{2}d\mu_{n,d}.$

By contractivity and using $|A_{ij}|^{2}\leq 1$,

$|P_{A}(\nabla_{W}f(A)(A\circ e_{ij}))|^{2}\leq|(\nabla_{W}f(A)(A\circ
e_{ij})|^{2}\leq d|\nabla_{W}f(A)|^{2}.$

Plugging in, summing up and noting that

$\sum_{i,j}|f_{W}^{\prime\prime}(A)e_{ij}|^{2}=\lVert
f_{W}^{\prime\prime}(A)\rVert_{\mathrm{HS}}^{2}$

(recall that $i\leq n$, $j\leq d$ and $f^{\prime\prime}_{W}(A)$ is an
$nd\times nd$ matrix) completes the proof. ∎

###### Proof of Theorem 1.1.

Obviously, we may argue as in the proof of Theorem 1.3 with (6.2) replaced by
(6.1) and [BGS19, Lemma 4.1] replaced by Proposition 4.1 (thus involving
intrinsic derivatives) to obtain

$\lVert
f\rVert_{p}^{2}\leq\frac{4(p-1)}{n-2}\lVert\nabla_{W}f\rVert_{2}^{2}+\Big{(}\frac{4(p-1)}{n-2}\Big{)}^{2}\lVert
f^{\prime\prime}_{W}\rVert_{\mathrm{op},\infty}^{2},$

which is the “intrinsic analogue” of (6) for $k=2$. Therefore, to see (1) and
(2), we may just copy the proof of Theorem 1.3. Moreover, (3) follows by
applying Lemma 6.3. ∎

Finally, to address Theorem 1.2 and Theorem 1.4, note that for any locally
Lipschitz function $g\colon G_{n,d}\to\mathbb{R}$ and for any $p\geq 2$,

$\lVert g\rVert_{p}^{2}\leq\lVert
g\rVert_{2}^{2}+8(n-2)^{-1}(p-2)\lVert\nabla_{W}Gg\rVert_{p}^{2},$

which is an analogue of (6.1). From here, we may essentially copy the proofs
of Theorem 1.1 and Theorem 1.3 with the obvious adaptions (which, in
particular, lead to slightly different constants).

##  7\. Concentration results for polynomial chaos

Arguably the most classical object in the study of higher order concentration
is polynomial chaos. In the context of Stiefel manifolds, by a polynomial
chaos of order $k$ we refer to functionals of the form

$f_{k}(A):=\sum_{i_{1},j_{1}=1}^{n,d}\cdots\sum_{i_{k},j_{k}=1}^{n,d}c_{i_{1}j_{1},\ldots,i_{k}j_{k}}A_{i_{1}j_{1}}\cdots
A_{i_{k}j_{k}},$

where $c_{i_{1}j_{1},\ldots,i_{k}j_{k}}$ are real-valued coefficients. For
$k=1,2$, these are linear or quadratic forms, respectively. Functionals of
this type are for instance related to angles and distances between such random
subspaces like $\det(A^{T}C)$, where $A\in W_{n,d}$ and
$C\in\mathbb{R}^{n\times d}$ is a matrix of coefficients which might be random
with, say, independent entries. (In this case, $\det(A^{T}C)$ is the scalar
product of the exterior $d$-forms induced by $A$ and $C$ representing
subspaces of dimension $d$.) In this section, we will mainly be interested in
the case of $k=2$ but also provide some basic comments on other values of $k$.

In particular, information about the (mixed) moments of the Stiefel matrix
entries $A_{ij}$ of order up to $k$ is needed. To this end, note that the
$A_{ij}$ are identically distributed uncorrelated random variables with mean
zero and variance $1/n$. Indeed, using the invariance under orthogonal
transformations we clearly have $A_{ij}\sim A_{k\ell}$ and also, multiplying
suitable columns or rows by $-1$,

$\int_{W_{n,d}}A_{ij}A_{k\ell}d\mu_{n,d}=-\int_{W_{n,d}}A_{ij}A_{k\ell}d\mu_{n,d}=0$

for any $(i,j)\neq(k,\ell)$. Moreover, the $A_{ij}$ are centered since
$A\sim-A$, and we have

$d=\sum_{k,\ell=1}^{n,d}A_{k\ell}^{2}=\int_{W_{n,d}}\sum_{k,\ell=1}^{n,d}A_{k\ell}^{2}d\mu_{n,d}=nd\int_{W_{n,d}}A_{ij}^{2}d\mu_{n,d}$

for any $(i,j)$. Some of these relations may also deduced from the fact that
the marginal distribution of each column of $A$ is the uniform distribution on
the sphere.

These results also imply that the entries of a Grassmannian (projection)
matrix $P$ satisfy

$\int_{G_{n,d}}P_{ij}d\nu_{n,d}=\begin{cases}d/n,&i=j\\\ 0,&i\neq
j\end{cases}.$

To see this, it suffices to note that writing $P=P_{A}=AA^{T}$, we have
$P_{ij}=\sum_{k=1}^{d}A_{ik}A_{jk}$.

###  7.1. First order results

To start, let us briefly study linear forms on $W_{n,d}$, which we may rewrite
as

$f_{1}(A)=\langle V,A\rangle$

for some matrix of coefficients $V\in\mathbb{R}^{n\times d}$. Since
$\nabla_{W}f_{1}(A)=P_{A}(V)$ and the entries of $A$ are centered, (1.2)
immediately yields

(7.1) $\mu_{n,d}(|f_{1}|\geq t)\leq 2\exp\Big{(}-\frac{(n-1)t^{2}}{8\lVert
P_{A}(V)\rVert_{\mathrm{HS},\infty}^{2}}\Big{)}.$

One may also choose the vector of coefficients $V$ at random according to some
distribution with independent entries or dependent entries with higher order
uncorrelatedness conditions. In this case, results like (7.1) hold
conditionally given $V$, for instance. Functionals of this type may be
regarded as natural generalizations of weighted sums $\langle
X,\theta\rangle=\theta_{1}X_{1}+\dots+\theta_{n}X_{n}$ with uniformly
distributed weights $\theta=(\theta_{1},\ldots,\theta_{n})\in S^{n-1}$ and
some random vector $X$ with higher order uncorrelated entries (cf. [BCG18,
BCG20]), replacing the sphere by the Stiefel manifold.

###  7.2. Second order results

The central object of this section are quadratic forms on $W_{n,d}$. Recall
that a classical (second order) concentration result for quadratic forms in
independent subgaussian random variables is the famous Hanson–Wright
inequality, cf. [HW71]. It states that for a random vector
$X=(X_{1},\ldots,X_{n})$ with independent centered components with variance
$1$,

$\mathbb{P}(|X^{T}MX-\mathrm{tr}(M)|\geq t)\leq
2\exp\Big{(}-\frac{1}{C}\min\Big{(}\frac{t^{2}}{\lVert
M\rVert_{\mathrm{HS}}^{2}},\frac{t}{\lVert
M\rVert_{\mathrm{op}}}\Big{)}\Big{)}$

for any $t\geq 0$, where the constant $C>0$ depends on the subgaussian norm of
the coordinates $X_{i}$.

We derive analogues of the Hanson–Wright inequality for Stiefel manifolds, i.
e., we consider

(7.2) $f_{2}(A)=\mathrm{vec}(A)^{T}M\mathrm{vec}(A)\equiv A^{T}MA$

for some matrix $M\in\mathbb{R}^{nd\times nd}$ which, for simplicity, we
assume to be symmetric. Note that the informal notation $A^{T}MA$ extends
(2.1). For instance, to go back to the example sketched at the beginning of
this section, we may consider $\det(A^{T}C)$, where for simplicity we assume
$C\in\mathbb{R}^{n\times d}$ to be fixed, in the case of $d=2$. Here,

$\det(A^{T}C)=\Big{(}\sum_{i=1}^{n}a_{i1}c_{i1}\Big{)}\Big{(}\sum_{j=1}^{n}a_{i2}c_{i2}\Big{)}-\Big{(}\sum_{i=1}^{n}a_{i1}c_{i2}\Big{)}\Big{(}\sum_{j=1}^{n}a_{i2}c_{i1}\Big{)}=A^{T}MA$

for the matrix $M$ with entries $M_{ij,kl}=(c_{ij}c_{kl}-c_{il}c_{kj})/2$.
More advanced examples will be discussed in a further subsection.

In the next theorem, we present three different Hanson–Wright type
inequalities for Stiefel manifolds. Write

$U:=\Big{(}\sum_{k,l}M_{ij,kl}A_{kl}\Big{)}_{i,j},\qquad B:=M-(A\circ
U)\otimes I_{n}$

for $i,k\leq n$, $j,l\leq d$, noting that $\nabla_{W}f_{2}(A)=2P_{A}U$ and
$f^{\prime\prime}_{2W}(A)=2P_{A}BP_{A}$.

######  Theorem 7.1 (Hanson–Wright type inequalities for Stiefel manifolds).

  1. (1)

For any $t\geq 0$, we have

$\mu_{n,d}(|f_{2}-\mathrm{tr}(M)/n|\geq t)\leq
2\exp\Big{(}-\frac{1}{C}\min\Big{(}\frac{(n-2)^{2}t^{2}}{\lVert
M\rVert_{\mathrm{HS}}^{2}},\frac{(n-2)t}{\lVert
M\rVert_{\mathrm{op}}}\Big{)}\Big{)},$

where we may choose $C=128e^{2}/\log(2)$.

  2. (2)

For any $t\geq 0$, we have

$\mu_{n,d}(|f_{2}-\mathrm{tr}(M)/n|\geq t)\leq
2\exp\Big{(}-\frac{1}{C}\min\Big{(}\frac{(n-2)t^{2}}{\lVert
P_{A}U\rVert_{\mathrm{HS},2}^{2}},\frac{(n-2)t}{\lVert
P_{A}BP_{A}\rVert_{\mathrm{op},\infty}}\Big{)}\Big{)},$

where we may choose $C=32e^{2}/\log(2)$.

  3. (3)

For any $t\geq 0$, we have

$\mu_{n,d}(|f_{2}-\mathrm{tr}(M)/n|\geq t)\leq
2\exp\Big{(}-\frac{1}{C}\min\Big{(}\frac{(n-2-8d)^{2}t^{2}}{\lVert
P_{A}BP_{A}\rVert_{\mathrm{HS},2}^{2}},\frac{(n-2)t}{\lVert
P_{A}BP_{A}\rVert_{\mathrm{op},\infty}}\Big{)}\Big{)},$

where we may choose $C=256e^{2}/\log(2)$.

For the notation used in (2) and (3), cf. (3.7). The three different analogues
of the classical Hanson–Wright inequality presented in Theorem 7.1 reflect a
certain flexibility which can turn out to be useful in applications (in
particular, it would be hard to speak of a single canonical Stiefel variant of
the Hanson–Wright inequality). Certainly, (1) is the most immediate analogue,
involving quantities which can easily be calculated. However, unlike (2) and
(3), it does not make use of intrinsic derivatives and can therefore be less
accurate.

For instance, in the trivial case $M=I_{nd}$ (thus,
$f_{2}(A)-\mathrm{tr}(M)/n\equiv 0$) (1) does not yield an optimal result,
while (2) and (3) do. On the other hand, if $M_{ij,kl}=1$ for every $i,j,k,l$,
(1), (2) and (3) are not substantially different. As compared to (2), (3) is
nearer to the classical Hanson–Wright inequality, but this is the cost of an
additional $d$-dependence in the subgaussian term, which can be avoided in
(2).

For explicit calculations of the $L^{2}$ norms appearing in (2) and (3), note
that we may construct a uniformly distributed Stiefel element via
$A=O_{n}I_{n,d}O_{d}$, where $O_{n}$ and $O_{d}$ are distributed according to
the Haar measure on $SO(n)$ and on $SO(d)$, respectively, and $I_{n,d}$ is the
element in $W_{n,d}$ whose columns are the first $d$ unit vectors. Then
$P_{A}=AA^{T}=O_{n}I_{n,d}I_{n,d}^{T}O_{n}^{-1}=(\langle
p_{d}\theta_{j},p_{d}\theta_{k}\rangle)_{j,k=1,\dots,n}$ (where
$O_{n}=(\theta_{1},\ldots,\theta_{n})$ and $p_{d}$ is the projection to the
first $d$ coordinates). Thus rewriting $\lVert
P_{A}U\rVert_{\mathrm{HS},2}^{2}$ resp. $\lVert
P_{A}BP_{A}\rVert_{\mathrm{HS},2}^{2}$ as linear combinations of products of
components of orthogonal matrices, we may now average them over $O(n)$ and
$O(d)$ by applying formulas from the Weingarten calculus. The only non-
vanishing products in this average are those whose indices are equal subject
to a scheme of two pair partitions which will determine a so-called Weingarten
function which is a rational function of $n$ of order $O(n^{-2})$ or smaller.
For explicit formulas covering our cases see in particular [CM09], table VII.
For an introduction to the underlying theory we refer to [CMN21] and for more
details to e. g. [CS06, BCS11].

###### Proof of Theorem 7.1.

To see (1), we extend $f_{2}$ to $X\in\mathbb{R}^{n\times d}$ by the canonical
choice $f_{2}(X)=\mathrm{vec}(X)^{T}M\mathrm{vec}(X)$ and apply Theorem 1.3
(2). Clearly, $f^{\prime\prime}(A)=2M$, and in the subgaussian term, we use
that by Poincaré inequality,

$\lVert\nabla f\rVert_{\mathrm{HS},2}\leq\frac{4}{n-2}\lVert
f^{\prime\prime}\rVert_{\mathrm{HS},2}$

whenever $\nabla f$ has centered components (indeed, the proof is a
simplification of the proof of Lemma 6.3). To see (2) and (3), it remains to
apply Theorem 1.1 (2) and (3), respectively, using that
$\nabla_{W}f(A)=2P_{A}U$ and $f^{\prime\prime}_{W}(A)=2P_{A}BP_{A}$. ∎

###  7.3. Higher orders

Let us briefly comment on possible results for polynomial chaos of order
$k\geq 3$. In general, similarly as in Theorem 7.1 (1) one may derive
concentration bounds for $f_{k}$ by applying Theorem 1.3 (for order $k$).
Here, for $k=3$, by similar arguments as for orders $1$ and $2$ (including
symmetry) we may use that $\int_{W_{n,d}}A_{ij}A_{kl}A_{pq}d\mu_{n,d}=0$ for
any choice of the indices. However, already for $k=4$ the calculation of the
mixed moments (e. g. with each index appearing twice) can turn out to be quite
involved.

Using the $L^{p}$ norm inequalities from Section 6, it is also possible to
derive results in the style of [AW15] (not only for polynomial chaos but also
for general $\mathcal{C}^{k}$ functions) which yield highly elaborate and
accurate tail bounds in terms of a large family of tensor-type norms. We will
not pursue this direction in this note.

Note though that all these results will make use of Euclidean derivatives
only. In fact, the calculus of intrinsic derivatives gets increasingly
involved as the order increases. For functions on the sphere, in [BGS19]
higher order concentration results depending on spherical partial derivatives
have been derived, which can be thought of as a sort of intermediate object
between Euclidean and intrinsic derivatives.

###  7.4. Applications

Typical applications of concentration inequalities for Stiefel and Grassmann
functionals include bounds for distances between subspaces. If
$E,F\subset\mathbb{R}^{n}$ are two subspaces of dimension $d$, canonical
choices for the distance between them are $\lVert
P_{E}-P_{F}\rVert_{\mathrm{HS}}$ or $\lVert P_{E}-P_{F}\rVert_{\mathrm{op}}$.
In the sequel, we will always choose one subspace at random, while the other
one is fixed.

As a start, we consider the following simple example, in which the function
$G_{n,d}\ni P\mapsto\lvert P-P_{F}\rvert\equiv\lVert
P-P_{F}\rVert_{\mathrm{HS}}$ is studied.

######  Proposition 7.2.

Let $F\subset\mathbb{R}^{n}$ be any fixed $d$-dimensional subspace of
$\mathbb{R}^{n}$, and denote by $P_{F}$ the corresponding projection matrix.
Then, for any $t\geq 0$,

$\nu_{n,d}(\lvert\lvert P-P_{F}\rvert^{2}-2d(1-d/n)\rvert\geq t)\leq
2\exp(-(n-1)t^{2}/(64d)).$

###### Proof.

Note that $\lvert P-P_{F}\rvert^{2}=2(d-\langle P,P_{F}\rangle)$, which has
expectation $2d(1-d/n)$ since

$\int_{G_{n,d}}\langle P,P_{F}\rangle dP=d/n\mathrm{tr}(P_{F})=d^{2}/n.$

Furthermore, the function $\mathbb{R}^{n\times n}\ni X\mapsto 2(d-\langle
X,P_{F}\rangle)$ has derivative $-2P_{F}$, which has Euclidean
(Hilbert–Schmidt) norm $\sqrt{d}$. The result now follows immediately by
combining (1.2) (in the Grassmann version) and Proposition 5.1. ∎

It is possible to remove the square and consider fluctuations of $\lvert
P-P_{F}\rvert$ by adapting some of the arguments used in the proof of Lemma
7.4 below. We skip the details, however, especially since we are rather
interested in applying higher order concentration results in what follows.

To this end, let us consider another distance between two subspaces $E$ and
$F$ of dimension $d$ which has been studied in [WWF06] and subsequent papers.
Here, we chose any orthonormal basis of $E$, say, $e_{1},\ldots,e_{d}$, and
set

$\mathrm{dist}(E,F):=\Big{(}\sum_{j=1}^{d}\mathrm{dist}(e_{j},F)^{2}\Big{)}^{1/2},$

where for any $x\in\mathbb{R}^{n}$, $\mathrm{dist}(x,F)=\min_{y\in F}|x-y|$ is
the usual point-to-subspace distance induced by the Euclidean norm. It is not
hard to see that $\mathrm{dist}(E,F)$ does not depend on the choice of the
orthonormal basis of $E$, cf. [WWF06, Theorem 1].

As above, let us fix $F$ and consider the distance as a function of the
subspace $E$. In view of the definition via orthonormal bases (and the
invariance under change of the basis), we may then regard $\mathrm{dist}$ as a
function on the Stiefel manifold $W_{n,d}$. That is, writing $A_{\bullet j}$
for the columns of $A\in W_{n,d}$, we shall study the fluctuations of

$\mathrm{dist}(A,F):=\Big{(}\sum_{j=1}^{d}\mathrm{dist}(A_{\bullet
j},F)^{2}\Big{)}^{1/2}$

under $\mu_{n,d}$. In this situation, the function fits into a framework which
allows to derive concentration bounds via Hanson–Wright inequalities similarly
as in [RV13, Corollary 3.1], where distances of random vectors to a fixed
subspace have been stuied. Under the uniform distribution on the Stiefel
manifold, we obtain the following result.

######  Proposition 7.3.

Let $F\subset\mathbb{R}^{n}$ be any fixed $d$-dimensional subspace of
$\mathbb{R}^{n}$. Then, for any $t\geq 0$,

$\mu_{n,d}(|\mathrm{dist}(A,F)-d/\sqrt{n}|\geq t)\leq 2\exp(-(n-2)t^{2}/C)$

with $C=384e^{2}/\log(2)$.

To prove Proposition 7.3, we need the following lemma, which is itself of
independent interest since it generalizes [RV13, Theorem 2.1] (cf. also [S20+,
Proposition 2.2]). In essence, given any matrix $M\in\mathbb{R}^{nd\times nd}$
it controls the fluctuations of the Euclidean norm of $MA$ around the
Hilbert–Schmidt norm of $M$, rescaled by $n^{-1/2}$ according to the standard
deviation of the entries of $A\sim\mu_{n,d}$.

######  Lemma 7.4.

For any fixed $M\in\mathbb{R}^{nd\times nd}$, we have

$\mu_{n,d}(|\lvert MA\rvert-\lVert M\rVert_{\mathrm{HS}}/\sqrt{n}|\geq t)\leq
2\exp\Big{(}-\frac{(n-2)t^{2}}{C\lVert M\rVert_{\mathrm{op}}^{2}}\Big{)}$

with $C=384e^{2}/\log(2)$.

###### Proof.

Let $Q=M^{T}M$, so that $|MA|^{2}=A^{T}QA$ (recall (2.1) and (7.2)) and
$\mathrm{tr}(Q)=\lVert M\rVert_{\mathrm{HS}}^{2}$. Assuming that $\lVert
M\rVert_{\mathrm{HS}}=\sqrt{n}$, Theorem 7.1 (1) yields

$\displaystyle\mu_{n,d}(|\lvert MA\rvert^{2}-1|\geq t)$ $\displaystyle\leq
2\exp\Big{(}-\frac{1}{C}\min\Big{(}\frac{(n-2)^{2}t^{2}}{n\lVert
M\rVert_{\mathrm{op}}^{2}},\frac{(n-2)t}{\lVert
M\rVert_{\mathrm{op}}^{2}}\Big{)}\Big{)}$ $\displaystyle\leq
2\exp\Big{(}-\frac{n-2}{3C\lVert
M\rVert_{\mathrm{op}}^{2}}\min(t^{2},t)\Big{)},$

where the first step follows from $\lVert Q\rVert_{\mathrm{HS}}^{2}\leq\lVert
M\rVert_{\mathrm{op}}^{2}\lVert M\rVert_{\mathrm{HS}}^{2}=n\lVert
M\rVert_{\mathrm{op}}^{2}$ and $\lVert Q\rVert_{\mathrm{op}}\leq\lVert
M\rVert_{\mathrm{op}}^{2}$, and $C=128e^{2}/\log(2)$.

Now, as in [RV13], we use the inequality $\lvert z-1\rvert\leq\min(\lvert
z^{2}-1\rvert,\lvert z^{2}-1\rvert^{1/2})$, giving for any $t\geq 0$

$\mu_{n,d}(|\lvert MA\rvert-1|\geq t\Big{)}\leq\mu_{n,d}(|\lvert
MA\rvert^{2}-1|\geq\max(t,t^{2})).$

Combining this with the first step and using that
$\min(\max(t,t^{2})^{2},\max(t,t^{2}))=t^{2}$ yields

$\mu_{n,d}(|\lvert MA\rvert-1|\geq t)\leq
2\exp\Big{(}-\frac{(n-2)t^{2}}{3C\lVert M\rVert_{\mathrm{op}}^{2}}\Big{)},$

i. e. the claim for for $\lVert M\rVert_{\mathrm{HS}}=\sqrt{n}$. The general
case now follows by considering $\tilde{M}\coloneqq\sqrt{n}M\lVert
M\rVert_{\mathrm{HS}}^{-1}$, noting that

$\mu_{n,d}(|\lvert MA\rvert-\lVert M\rVert_{\mathrm{HS}}/\sqrt{n}|\geq
t)=\mu_{n,d}(|\lvert\tilde{M}A\rvert-1|\geq\sqrt{n}t\lVert
M\rVert_{\mathrm{HS}}^{-1}).$

∎

###### Proof of Proposition 7.3.

Let $P_{F}$ be the orthogonal projection onto $F$, and denote by $A_{\bullet
j}$ the columns of $A$. Then, $\mathrm{dist}(A_{\bullet j},F)=|P_{F}A_{\bullet
j}|$. In particular, if $M:=I_{d}\otimes P_{F}$ is the $nd\times nd$ block
diagonal matrix whose diagonal consists of $d$ copies of the matrix $P_{F}$,
$\mathrm{dist}(A,F)=|MA|$, so that we may apply Lemma 7.4 (1). Here we use
that $\lVert M\rVert_{\mathrm{HS}}^{2}=d\lVert
P_{F}\rVert_{\mathrm{HS}}^{2}=d^{2}$ and moreover that since $d$ eigenvalues
of $P_{F}$ are $1$ and all other eigenvalues are $0$, we have $\lVert
M\rVert_{\mathrm{op}}=\lVert P_{F}\rVert_{\mathrm{op}}=1$. ∎

In fact, it is not even necessary to consider subspaces $F$ of the same
dimension $d$. If $F\subset\mathbb{R}^{n}$ is a subspace of any dimension
$m<n$, the definitions of $\mathrm{dist}(E,F)$ and $\mathrm{dist}(A,F)$
continue to hold (even if $\mathrm{dist}(E,F)$ is no longer symmetric in $E$
and $F$ in this case), and with minimal adaptions in the proof of Proposition
7.3, one can show that

$\mu_{n,d}(|\mathrm{dist}(A,F)-\sqrt{md/n}|\geq t)\leq 2\exp(-(n-2)t^{2}/C)$

with $C=384e^{2}/\log(2)$.

##  Appendix A Proofs of the results from Section 5

###### Proof of Proposition 5.3.

Considering the function

$\psi_{V}(P):=\langle\nabla_{G}f(P),V\rangle=\langle[P,[P,\nabla
f(P)]],V\rangle,$

let us calculate $\nabla\psi_{V}(P)$. Choosing some extension of $f$ onto an
open neighborhood of $G_{n,d}$ in $\mathbb{R}^{n\times n}$, the function

$\displaystyle\psi_{V}(X)$
$\displaystyle:=\langle[X,[X,\pi_{\mathrm{sym}}(\nabla f(X))]],V\rangle$
$\displaystyle=\mathrm{tr}(V^{T}(X\pi_{\mathrm{sym}}(\nabla
f(X))+\pi_{\mathrm{sym}}(\nabla f(X))X-2P\pi_{\mathrm{sym}}(\nabla f(X))X))$

is a smooth extension of $\psi_{V}(P)$. Note that $\pi_{\mathrm{sym}}$ has
derivative

$\frac{d\pi_{\mathrm{sym}}(Y)}{dY}=\frac{1}{2}\frac{d(Y+Y^{T})}{dY}=\frac{1}{2}(I_{n^{2}}+K_{n,n}).$

As a consequence, by chain rule,

$\frac{d\pi_{\mathrm{sym}}(\nabla
f(X))}{dX}=\frac{1}{2}(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X).$

Let us now calculate

$\frac{d}{dX}[V^{T}(X\pi_{\mathrm{sym}}(\nabla f(X))+\pi_{\mathrm{sym}}(\nabla
f(X))X-2X\pi_{\mathrm{sym}}(\nabla f(X))X)].$

First, by Leibniz rule,

$\displaystyle\quad\frac{d}{dX}[V^{T}X\pi_{\mathrm{sym}}(\nabla f(X))]$
$\displaystyle=(\pi_{\mathrm{sym}}(\nabla f(X))\otimes I_{n})(I_{n}\otimes
V^{T})+\frac{1}{2}(I_{n}\otimes(V^{T}X))(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X)$
$\displaystyle=\pi_{\mathrm{sym}}(\nabla f(X))\otimes
V^{T}+\frac{1}{2}(I_{n}\otimes(V^{T}X))(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X).$

Next, note that similarly,

$\frac{d(\pi_{\mathrm{sym}}(\nabla f(X))X)}{dX}=\frac{1}{2}(X^{T}\otimes
I_{n})(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X)+I_{n}\otimes\pi_{\mathrm{sym}}(\nabla
f(X)).$

In particular, we obtain that

$\displaystyle\quad\frac{d}{dX}[V^{T}\pi_{\mathrm{sym}}(\nabla f(X))X]$
$\displaystyle=(I_{n}\otimes V^{T})\Big{(}\frac{1}{2}(X^{T}\otimes
I_{n})(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X)+I_{n}\otimes\pi_{\mathrm{sym}}(\nabla
f(X))\Big{)}$ $\displaystyle=\frac{1}{2}(X^{T}\otimes
V^{T})(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X)+I_{n}\otimes(V^{T}\pi_{\mathrm{sym}}(\nabla
f(X))).$

Finally,

$\displaystyle\quad\frac{d}{dX}[V^{T}X\pi_{\mathrm{sym}}(\nabla f(X))X]$
$\displaystyle=((X^{T}\pi_{\mathrm{sym}}[\nabla f(X)])\otimes
I_{n})\frac{d(V^{T}X)}{dX}+(I_{n}\otimes(V^{T}X))\frac{d}{dX}[\pi_{\mathrm{sym}}(\nabla
f(X))X]$ $\displaystyle=((X^{T}\pi_{\mathrm{sym}}[\nabla f(X)])\otimes
I_{n})(I_{n}\otimes V^{T})$
$\displaystyle\quad+(I_{n}\otimes(V^{T}X))\Big{(}\frac{1}{2}(X^{T}\otimes
I_{n})(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X)+I_{n}\otimes\pi_{\mathrm{sym}}(\nabla
f(X))\Big{)}$ $\displaystyle=(X^{T}\pi_{\mathrm{sym}}[\nabla f(X)])\otimes
V^{T}+\frac{1}{2}(X^{T}\otimes(V^{T}X))(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X)$
$\displaystyle\quad+I_{n}\otimes(V^{T}X\pi_{\mathrm{sym}}(\nabla f(X))).$

Consequently, by chain rule,

$\displaystyle\quad\frac{d}{dX}\mathrm{tr}(V^{T}(X\pi_{\mathrm{sym}}(\nabla
f(X))+\pi_{\mathrm{sym}}(\nabla f(X))X-2X\pi_{\mathrm{sym}}(\nabla f(X))X))$
$\displaystyle=\mathrm{vec}(I_{n})^{T}\Big{(}\pi_{\mathrm{sym}}(\nabla
f(X))\otimes
V^{T}+\frac{1}{2}(I_{n}\otimes(V^{T}X))(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X)$
$\displaystyle\quad+\frac{1}{2}(X^{T}\otimes
V^{T})(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X)+I_{n}\otimes(V^{T}\pi_{\mathrm{sym}}(\nabla
f(X)))$ $\displaystyle\quad-2(X^{T}\pi_{\mathrm{sym}}[\nabla f(X)])\otimes
V^{T}-(X^{T}\otimes(V^{T}X))(I_{n^{2}}+K_{n,n})f^{\prime\prime}(X)$
$\displaystyle\quad-2I_{n}\otimes(V^{T}X\pi_{\mathrm{sym}}(\nabla
f(X)))\Big{)}.$

Switching from $d/(dX)$ to $\mathrm{vec}(\nabla)$ (and thus, transposing), we
obtain

$\displaystyle\mathrm{vec}(\nabla\psi_{V}(X))$
$\displaystyle=\Big{(}\pi_{\mathrm{sym}}(\nabla f(X))\otimes
V+\frac{1}{2}f^{\prime\prime}(X)(I_{n^{2}}+K_{n,n})(I_{n}\otimes(X^{T}V))$
$\displaystyle\quad+\frac{1}{2}f^{\prime\prime}(X)(I_{n^{2}}+K_{n,n})(X\otimes
V)+I_{n}\otimes(\pi_{\mathrm{sym}}(\nabla f(X))V)$
$\displaystyle\quad-2(\pi_{\mathrm{sym}}(\nabla f(X))X)\otimes
V-f^{\prime\prime}(X)(I_{n^{2}}+K_{n,n})(X\otimes(X^{T}V))$
$\displaystyle\quad-2I_{n}\otimes(\pi_{\mathrm{sym}}(\nabla
f(X))X^{T}V)\Big{)}\mathrm{vec}(I_{n}).$

Let us simplify this further. We have

$\displaystyle(\pi_{\mathrm{sym}}(\nabla f(X))\otimes V)\mathrm{vec}(I_{n})$
$\displaystyle=\mathrm{vec}[V\pi_{\mathrm{sym}}(\nabla f(X))],$
$\displaystyle(I_{n}\otimes(\pi_{\mathrm{sym}}(\nabla
f(X))V))\mathrm{vec}(I_{n})$
$\displaystyle=\mathrm{vec}[(\pi_{\mathrm{sym}}(\nabla f(X))V)],$
$\displaystyle((\pi_{\mathrm{sym}}(\nabla f(X))X)\otimes
V)\mathrm{vec}(I_{n})$
$\displaystyle=\mathrm{vec}[VX^{T}\pi_{\mathrm{sym}}(\nabla f(X))],$
$\displaystyle(I_{n}\otimes(\pi_{\mathrm{sym}}(\nabla
f(X))X^{T}V))\mathrm{vec}(I_{n})$
$\displaystyle=\mathrm{vec}[\pi_{\mathrm{sym}}(\nabla f(X))X^{T}V)].$

Recalling that $\pi_{\mathrm{sym}}=(I_{n^{2}}+K_{n,n})/2$, we may further
rewrite

$\displaystyle\frac{1}{2}f^{\prime\prime}(X)(I_{n^{2}}+K_{n,n})(I_{n}\otimes(X^{T}V))\mathrm{vec}(I_{n})$
$\displaystyle=f^{\prime\prime}(X)\pi_{\mathrm{sym}}\mathrm{vec}(X^{T}V),$
$\displaystyle\frac{1}{2}f^{\prime\prime}(X)(I_{n^{2}}+K_{n,n})(X\otimes
V)\mathrm{vec}(I_{n})$
$\displaystyle=f^{\prime\prime}(X)\pi_{\mathrm{sym}}\mathrm{vec}(VX^{T}),$
$\displaystyle
f^{\prime\prime}(X)(I_{n^{2}}+K_{n,n})(X\otimes(X^{T}V))\mathrm{vec}(I_{n})$
$\displaystyle=f^{\prime\prime}(X)\pi_{\mathrm{sym}}\mathrm{vec}(X^{T}VX^{T})$

Summing up and restricting to $P\in G_{n,d}$ (which is symmetric, in
particular),

$\displaystyle\mathrm{vec}(\nabla\psi_{V}(P))$
$\displaystyle=f^{\prime\prime}(P)\pi_{\mathrm{sym}}\mathrm{vec}([P,[P,V]])+2\mathrm{vec}[\pi_{\mathrm{sym}}(V\pi_{\mathrm{sym}}(\nabla
f(P)))]$
$\displaystyle\quad-4\mathrm{vec}[\pi_{\mathrm{sym}}(VP\pi_{\mathrm{sym}}(\nabla
f(P)))].$

Next we calculate the intrinsic derivatives. First, we need to project onto
$\mathbb{R}^{n\times n}_{\mathrm{sym}}$ by applying $\pi_{\mathrm{sym}}$.
Using the notation from Proposition 5.2, this yields

$\displaystyle\pi_{\mathrm{sym}}\nabla\psi_{V}(P)$
$\displaystyle=\pi_{\mathrm{sym}}f^{\prime\prime}(P)\pi_{\mathrm{sym}}\mathrm{vec}([P,[P,V]])+2\mathrm{vec}[\pi_{\mathrm{sym}}(V\pi_{\mathrm{sym}}(\nabla
f(P)))]$
$\displaystyle\quad-4\mathrm{vec}[\pi_{\mathrm{sym}}(VP\pi_{\mathrm{sym}}(\nabla
f(P)))].$

Now we project onto $T_{P}$ by applying $\pi_{P}$. This yields

$\displaystyle\nabla_{G}\psi_{V}(P)$
$\displaystyle=\pi_{P}\pi_{\mathrm{sym}}f^{\prime\prime}(P)\pi_{\mathrm{sym}}\pi_{P}(V)+2\pi_{P}\mathrm{vec}[\pi_{\mathrm{sym}}(V\pi_{\mathrm{sym}}(\nabla
f(P)))]$
$\displaystyle\quad-4\pi_{P}\mathrm{vec}[\pi_{\mathrm{sym}}(VP\pi_{\mathrm{sym}}(\nabla
f(P)))].$

By Proposition 5.2, it follows that

$\displaystyle f_{G}^{\prime\prime}(P)V$
$\displaystyle=\nabla_{G}\psi_{V}(P)-2\pi_{P}\mathrm{vec}[\pi_{\mathrm{sym}}(V\pi_{\mathrm{sym}}(\nabla
f(P)))]$
$\displaystyle\quad+4\pi_{P}\mathrm{vec}[\pi_{\mathrm{sym}}(VP\pi_{\mathrm{sym}}(\nabla
f(P)))]-[P,[\pi_{\mathrm{sym}}(\nabla f(P)),\pi_{P}V]].$

Now, a direct calculation yields

$\displaystyle\quad-2\pi_{P}\mathrm{vec}[\pi_{\mathrm{sym}}(V\pi_{\mathrm{sym}}(\nabla
f(P)))]+4\pi_{P}\mathrm{vec}[\pi_{\mathrm{sym}}(VP\pi_{\mathrm{sym}}(\nabla
f(P)))]$ $\displaystyle\quad-[P,[\pi_{\mathrm{sym}}(\nabla f(P)),\pi_{P}V]]$
$\displaystyle=-[P,[\pi_{P}\nabla f(P),V]]=-[P,[\nabla_{G}f(P),V]],$

which finishes the proof. ∎

###### Proof of Proposition 5.4.

Similarly as in the proof of Proposition 4.1, it follows from Proposition 5.3
applied to any $V\in\mathbb{R}^{n\times n}_{\mathrm{sym}}$ such that
$|V|\equiv\lVert V\rVert_{\mathrm{HS}}=1$ together with compactness arguments
that $|\nabla_{G}f|$ has finite Lipschitz semi-norm.

To show the identity for second order modulus of gradient, fix $P\in G_{n,d}$
and note that by the definition of the intrinsic gradient and Proposition 5.3,
we have

$\langle\nabla_{G}f(P^{\prime}),V\rangle=\langle\nabla_{G}f(P),V\rangle+\langle\tilde{V},P^{\prime}-P\rangle+o(|P^{\prime}-P|),$

where

$\tilde{V}=f^{\prime\prime}_{G}(P)V+[P,[\nabla_{G}f(P),V]].$

Moreover, using the integral form of the Taylor formula and the compactness of
$G_{n,d}$, which implies that every continuous function is already uniformly
continuous, we see that the remainder term in the Taylor expansion can be
bounded independently of $V\in\mathbb{R}^{n\times n}_{\mathrm{sym}}$ such that
$|V|=1$, i. e.

$\sup_{|V|=1}|\langle\nabla_{G}f(P^{\prime}),V\rangle-\langle\nabla_{G}f(P),V\rangle-\langle\tilde{V},P^{\prime}-P\rangle|\leq\varepsilon(|P^{\prime}-P|),$

where $\varepsilon(t)$ is some function which satisfies $\varepsilon(t)\to 0$
as $t\to 0$.

We now rewrite the Taylor formula as

(A.1)
$\langle\nabla_{G}f(P^{\prime}),V\rangle=\langle\nabla_{G}f(P)+L,V\rangle+o(|P^{\prime}-P|)$

by choosing a suitable $L$ such that $\langle
L,V\rangle=\langle\tilde{V},P^{\prime}-P\rangle$. Obviously,

$\langle f_{G}^{\prime\prime}(P)V,P^{\prime}-P\rangle=\langle
f^{\prime\prime}_{G}(P)(P^{\prime}-P),V\rangle.$

Moreover, write

$[P,[\nabla_{G}f(P),V]]=P\nabla_{G}f(P)V-PV\nabla_{G}f(P)-\nabla_{G}f(P)VP+V\nabla_{G}f(P)P.$

Adapting the identities (4.2) to $U,W\in\mathbb{R}^{n\times n}$ and
$V,X(=P^{\prime}-P)\in\mathbb{R}^{n\times n}_{\mathrm{sym}}$, we obtain

$\langle U^{T}XW^{T},V\rangle=\langle UVW,X\rangle=\langle WXU,V\rangle,$

where we have used the symmetry of $V$ and $X$. If we apply the first identity
to $P\nabla_{G}f(P)V$ and $PV\nabla_{G}f(P)$ and the second one to
$\nabla_{G}f(P)VP$ and $V\nabla_{G}f(P)P$, we obtain

$\langle[P,[\nabla_{G}f(P),V]],P^{\prime}-P\rangle=2\langle[\nabla_{G}f(P),P(P^{\prime}-P)],V\rangle,$

so that altogether, we may also set

$L:=f^{\prime\prime}_{G}(P)(P^{\prime}-P)+2[\nabla_{G}f(P),P(P^{\prime}-P)].$

Now we take an absolute value on both sides of the Taylor formula (A.1) and
take the supremum over all symmetric $V$ such that $\mathrm{vec}(V)\in
S^{nd-1}$. This leads to

$|\nabla_{G}f(P^{\prime})|=|\nabla_{G}f(P)+L|+o(|P^{\prime}-P|).$

Next, we write

$|\nabla_{G}f(P)+L|^{2}=|\nabla_{G}f(P)|^{2}+2\langle\nabla_{G}f(P),L\rangle+|L|^{2}.$

Note that as a trace of the product of a symmetric and an anti-symmetric
matrix, $\langle\nabla_{G}f(P),[\nabla_{G}f(P),P(P^{\prime}-P)]\rangle=0$.
Therefore,

$\langle\nabla_{G}f(P),L\rangle=\langle\nabla_{G}f(P),f_{G}^{\prime\prime}(P)(P^{\prime}-P)\rangle=\langle
U,P^{\prime}-P\rangle$

with $U:=f_{G}^{\prime\prime}(P)\nabla_{G}f(P)$. Since
$|L|^{2}=O(|P^{\prime}-P|^{2})$, we obtain

$|\nabla_{G}f(P)+L|^{2}=|\nabla_{G}f(P)|^{2}+2\langle
U,P^{\prime}-P\rangle+o(|P^{\prime}-P|).$

If $|\nabla_{G}f(P)|>0$, it therefore follows that

$|\nabla_{G}f(P)+L|=|\nabla_{G}f(P)|+|\nabla_{G}f(P)|^{-1}\langle
U,P^{\prime}-P\rangle+o(|P^{\prime}-P|).$

Hence,

$|\nabla_{G}f(P^{\prime})|-|\nabla_{G}f(P)|=|\nabla_{G}f(P)|^{-1}\langle
U,P^{\prime}-P\rangle+o(|P^{\prime}-P|)$

and thus

$\displaystyle\limsup_{P^{\prime}\to
P}\frac{\big{|}|\nabla_{G}f(P^{\prime})|-|\nabla_{G}f(P)|\big{|}}{|P^{\prime}-P|}$
$\displaystyle=|\nabla_{G}f(P)|^{-1}\limsup_{P^{\prime}\to
P}\frac{\big{|}\langle U,P^{\prime}-P\rangle\big{|}}{|P^{\prime}-P|}$
$\displaystyle=|\nabla_{G}f(X)|^{-1}|\nabla_{G}\psi_{U}(P)|,$

where $\psi_{U}(P):=\langle U,P\rangle$. As noted after Proposition 5.2, it
holds that $U\in T_{P}$, so that $\nabla_{G}\psi_{U}(P)=U$. Thus, we arrive at

$|\nabla^{(2)}_{G}f(P)|=|\nabla_{G}f(P)|^{-1}|f_{G}^{\prime\prime}(P)\nabla_{G}f(P)|$

if $|\nabla_{W}f(P)|>0$.

It remains to consider the case where $|\nabla_{G}f(P)|=0$. Here,
$L=f_{G}^{\prime\prime}(P)(P^{\prime}-P)$, and the Taylor formula reads

$|\nabla_{G}f(P^{\prime})|=|L|+o(|P^{\prime}-P|).$

It follows that

$\displaystyle|\nabla^{(2)}_{G}f(P)|$ $\displaystyle=\limsup_{P^{\prime}\to
P}\frac{|\nabla_{G}f(P^{\prime})|}{|P^{\prime}-P|}=\limsup_{P^{\prime}\to
P}\frac{|f_{G}^{\prime\prime}(P)(P^{\prime}-P)|}{|P^{\prime}-P|}$
$\displaystyle=\limsup_{V\to 0,V\in
T_{P}^{\perp}}\frac{|f_{G}^{\prime\prime}(P)V|}{|V|}=\lVert
f_{G}^{\prime\prime}(P)\rVert_{\mathrm{op}},$

which finishes the proof. ∎

## References

  * [AL12] Adamczak, R., and Latała, R.; _Tail and moment estimates for chaoses generated by symmetric random variables with logarithmically concave tails_ , Ann. Inst. Henri Poincaré Probab. Stat. 48(4) (2012), 1103–1136.
  * [AW15] Adamczak, R., and Wolff, P.: _Concentration inequalities for non-Lipschitz functions with bounded derivatives of higher order_ , Probab. Theory Relat. Fields 162(3–4) (2015), 531–586.
  * [Af57] Afriat, S. N.: _Orthogonal and oblique projectors and the characteristics of pairs of vector spaces_ , Math. Proc. Camb. Philos. Soc. 53 (1957), 800–816.
  * [AS94] Aida, S., and Stroock, D.: _Moment estimates derived from Poincaré and logarithmic Sobolev inequalities_ , Math. Res. Lett. 1(1) (1994), 75–86.
  * [BCS11] Banica, T., Collins, B., and Schlenker, J. M.: _On polynomial integrals over the orthogonal group_ , J. Comb. Theory Ser. A. 118(3) (2001), 778–795.
  * [BZA20] Bendokat, T., Zimmermann, R., and Absil, P.-A.: _A Grassmann Manifold Handbook: Basic Geometry and Computational Aspects_ , arXiv preprint, arXiv:2011.13699.
  * [BCG17] Bobkov, S. G., Chistyakov, G. P., and Götze, F.: _Second-order concentration on the sphere_ , Commun. Contemp. Math. 19(5) (2017), Article 1650058, 20 pp.
  * [BCG18] Bobkov, S. G., Chistyakov, G. P., and Götze, F.: _Berryâ€“Esseen bounds for typical weighted sums_ , Electron. J. Probab. 23 (2018), no. 23, 22 pp.
  * [BCG20] Bobkov, S. G., Chistyakov, G. P., and Götze, F.: _Normal approximation for weighted sums under a second-order correlation condition_ , Ann. Probab. 48(3) (2020), 1202–1219.
  * [BGS19] Bobkov, S. G., Götze, F., and Sambale, H.: _Higher order concentration of measure_ , Commun. Contemp. Math. 21(3) (2019), Article 1850043, 36 pp.
  * [CM09] Collins, B., and Matsumoto, S.: _On some properties of orthogonal Weingarten functions_ , J. Math. Phys. 50(11) (2009), Article 113516, 14 pp.
  * [CMN21] Collins, B., Matsumoto, S., and Novak, J.: _Th Weingarten Calculus_ , arXiv preprint, arXiv:2109.14890.
  * [CS06] Collins, B., and Śniady, P.: _Integration with respect to the Haar measure on unitary, orthogonal and symplectic group_ , Commun. Math. Phys. 264(3) (2006), 773–795.
  * [EAS98] Edelman, A., Arias, T. A., and Smith, S. T.: _The geometry of algorithms with orthogonality constraints_ , SIAM J. Matrix Anal. Appl. 20(2) (1998), 303–353.
  * [Ga08] Galántai, A.: _Subspaces, angles and pairs of orthogonal projections_ , Linear Multilinear Algebra 56(3) (2008), 227–260.
  * [GKZ21] Götze, F., Kabluchko, Z., Zaporozhets, D: _Grassmann angles and absorption probabilities of Gaussian convex hulls_ , Zap. Nauchn. Sem. POMI 501 (2021), 126–148.
  * [GS20] Götze, F., and Sambale, H.: _Second order concentration via logarithmic Sobolev inequalities_ , Bernoulli 26(1) (2020), 93–126.
  * [GSS19] Götze, F., Sambale, H., and Sinulis, A.: _Higher order concentration for functions of weakly dependent random variables_ , Electron. J. Probab. 24 (2019), no. 85, 19 pp.
  * [GSS21b] Götze, F., Sambale, H., and Sinulis, A.: _Concentration Inequalities for Bounded Functionals via Log-Sobolev-Type Inequalities_ , J. Theor. Probab. 34(3) 34(3) (2021), 1623–1652.
  * [GSS21a] Götze, F., Sambale, H., and Sinulis, A.: _Concentration inequalities for polynomials in $\alpha$-sub-exponential random variables_, Electron. J. Probab. 26 (2021), no. 48, 22 pp.
  * [HL08] Hamm, J., and Lee, D. D.: _Grassmann discriminant analysis: a unifying view on subspace-based learning_ , In: Proceedings of the 25th international conference on Machine learning, 2008, 376–383.
  * [HW71] Hanson, D. L., and Wright, F. T.: _A bound on tail probabilities for quadratic forms in independent random variables_ , Ann. Math. Stat. 42 (1971), 1079–1083.
  * [HHT07] Helmke, U., Hüper, K., and Trumpf, J.: _Newton’s method on Graßmann manifolds_ , arXiv preprint, arXiv:0709.2205.
  * [KP21] Kabluchko, Z., and Prochno, J.: _Large deviations for random matrices in the orthogonal group and Stiefel manifold with applications to random projections of product distributions_ , arXiv preprint, arXiv:2110.12977.
  * [KPT20] Kabluchko, Z., Prochno, J., and Thäle, C.: _A new look at random projections of the cube and general product measures_ , Bernoulli 27(3) (2021), 2117–2138.
  * [KV00] Kim, J. H., and Vu, V. H.: _Concentration of multivariate polynomials and its applications_ , Combinatorica 20(3) (2000), 417–434.
  * [KL15] Kolesko, K., and Latała, R.: _Moment estimates for chaoses generated by symmetric random variables with logarithmically convex tails_ , Stat. Probab. Letters 107 (2015), 210–214.
  * [LLY20] Lai, Z., Lim, L. H., and Ye, K.: _Simpler Grassmannian optimization_ , arXiv preprint, arXiv:2009.13502.
  * [La06] Latała, R.: _Estimates of moments and tails of Gaussian chaoses_ , Ann. Probab. 34(6) (2006), 2315–2331.
  * [Le01] Ledoux, M.: _The concentration of measure phenomenon_ , American Mathematical Society, 2001.
  * [LPT06] Litvak, A. E., Pajor, A., and Tomczak-Jaegermann, N.: _Diameters of sections and coverings of convex bodies_ , J. Funct. Anal. 231(2) (2006), 438–457.
  * [Me19] Meckes, E.: _The Random Matrix Theory of the Classical Compact Groups_ , Cambridge University Press, 2019.
  * [MW82] Mueller, C. E., and Weissler, F. B.: _Hypercontractivity for the heat semigroup for ultraspherical polynomials and on the $n$-sphere_, J. Funct. Anal. 48 (1992), 252–283.
  * [RV13] Rudelson, M., and Vershynin, R.: _Hanson–Wright inequality and sub-Gaussian concentration_ , Electron. Commun. Probab. 18 (2013), no. 82, 9 pp.
  * [S20+] Sambale, H.: _Some notes on concentration for $\alpha$-subexponential random variables_, arXiv preprint, arXiv:2002.10761, to appear in: High dimensional probability IX.
  * [SS21] Sambale, H., and Sinulis, A.: _Modified log-Sobolev inequalities and two-level concentration_ , ALEA, Lat. Am. J. Probab. Math. Stat. 18 (2021), 855–885.
  * [SS12] Schudy, W., and Sviridenko, M.: _Concentration and moment inequalities for polynomials of independent random variables_ , In: Proceedings of the twenty-third annual ACM-SIAM symposium on Discrete Algorithms, Society for Industrial and Applied Mathematics, 2012, 437–446.
  * [TVC08] P. Turaga, A. Veeraraghavan, and R. Chellappa: _Statistical analysis on Stiefel and Grassmann manifolds with applications in computer vision_ , 2008 IEEE Conference on Computer Vision and Pattern Recognition. (2008), 1–8.
  * [WLT11] Wang, X., Li, Z., and Tao, D.: _Subspaces indexing model on Grassmann manifold for image search_ , IEEE Trans. Image Process. 20(9) (2011), 2627–2635.
  * [WWF06] Wang, L., Wang, X., and Feng, J.: _Subspace Distance Analysis with Application to Adaptive Bayesian Algorithm for Face Recognition_ , Pat. Rec. 39(3) (2006), 456–464.
  * [ZZJH18] Zhang, J., Zhu, G., Heath Jr., R. W., and Huang, K.: _Grassmannian Learning: Embedding Geometry Awareness in Shallow and Deep Learning_ , arXiv preprint, arXiv:1808.02229.