# Equivariant Graph Neural Networks for Prediction of Tensor Material
Properties of Crystals

Alex Heilman∗ Northeastern University  Claire Schlesinger∗ Northeastern
University  Qimin Yan  Northeastern University

**footnotetext: These authors contributed equally to this work

##  1 Introduction

Traditional machine learning methods applied to the material sciences have
often predicted invariant, scalar properties of material systems to great
effect. Newer, coordinate equivariant models promise to provide a coordinate
system dependent output in a well defined manner, but recent applications
often neglect a direct prediction of directional (i.e. coordinate system
dependent) quantities and instead are used to predict still just invariant
quantities.

In such rotationally equivariant models, features are often associated with
irreducible representations of the rotation group. This association may be
leveraged to predict tensorial quantities directly from the outputs of such
models. This component-wise prediction of tensorial properties is achieved by
decomposing tensors into harmonic subspaces via a tensor spherical harmonic
decomposition, by which we may also associate arbitrary tensors with the
irreducible representations of the rotation group. This essentially allows us
to read off tensors component-wise from the output representations of these
equivariant models.

In this work, we present results for the prediction of various material
property tensors directly from crystalline structures. Namely, given some
material’s crystalline structure, we may predict tensor components of
dielectric, piezoelectric, and elasticity tensors directly from the output of
a $SE(3)$ equivariant model.

Below, we give explicit decompositions of the relevant tensors and a brief
overview of the tools used. We then present results for some basic
implementations of such an approach, and attempt some transfer learning
applications.

##  2 Related Works

CGCNN [XG18] is an early graph neural network design used to predict seven
different invariant properties of crystals. CGCNN was constructed using graph
convolutions where the node features were atom properties and the edge
features were distances. EMPNN [Kli+23] came later and utilized the binding
direction as the equivariant input. It combined both invariant and equivariant
features allowing an output of a pair of equivariant and invariant properties.

Prediction of equivariant properties arose with the development of more
sophisticated equivariant graph neural networks. StrainNet [PEC23] utilized a
spherical equivariant graph transformer and additional information about the
strained structure of a crystal to predict the elastic tensor. MatTen [Wen+23]
utilized a steerable equivariant convolutional neural network to predict the
elastic tensor by decomposing the elastic tensor into spherical harmonics and
attempting to predict those spherical harmonics.

##  3 $SO(3)$ Equivariant Neural Networks

Neural networks are a class of universal function approximators, composed
layer-wise by functions
$\mathcal{L}^{1}\circ\mathcal{L}^{2}\circ...\circ\mathcal{L}^{n}$, where each
layer-to-layer transition map $\mathcal{L}^{i}$ is a trainable function from
some feature space associated with layer $L$ to a feature space associated
with layer $L+1$.

Many modern approaches use a specific type of neural network, referred to as a
Graph Neural Network (GNN), which acts on data encoded in features associated
with some representative graph (i.e. a collection of nodes and connections
between them), while preserving the underlying connectivity of the graph
representation.

Basic graph networks however are, in general, rotationally invariant. As such,
these models are incapable of predicting any coordinate-system quantities,
such as tensor components.

In an $SO(3)$-network, features are further associated with $SO(3)$
irreducible representation: namely, spherical harmonics $Y_{\ell}^{m}$, which
are doubly indexed by the rotational order $\ell\geq 0$, and the azimuthal
order $-\ell\leq m\leq\ell$.

Thus, a traditional feature set $V^{(n)a}$ of channel $a$ and associated with
object $n$, has an additional two indices $\ell$ and $m$ in an $SO(3)$
network, corresponding to the aformentioned indices of harmonic
representations.

Figure 1: Example diagram showcasing a rotationally equivariant update scheme
where features are associated with spherical harmonics (indexed by $\ell$ and
$m$). Note that the function $f$’s output also has a corresponding $\ell$
index, as well as the rotation operator $R^{\ell}$.

Furthermore, compositions of equivariant functions are themselves equivariant
functions. As such, we may form an equivariant network by composing it layer-
wise from a set of common equivariant functions.

Here, we consider four types of $SO(3)$-equivariant functions from which we
may compose our equivariant networks: namely, $SO(3)$-feature convolutions,
$\ell$-wise self-interactions and non-linearities, and pooling across nodes. A
brief overview of each is given below.

###  3.1 $SO(3)$ Convolution

As mentioned above, features and filters in $SO(3)$ networks are both
associated with spherical harmonics. Tensor products of these two
representation spaces are equivariant under transformation of the two
subspaces, i.e.:

$\mathcal{D}^{V}\otimes\mathcal{D}^{W}=\mathcal{D}^{V\otimes W}$

By way of Clebsch-Gordan coefficients
$c^{\ell_{3}m_{3}}_{\ell_{1}m_{1}\ell_{2}m_{2}}$, these tensor products of
$SO(3)$ irreducible representations may be related to a third set of
irreducible representations as:

$(u\otimes
v)_{\ell_{o}}^{m_{o}}=c_{\ell_{1}m_{1}\ell_{2}m_{2}}^{\ell_{o}m_{o}}u_{\ell_{1}}^{m_{1}}v_{\ell_{2}}^{m_{2}}$

where $u$ and $v$ are harmonic vectors of order $\ell_{1}$ and $\ell_{2}$,
respectively.

Thus, we maintain $SO(3)$ equivariance through convolution by defining $SO(3)$
convolution to be the scaled tensor product of the two representation spaces
(i.e. that of the input feature space, and the filter space), so that layer to
layer convolutional maps $\mathcal{L}$ may be defined component-wise
as[Tho+18]:

$\mathcal{L}^{\ell_{o}}_{acm_{o}}\big{(}\vec{r}_{a},V_{acm_{i}}^{\ell_{i}}\big{)}=\sum_{m_{f},m_{i}}c_{\ell_{i}m_{i}\ell_{f}m_{f}}^{\ell_{o}m_{o}}\sum_{b}F^{\ell_{f}\ell_{i}}_{cm_{f}}(r_{ab})V_{bcm_{i}}^{\ell_{i}}$

where the filter function $F^{\ell_{f}\ell_{i}}_{cm_{f}}(r_{ab})$ depends only
on the distance between point $a$ and $b$ (as opposed to directional
dependence, to maintain equivariance), but has independent, trainable
parameters for different rotational orders $\ell_{f},\ell_{i}$, azimuthal
orders $m$, and channels $c$.

###  3.2 Self-Interaction

Feature sets for individual objects may also update according to themselves as
long as they act across $m$ for every $\ell$ and only update according to the
different channels $c$. That is, functions of the form:

$V_{acm}^{\ell}\rightarrow\sum_{c}W^{\ell}_{c^{\prime}c}V_{acm}^{\ell}$

are also equivariant. Such functions we refer to as self-interaction layers.

###  3.3 Non-Linearities

We can also apply point-wise non linearities and maintain equivariance, as
long as they also respect the across $m$ indices for every order feature
$\ell$.

###  3.4 Pooling

Pooling, or aggregation, across all elements or objects (index $a$) while
preserving the $m$ and $\ell$ indices is itself equivariant. Thus, functions
of the form:

$M_{cm}^{\ell}=\text{AGG}_{a}(\\{V_{acm}^{\ell}\\})$

where AGG is an arbitrary aggregation function performed only over the object
index $a$, are also available in the construction of $SO(3)$ networks.

#### SO(3) Equivariant Outputs

The outputs of these networks are then also associated with some indices
$\ell$ and $m$, and thus are also associated with some set of spherical
harmonics.

These $SO(3)$ equivariant networks are then naturally well suited for the
prediction of tensorial properties. As will be shown below, tensors may
generally be decomposed into a set of $SO(3)$ invariant subspaces which can
each be associated with a set of spherical harmonic tensors themselves. Since
the outputs of $SO(3)$ networks naturally transform like spherical harmonic
tensor coefficients indexed by $\ell$ and $m$, we may then essentially read
them off as such and convert these into Cartesian tensor components.

##  4 Spherical Harmonic Tensors and $SO(3)$ Invariant Subspaces

We may construct a set of spherical harmonic tensors $\hat{Y}_{\ell}^{m}$ by
first defining the $J_{z}$ vector basis and then considering symmetric
products of such vectors. We define the $J_{z}$ basis:

$\begin{bmatrix}a_{+}\\\ a_{0}\\\
a_{-}\end{bmatrix}=\begin{bmatrix}-\frac{1}{\sqrt{2}}&-\frac{i}{\sqrt{2}}&0\\\
0&0&1\\\ -\frac{1}{\sqrt{2}}&+\frac{i}{\sqrt{2}}&0\\\
\end{bmatrix}\begin{bmatrix}x\\\ y\\\ z\end{bmatrix}$

where $x,y,z$ are Cartesian components of the same vector. In this basis, the
unit vectors associated with these components correspond to unit vector
spherical harmonics $\hat{Y}_{\ell=1}^{m}$ (where we will assume Racah
normalization for all definitions).

Now, recall the Clebsch-Gordon expansion of products of spherical harmonics
(again, in the Racah normalization):

$Y_{\ell_{1}}^{m_{1}}\otimes
Y_{\ell_{2}}^{m_{2}}=\sum_{L=-|\ell_{1}-\ell_{2}|}^{\ell_{1}+\ell_{2}}\sum_{M=-L}^{L}c_{\ell_{1}0\ell_{2}0}^{L0}c_{\ell_{1}m_{1}\ell_{2}m_{2}}^{LM}Y_{L}^{M}.$

This may be used to draw a correspondence between symmetrized tensor products
of $n$ vectors in the $J_{z}$ and rank $\leq n$ spherical harmonic tensors.
The coefficients for such a transformation relevant for the tensors considered
here are listed in Appendix B

The spherical harmonics $Y_{\ell}^{m}$ canonically refer to a complex basis,
as should be clear from our definition of the $J_{z}$ basis. Many tensors
describing macroscopic responses of materials are strictly real-valued
however. Hence, we may desire to then transform into a set of real-valued
spherical harmonics.

Thus, we may decompose an arbitrary tensor of rank-$n$ into a set of symmetric
tensors of rank-$\ell$ with $0\leq\ell\leq n$, and then use the relations
between the $J_{z}$ basis components and the spherical harmonic components
$y_{\ell}^{m}$. We then may arrive at a set of real spherical harmonic
components $y_{\ell m}$ using the relations specified in Appendix C.

However, since this correspondence can only be drawn for symmetric tensors,
for any arbitrary tensor we first need some way to describe it in terms of a
set of symmetric tensors. The above approach can then be applied to this set
of symmetric components.

This decomposition of an arbitrary rank tensor into a set of invariant
symmetric subspaces is equivalent to an $SO(3)$ decomposition of a tensor
space. Such $SO(3)$ invariant subspaces can be constructed via a compound
decomposition with respect to $GL$, $SL$, and $O$: which result from
application of Young symmetrizers, contractions with the totally antisymmetric
$\epsilon$, and contractions with the totally symmetric $g$, respectively.

For example, for a rank-two tensor, we may always decompose it with respect to
GL first (by using the corresponding Young symmetrizers), resulting in a
symmetric tensor $S$ and antisymmetric tensor $A$.

The symmetric $S$ then has an invariant rank-0 space, commonly referred to as
it’s trace, and constructed via total contraction with the metric tensor
$g_{ij}$. The traceless residue of $S$ then is a symmetric, rank-2 tensor
which is itself an invariant subspace. The antisymmetric $A$ then may be fully
contracted with the totally antisymmetric $\epsilon_{ij}$ to return another
rank-0 invariant subspace.

The relevance of these facts is that, in the manner described above, we are
able to decompose an arbitrary tensor into a set of components that transform
like spherical harmonics, just like the representations in $SO(3)$ equivariant
models. We now give spherical harmonic decompositions of several tensors
describing common material properties, and then use these decompositions to
predict these tensors directly from material structure.

##  5 Spherical Harmonic Decomposition of Common Material Property Tensors

We now give a brief overview of the harmonic decomposition of three common
material properties described by tensors: the dielectric tensor $\epsilon$,
the piezoelectric strain tensor $d$, and the elasticity tensor $C$. A full
overview of each decomposition is given in Appendix A.

###  5.1 Dielectric Tensors

The dielectric permittivity tensor $\mathbf{\epsilon}$ of some material is a
linear model of it’s electric displacement $\vec{D}$ in response to an
external electric field $\vec{E}$:

$\vec{D}=\mathbf{\epsilon}\vec{E}$

Note that here we focus only on static responses, that is, $(\partial/\partial
t)\vec{E}=0$, due to the restriction of data to this case.

The dielectric tensor $\epsilon$ then is a rank-two tensor formed from vector
spaces over the field $\mathbb{R}^{3}$, where we henceforth ignore the
distinction between co- and contravariant components due to the Euclidean
structure $g_{ij}=\delta_{ij}$.

The dielectric tensor $\epsilon$ is symmetric under permutation of it’s
indices, such that:

$\epsilon_{ij}=\epsilon_{ji}.$

The dielectric tensor then may be decomposed further into it’s trace
$T=g_{ij}\epsilon_{ij}$ and it’s traceless residue $R$, defined by:

$R_{ij}=\epsilon_{ij}-Tg_{ij}$

resulting in two invariant subspaces of rank-0 and 2, respectively. These two
symmetric subspaces then admit a decomposition in the manner described in
Section 4.

###  5.2 Piezoelectric Tensors

The piezoelectric strain constants $(d_{ijk})_{T}$ are defined (at constant
temperature) by the thermodynamic relation:

$\big{(}d_{ijk}\big{)}_{T}=\left(\frac{\partial\epsilon_{ij}}{\partial
E_{k}}\right)_{\sigma,T}$

where here $\epsilon$ is the strain tensor, and the partial derivative is
taken at constant stress $\sigma$ and temperature $T$.

These strain constants $d_{ijk}$ are related to the piezoelectric stress
constants $e_{ijk}$ via the elastic tensor $C_{ijkl}$ according to:

$\big{(}e_{ijk}\big{)}_{T}=\big{(}d_{ilm}\big{)}\big{(}C_{ijkl}\big{)}_{E,T}$

with $e_{ijk}$ defined as:

$\big{(}e_{ijk}\big{)}_{T}=\left(\frac{\partial
D_{i}}{\partial\epsilon_{jk}}\right)_{\sigma,T}$

and where $D$ is the resulting electric displacement vector in the material
(we now generally neglect the explicit notation of constant parameters).

The piezoelectric strain components $d_{ijk}$ are symmetric under $i,j$ due to
the symmetry of the strain tensor $\epsilon_{ij}$, so that we have:

$d_{ijk}=d_{jik}.$

This symmetry gives us a natural $GL$ decomposition into the totally symmetric
$S$ and the mixed-symmetry $A$, defined component-wise from $d$ as[IR22]:

$\displaystyle S_{ijk}$
$\displaystyle=\frac{1}{3}\big{(}d_{ijk}+d_{ikj}+d_{kji}\big{)}$
$\displaystyle A_{ijk}$
$\displaystyle=\frac{1}{3}\big{(}2d_{ijk}-d_{ikj}-d_{kji}\big{)}.$

The fully symmetric part $S$ then consists of a trace vector
$s_{k}=g_{ij}S_{ijk}$, and a symmetric residue $W_{ijk}$ defined by:

$W_{ijk}=S_{ijk}-\frac{1}{5}(g_{ij}v_{k}+g_{ik}v_{j}+g_{jk}v_{i})$

These correspond to the spaces $\mathcal{H}^{(1)}$ and $\mathcal{H}^{(3)}$,
respectively.

The mixed symmetry part $A$ also requires decomposition with respect to
$SO(3)$, so that we have a set of symmetric tensors describing it. It’s 8
independent components can be described by a $5\oplus 3$ dimensional space
consisting of a symmetric rank-2 tensor and a trace vector.

The trace vector $a^{i}$, describing $A$’s 3-dimensional $SO(3)$ invariant
subspace, can be formed from the contraction of the metric tensor $g_{ij}$
along $A$’s first and second indices. That is, we define:

$a^{i}=g_{jk}A^{ijk}$

Note that this choice is somewhat arbitrary, since we could define the trace
part to correspond to the contraction along the first and third indices, or
the second and third. However, it can be shown that for the mixed symmetry of
$A$, these two trace vectors are linearly dependent (related by an overall
factor of $1$ and $-2$, respectively).

The rank-2 invariant subspace of $A$ then may be constructed by symmetrizing
the partial contraction with $\epsilon$ along the first and third indices (the
anti-symmetric pair). Note that the antisymmetric part of this partial
contraction corresponds to the trace vector space accounted for here by $u$.
Explicitly, we define:

$b_{ij}=\frac{1}{2}\big{(}\epsilon_{nim}A_{njm}+\epsilon_{njm}A_{nim}\big{)}$

which is a traceless symmetric rank-2 tensor.

And so, as mentioned above, the symmetric part $S$ is readily decomposed in
the spherical bases into $\mathcal{H}^{(1)}\oplus\mathcal{H}^{(3)}$. And then
the mixed-symmetry part inhabits the space
$\mathcal{H}^{(1)}\oplus\mathcal{H}^{(2)}$.

###  5.3 Elasticity Tensor

The elasticity (or simply, elastic) tensor $C$ of a material relates it’s
Cauchy strain to some applied infinitesimal stress. This may be described,
component-wise, as the linear relation below:

$\epsilon_{ij}=C_{ijkl}\tau_{kl}$

As should be clear by the number of indices, the elastic tensor is a fourth
rank tensor. The symmetries of the elastic tensor are discussed below, as they
are relevant for our application. However, it is worth noting here that the
total number of independent components of the elastic tensor is 21 in general.

The elastic tensor has several symmetries, but is not, in general, symmetric
upon any swapping of indices. For all systems, $C$ satisfies the so-called
’minor symmetries’ below:

$C_{ijkl}=C_{jikl}$

$C_{ijkl}=C_{ijlk}$

resulting from the symmetry of the strain and stress tensors
($\tau_{ij}=\tau_{ji}$ and $\epsilon_{ij}=\epsilon_{ji}$) under the assumption
of equilibrium. This reduces the number of independent components from 81 to
36.

Furthermore, for conservative systems in which the elastic deformation is
describable in terms of some potential energy function (and which we shall
henceforth assume for our application), $C$ has the additional ’major
symmetry’:

$C_{ijkl}=C_{klij}$

This further reduces the number of independent components from 36 (from the
minor symmetries) to 21 in total.

Note that this mixed-symmetry subspace $A$ corresponds to Backus’ [Bac70]
asymmetric tensor $A$. With $S$ and $A$ being defined component-wise
as[Iti20]:

$S_{ijkl}=\frac{1}{3}\big{(}C_{ijkl}+C_{ikjl}+C_{klij}\big{)}$
$A_{ijkl}=\frac{1}{3}\big{(}2C_{ijkl}-C_{ikjl}-C_{klij}\big{)}$

The fully symmetric part can of course be converted to spherical harmonic
components via a Clebsch-Gordon expansion. The mixed symmetry $A$, however
requires further decomposition.

All six independent components of $A$ can be described by the symmetric (but
not traceless) tensor $t$, defined as the double partial contraction of $A$
with the totally antisymmetric tensor $\epsilon$ as:

$t_{ij}=\epsilon_{i}^{mk}\epsilon_{j}^{nl}A_{mnkl}$

This tensor $t$ then has a harmonic decomposition according to the rank-two
Clebsch-Gordon transformation between the $J_{z}$ basis and the harmonic basis
$y_{\ell}^{m}$.

As such, the elastic tensor $C$ admits a decomposition into $S$ and $A$, which
inhabit spaces of
$\mathcal{H}^{(4)}\oplus\mathcal{H}^{(2)}\oplus\mathcal{H}^{(0)}$ and
$\mathcal{H}^{(2)}\oplus\mathcal{H}^{(0)}$, respectively.

##  6 Approach

We investigated the performance of three distinct $SO(3)$ equivariant models
in the direct prediction of tensor components from crystal structure. This is
performed in the manner outlined above, wherein we read off the equivariant
model’s output as a set of spherical harmonic coefficients and then convert
these to Cartesian tensor components. This approach is naturally equivariant.

The three models tested were: the Steerable Equivariant Graph Neural Network
(SEGNN) [Bra+21]; the Steerable Equivariant Convolutional network (SEConv)
[Bra+21]; and the Steerable Equivariant Transformer (SETransformer) [Fuc+20].
All are based on the popular e3nn [GS22], and have outputs associated with a
set of spherical harmonics by design. Note that the SETransformer was unique
in that it alone used attention in the filter function
$\mathcal{F}_{acm_{i}m_{o}}^{\ell_{o}\ell_{i}}(r_{ab})$ such that it could
learn weights to associate with edges from connecting node features.

###  6.1 Data

We utilized the data available in the Materials Project [Jai+13] which
includes $\sim$150k materials. Of which, almost all have calculated scalar
values for band gap and formation energy, $\sim$12k have a calculated elastic
tensor, $\sim$6k have a calculated dielectric tensor, and $\sim$3k have a
calculated piezoelectric tensor. The data was split so that 70% was in the
training set, 15% was in the validation set, and the remaining 15% was held
out for testing.

The input features were the provided crystal graphs. Each node’s initial
feature encoded the following atomic information: atomic number111These
properties were one-hot encoded., group1, period1, block1, electronegativity,
atomic radius, number of valence electrons, electron affinity, ionization
energy, and atomic mass. Each edge contained the spherical harmonics produced
by the direction vector and the distance between atoms. For the purpose of
steerable equivariant neural networks, any invariant property was considered
to be an $Y^{0}_{0}$ spherical harmonic.

###  6.2 Model Architecture

Figure 2: Diagram depicting the general architecture of the model used in this
work.

Our model architecture was based on the design of the SE(3) Equivariant graph
neural networks found in [Bra+21] where we utilized the SETransformer [Fuc+20]
from E3NN [GS22] for the SETransformer model. A visualization of the model
architecture can be found in figure 2.

The model is built by first embedding the input atom features into spherical
harmonics up to order $\ell_{max}$. Then there are $R$ graph convolution
layers (either message passing, convolution, or transformer layers) where
nodes are updated by their neighbors, $N$ self interaction layers in which
individual nodes update themselves, a pooling layer (either mean or max
pooling), and $M$ MLP layers to transform the pooled representation into the
desired output spherical harmonics.

##  7 Results

The biggest issue with using machine learning to predict equivariant
properties is the lack of sufficient training data. To make up for this gap,
we attempted to pretrain in a variety of ways. We gathered two different
pretraining datasets to try: band gap and formation energy. We initially
trained each of the models on both pretraining sets and then performed a
series of experiments to see if learning on multiple datasets could improve
performance.

###  7.1 Direct Prediction

First, we consider a straight-forward prediction of the target tensor by
training on the target set and then testing on some as-yet-unseen validation
data. Results for these predictions are given in Figure 4, with component-wise
MAE presented in Figure 3.

While the average MAE for tensor components beats previous results as reported
in [PEC23], the heatmap displaying component-wise MAE suggest further
incorporation of symmetry class information is necessary to further improve
results.

Figure 3: Heatmaps showing the component-wise MAE in Voigt notation for the all direct prediction tasks. Shown in order of dielectric, piezoelectric, and elastic tensor components. | MAE (Averaged Over Components)
---|---
Target Components | SEGNN | SEConv | SETransformer
Elastic [log${}_{1}0$(GPa)] | 8.139 | 7.689 | 7.941
Dielectric | 4.82 | 4.702 | 4.718
Piezoelectric [C/$m^{2}$] | 0.170 | 0.170 | 0.1714
Figure 4: Results for direct prediction of elastic, piezoelectric, and
dielectric tensor components.

###  7.2 Transfer Learning Applications

As a basic investigation into the broader usefulness of the trained models, we
also test pretrained models on successive tasks and compare performance.

Pretraining may be particularly useful in the case of target sets with a
limited number of data. As such, our experiments focus on a filter-down
approach, in which we start training on larger datasets and then transfer the
trained graph network weights to a downstream task with a smaller dataset
(while continuing to train the transfered graph network).

Since all tensorial data is relatively restricted in terms of the total number
of samples, the first transfer learning experiment consisted of pretraining a
graph network on as large a dataset as possible, namely scalar data, and then
transferring this model to different tensorial tasks. That is, we first
consider a training progression:

$\textit{Scalar Target}\rightarrow\textit{Tensor Target},$

and compare results in Table 5. The larger scalar datasets were the predicted
formation energy per atom for a crystal, and the calculated band gap. Note
that in these progressions, no data was witheld in the scalar target
pretraining process.

In general though, pretraining of this nature had little effect on down-stream
results. This may result from a number of factors. Perhaps there is too little
overlap between the crystal structures of the smaller datasets and those of
the larger dataset, so that the pretraining overgeneralizes the model to a
larger class of materials than are relevant. The difference in rotational
order between tasks may furthermore result in little overlap between the
pretrained filter weights and the down-stream task’s pathway through the
model. And, the relevance of certain tasks to one another may be hard to
determine a priori.

|  | MAE (Avg. Over Components)
---|---|---
Pretraining Task: | Tensor Target | SEGNN | SEConv | SETransformer
| Elastic [log10(GPa)] | 8.19 (8.14) | 7.98 (7.69) | 7.65 (7.94)
Formation Energy | Dielectric | 5.02 (4.82) | 4.52 (4.71) | 4.41 (4.72)
| Piezoelectric [C$/m^{2}$] | 0.170 (0.170) | 0.170 (0.170) | 0.171 (0.171)
| Elastic [log10(GPa)] | 8.39 (8.14) | 7.87 (7.69) | 8.34 (7.94)
Band Gap | Dielectric | 4.74 (4.82) | 4.35 (4.71) | 4.56 (4.72)
| Piezoelectric [C$/m^{2}$] | 0.170 (0.170) | 0.170 (0.170) | 0.171 (0.171)
Figure 5: Results for tensor component prediction accuracy after pretraining
on a scalar-target task for materials. Numbers italicized in parentheses are
the original accuracies (without pretraining).

The smallest dataset considered here is that of piezoelectric strain
constants. Performance did not significantly improve with any scalar target
pretraining in any of the models for this task. The scarcity of data for this
target motivates a search for further improvement from larger, adjacent
datasets.

All failed to significantly improve performance in terms of MAE across tensor
components.

###  7.3 Chained Pretraining

One suspected reason for failure was that the pretraining on scalar-only
targets did not provide enough information for the models to learn how to
properly utilize the higher order spherical harmonics. So, we attempted to
chain training over each dataset together. The experiments performed to this
end focused on three different training progressions of tasks:

  1. 1.

Formation energy/Band gap $\rightarrow$ Elastic $\rightarrow$ Dielectric
$\rightarrow$ Piezoelectric

  2. 2.

Formation energy/Band gap $\rightarrow$ Elastic $\rightarrow$ Piezoelectric

  3. 3.

Formation energy/Band gap $\rightarrow$ Dielectric $\rightarrow$ Piezoelectric

We believed that by chaining these datasets together, the model would be able
to learn better how to use higher order spherical harmonics and use the prior
knowledge gathered from the elastic and dielectric tensor to better predict
the piezoelectric tensor. The result of these attempts can be found in figure
6

|  | MAE (Avg. Over Components) [C$/m^{2}$]
---|---|---
Scalar Dataset: | Experiment # | SEGNN | SEConv | SETransformer
| Exp. 1 | 0.170 (0.170) | 0.170 (0.170) | 0.171 (0.171)
Formation Energy | Exp. 2 | 0.170 (0.170) | 0.170 (0.170) | 0.171 (0.171)
| Exp. 3 | 0.170 (0.170) | 0.170 (0.170) | 0.171 (0.171)
| Exp. 1 | 0.170 (0.170) | 0.170 (0.170) | 0.171 (0.171)
Band Gap | Exp. 2 | 0.170 (0.170) | 0.170 (0.170) | 0.171 (0.171)
| Exp. 3 | 0.170 (0.170) | 0.170 (0.170) | 0.171 (0.171)
Figure 6: Results for prediction accuracy on the piezoelectric tensor after
chained pretraining.

Overall, there was no meaningful change to the prediction of the piezoelectric
tensor. Because there was only a total of three thousand data points, any
model would be unable to make a proper approximation.

In experiment 1, the dielectric tensor model was also part of a pretraining
chain by adding the elastic tensor along with the scalar pretraining. As shown
in figure 7, this pretraining method hindered the performance of the model in
comparison to just pretraining on the scalar dataset.

| MAE (Avg. Over Components)
---|---
Pretraining Task: | SEGNN | SEConv | SETransformer
Formation Energy | 4.67 (4.82) | 4.69 (4.71) | 4.45 (4.72)
Band Gap | 4.66 (4.82) | 4.64 (4.71) | 4.74 (4.72)
Figure 7: Results for predicting the components of the dielectric tensor in
experiment 1.

### Conclusion

In a certain sense, the component-wise results and lack of transferability may
indicate that the spherical harmonic approach is potentially ’too big’ or too
expressive to effectively predict crystalline material properties in a way
that naturally respects crystalline symmetries. Indeed, $SO(3)$ convolution,
as described here, generates a large set of unwanted and consequently
discarded combinations of spherical harmonics.

A simple approach would be to mask results according to crystal system, or
build an individual model for each different crystal system with a unique
output structure. However, a more unified approach might be to restrict
convolution specifically to the crystal group elements. Future works may
consider the symmetry of crystalline systems in such a way, so that we may
avoid predicting tensor components that should be identically zero a priori.

## References

  * [Bac70] George Backus  “A geometrical picture of anisotropic elastic tensors”  In _Reviews of geophysics_ 8.3 Wiley Online Library, 1970, pp. 633–671  URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/RG008i003p00633
  * [Bra+21] Johannes Brandstetter et al.  “Geometric and Physical Quantities improve E(3) Equivariant Message Passing”, 2021  arXiv:2110.02905 [cs.LG]
  * [Fuc+20] Fabian B. Fuchs, Daniel E. Worrall, Volker Fischer and Max Welling  “SE(3)-transformers: 3D roto-translation equivariant attention networks”  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS ’20  Vancouver, Canada: Curran Associates Inc., 2020
  * [GS22] Mario Geiger and Tess Smidt  “e3nn: Euclidean Neural Networks”  arXiv, 2022  DOI: 10.48550/ARXIV.2207.09453
  * [Gei+22] Mario Geiger et al.  “Euclidean neural networks: e3nn”  Zenodo, 2022  DOI: 10.5281/zenodo.6459381
  * [HG22] Hongyu Yu Hongjun Xiang and Xinggao Gong  “A general tensor prediction framework based on graph neural networks”, 2022  URL: https://doi.org/10.21203/rs.3.rs-1900756/v1
  * [Iti20] Yakov Itin  “Irreducible matrix resolution for symmetry classes of elasticity tensors”  In _Mathematics and Mechanics of Solids_ 25.10, 2020, pp. 1873–1895  DOI: 10.1177/1081286520913596
  * [IR22] Yakov Itin and Shulamit Reches  “Decomposition of third-order constitutive tensors”  In _Mathematics and Mechanics of Solids_ 27.2, 2022, pp. 222–249  DOI: 10.1177/10812865211016530
  * [Jai+13] Anubhav Jain et al.  “Commentary: The Materials Project: A materials genome approach to accelerating materials innovation”  In _APL Materials_ 1.1, 2013, pp. 011002  DOI: 10.1063/1.4812323
  * [Kli+23] Astrid Klipfel et al.  “Equivariant message passing neural network for crystal material discovery”  In _Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence_ , AAAI’23/IAAI’23/EAAI’23  AAAI Press, 2023  DOI: 10.1609/aaai.v37i12.26673
  * [LM20] Francisco Gonzalez Ledesma and Matthew Mewes  “Spherical Harmonic Tensors”, 2020  arXiv: https://arxiv.org/pdf/2010.09433
  * [Moc88] Eiji Mochizuki  “Spherical harmonic decomposition of an elastic tensor”  In _Geophysical Journal International_ 93.3 Blackwell Publishing Ltd Oxford, UK, 1988, pp. 521–526  URL: https://academic.oup.com/gji/article/93/3/521/662002
  * [Moc96] Eiji Mochizuki  “Spherical Harmonic Decomposition of an Elastic Tensor II”  In _Journal of Physics of the Earth_ 44.1, 1996, pp. 79–84  URL: https://www.jstage.jst.go.jp/article/jpe1952/44/1/44_1_79/_article/-char/ja/
  * [NL22] Vu Ha Anh Nguyen and Alessandro Lunghi  “Predicting tensorial molecular properties with equivariant machine learning models”  In _Phys. Rev. B_ 105 American Physical Society, 2022, pp. 165131  DOI: 10.1103/PhysRevB.105.165131
  * [PEC23] Teerachote Pakornchote, Annop Ektarawong and Thiparat Chotibut  “StrainTensorNet: Predicting crystal structure elastic properties using SE(3)-equivariant graph neural networks”, 2023  arXiv: https://arxiv.org/abs/2306.12818
  * [Tho+18] Nathaniel Thomas et al.  “Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds”, 2018  arXiv:1802.08219 [cs.LG]
  * [VWP23] Maxwell C Venetos, Mingjian Wen and Kristin A Persson  “Machine learning full NMR chemical shift tensors of silicon oxides with equivariant graph neural networks”  In _The Journal of Physical Chemistry A_ 127.10 ACS Publications, 2023, pp. 2388–2398
  * [Wen+23] Mingjian Wen et al.  “A universal equivariant graph neural network for the elasticity tensors of any crystal system”, 2023  arXiv: https://arxiv.org/abs/2307.15242
  * [XG18] Tian Xie and Jeffrey C. Grossman  “Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties”  In _Phys. Rev. Lett._ 120 American Physical Society, 2018, pp. 145301  DOI: 10.1103/PhysRevLett.120.145301

##  Appendix A Full Decompositions of Tensors

Below, we give the total decomposition of the three tensors of interest here.
Namely, the rank-two dielectric tensor $\epsilon$, the rank-three
piezoelecrtic strain tensor $d$, and the rank-four elastic tensor $C$. Each
begins with the Young diagrams for the total rank tensor space and then uses
the symmetries of the respective tensor to restrict down to the relevant Young
tableaux. These are used to form symmetrizers and then further decomposed with
contractions with the metric tensor $g$ and the levi-civita tensor $\epsilon$
until all that remains are a set of fully symmetric sub-tensors.

###  A.1 Dielectric Tensor Decomposition

The decomposition of the dielectric tensor is essentially trivial since it is
already symmetric. However, we give a detailed overview here for instructive
purposes.

The GL decomposition of a rank-two tensor space is the usual symmetric-
antisymmetric decomposition of a matrix $M$.

$M_{ij}=S_{ij}+A_{ij}\quad$

with:

$\displaystyle S_{ij}$
$\displaystyle=\frac{1}{2}(M_{ij}+M_{ji})\quad\Rightarrow\quad S_{ij}=S_{ji}$
$\displaystyle A_{ij}$
$\displaystyle=\frac{1}{2}(M_{ij}-M_{ji})\quad\Rightarrow\quad A_{ij}=-A_{ji}$

which correspond to the Young diagrams below:

.

Since the dielectric tensor $\epsilon$ is symmetric, we are left then only
with the unique tableaux below, corresponding to the fully symmetric part $S$.

We can then further decompose this rank-two symmetric into a rank-zero space
and a rank-two space by taking the rank-0 trace $t=g_{ij}\epsilon_{ij}$ and
then forming the rank-2 traceless residue $R_{ij}=\epsilon_{ij}-t\delta_{ij}$.

###  A.2 Piezoelectric Tensor Decomposition

The Young diagrams for a rank-three tensor are as follows:

according to the symmetry$d_{ijk}=d_{ikj}$, we can again see that all Young
tableaux but the following will disappear, since the rest are asymmetric in
the last two indices.

where the totally symmetric component tensor here we define as $S$ and the
mixed symmetry component tensor we define $A$, so that we have the Young
decomposition:

$d_{ijk}=S_{ijk}+A_{ijk}$

defined component-wise in terms of strain components:

$\displaystyle S_{ijk}$
$\displaystyle=\frac{1}{3}\big{(}d_{ijk}+d_{ikj}+d_{kji}\big{)}$
$\displaystyle A_{ijk}$
$\displaystyle=\frac{1}{3}\big{(}2d_{ijk}-d_{ikj}-d_{kji}\big{)}$

The fully symmetric part $S$ is of an adequate form for harmonic
decomposition, with those relations given in the next section.

The mixed symmetry part $A$ however requires further decomposition with
respect to $SO(3)$, so that we have a set of symmetric tensors describing it.
It’s 8 independent components can be described by a $5\oplus 3$ dimensional
space consisting of a symmetric rank-2 tensor and a trace vector.

The trace vector $v^{i}$, describing $A$’s 3-dimensional $SO(3)$ invariant
subspace, can be formed from the contraction of the metric tensor $g$ along
$A$’s first and second indices. That is, we define:

$v^{i}=g_{jk}A^{ijk}$

Note that this choice is somewhat arbitrary, since we could define the trace
part to correspond to the contraction along the first and third indices, or
the second and third. However, it can be shown that for the mixed symmetry of
$A$, these two potential trace vectors are linearly dependent (related by an
overall factor of $1$ and $-2$, respectively).

Rather simply then, we can reconstruct a third-rank tensor $V$, corresponding
to this rank-one invariant subspace, by defining $V$ component-wise as:

$V_{ijk}=\frac{1}{4}\big{[}v_{i}g_{jk}+v_{j}g_{ik}-2v_{j}g_{ik}\big{]}$

The rank-2 invariant subspace of $A$ then may be constructed by symmetrizing
the partial contraction with $\epsilon$ along the first and second indices
(the anti-symmetric pair). Note that the antisymmetric part of this partial
contraction corresponds to the trace vector space accounted for here by $u$.
Explicitly, we define:

$b_{ij}=\frac{1}{2}\big{(}\epsilon_{i}^{mk}A_{mkj}+\epsilon_{j}^{mk}A_{mki}\big{)}$

which is a traceless symmetric rank-2 tensor. And from which we may
reconstruct the corresponding rank-3 tensor $B$, defined by:

$B_{ijk}=\frac{1}{3}\big{[}\epsilon_{ik}^{p}b_{pj}+\epsilon_{ij}^{p}b_{pk}\big{]}$

Thus, we may provide a total harmonic decomposition for $A$ in terms of the
invariant subspaces of the rank-1 $u$ and the rank-2 $b$, such that:

$A_{ijk}=V_{ijk}+B_{ijk}$

And so, as mentioned above, the symmetric part $S$ is readily decomposed in
the spherical bases into $\mathcal{H}^{(1)}\oplus\mathcal{H}^{(3)}$. And then
the mixed-symmetry part inhabits the space
$\mathcal{H}^{(1)}\oplus\mathcal{H}^{(2)}$.

###  A.3 Elastic Tensor Decomposition

If we examine the Young diagrams of a fourth rank tensor, as below:

We can immediately notice that the symmetries of $C$ require that all but the
totally symmetric tensor $S$ and one mixed symmetry tensor $A$ must vanish.
$S$ has 15 independent components, and $A$ has 6. Furthermore, $A$ has
corresponding symmetrizer
$\mathcal{C}S(x_{1}x_{2})S(x_{3}x_{4})A(x_{1}x_{3})A(x_{2}x_{4})$. Thus, $A$
is exactly the tensor symmetric under permutation of $i,j$ and $k,l$ but
antisymmetric under exchanges $i,k$ and $j,l$.

Note that this mixed-symmetry subspace $A$ corresponds to Backus’ [Bac70]
asymmetric tensor $A$. With $S$ and $A$ being defined component-wise as:

$S_{ijkl}=\frac{1}{3}\big{(}C_{ijkl}+C_{ikjl}+C_{klij}\big{)}$
$A_{ijkl}=\frac{1}{3}\big{(}2C_{ijkl}-C_{ikjl}-C_{klij}\big{)}$

The fully symmetric part can of course be converted to spherical harmonic
components via a Clebsch-Gordon expansion. The mixed symmetry $A$, however
requires further decomposition.

All six components can be described by the symmetric (but not traceless)
tensor $t$, defined as the double partial contraction of $A$ with the totally
antisymmetric tensor $\epsilon$ as:

$t_{ij}=\epsilon_{i}^{mk}\epsilon_{j}^{nl}A_{mnkl}$

which can be reconstructed as the subtensor $N$ as:

$N_{ijkl}=\frac{1}{2}\big{(}\epsilon\epsilon-\epsilon\epsilon\big{)}t_{mn}$

This tensor $t$ then has a harmonic decomposition according to the rank-two
Clebsch-Gordon transformation between the $J_{z}$ basis and the harmonic basis
$y_{\ell}^{m}$.

##  Appendix B Table of Coefficients for $J_{z}\rightarrow y_{\ell}^{m}$
Transformation

$\mathbf{Rank\ 2}$ |  |  |  |  |  |
---|---|---|---|---|---|---
| $a_{00}$ | $a_{+-}$ | $a_{0\pm}$ | $a_{\pm\pm}$ |  |
$y_{0}^{0}$ | $1$ | $-2$ |  |  |  |
$y_{2}^{0}$ | $1$ | $1$ |  |  |  |
$y_{2}^{\pm 1}$ |  |  | $\sqrt{3}$ |  |  |
$y_{2}^{\pm 2}$ |  |  |  | $\sqrt{\frac{3}{2}}$ |  |
$\mathbf{Rank\ 3}$ |  |  |  |  |  |
| $a_{000}$ | $a_{0+-}$ | $a_{00\pm}$ | $a_{+-\pm}$ | $a_{0\pm\pm}$ | $a_{\pm\pm\pm}$
$y_{1}^{0}$ | $3$ | $-6$ |  |  |  |
$y_{3}^{0}$ | $\frac{15}{7}$ | $-\frac{5}{7}$ |  |  |  |
$y_{1}^{\pm 1}$ |  |  | $\sqrt{\frac{3}{2}}$ | $-2\sqrt{\frac{3}{2}}$ |  |
$y_{3}^{\pm 1}$ |  |  | $2\sqrt{\frac{3}{2}}$ | $\sqrt{\frac{3}{2}}$ |  |
$y_{3}^{\pm 2}$ |  |  |  |  | $\sqrt{\frac{15}{2}}$ |
$y_{3}^{\pm 3}$ |  |  |  |  |  | $\sqrt{\frac{5}{2}}$
Elastic |  |  |  |  |  |
(Rank 4) |  |  |  |  |  |
$(m=0)$ | $a_{0000}$ | $a_{00+-}$ | $a_{0+0-}$ | $a_{+-+-}$ | $a_{++--}$ |
$y_{0}^{0}$ | $1$ | $2$ | $-6$ | 1 | 3 |
$y_{2}^{0}$ | 1 | $2$ | $-3$ | 1 | $-3$ |
$y_{4}^{0}$ | 1 | $2$ | $4$ | 1 | $\frac{1}{2}$ |
$(m=1,2)$ | $a_{000\pm}$ | $a_{+-0\pm}$ | $a_{\mp 0\pm\pm}$ | $a_{00\pm\pm}$ | $a_{+-\pm\pm}$ | $a_{\mp\pm\mp\pm}$
$y_{2}^{\pm 1}$ | $\sqrt{3}$ | $\sqrt{3}$ | $-3\sqrt{3}$ |  |  |
$y_{4}^{\pm 1}$ | $2\sqrt{\frac{5}{2}}$ | $2\sqrt{\frac{5}{2}}$ | $\sqrt{\frac{5}{2}}$ |  |  |
$y_{2}^{\pm 2}$ |  |  |  | $-2\sqrt{\frac{3}{2}}$ | $-2\sqrt{\frac{3}{2}}$ | $3\sqrt{\frac{3}{2}}$
$y_{4}^{\pm 2}$ |  |  |  | $\sqrt{\frac{5}{2}}$ | $\sqrt{\frac{5}{2}}$ | $2\sqrt{\frac{5}{2}}$
$(m=3,4)$ | $a_{\pm\pm\pm 0}$ | $a_{\pm\pm\pm\pm}$ |  |  |  |
$y_{4}^{\pm 3}$ | $\sqrt{\frac{35}{2}}$ |  |  |  |  |
$y_{4}^{\pm 4}$ |  | $\frac{1}{2}\sqrt{\frac{35}{2}}$ |  |  |  |

Figure 8: Coefficients for transformation between spherical basis components
$a_{\alpha\beta\gamma\delta}$ and harmonic components $y^{m}_{l}$ for a
rank-2,3,4 symmetric tensor. Note that the rank-4 is only relevant for
symmetric components of elastic tensors and correspond to Mochizuki’s
transformation [Moc88] between the $J_{z}$ basis and their symmetric tensor
$s$.

##  Appendix C Real Spherical Harmonics

The (complex) spherical harmonics $Y_{\ell}^{m}$ can be transformed into a set
of real spherical harmonics $Y_{\ell m}$ according the following relations:

$Y_{\ell
m}=\begin{cases}\frac{i}{\sqrt{2}}\big{(}Y_{\ell}^{-|m|}-(-1)^{m}Y_{\ell}^{|m|}\big{)}:\
\ m<0\\\ \quad\quad\quad\quad Y_{\ell}^{0}:\quad\quad\quad\quad\quad\ \ \ \
m=0\\\
\frac{1}{\sqrt{2}}\big{(}Y_{\ell}^{-|m|}+(-1)^{m}Y_{\ell}^{|m|}\big{)}:\ \
m>0\\\ \end{cases}$

(of course, this choice is not entirely unique, with the choice made here
assuming a Condon-Shortley phase included in the definition of
$Y_{\ell}^{m}$).

##  Appendix D Young Diagrams, Tableaux and Symmetrizers

##  Appendix E Elastic Tensor Component-wise MAE

###  E.1 Elastic Tensor Prediction from Band Gap

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

###  E.2 Elastic Tensor Prediction from Formation Energy

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

##  Appendix F Dielectric Tensor Component-wise MAE

###  F.1 Dielectric Tensor Prediction from Band Gap

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

###  F.2 Dielectric Tensor Prediction from Formation Energy

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

###  F.3 Dielectric Tensor Prediction From Experiment 1

####  F.3.1 Band Gap as the Scalar Pretraining

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

####  F.3.2 Formation Energy as the Scalar Pretraining

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

##  Appendix G Piezoelectric Tensor Component-wise MAE

###  G.1 Piezoelectric Tensor Prediction from Band Gap

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

###  G.2 Piezoelectric Tensor Prediction from Formation Energy

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

###  G.3 Piezoelectric Tensor Prediction From Experiment 1

####  G.3.1 Band Gap as the Scalar Pretraining

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

####  G.3.2 Formation Energy as the Scalar Pretraining

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

###  G.4 Piezoelectric Tensor Prediction From Experiment 2

####  G.4.1 Band Gap as the Scalar Pretraining

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

####  G.4.2 Formation Energy as the Scalar Pretraining

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

###  G.5 Piezoelectric Tensor Prediction From Experiment 3

####  G.5.1 Band Gap as the Scalar Pretraining

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance

####  G.5.2 Formation Energy as the Scalar Pretraining

(a) SEConv Component-wise Performance

(b) SETransformer Component-wise Performance

(c) SEGNN Component-wise Performance


\bigl[ Q_1(\vect{s}_i^{\textnormal{av}}, a_i^{\prime}) -
a_i^{\prime}) \bigr]^2 \\
& = \sum\nolimits_{i=1}^{N_{\textnormal{av}}} \bigl[
\mu^{\prime}(\vect{s}_i^{\textnormal{av}}) ) -
\mu^{\prime}(\vect{s}_i^{\textnormal{av}})) \bigr]^2 \\
& = \norm{
\vectgr{\Phi}_{\mu^{\prime}}^{\textnormal{av}}{}^{\intercal}
(Q_1 - Q_2) }^2 \\
& =
\innerp{\vectgr{\Phi}_{\mu^{\prime}}^{\textnormal{av}}{}^{\intercal}
(Q_1 - Q_2)}
(Q_1 - Q_2)} \\
& = \innerp{Q_1 - Q_2}
{ \vectgr{\Phi}_{\mu^{\prime}}^{\textnormal{av}}
\vectgr{\Phi}_{\mu^{\prime}}^{\textnormal{av}}{}^{\intercal}
(Q_1 - Q_2)}_{\mathcal{H}} \\
& \leq \norm{ \vect{K}_{\mu^{\prime}}^{\textnormal{av}}
}_2 \, \norm{ Q_1 - Q_2 }_{\mathcal{H}}^2 \,,
\end{align*}

so that (<ref>) results in

\begin{align*}
& \norm{T(Q_1) - T(Q_2)}_{\mathcal{H}}^2 \notag \\
& \leq
\alpha^2 \norm{ \vect{K}_{\Psi} }_2
\sup_{\mu^{\prime}\in \mathcal{H}} \norm{
\vect{K}_{\mu^{\prime}}^{\textnormal{av}} }_2 \, \norm{
Q_1 - Q_2 }_{\mathcal{H}}^2
+ \epsilon \alpha^2 N_{\textnormal{av}} \norm{
\vect{K}_{\Psi} }_2 \\
& = \beta^2 \norm{ Q_1 - Q_2 }_{\mathcal{H}}^2
+ \epsilon \alpha^2 N_{\textnormal{av}} \norm{
\vect{K}_{\Psi} }_2 \,.
\end{align*}

Since $\epsilon$ can be made arbitrarily small, this last
inequality establishes (<ref>).

§ PROOF OF <REF>

The proof is built on arguments
of <cit.>. In particular, following <cit.> and for an arbitrarily fixed policy
$\mu(\cdot) \in \mathcal{M}$, define the linear covariance
$\Sigma_{zz}, \Sigma_{s^{\prime} z}^{\mu}, \Sigma_{
s^{\prime} \given z }^{\mu} \colon \mathcal{H} \to
\mathcal{H}$ by

Σ_zz (Q)

φ(z)} ,

Σ_s^' z^μ (Q)

_ (s^', z)  {
φ(s^', μ(s^'))Q_ℋ  φ(z)
} ,

Σ_ s^' z ^μ (Q)
Σ_s^' z^μ (Q)

where $\expect_{ \vect{z} }\{\cdot\}$ denotes expectation
with respect to (w.r.t.) the $\sigma$-subalgebra generated
by $\vect{z}$ <cit.>, similarly for
$\expect_{ (\vect{s}^{\prime}, \vect{z}) }\{ \cdot\}$, and
$\Sigma_{zz}^{-1}$ stands for the inverse of
$\Sigma_{zz}$. It is important to stress here that the
expectation symbols in (<ref>) are considered in
the following sense. In (<ref>), for example, for
arbitrarily fixed $Q\in \mathcal{H}$, $\Sigma_{zz} (Q)$
stands for the unique point of $\mathcal{H}$ which
satisfies, according to the Riesz representation
theorem <cit.>,
$\innerp{\Sigma_{zz} (Q)} {h} = L_{zz} ( h )$,
$\forall h\in \mathcal{H}$, where $L_{zz} (\cdot)$ is the
linear continuous operator defined by
$L_{zz} (\cdot) \colon \mathcal{H} \to \Real \colon h
\mapsto L_{zz} ( h ) \coloneqq \expect_{\vect{z}} \Set{
\innerp{ \innerp{Q} { \varphi(\vect{z}) }_{\mathcal{H}}
\cdot \varphi(\vect{z})} {h}_{\mathcal{H}} }$, and where
expectation is taken here in the usual
sense <cit.>. Operators (<ref>)
and (<ref>) are defined in a similar

Recall here also the definitions for the minimum
$\sigma_{\min} ( \mathcal{A} )$ and maximum
$\sigma_{\max} ( \mathcal{A} )$ spectral values of a linear
bounded and self-adjoint operator
$\mathcal{A} \colon \mathcal{H} \to
\mathcal{H}$ <cit.>:

σ_min ( 𝒜 )

inf_ h_ℋ = 1  h
𝒜 (h)_ℋ

σ_max ( 𝒜 )

sup_ h_ℋ = 1  h
𝒜 (h)_ℋ =  𝒜

It can be verified
by <cit.> and
<cit.> that
$\expect_{ \vect{s}^\prime \given \vect{z} } \{
Q(\vect{s}^{\prime}, \mu(\vect{s}^{\prime})) \} = \innerp{
\Sigma_{ s^{\prime} \given z }^{\mu\, *} (
\varphi(\vect{z}) ) }{ Q }_{\mathcal{H}}$, where $*$
denotes the adjoint of a linear
operator <cit.>. Hence, by the reproducing property
of the kernel in $\mathcal{H}$,
$\expect_{ \vect{s}^\prime \given \vect{z} } \{
Q(\vect{s}^{\prime}, \mu(\vect{s}^{\prime})) \} =
\innerp{\varphi( \vect{z} )}{ \Sigma_{ s^{\prime} \given z
}^{\mu} (Q) }_{\mathcal{H}} = \Sigma_{ s^{\prime} \given z
}^{\mu} (Q) (\vect{z})$, and the classical
B-Maps (<ref>) take the following
equivalent form:

(T_μ^♢ Q)(z)

g(z) + αΣ_
s^' z ^μ (Q)

(T^♢ Q)(z)
g(z) + αΣ_ s^'
z ^μ_Q (Q) (z)

where the stationary policy $\mu_Q(\cdot)$ is defined as
in (<ref>). Along the lines of
<ref> and <cit.>,

\begin{align}
\hat{\Sigma}_{ s^{\prime} \given z }^{\mu}
& \coloneqq \hat{\Sigma}_{ s^{\prime} \given z }^{\mu} (N)
\notag \\
& \coloneqq
\vectgr{\Phi}_{\mathcal{T}_N} ( \vect{K}_{\mathcal{T}_N} +
N \sigma^{\prime}_N \vect{I}_N )^{-1}
\vectgr{\Phi}_{\mu}^{\textnormal{av}\intercal} \notag \\
& = \tfrac{1}{\sqrt{N}} \vectgr{\Phi}_{\mathcal{T}_N}
\left( \tfrac{1}{N} \vect{K}_{\mathcal{T}_N} +
\sigma^{\prime}_N \vect{I}_N \right)^{-1}
\tfrac{1}{\sqrt{N}}
\vectgr{\Phi}_{\mu}^{\textnormal{av}\intercal}
\,, \label{hat.classical.Bellman.mu.alternative}
\end{align}

so that (<ref>) is recast as

\begin{align}
(T_{\mu} Q) (\vect{z})
& \coloneqq g(\vect{z}) + \alpha \hat{\Sigma}_{ s^{\prime}
\given z }^{\mu} (Q) (\vect{z})
\,. \label{new.Bellman.mu.alternative}
\end{align}

The following theorem asserts that the previous quantity is
a consistent finite-sample estimate
of (<ref>).

<ref> means that

\begin{align*}
\sum\nolimits_{i=1}^{+\infty} \norm{ \Sigma_{zz}^{-3/2}
\Sigma_{ s^{\prime}z }^{\mu} e_i}_{\mathcal{H}}^2 <
\end{align*}

for a countable orthonormal basis $(e_i)_{i=1}^{+\infty}$
of $\mathcal{H}$ <cit.>. Under also
$\Plim_{N\to \infty} \norm{ \Sigma_{ s^{\prime} \given z
}^{\mu} - \hat{\Sigma}_{ s^{\prime} \given z }^{\mu} (N)
} = 0$.

<cit.> yields
$\Plim_{N\to \infty} \norm{ \Sigma_{ s^{\prime} \given z
}^{\mu} - \hat{\Sigma}_{ s^{\prime} \given z }^{\mu} (N)
}_{\textnormal{HS}} = 0$, where
$\norm{\cdot}_{\textnormal{HS}}$ stands for the
Hilbert-Schmidt norm of an
operator <cit.>. Consequently, the fact
$\norm{\cdot} \leq
\norm{\cdot}_{\textnormal{HS}}$ <cit.>
establishes the claim of the theorem.

The claim of <ref> that
$T_{\mu}$ and $T$ are contractions follows directly by
(<ref>) and assumption
$\beta(N) \leq \beta_{\infty} < 1$, a.s. Now, the triangle
inequality and <ref> suggest

≤  T_μ^♢
(Q_1) - T_μ^♢

T_μ^♢ (Q_1) - T_μ
(Q_1)_ℋ +
T_μ^♢ (Q_2) - T_μ

+ T_μ (Q_1) - T_μ (Q_2)_ℋ

α Σ_ s^' z
^μ - Σ̂_ s^' z ^μ (N)

+ α Σ_ s^' z
^μ - Σ̂_ s^' z
^μ (N)    Q_2

+ β_∞ Q_1 - Q_2_ℋ  .

By applying $\Plim_{N\to \infty}$ to
(<ref>) and by
<ref>, it can be verified that
$\norm{T_{\mu}^{\diamond} (Q_1) - T_{\mu}^{\diamond}
(Q_2)}_{\mathcal{H}} \leq \beta_{\infty} \norm{Q_1 -
Q_2}_{\mathcal{H}}$, $\forall Q_1, Q_2\in \mathcal{H}$,
a.s. The claim
$\norm{T^{\diamond} (Q_1) - T^{\diamond}
(Q_2)}_{\mathcal{H}} \leq \beta_{\infty} \norm{Q_1 -
Q_2}_{\mathcal{H}}$, $\forall Q_1, Q_2\in \mathcal{H}$,
a.s., can be established in a similar way to
(<ref>), but with $\mu_{Q_1}$ and
$\mu_{Q_2}$, whose definitions are provided
below (<ref>), taking the place of
$\mu$ in (<ref>) and
(<ref>), respectively.

Consider now the fixed points $Q_{\mu}^{\diamond}$ and
$Q_{\mu}$ of $T_{\mu}^{\diamond}$ and $T_{\mu}$,
respectively. Notice now that

=   Q_μ^♢ -

T_μ^♢ (Q_μ^♢) -
T_μ(Q_μ) _ℋ

≤       T_μ^♢ (Q_μ^♢)
- T_μ (Q_μ^♢)_ℋ +
T_μ (Q_μ^♢) -

≤       α ( Σ_ s^' z
^μ - Σ̂_ s^' z
^μ (N) ) (Q_μ^♢) _ℋ +
βQ_μ^♢- Q_μ_ℋ

≤       α Σ_ s^' z
^μ - Σ̂_ s^' z
^μ (N)     Q_μ^♢
_ℋ + β_∞ Q_μ^♢-
Q_μ_ℋ ,

which yields

\begin{align*}
& \norm{Q_{\mu}^{\diamond} - Q_{\mu}(N)}_{\mathcal{H}}
\leq \frac{\alpha \norm{ Q_{\mu}^{\diamond}
}_\mathcal{H} }{1-\beta_{\infty}} \norm{ \Sigma_{
s^{\prime} \given z }^{\mu} - \hat{\Sigma}_{ s^{\prime}
\given z }^{\mu} (N)} \,,
\end{align*}

that establishes in turn (<ref>) by

Notice now that $Q_*^{\diamond} - Q_* =
T^{\diamond}(Q_*^{\diamond}) - T(Q_*) =
T^{\diamond}(Q_*^{\diamond}) - T(Q_*^{\diamond}) +
T(Q_*^{\diamond}) - T(Q_*)$, that

\begin{align*}
T^{\diamond}(Q_*^{\diamond}) - T(Q_*^{\diamond})
= ( \Sigma_{
s^{\prime} \given z }^{ \mu_{Q_*^{\diamond}} } -
\hat{\Sigma}_{ s^{\prime} \given z }^{
\mu_{Q_*^{\diamond}} } (N) ) ( Q_*^{\diamond} ) \,,
\end{align*}

and follow steps like those in (<ref>) to
establish (<ref>) by

§ PROOF OF <REF>

The following discussion is performed for any $\omega$
chosen arbitrarily from $E^{(\epsilon)}$ of
<ref>, after possibly excluding from
$E^{(\epsilon)}$ the union of zero-probability events which
appear via the qualifier “a.s.” in
<Ref>. By the definition of
$E^{(\epsilon)}$, $\omega \in E^{(\epsilon)}$ implies that
there exists a sufficiently large $n_0$ s.t. for any
$n\geq n_0$, there exists a sufficiently large $N[n]$ with
$\omega \in E_{n,N[n]}^{(\epsilon)}$.

By <ref>, $T_{\mu_n}^{\diamond}$ is
$\beta_{\infty}$-Lipschitz continuous, and thus a
contraction for all sufficiently large $n$ by
<ref> and the discussion
after (<ref>). Recall then the
Banach-Picard fixed-point theorem <cit.>, which
guarantees that $\forall Q\in \mathcal{H}$,
$\lim_{K\to \infty} (T_{\mu_{n+1}}^{\diamond})^K(Q) =
Q_{\mu_{n+1}}^{\diamond}$, with $Q_{\mu_{n+1}}^{\diamond}$
being the unique fixed point of

For all sufficiently large $n$, a.s.,

\begin{align*}
\norm{Q_{\mu_{n+1}}^{\diamond} - T_{\mu_{n+1}}^{\diamond}
(Q_{\mu_n}^\diamond)}_{\mathcal{H}} \leq
\frac{\Delta_2}{1-\beta_\infty} \,.
\end{align*}

For any $k\in \IntegerPP$,

\begin{align*}
& \norm{(T_{\mu_{n+1}}^{\diamond})^k(Q_{\mu_n}^\diamond) -
\\
& \leq \beta_\infty \norm{(T_{\mu_{n+1}}^\diamond)^{k-1}
(Q_{\mu_n}^\diamond) - (T_{\mu_{n+1}}^\diamond)^{k-2}
(Q_{\mu_n}^\diamond)}_{\mathcal{H}} \\
& \leq \beta_\infty^{k-1} \norm{(T_{\mu_{n+1}}^\diamond
- \Id) (Q_{\mu_n}^\diamond) }_\mathcal{H} \leq
\beta_\infty^{k-1} \Delta_2 \,,
\end{align*}

by <ref>. Hence, for any
$K\in \IntegerPP$,

\begin{align}
& \norm{(T_{\mu_{n+1}}^\diamond)^K (Q_{\mu_n}^\diamond) -
\notag\\
& \leq \sum\nolimits_{k=1}^K
\norm{(T_{\mu_{n+1}}^\diamond)^k (Q_{\mu_n}^\diamond)
- (T_{\mu_{n+1}}^\diamond)^{k-1}
(Q_{\mu_n}^\diamond)}_{\mathcal{H}} \notag\\
& \leq \sum\nolimits_{k=1}^K \beta_\infty^{k-1} \Delta_2
\leq \Delta_2
\sum \nolimits_{k=0}^{\infty} \beta_\infty^{k} =
\tfrac{\Delta_2}{1-\beta_\infty}
\,. \label{bound.Id-T.1}
\end{align}

$\lim_{K\to \infty}(T_{\mu_{n+1}}^\diamond)^K
(Q_{\mu_n}^\diamond) = Q_{\mu_{n+1}}^\diamond$, the
application of $\lim_{K\to \infty}$
to (<ref>) establishes

For all sufficiently large $n$,

Q_*^♢_ℋ + 2 β_∞(Δ_0 + ϵ)

+ Δ_1 + Δ_2/1-β_∞  .

By <ref> and by following again the
discussion after (<ref>), it can be
verified that $T^\diamond$ is a
$\beta_{\infty}$-contraction. Observe also by
<ref> and <ref> that

T^♢(Q_n)-Q_*^♢_ℋ +

T^♢(Q_n) - T^♢(Q_*^♢)
_ℋ +

≤       β_∞Q_n-Q_*^♢_ℋ +

+ T_μ_n+1^♢(Q_n) -

≤       β_∞ Q_μ_n^♢ -
Q_*^♢_ℋ +
β_∞ Q_n -

+ β_∞ Q_μ_n(N[n]) -

+ Q_μ_n+1^♢-

+ T_μ_n+1^♢(Q_μ_n^♢) -

+ T_μ_n+1^♢(Q_n) -

≤       β_∞ Q_μ_n^♢ -
Q_*^♢_ℋ +
β_∞ (Δ_0 + ϵ)

+ Δ_21-β_∞ +
β_∞ Q_μ_n^♢-
Q_n_ℋ + Δ_1

≤       β_∞ Q_μ_n^♢ -
Q_*^♢_ℋ +
β_∞ (Δ_0+ϵ)

+ Δ_21-β_∞
+ β_∞ Q_μ_n(N[n]) -

+ β_∞ Q_μ_n^♢-
Q_μ_n(N[n])_ℋ + Δ_1

≤       β_∞ Q_μ_n^♢ -
Q_*^♢_ℋ +
2 β_∞ (Δ_0+ϵ) + Δ_1
+ Δ_21-β_∞  ,

which establishes <ref>.

For a sufficiently large $n$, the application of
<ref> recursively for $K$ times yields

Q_μ_n+K^♢ -

≤        β_∞^K
Q_μ_n^♢ -

+ ∑_k=0^K-1 β_∞^k (
2β_∞(Δ_0 + ϵ) + Δ_1 +
Δ_21-β_∞ )

≤       β_∞^K

+ 11-β_∞ (
2β_∞(Δ_0 + ϵ) + Δ_1 +
Δ_21- β_∞
)_Δ^'  ,

and because of $\beta_{\infty} < 1$,

\begin{align*}
\limsup_{n \to \infty} \norm{Q_{\mu_{n}}^\diamond -
Q_*^\diamond}_{\mathcal{H}} = \limsup_{K \to \infty}
\norm{Q_{\mu_{n+K}}^\diamond - Q_*^\diamond}_{\mathcal{H}}
\leq \Delta^{\prime} \,.
\end{align*}

Now, the triangle inequality suggests

≤  Q_n -

Q_n - Q_μ_n(N[n])_ℋ +
Q_μ_n(N[n]) - Q_μ_n^♢_ℋ

+ Q_μ_n^♢ -

≤       Δ_0 + ϵ+ Q_μ_n^♢ -
Q_*^♢_ℋ ,

and an application of $\limsup_{n\to \infty}$ to the
previous inequality yields
$\limsup_{n\to \infty} \norm{Q_{n} -
Q_*^{\diamond}}_{\mathcal{H}} \leq \Delta_0 + \epsilon +
\Delta^{\prime}$, which establishes <ref>.

§ PROOF OF <REF>

First, recall operators (<ref>)
and (<ref>). Define

\begin{align}
\xi_n \coloneqq ( \hat{\Sigma}_{ s^{\prime} \given z
}^{\mu_n} (N) - \Sigma_{ s^{\prime} \given z
}^{\mu_n} )^* (\varphi(\vect{z}_{n-1})) \,, \label{xi.def}
\end{align}

where superscript $*$ over a bounded linear operator denotes
its adjoint operator <cit.>.

The linear covariance operators
$\Sigma_{zz}^{(n)}, \Sigma_{\xi z}^{(n)}, \Sigma_{\xi
\xi}^{(n)} \colon \mathcal{H} \to \mathcal{H}$ are
introduced next; $\forall Q\in \mathcal{H}$,

Σ_zz^(n) (Q)

·φ(z_n) }

Σ_ξz^(n) (Q)
·ξ_n }  ,

·( Σ̂_ s^' z
^μ_n (N) - Σ_ s^' z
^μ_n )^* (φ(z_n-1)) }

Σ_ξξ^(n) (Q)
ξ_nQ_ℋ ·ξ_n }

( Σ̂_ s^'
z ^μ_n (N) - Σ_ s^' z
^μ_n ) (Q) _ℋ

·( Σ̂_ s^' z
^μ_n (N) - Σ_ s^' z
^μ_n )^* (φ(z_n-1)) }  ,

where expectations in (<ref>) are
considered along the lines of (<ref>). It can be
verified that $\Sigma_{zz}^{(n)}, \Sigma_{\xi\xi}^{(n)}$ are
self adjoint, ,
$\Sigma_{zz}^{(n)}{}^* = \Sigma_{zz}^{(n)}$ and
$\Sigma_{\xi\xi}^{(n)}{}^* = \Sigma_{\xi\xi}^{(n)}$.

With $\mathcal{L}_{\mu_n}^{(n)} [\vect{z}_{n-1}] (\cdot)$
defined in (<ref>), its expected loss
$G_{\mu_n} (\cdot) \coloneqq \expect
\{\mathcal{L}_{\mu_n}^{(n)} [ \vect{z}_{n-1} ] (\cdot) \}$
takes the following form for all sufficiently large $n$:

G_μ_n (Q)
=         12 Q
𝒜_μ_n (Q)_ℋ +

+ 12
gΣ_zz(g)_ℋ  ,   ∀Q∈ℋ ,

where the linear
$\mathcal{A}_{\mu_n}, \mathcal{B}_{\mu_n} \colon
\mathcal{H} \to \mathcal{H}$ are defined by

(αΣ_ s^' z ^μ_n - )^*
Σ_zz (αΣ_ s^' z
^μ_n - ) + α^2

+ αΣ_ξz (αΣ_
s^' z ^μ_n - ) + α(αΣ_ s^' z ^μ_n - )^*
Σ_ξz^*   ,

(αΣ_ s^' z ^μ_n -
)^* Σ_zz + αΣ_ξz

Loss (<ref>) takes the following form:
$\forall Q\in \mathcal{H}$,

ℒ_μ_n^(n) [z_n-1] (Q)
=  12
T_μ_n^(n)(Q) -
Qφ( z_n-1 )_ℋ^2

(αΣ_ s^' z
^μ_n - )(Q)

+ αξ_nQ_ℋ
+ g
φ(z_n-1)_ℋ ]^2

=                12 [
term_1 + term_2 +
term_3  ,

12 [

12 [
(αΣ_ s^' z ^μ_n
- )
+ α^2 ξ_nQ_ℋ^2

+ 2 α(αΣ_ s^'
z ^μ_n - )(Q)
Q_ℋ ]  ,

12 [

12 [ g
φ(z_n-1)_ℋ ·(αΣ_ s^' z
^μ_n - ) (Q)

12 [ + αg
Q_ℋ  ,

12 [
term_3  12
g φ(z_n-1)_ℋ^2

A closer look at $\textnormal{term}_1$
via (<ref>) suggests that

=   { (αΣ_ s^' z ^μ_n - ) (Q)
φ(z_n-1)_ℋ^2 }

{ ⟨(αΣ_ s^' z ^μ_n - )(Q)
(αΣ_ s^' z
^μ_n - )(Q)

_z_n-1 {  ·φ(z_n-1) ⟩_ℋ }

⟨(αΣ_ s^' z
^μ_n - )(Q) { (αΣ_ s^'
z ^μ_n - )(Q)

⟨(αΣ_ s^'
z ^μ_n - )(Q) _z_n-1
{  ·φ(z_n-1) }

Q(αΣ_ s^'
z ^μ_n - )^*
Σ_zz (αΣ_ s^'
z ^μ_n - ) (Q)_ℋ  ,

\begin{align*}
\alpha^2 \expect \{\innerp{\xi_n}
& = \alpha^2 \innerp{Q} {\expect \{ \innerp{\xi_n}
{Q}_{\mathcal{H}} \xi_n \}}_{\mathcal{H}} \\
& = \innerp{Q}
{\alpha^2 \Sigma_{\xi\xi} (Q)}_{\mathcal{H}} \,,
\end{align*}

=   2 α{(αΣ_ s^'
z ^μ_n -
)(Q) φ(z_n-1)_ℋ
ξ_n Q_ℋ}

2 αQ {(αΣ_ s^'
z ^μ_n - )(Q)
ξ_n }_ℋ

αQΣ_ξz (αΣ_
s^' z ^μ_n - )(Q)_ℋ

+ αQ Σ_ξz (αΣ_
s^' z ^μ_n - )

αQ Σ_ξz
s^' z ^μ_n - ) (Q)_ℋ

+ αQ(αΣ_
s^' z ^μ_n -
)^* Σ_ξz^*(Q)_ℋ

Q  ( αΣ_ξz
s^' z ^μ_n - ) + α( αΣ_
s^' z ^μ_n - )^*
Σ_ξz^*) (Q)_ℋ  .

$\expect\{ \textnormal{term}_1 \} = (1/2)
\innerp{Q}{\mathcal{A}_{\mu_n}
(Q)}_{\mathcal{H}}$. Moreover,

=   { term_2 }

{ g
φ(z_n-1)_ℋ ·(αΣ_ s^'
z ^μ_n - ) (Q)

{ + αg
φ(z_n-1)_ℋ ·ξ_nQ_ℋ}

(αΣ_ s^'
z ^μ_n - )(Q)
φ(z_n-1) } _ℋ

+ αQ { g
φ(z_n-1)_ℋ   ξ_n }

Q(αΣ_ s^'
z ^μ_n - )^*
Σ_zz (g)_ℋ +
αQΣ_ξz (g)_ℋ

Qℬ_μ_n (g)_ℋ

and finally,

\begin{align*}
\expect\{ \textnormal{term}_3 \}
& = \tfrac{1}{2} \expect \{ \innerp{g}
{\varphi(\vect{z}_{n-1})}_{\mathcal{H}}^2 \} \\
& = \tfrac{1}{2} \innerp{g}
{\expect \{ \innerp{g}
\varphi (\vect{z}_{n-1})\}}_{\mathcal{H}}\\
& = \tfrac{1}{2} \innerp{g} {\Sigma_{zz}
(g)}_{\mathcal{H}} \,,
\end{align*}

which completes the proof of <ref>.

It is worth noting here that by the adopted assumptions and
the observation
$\mathcal{A}_{\mu_n} = \mathcal{A}_{\mu_n}^*$, operator
$\mathcal{A}_{\mu_n}$ turns out to be bounded linear and

The expected loss $G_{\mu_n}(\cdot)$ (<ref>) is
$\sigma_{\min} (\mathcal{A}_{\mu_n})$-strongly convex for
all sufficiently large $n$.

Verify that $\forall Q_1, Q_2 \in \mathcal{H}$,
$\forall \gamma \in (0,1)$,

=   γG_μ_n (Q_1) + (1 -
γ) G_μ_n(Q_2) - G_μ_n( γQ_1 + (1 -
γ) Q_2 )

12 γ(1 - γ) [
Q_1𝒜_μ_n (Q_1)_ℋ +

12 γ(1 - γ) [  - 2
(Q_2)_ℋ ]

=        12 γ( 1 - γ)
(Q_1 - Q_2)_ℋ

≥       12 γ(1 - γ)
σ_min (𝒜_μ_n)
Q_1 - Q_2_ℋ  ,

and recall that $\sigma_{\min} (\mathcal{A}_{\mu_n})$ is
assumed to be positive for all sufficiently large $n$.

Given the assertion of <ref>, define
the minimizer

\begin{align}
\breve{Q}_{\mu_n}^{\diamond} \coloneqq
\argmin\nolimits_{ Q\in\mathcal{H} } G_{\mu_n} (Q)
\,, \label{breve.Q.def}
\end{align}

which is well defined and unique because of the coercivity
and strongly convexity of $G_{\mu_n}$ <cit.>.

For any $h_1, h_2 \in \mathcal{H}$ and any
$Q\in \mathcal{H}$,

\begin{align*}
\nabla \left( \innerp{ \cdot }{\innerp{h_1}{ \cdot
}_{\mathcal{H}}\, h_2}_{\mathcal{H}} \right) (Q) =
\innerp{Q}{h_1}_{\mathcal{H}}\, h_2 +
\innerp{Q}{h_2}_{\mathcal{H}}\, h_1 \,,
\end{align*}

where $\nabla$ stands for the Fréchet
gradient <cit.>.

Notice that for any $q\in \mathcal{H}$,

=   Q+q
h_1Q+q_ℋ  h_2_ℋ -

h_1Q_ℋ  h_2_ℋ +
Q h_1q_ℋ

+ q
h_1q_ℋ  h_2_ℋ

Qh_1_ℋ  h_2 +
Qh_2_ℋ  h_1 _ℋ

+ q h_1q_ℋ
h_2_ℋ  ,

$\lim_{0\neq \norm{q}_{\mathcal{H}} \to 0} \innerp{q}
{\innerp{h_1}{q}_{\mathcal{H}}\, h_2}_{\mathcal{H}} /
\norm{q}_{\mathcal{H}} = 0$, and recall the definition of
the Fréchet gradient <cit.> to establish

*  For all sufficiently large
$n$ and for any $Q\in \mathcal{F}_{n-1}$, a.s.,

\begin{align*}
\nabla G_{\mu_n} (Q) = \expect_{ \given
\mathcal{F}_{n-1} } \{ \nabla \mathcal{L}_{\mu_n}^{(n)}
[\vect{z}_{n-1}] (Q) \} \,,
\end{align*}

where $\expect_{ \given \mathcal{F}_{n-1} } \{ \cdot \}$
stands for the conditional expectation, conditioned on
the filtration $\mathcal{F}_{n-1}$.

*  $\nabla G_{\mu_n}$
is $\norm{ \mathcal{A}_{\mu_n} }$-Lipschitz continuous.

Because of (<ref>),
$\nabla G_{\mu_n}(Q) = \mathcal{A}_{\mu_n} (Q) +
\mathcal{B}_{\mu_n} (g)$. By (<ref>),
$\nabla \mathcal{L}_{\mu_n}^{(n)} [\vect{z}_{n-1}] (Q) =
\nabla \textnormal{term}_1 + \nabla
\textnormal{term}_2$. Following the lines of the proof of
<ref>, notice by <ref> that

\begin{align}
& \expect_{ \given
\mathcal{F}_{n-1} } \{ \nabla \langle (\alpha
\Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)(Q)
\given \innerp{(\alpha
\Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)(Q)}
{\varphi(\vect{z}_{n-1})}_{\mathcal{H}} \notag \\
& \phantom{ = \expect_{ \given
\mathcal{F}_{n-1} } \{ } \cdot
\varphi(\vect{z}_{n-1}) \rangle_{\mathcal{H}} \}
\notag \\
& = \expect_{ \given
\mathcal{F}_{n-1} } \{ \nabla \langle Q \given \innerp{Q}
{(\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)^*
\varphi(\vect{z}_{n-1})}_{\mathcal{H}} \notag \\
& \phantom{ = \expect_{ \given \mathcal{F}_{n-1} \{ } }
\cdot (\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)^*
\varphi(\vect{z}_{n-1}) \rangle_{\mathcal{H}} \}
\notag \\
& = \expect_{ \given
\mathcal{F}_{n-1} } \{ 2 \innerp{Q}{(\alpha
\Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)^*
\varphi( \vect{z}_{n-1}
)}_{\mathcal{H}} (\alpha \Sigma_{ s^{\prime} \given z
}^{\mu_n} - \Id)^* \notag \\
& \phantom{ = \expect_{ \given \mathcal{F}_{n-1} } \{ }
\cdot \varphi( \vect{z}_{n-1} ) \} \notag \\
& = 2 (\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} -
\Id)^* \notag \\
& \phantom{ = \expect_{ \given \mathcal{F}_{n-1} } \{ }
\cdot \expect_{ \given \mathcal{F}_{n-1} }
\{\innerp{(\alpha \Sigma_{ s^{\prime} \given z
}^{\mu_n} - \Id)(Q)}{\varphi( \vect{z}_{n-1}
)}_{\mathcal{H}} \varphi( \vect{z}_{n-1} )\} \notag \\
& = 2 (\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} -
\Id)^* \notag \\
& \phantom{ = \expect_{ \given \mathcal{F}_{n-1} } \{ }
\cdot \expect \{\innerp{(\alpha \Sigma_{ s^{\prime}
\given z }^{\mu_n} - \Id)(Q)}{\varphi( \vect{z}_{n-1}
\varphi( \vect{z}_{n-1} )\} \label{use.independency} \\
& = 2 (\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)^*
\Sigma_{zz} (\alpha \Sigma_{ s^{\prime} \given z
}^{\mu_n} - \Id)(Q) \,, \notag
\end{align}

where <ref> was used
in (<ref>). Furthermore,

\begin{align*}
& \alpha^2 \expect_{ \given \mathcal{F}_{n-1} } \{ \nabla
\innerp{Q} {\innerp{Q}{\xi_n}_{\mathcal{H}}
\xi_n}_{\mathcal{H}} \} \\
& = \alpha^2 \expect_{ \given \mathcal{F}_{n-1} } \{ 2
\innerp{Q}{\xi_n}_{\mathcal{H}} \xi_n\} \\
& = \alpha^2 \expect \{ 2
\innerp{Q}{\xi_n}_{\mathcal{H}} \xi_n\} \\
& = 2 \alpha^2 \Sigma_{\xi\xi}(Q) \,,
\end{align*}

=   α_ ℱ_n-1  { ∇Q (αΣ_ s^'
z ^μ_n - )(Q) φ(
z_n-1 )_ℋ ξ_n_ℋ}

α_ ℱ_n-1  { ∇Q Q  (αΣ_ s^' z ^μ_n - )^*
φ( z_n-1 )_ℋ

α_ ℱ_n-1  {
Q (αΣ_ s^' z
^μ_n - )^* (φ(
z_n-1 ))_ℋ ξ_n

α_ ℱ_n-1
{  + Qξ_n_ℋ ( αΣ_ s^' z ^μ_n - )^* (φ(
z_n-1 ))}

α_ ℱ_n-1  {
(αΣ_ s^' z
^μ_n - )(Q) φ(
z_n-1 )_ℋ ξ_n }

α_ ℱ_n-1
{  + α(αΣ_ s^' z
^μ_n - )
_ ℱ_n-1
{Qξ_n_ℋ φ(
z_n-1 )}

(αΣ_ s^' z
^μ_n - )(Q) φ(
z_n-1 )_ℋ ξ_n }

α{  + α(αΣ_ s^' z ^μ_n - )
{Qξ_n_ℋ φ(
z_n-1 )}

( αΣ_ξz (αΣ_ s^' z
^μ_n - )
+ α(αΣ_ s^' z
^μ_n - )^* Σ_ξz^* ) (Q)  ,

where <ref> was used again as
in (<ref>) to replace conditional
expectations by $\expect\{ \cdot \}$. The previous
derivations suggest
$\expect_{ \given \mathcal{F}_{n-1} } \{ \nabla
\textnormal{term}_1 \} =
\mathcal{A}_{\mu_n}(Q)$. Moreover,

=   _ ℱ_n-1  { ∇term_2 }

_ ℱ_n-1  { ∇g φ( z_n-1
)_ℋ  (αΣ_
s^' z ^μ_n - )(Q) φ(
z_n-1 )_ℋ

_ ℱ_n-1
{  + α∇g
φ( z_n-1 )_ℋ

(αΣ_ s^' z
^μ_n - )^* _
{g φ( z_n-1 )_ℋ
φ( z_n-1 ) }

_ ℱ_n-1
{  + α_ ℱ_n-1
{ g φ( z_n-1 ) ξ_n }

(αΣ_ s^' z
^μ_n - )^* {g φ( z_n-1 )_ℋ
φ( z_n-1 ) }

_ ℱ_n-1
{  + α{ g φ(
z_n-1 ) ξ_n }

=        (αΣ_ s^' z
^μ_n - )^* Σ_zz
(g) + αΣ_ξz (g) =
ℬ_μ_n (g)  .

Gathering all of the previous results,
$\expect_{ \given \mathcal{F}_{n-1} } \{ \nabla
\textnormal{term}_1 + \nabla \textnormal{term}_2 \} =
\mathcal{A}_{\mu_n}(Q) + \mathcal{B}_{\mu_n} (g) = \nabla
G_{\mu_n}(Q)$, which establishes the proof of

The proof of <ref> follows
directly from the observation that $\forall Q_1, Q_2\in
\mathcal{H}$,

\begin{align*}
\norm{ \nabla G_{\mu_n}(Q_1) - \nabla G_{\mu_n}(Q_2)
& = \norm{ \mathcal{A}_{\mu_n}(Q_1) -
\mathcal{A}_{\mu_n}(Q_2) }_{\mathcal{H}} \\
& \leq \norm{ \mathcal{A}_{\mu_n} } \, \norm{ Q_1 - Q_2
}_{\mathcal{H}} \,,
\end{align*}

where it can be also observed
by (<ref>) that

\begin{align*}
\norm{ \mathcal{A}_{\mu_n} } {} \leq {}
& \norm{ \Sigma_{zz} }\,
\norm{ \alpha \Sigma_{ s^{\prime} \given z
}^{\mu_n} - \Id}^2 + \alpha^2
\norm{ \Sigma_{\xi\xi} } \\
& + 2\alpha \norm{ \Sigma_{\xi z} }\, \norm{ \alpha
\Sigma_{ s^{\prime} \given z
}^{\mu_n} - \Id} \,.
\end{align*}

There exist $c_1, c_2 \in \RealPP$ s.t. for all
sufficiently large $n$ and for any
$Q \in \mathcal{F}_{n-1}$, a.s.,

\begin{align*}
\expect_{ \given \mathcal{F}_{n-1} } \{ \norm{
\nabla \mathcal{L}_{\mu_n}^{(n)} [\vect{z}_{n-1}] (Q) -
\nabla G(Q) }_{\mathcal{H}}^2 \} \leq c_1
\norm{Q}_{\mathcal{H}}^2 + c_2 \,.
\end{align*}

Because of <ref>,

\begin{align*}
& \expect_{ \given \mathcal{F}_{n-1} } \{ \norm{
\nabla \mathcal{L}_{\mu_n}^{(n)} [\vect{z}_{n-1}] (Q)
- \nabla G(Q) }_{\mathcal{H}}^2 \} \notag\\
& = \expect_{ \given \mathcal{F}_{n-1} } \{ \norm{
\nabla \mathcal{L}_{\mu_n}^{(n)}
[\vect{z}_{n-1}] (Q) }_{\mathcal{H}}^2 \} -
\norm{ \nabla G(Q)
}_\mathcal{H}^2 \\
& \leq \expect_{ \given \mathcal{F}_{n-1} } \{
\norm{ \nabla \mathcal{L}_{\mu_n}^{(n)}
[\vect{z}_{n-1}] (Q) }_{\mathcal{H}}^2 \} \,.
\end{align*}

By following the proof of <ref>, it
can be readily verified that $\nabla
\mathcal{L}_{\mu_n}^{(n)} [\vect{z}_{n-1}] (Q) =
\mathcal{A}_{\mu_n}^{(n)} [ \vect{z}_{n-1} ] (Q) +
\mathcal{B}_{\mu_n}^{(n)} [ \vect{z}_{n-1} ] (g)$, where
the mappings

𝒜_μ_n^(n) [ z_n-1 ] (·)

·(αΣ_ s^' z
^μ_n - )^* φ(z_n-1)_ℋ

·(αΣ_ s^' z
^μ_n - )^* φ(z_n-1)

+ α·(αΣ_ s^' z
^μ_n - )^* φ(z_n-1)_ℋ ξ_n

+ α· ξ_n_ℋ
(αΣ_ s^' z ^μ_n - )^*

+ α^2 · ξ_n_ℋ ξ_n

ℬ_μ_n^(n) [ z_n-1 ] (·)
(αΣ_ s^' z
^μ_n - )^* ·  φ(z_n-1)_ℋ φ(z_n-1)

+ α· φ(z_n-1)_ℋ ξ_n  .

Notice now by <ref> that for
any $Q\in \mathcal{H}$,

\begin{align*}
\norm{ \Sigma_{ s^{\prime} \given z }^{\mu_n} (Q)
& = \norm{ \Sigma_{ s^{\prime} \given z }^{\mu_n} (Q) -
\Sigma_{ s^{\prime} \given z }^{\mu_n} (0) }_{\mathcal{H}} \\
& = \tfrac{1}{\alpha} \norm{ T_{\mu_n}^{\diamond} (Q) -
T_{\mu_n}^{\diamond}(0) }_{\mathcal{H}} \\
& \leq \tfrac{1}{\alpha} \beta_{\infty} \norm{ Q - 0
}_{\mathcal{H}} = \tfrac{1}{\alpha} \beta_{\infty}
\norm{ Q }_{\mathcal{H}} \,,
\end{align*}

which suggests that
$\norm{ \Sigma_{ s^{\prime} \given z }^{\mu_n} } \leq
\beta_{\infty} / \alpha$. This implies in turn
$\norm{\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} -
\Id}_\mathcal{H} \leq \alpha \norm{ \Sigma_{ s^{\prime}
\given z }^{\mu_n} } + \norm{\Id} \leq \alpha
\beta_{\infty} / \alpha + 1 \leq \beta_{\infty} +
1$. Moreover, the reproducing property of the kernel
$\kappa$ yields
$\norm{\varphi (\vect{z}_{n-1})}_{\mathcal{H}}^2 = \kappa(
\vect{z}_{n-1}, \vect{z}_{n-1}) \leq B_{\kappa}$. Notice
also that

𝒜_μ_n^(n) [ z_n-1 ]

[ αΣ_ s^' z
^μ_n - ^2

+ 2ααΣ_ s^' z
^μ_n -
φ(z_n-1) _ℋ

+ α^2  ξ_n _ℋ^2 ]

≤       [ (β_∞ + 1)^2
B_κ + 2α(β_∞ + 1) √(
B_κ )  ξ_n _ℋ

+ α^2  ξ_n
_ℋ^2 ] Q_ℋ  ,

ℬ_μ_n^(n) [ z_n-1 ]

≤       αΣ_ s^' z ^μ_n
φ(z_n-1) _ℋ^2 + αξ_n_ℋ  g_ℋ

≤       (β_∞ + 1) B_κ + α
ξ_n _ℋ  g_ℋ   .

Observe that function
$(\cdot)^{4/i} \colon \Real \to \Real$ is convex
$\forall i\in \{1, 2, 3\}$, and recall Jensen's inequality
for conditional expectation <cit.>
to verify that
$( \expect_{ \given \mathcal{F}_{n-1} } \{ \norm{ \xi_n
}_{\mathcal{H}}^i \} )^{4/i} \leq \expect_{ \given
\mathcal{F}_{n-1} } \{ \norm{ \xi_n
}_{\mathcal{H}}^{i\cdot 4/i} \} = \expect \{ \norm{ \xi_n
}_{\mathcal{H}}^4 \} = \mathfrak{m}_{\xi}^{(4)}$ for all
sufficiently large $n$, $\forall i\in \{1, 2, 3\}$. These
arguments suggest that there exist
$\{ \varrho_i \}_{i=0}^4 \subset \RealP$ s.t.
≤  _ ℱ_n-1  {
𝒜_μ_n^(n) [ z_n-1 ] (Q)
_ℋ^2 }

_ ℱ_n-1  { [ (β_∞ +
1)^2 B_κ + 2α(β_∞ + 1)
√( B_κ )  ξ_n _ℋ

_ ℱ_n-1  {  + α^2
ξ_n _ℋ^2 ]^2 }

_ ℱ_n-1  { ∑_i=0^4
ϱ_i ξ_n_ℋ^i }

≤       [ ∑_i=0^4
ϱ_i ( _ ℱ_n-1  {
ξ_n_ℋ^4 } )^i/4
] Q_ℋ^2

( ∑_i=0^4
ϱ_i ( 𝔪_ξ^(4) )^i/4 )
_ c_1^'
Q_ℋ^2 ≤c_1^' Q_ℋ^2

≤  _ ℱ_n-1  {
ℬ_μ_n^(n) [ z_n-1 ]
(g)_ℋ^2 }

≤       (β_∞ + 1)^2 B_κ^2

+ 2α(β_∞ + 1) B_κ
g_ℋ  _ ℱ_n-1  {  ξ_n
_ℋ }

+ α^2 g_ℋ^2  _
ℱ_n-1  {  ξ_n
_ℋ^2 }

≤       (β_∞ + 1)^2 B_κ^2

+ 2α(β_∞ + 1) B_κ
g_ℋ   ( _ ℱ_n-1  {  ξ_n
_ℋ^4 } )^1/4

+ α^2 g_ℋ^2   ( _
ℱ_n-1  {  ξ_n
_ℋ^4 } )^2/4

(β_∞ + 1)^2 B_κ^2 + 2α(β_∞ + 1) B_κ
g_ℋ   ( 𝔪_ξ^(4)

+ α^2 g_ℋ^2   (
𝔪_ξ^(4) )^1/2

c_2^'  .

≤  _ ℱ_n-1  {  ∇ℒ_μ_n^(n) [z_n-1] (Q)
_ℋ^2 }
2 c_1^'^2
Q_ℋ^2 + 2 c_2^'^2

c_1 Q_ℋ^2 + c_2  ,

where $c_1 \coloneqq 2 c_1^{\prime}{}^2$ and $c_2
\coloneqq 2 c_2^{\prime}{}^2$. This completes the proof.

An inspection of <cit.>,
under the light of <Ref>, suggests that for any
sufficiently small step size $\eta$, or more specifically,
for any

\begin{align*}
\eta < \frac{2 \sigma_{\min}( \mathcal{A}_{\mu_n} ) } {
\sigma_{\max}^2 ( \mathcal{A}_{\mu_n} ) + 2 c_1 } \,,
\end{align*}

there exists $c_3 \in \RealPP$ s.t.
\begin{align}
\limsup \nolimits_{n \to \infty}\, \expect\{ \norm{ Q_n -
\breve{Q}_{\mu_n}^{\diamond} }_{\mathcal{H}}^2 \} \leq c_3
\eta \,. \label{Qn.to.Qmin}
\end{align}

Now, because $T_{\mu_n}^{\diamond}$ is a contraction, its
fixed point $Q_{\mu_n}^{\diamond}$ is unique and satisfies
$(\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)
(Q_{\mu_n}^{\diamond}) = -g$. This implies that the linear
$(\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)$ is
non-singular, because any $Q$ in the null space of
$(\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)$
$(\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)
(Q_{\mu_n}^{\diamond} - Q) = -g$, and thus
$Q = 0$ <cit.>. Therefore,
$(\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)^*
(\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)$ is
also non-singular. Consequently, with
$\sigma_{\min} (\cdot)$ defined by (<ref>),
$\sigma_{\min}( (\alpha \Sigma_{ s^{\prime} \given z
}^{\mu_n} - \Id)^* (\alpha \Sigma_{ s^{\prime} \given z
}^{\mu_n} - \Id) ) > 0$, because otherwise, and
by (<ref>), there would exist a sequence
$(h_k)_{k\in \IntegerP} \subset \mathcal{H}$, with
$\norm{ h_k }_{\mathcal{H}} = 1$, s.t.$\lim_{k \to \infty} \norm{ (\alpha \Sigma_{ s^{\prime}
\given z }^{\mu_n} - \Id) h_k}_{\mathcal{H}}^2 = 0
\Leftrightarrow \lim_{k \to \infty} ( h_k^{\prime} \coloneqq
(\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id) h_k ) =
0 \Leftrightarrow \lim_{k \to \infty} ( h_k = (\alpha \Sigma_{
s^{\prime} \given z }^{\mu_n} - \Id)^{-1} h_k^{\prime} ) = 0
\Rightarrow 0 = \lim_{k \to \infty} \norm{h_k}_{\mathcal{H}}
= 1$, which is absurd. Moreover, notice that because of
<ref>, $G_{\mu_n}$ is
coercive <cit.>, and there exists thus
$B_{\breve{q}} \in \RealPP$ s.t.$\norm{ \breve{Q}_{\mu_n} }_{\mathcal{H}} \leq
B_{\breve{q}}$, for all sufficiently large $n$, via

Notice that there exists $\mathfrak{m}_{\xi}^{(2)} \in
\RealPP$ s.t.

sup_h_ℋ = 1
h _ξ_n
{hξ_n_ℋ   ξ_n
} _ℋ

sup_h_ℋ = 1 _ξ_n
{ h hξ_n_ℋ   ξ_n
_ℋ }

{  ξ_n _ℋ^2 }
𝔪_ξ^(2)  ,

sup_h_ℋ = 1 h
_z_n-1, ξ_n
{h φ(z_n-1) _ℋ
ξ_n }_ℋ

sup_h_ℋ = 1
_z_n-1, ξ_n { h
h φ(z_n-1) _ℋ
ξ_n _ℋ }

≤       sup_h_ℋ = 1
h_ℋ^2 _z_n-1, ξ_n
{  ξ_n _ℋ    φ(
z_n-1 ) _ℋ }

≤       √(B_κ)
{  ξ_n _ℋ }
≤( B_κ 𝔪_ξ^(2) )^1/2

where Jensen's inequality
$\expect \{ \norm{\xi_n}_{\mathcal{H}} \} \leq (
\mathfrak{m}_{\xi}^{(2)})^{1/2}$ <cit.>,
propelled by the convexity of the function $(\cdot)^2$, was
used in (<ref>). Moreover, define

𝒜_μ_n - (αΣ_ s^'
z ^μ_n -
)^* Σ_zz ( αΣ_ s^' z ^μ_n - )

αΣ_ξz (αΣ_ s^'
z ^μ_n - ) + α(αΣ_
s^' z ^μ_n - )^* Σ_ξz^* + α^2 Σ_ξξ  ,

ℬ_μ_n - (αΣ_ s^'
z ^μ_n - )^* Σ_zz =
αΣ_ξz  .

Via (<ref>),

2α( B_κ 𝔪_ξ^(2) )^1/2
(αΣ_ s^' z ^μ_n -
) + α^2 𝔪_ξ^(2)

≤       2α( B_κ 𝔪_ξ^(2) )^1/2
( β_∞ + 1 ) + α^2
𝔪_ξ^(2)  ,

≤       α( B_κ 𝔪_ξ^(2) )^1/2

Recall $(\alpha \Sigma_{ s^{\prime} \given z }^{\mu_n} - \Id)
(Q_{\mu_n}^{\diamond}) + g = 0$, and observe
$\nabla G (\breve{Q}_{\mu_n}^{\diamond}) = 0$, because of
the convexity of $G_{\mu_n} (\cdot)$. Hence,

0      =
(αΣ_ s^' z ^μ_n -
)^* Σ_zz (αΣ_ s^' z ^μ_n - ) (Q_μ_n^♢)

+ (αΣ_ s^' z ^μ_n -
)^* Σ_zz(g)  ,

0     =
𝒜_μ_n ( Q̆_μ_n^♢) +
ℬ_μ_n (g)  ,

which lead in turn to

0      =
‖(αΣ_ s^' z ^μ_n - )^*
Σ_zz (αΣ_ s^' z
^μ_n - ) (Q_μ_n^♢)

‖ + (αΣ_ s^'
z ^μ_n - )^*
Σ_zz(g) - [ 𝒜_μ_n(
Q̆_μ_n^♢) + ℬ(g) ]

=        ‖(αΣ_ s^' z
^μ_n - )^*
Σ_zz (αΣ_ s^' z
^μ_n - ) (Q_μ_n^♢-

‖ - [ 𝒜̃_μ_n
(Q̆_μ_n^♢) +
ℬ̃_μ_n (g) ] ‖_ℋ

≥       (αΣ_ s^' z ^μ_n - )^*
Σ_zz (αΣ_ s^' z
^μ_n - ) (Q_μ_n^♢ -

‖ -  𝒜̃_μ_n
(Q̆_μ_n^♢) +
ℬ̃_μ_n (g) _ℋ

≥       σ_min ( (αΣ_ s^' z
^μ_n - )^* (αΣ_ s^' z
^μ_n - ) )  σ_min ( Σ_zz )

‖ ·Q_μ_n^♢
- Q̆_μ_n^♢_ℋ -
Q̆_μ_n^♢_ℋ -
g_ℋ  .

The previous discussion suggests that there exists $c_4
\in \RealPP$ s.t.
≤  Q_μ_n^♢ -

Q̆_μ_n^♢_ℋ +
g_ℋ/σ_min ( (αΣ_ s^' z ^μ_n - )^*
(αΣ_ s^' z ^μ_n - )
)  σ_min ( Σ_zz )

≤        [ 2α√(B_κ)
( β_∞ + 1 ) + α^2
(𝔪_ξ^(2))^1/2 ] B_q̆
/σ_min ( (αΣ_ s^' z ^μ_n - )^* (αΣ_ s^'
z ^μ_n - ))
σ_min ( Σ_zz )

+  α√(B_κ)
g_ℋ /σ_min ( (αΣ_ s^' z ^μ_n - )^*
(αΣ_ s^' z ^μ_n - ) )
σ_min ( Σ_zz )

≤       c_4 (𝔪_ξ^(2))^1/2  .

Notice also that

{  ξ_n _ℋ^2 }

{ ⟨( Σ̂_ s^'
z ^μ_n - Σ_ s^' z
^μ_n )^* (φ(z_n-1))

{  ( Σ̂_
s^' z ^μ_n - Σ_
s^' z ^μ_n )^*
(φ(z_n-1)) ⟩_ℋ }

{ ⟨φ(z_n-1)

( Σ̂_
s^' z ^μ_n - Σ_
s^' z ^μ_n ) ( Σ̂_
s^' z ^μ_n - Σ_
s^' z ^μ_n )^*
(φ(z_n-1)) ⟩_ℋ }

≤       { σ_max ( ( Σ̂_
s^' z ^μ_n - Σ_
s^' z ^μ_n ) ( Σ̂_
s^' z ^μ_n - Σ_
s^' z ^μ_n )^* )

{  ·κ( z_n-1,
z_n-1) }

≤       B_κ {  ( Σ̂_
s^' z ^μ_n - Σ_
s^' z ^μ_n ) ( Σ̂_
s^' z ^μ_n - Σ_
s^' z ^μ_n )^*  }

B_κ {  Σ̂_
s^' z ^μ_n( N[n] ) - Σ_
s^' z ^μ_n ^2 }  ,

so that

=   lim sup_ n→∞
{ Q_μ_n^♢ -

lim sup_ n→∞
Q_μ_n^♢ -

≤       c_4^2 𝔪_ξ^(2)

≤       c_4^2 B_κ lim sup_ n→∞ {
Σ̂_ s^' z ^μ_n(
N[n] ) - Σ_ s^' z ^μ_n ^2
}  .

Combine now (<ref>) with (<ref>) to

≤  lim sup_
n→∞   {  Q_n -
Q_μ_n^♢ _ℋ^2 }

2 lim sup_ n→∞   {
Q_n - Q̆_μ_n^♢ _ℋ^2 }

+ 2 lim sup_ n→∞   {
Q_μ_n^♢ -
Q̆_μ_n^♢ _ℋ^2 }

≤       2c_3 η+ 2 c_4^2 B_κ lim sup_ n→∞
{  Σ̂_ s^' z
^μ_n( N[n] ) - Σ_ s^' z
^μ_n ^2 }  ,
and hence,
≤  lim sup_
n→∞   {  Q_n -
Q_*^♢ _ℋ^2 }

2 lim sup_
n→∞   {  Q_n -
Q_μ_n^♢ _ℋ^2 }

+ 2 lim sup_
n→∞   {  Q_μ_n^♢
- Q_*^♢ _ℋ^2 }

≤       4c_3 η+ 4c_4^2 B_κ lim sup_n→∞
{  Σ̂_ s^' z
^μ_n( N[n] ) - Σ_ s^' z
^μ_n ^2 }

+ 2 lim sup_
n→∞   {  Q_μ_n^♢
- Q_*^♢ _ℋ^2 }  ,

which establishes <ref> with
$\Delta_4 \coloneqq 4c_3$ and
$\Delta_5 \coloneqq 4 c_4^2 B_{\kappa}$.

§ PROOF OF <REF>

There exist $c_1 \in \RealPP$ and $c_2\in \RealP$ s.t.  the following inequalities hold true for all
$m \in \{ n - M_{\textnormal{av}} + 1, \ldots, n \}$ and
for all sufficiently large $n$, a.s.,

\begin{align*}
c_1 (y_m - \vectgr{\theta}_{n+1}^{\intercal} \vect{x}_m )^2 +
& \leq \log\, \lvert y_m -
\vectgr{\theta}_{n+1}^{\intercal} \vect{x}_m
\rvert^2 \\
& \leq 1 + ( y_m -
\vectgr{\theta}_{n+1}^{\intercal} \vect{x}_m )^2 \,.
\end{align*}

By the concavity of $\log (\cdot)$,
$\log \varpi \leq 1 + \varpi$,
$\forall \varpi\in \RealPP$. Hence,
$\log\, \lvert y_m -
\vectgr{\theta}_{n+1}^{\intercal} \vect{x}_m
\rvert^2 \leq 1 + (y_m -
\vectgr{\theta}_{n+1}^{\intercal}
\vect{x}_m)^2$. The concavity of $\log (\cdot)$
suggests also that
$\forall \varpi\in ( \Delta_6^2, \Delta_7^2 )$,

\begin{align*}
\log \varpi
& \geq \frac{ \log \Delta_7^2 - \log \Delta_6^2 } {
\Delta_7^2 - \Delta_6^2 } (\varpi -
\Delta_6^2 ) + \log \Delta_6^2 =
c_1 \varpi + c_2 \,,
\end{align*}

$c_1 \coloneqq ( \log \Delta_7^2 - \log \Delta_6^2 ) / (
\Delta_7^2 - \Delta_6^2 )$ and
$c_2 \coloneqq \log \Delta_6^2 - \Delta_6^2 ( \log
\Delta_7^2 - \log \Delta_6^2 ) / ( \Delta_7^2 - \Delta_6^2
) $. Substituting $\varpi$ in the previous inequality by
$\lvert y_m - \vectgr{\theta}_{n+1}^{\intercal} \vect{x}_m
\rvert^2$ establishes <ref>.

Recall now the data model in <ref> to verify
$y_m - \vectgr{\theta}_{n+1}^{\intercal} \vect{x}_m =
\vectgr{\theta}_*^{\intercal} \vect{x}_m + o_m -
\vectgr{\theta}_{n+1}^{\intercal} \vect{x}_m = (
\vectgr{\theta}_* - \vectgr{\theta}_{n+1} )^{\intercal}
\vect{x}_m + o_m$. Moreover,
recall (<ref>) and (<ref>) to
verify that the chosen one-step loss satisfies

\begin{align*}
g( \vect{s}_n, \mu (\vect{s}_n) )
& = g( \vect{s}_n, a_n ) = g( \vect{z}_n )\\
& = \tfrac{1}{M_{\textnormal{av}}} \sum_{m = n -
M_{\textnormal{av}} + 1}^{n}
\log \frac{\lvert y_m -
\vectgr{\theta}_{n+1}^{\intercal} \vect{x}_m \rvert^2}
{\norm{\vect{x}_m }_2^2} \,.
\end{align*}

Observe then via <ref> that

M_av + ∑_m = n -
M_av + 1^n ( y_m -
θ_n+1^⊺ x_m )^2

- ∑_m = n - M_av +
1^n log  x_m

∑_m = n - M_av + 1^n
log  |y_m -
θ_n+1^⊺ x_m

- ∑_m = n - M_av +
1^n log  x_m _2^2

∑_m = n - M_av +
log|y_m -
θ_n+1^⊺ x_m |^2/x_m _2^2

M_av g( s_n,
μ(s_n) )

≥       c_1 ∑_m = n - M_av +
1^n [ (θ_* -
θ_n+1 )^⊺ x_m +
o_m ]^2

+ M_av c_2 - ∑_m = n - M_av +
1^n log  x_m_2^2

c_1 ∑_m = n - M_av + 1^n
(θ_* -
θ_n+1 )^⊺
x_m x_m^⊺
(θ_* - θ_n+1 )

+ 2c_1 ∑_m = n - M_av + 1^n
(θ_* -
θ_n+1 )^⊺
x_m o_m

+ c_1 ∑_m = n - M_av + 1^n
o_m^2 + M_av c_2

- ∑_m = n - M_av + 1^n
log  x_m_2^2   .

Notice also by
<Ref> that there
exists $c_3 \in \Real$ s.t.$\expect_{ \given \vectgr{\theta}_{n+1} } \{ \log\,
\norm{\vect{x}_m }_2^2 \} = \expect \{ \log\,
\norm{\vect{x}_m }_2^2 \} \geq c_3$, for all sufficiently
large $m$, a.s. Observe also that
$\expect\{ \norm{ \vect{x}_m }_2^2 \} = \expect\{ \trace
(\vect{x}_m \vect{x}_m^{\intercal}) \} = \trace ( \expect\{
\vect{x}_m \vect{x}_m^{\intercal} \} ) = \trace (
\Sigma_{xx} )$. Hence, by (<ref>), a.s.,

M_av  M_av
+ ∑_m = n - M_av + 1^n
_ θ_n+1  { ( y_m -
θ_n+1^⊺ x_m )^2

- M_av c_3

+ ∑_m = n - M_av + 1^n
_ θ_n+1  { ( y_m -
θ_n+1^⊺ x_m )^2

- ∑_m = n - M_av +
1^n _ θ_n+1
{ log  x_m _2^2 }

≥       M_av _ θ_n+1  { g( s_n,
μ(s_n) ) }

≥       c_1 ∑_m = n - M_av + 1^n
(θ_* -
θ_n+1 )^⊺
_ θ_n+1  {
x_m x_m^⊺ }
(θ_* - θ_n+1 )

+ 2c_1 ∑_m = n - M_av + 1^n
(θ_* -
θ_n+1 )^⊺
_ θ_n+1  {
x_m o_m }

+ c_1 ∑_m = n - M_av + 1^n
_ θ_n+1  { o_m^2 }
+ M_av c_2

- ∑_m = n - M_av + 1^n
_ θ_n+1  { log
x_m_2^2 }

≥       c_1 ∑_m = n - M_av + 1^n
(θ_* -
θ_n+1 )^⊺
x_m x_m^⊺ }
(θ_* - θ_n+1 )

+ 2c_1 ∑_m = n - M_av + 1^n
(θ_* -
θ_n+1 )^⊺
{ x_m } { o_m }

+ c_1 ∑_m = n - M_av + 1^n
{ o_m^2 }
+ M_av c_2

- ∑_m = n - M_av + 1^n
log_ θ_n+1  {
x_m_2^2 }

≥       c_1 λ_min (Σ_xx)
∑_m = n - M_av + 1^n
θ_* - θ_n+1

+ c_1 M_av σ_o^2 +
M_av c_2

- ∑_m = n - M_av +
1^n log{ x_m_2^2 }

c_1 M_av λ_min
(Σ_xx)  θ_* -
θ_n+1 _2^2
+ c_1 M_av σ_o^2

+ M_av c_2 -
M_av log( Σ_xx )  ,

which yield in turn

≥  M_av( 1 - c_3
) + M_av Δ_7^2

M_av + ∑_m = n - M_av + 1^n
{ ( y_m -
θ_n+1^⊺ x_m )^2

≥       M_av { g( s_n,
μ(s_n) ) }

≥       c_1 M_av λ_min
(Σ_xx) {  θ_* -
θ_n+1 _2^2 }
+ c_1 M_av σ_o^2

+ M_av c_2 -
M_av log( Σ_xx )  ,

or equivalently,

≥  1 - c_3 +

{ g( s_n,
μ(s_n) ) }

≥       c_1 λ_min
(Σ_xx) {  θ_* -
θ_n+1 _2^2 }
+ c_1 σ_o^2

+ c_2 - log( Σ_xx )

Recall now that
$Q_{\mu}^{\diamond} = T_{\mu}^{\diamond} (
Q_{\mu}^{\diamond} )$. Hence,
by (<ref>), $\forall n\geq n_0$,

\begin{align*}
& Q_{\mu}^{\diamond} ( \vect{s}_n, \mu( \vect{s}_n ) ) \\
& = T_{\mu}^{\diamond} ( Q_{\mu}^{\diamond} ) (
\vect{s}_n, \mu( \vect{s}_n ) ) \\
& = g( \vect{s}_n, \mu( \vect{s}_n ) ) + \alpha \expect_{
\vect{s}_{n+1} \given \vect{s}_n} \{
Q_{\mu}^{\diamond} ( \vect{s}_{n+1}, \mu( \vect{s}_{n+1}
) ) \} \,.
\end{align*}

It can be directly verified by this last recursion and
induction that for any $K\in \IntegerPP$, a.s.,

=   Q_μ^♢ (
s_n_0, μ( s_n_0 ) )

g( s_n_0, μ( s_n_0 ) )
+ ∑_ ν= n_0 + 1 ^n_0 + K - 1
α^ν- n_0 _
s_ν s_n_0   {
g ( s_ν, μ( s_ν ) ) }

+ α^K _
s_n_0 + K s_n_0  { Q_μ^♢ (
s_n_0+K, μ(
s_n_0+K ) ) }  ,

and hence,

{ Q_μ^♢ (
s_n_0, μ( s_n_0 ) ) }
∑_ ν= n_0  ^n_0 + K - 1
α^ν- n_0 { g ( s_ν, μ(
s_ν ) ) }

+ α^K { Q_μ^♢ (
s_n_0+K, μ( s_n_0+K ) ) }

By (<ref>), $\forall K\in \IntegerPP$,

≤  ∑_ ν= n_0
^n_0 + K - 1 α^ν- n_0 { g (
s_ν, μ( s_ν ) ) }

( 1 - c_3 + Δ_7^2 ) ∑_ ν= n_0
^ n_0 + K - 1  α^ν- n_0

≤       ( 1 - c_3 + Δ_7^2 ) ∑_ ν= n_0
^ +∞ α^ν- n_0 =
1 /1 - α ( 1 - c_3 + Δ_7^2 )  ,

so that
$\sum_{ \nu = n_0 }^{+\infty} \alpha^{\nu - n_0} \expect \{
g ( \vect{s}_{\nu}, \mu( \vect{s}_{\nu} ) ) \} < +
\infty$. Thus, by applying $\limsup_{ K \to \infty}$
to (<ref>) and by recalling

\begin{align*}
\expect\{ Q_{\mu}^{\diamond} (
\vect{s}_{n_0}, \mu( \vect{s}_{n_0} ) ) \} =
\sum\nolimits_{ n = n_0 }^{+\infty} \alpha^{n - n_0}
\expect \{ g ( \vect{s}_{n}, \mu( \vect{s}_{n} ) ) \}
\,.
\end{align*}

Hence, by (<ref>),

{ Q_μ^♢( s_n_0,
μ(s_n_0) ) }  =
∑_n = n_0^+∞ α^n - n_0
{ g( s_n, μ(s_n) ) }

c_1 λ_min
(Σ_xx)  ∑_n = n_0^+∞ α^n - n_0
{  θ_* -
θ_n+1 _2^2 }

+ ∑_n = n_0^+∞ α^n - n_0
[ c_1 σ_o^2 + c_2 - log( Σ_xx ) ]

c_1 λ_min
(Σ_xx)  1/α^n_0+1 ∑_n = n_0 + 1^+∞ α^n
{  θ_* -
θ_n _2^2 }

+ 1/1 - α [ c_1 σ_o^2 + c_2 - log( Σ_xx ) ]  , which yields
∑_n= n_0 + 1^∞ α^n {
θ_* - θ_n (
μ( s_n-1 ) ) _2^2 }

α^n_0+1/c_1 λ_min
(Σ_xx) {
Q_μ^♢( s_n_0,
μ(s_n_0) ) }

+ 1/c_1 λ_min
(Σ_xx) ·α^n_0+1 / 1 - α [
log( Σ_xx )  - c_1 σ_o^2 - c_2 ]

Observe that the way to update $\vectgr{\theta}_n$,
$\forall n\geq n_0$, was not specified throughout the
previous analysis. As such, set
$\vectgr{\theta}_n \coloneqq \vectgr{\theta}_{n_0}$,
$\forall n\geq n_0$, $\Delta_8 \coloneqq c_1$,
$\Delta_9 \coloneqq c_2$ in the last inequality, and finally
substitute $n_0$ by $n$ to establish