marginparsep has been altered.
topmargin has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

A Dual Formulation for Probabilistic Principal Component Analysis

Henri De Plaen 1  Johan A. K. Suykens 1

††footnotetext:  1KU Leuven, Department of Electrical Engineering (ESAT),
STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics,
Kasteelpark Arenberg 10, 3001 Leuven, Belgium. Correspondence to: Henri De
Plaen<EMAIL_ADDRESS>
ICML 2023 Workshop on Duality for Modern Machine Learning, Honolulu, Hawaii,
USA. Copyright 2023 by the author(s).

###### Abstract

In this paper, we characterize _Probabilistic Principal Component Analysis_ in
Hilbert spaces and demonstrate how the optimal solution admits a
representation in dual space. This allows us to develop a generative framework
for kernel methods. Furthermore, we show how it englobes _Kernel Principal
Component Analysis_ and illustrate its working on a toy and a real dataset.

##  1 Introduction

Classical datasets often consist of many features, making dimensionality
reduction methods particularly appealing. _Principal Component Analysis_ (PCA)
is one of the most straightforward frameworks to that goal and it is hard to
find a domain in machine learning or statistics where it has not proven to be
useful. PCA considers new decorrelated features by computing the
eigendecomposition of the covariance matrix.

Probabilistic models on another side participate to the building of a stronger
foundation for machine learning models. By considering models as probability
distributions, we are able to natively access notions such as variance or
sampling, _i.e._ generation. A probabilistic approach to PCA, known as
_Probabilistic Principal Component Analysis_ (Prob. PCA), has been formulated
by Tipping & Bishop (1999). Its principles can be visualized in the primal
part of Table 1.

Even endowed with a probabilistic interpretation, PCA remains restricted to
linear relations between the different features. _Kernel Principal Component
Analysis_ (KPCA) Mika et al. (1998); Schölkopf et al. (1998) was an attempt to
give a non-linear extension to (non-probabilistic) PCA by decomposing a kernel
matrix instead of the covariance matrix. An earlier attempt to give a
probabilistic formulation of KPCA has been done by Zhang et al. (2004). As
developed further, the latter model does not consist in a kernel equivalent of
the Prob. PCA, but rather in another model based on similar principles.

More recently, _Restricted Kernel Machines_ Suykens (2017) opened a new door
for a probabilistic version of PCA both in primal and dual. They essentially
use the Fenchel-Young inequality on a variational formulation of KPCA Suykens
et al. (2003); Alaíz et al. (2018) to obtain an energy function, closely
resembling to _Restricted Boltzmann Machines_. The framework has been further
extended to generation Schreurs & Suykens (2018); Winant et al. (2020),
incorporating robustness Pandey et al. (2020), multi-view models Pandey et al.
(2021), deep explicit feature maps Pandey et al. (2022b) or times-series
Pandey et al. (2022a).

###  1.1 Contributions

  1. 1.

We characterize the Prob. PCA framework in Hilbert spaces and give a dual
interpretation to the model.

  2. 2.

We develop a new extension of KPCA incorporating a noise assumption on the
explicit feature map.

  3. 3.

We give a probabilistic interpretation of the generation in KPCA.

  4. 4.

We illustrate how the dual model works on a toy and a real dataset and show
its connections to KPCA222Resources: https://hdeplaen.github.io/kppca..

$\mathcal{H}$
---
$\mathcal{E}$
---
$\mathcal{L}$
---
$\mathcal{X}$
---
$\mathcal{H}_{\mathcal{E}}$
---
$\mathcal{H}_{\mathcal{L}}$
---
$\bm{x}$
---
$\bm{\varphi}-\bm{\varphi}_{c}$
---
$\bm{\phi}-\bm{\phi}_{c}$
---
$\bm{\phi}_{\mathrm{MAP}}-\bm{\phi}_{c}$
---
$\bm{h}_{\mathrm{MAP}}$
---
$\left(\bm{k}_{c}\right)_{\mathrm{MAP}}$
---
$\bm{k}_{c}$
---
Dimensionality
---
Reduction
Dimensionality
---
Reduction
$\bm{\Phi}_{c}^{\ast}$
---
$\bm{\Phi}_{c}^{\ast}$
---
$\left(\bm{W}_{\mathrm{ML}}^{\ast}\circ\bm{W}_{\mathrm{ML}}+\sigma^{2}\bm{I}_{\mathcal{L}}\right)^{-1}\circ\bm{W}_{\mathrm{ML}}^{\ast}$
---
$\bm{W}_{\mathrm{ML}}$
---
$\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\circ\bm{A}_{\mathrm{ML}}$
---
$\left(\bm{A}_{\mathrm{ML}}^{\ast}\circ\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\circ\bm{A}_{\mathrm{ML}}+\sigma^{2}\bm{I}_{\mathcal{L}}\right)^{-1}\circ\bm{A}_{\mathrm{ML}}^{\ast}$
---
$\phi_{c}$
---
$\sum_{i=1}^{N}k_{c}(\bm{x}_{i},\cdot)$
---
$\varphi_{c}$
---
$\bm{I}_{\mathcal{H}_{\mathcal{E}}}$
---
Figure 1: Global overview of the Probabilistic Principal Component Analysis in
both primal and dual formulations. The primal spaces, or feature
$\mathcal{H}$, $\mathcal{H}_{\mathcal{E}}$ and $\mathcal{H}_{\mathcal{L}}$ are
in blue. The dual, or kernel and latent spaces $\mathcal{E}$ and $\mathcal{L}$
are in brown. The input space $\mathcal{X}$ is in green. The color or the
applications (arrows) is just for the readability and has nothing to do with
the color of the spaces.

Distribution | Interpretation | Primal (features) | Dual (kernels)
---|---|---|---
latent $|$ observation | latent projection | $\bm{h}|\bm{\phi}\sim\mathcal{N}\bigl{(}\bm{\Sigma}_{\bm{h}|\bm{\phi}}^{-1}\circ\bm{W}_{\mathrm{ML}}^{\ast}(\bm{\phi}-\bm{\phi}_{c}),\sigma^{2}\bm{\Sigma}_{\bm{h}|\bm{\phi}}^{-1}\bigr{)}$ | $\bm{h}|\bm{k}_{c}\sim\mathcal{N}\bigl{(}\bm{\Sigma}_{\bm{h}|\bm{k}_{c}}^{-1}\circ\bm{A}_{\mathrm{ML}}\bm{k}_{c},\bm{\Sigma}^{-1}_{\bm{h}|\bm{k}_{c}}\bigr{)}$
observation $|$ latent | latent-based generation | $\bm{\phi}|\bm{h}\sim\mathcal{N}\bigl{(}\bm{W}_{\mathrm{ML}}\bm{h}-\bm{\phi}_{c},\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}\bigr{)}$ | $\bm{k}_{c}|\bm{h}\sim\mathcal{N}\bigl{(}(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c})\circ\bm{A}_{\mathrm{ML}}\bm{h},\sigma^{2}\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\bigr{)}$
latent | latent prior | $\bm{h}\sim\mathcal{N}(\bm{0},\bm{I}_{\mathcal{L}})$ | $\bm{h}\sim\mathcal{N}(\bm{0},\bm{I}_{\mathcal{L}})$
observation | absolute generation | $\bm{\phi}\sim\mathcal{N}(\bm{\mu},\bm{W}_{\mathrm{ML}}\circ\bm{W}_{\mathrm{ML}}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}})$ | $\bm{k}_{c}\sim\mathcal{N}\bigl{(}\bm{0},\bm{A}_{\mathrm{ML}}^{\ast}\circ\bm{A}_{\mathrm{ML}}+\sigma^{2}\left(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\right)^{-1}\bigr{)}$

Table 1: Interpretation of the different distributions of the Prob. PCA
framework after training, in both primal and dual formulations. The covariance
operators are given by
$\bm{\Sigma}_{\bm{h}|\bm{\phi}}=(\bm{W}_{\mathrm{ML}}^{\ast}\circ\bm{W}_{\mathrm{ML}}+\sigma^{2}\bm{I}_{\mathcal{\mathcal{L}}})^{-1}$
and
$\bm{\Sigma}_{\bm{h}|\bm{k}_{c}}=\bigl{(}\bm{A}_{\mathrm{ML}}^{\ast}\circ(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c})\circ\bm{A}_{\mathrm{ML}}+\sigma^{2}\bm{I}_{\mathcal{L}}\bigr{)}^{-1}$,
with maximum likelihood estimators for the primal and dual interconnection
operators $\bm{W}_{\mathrm{ML}}$ and $\bm{A}_{\mathrm{ML}}$.

##  2 Primal and Dual Spaces

The key idea behind the duality in PCA is that outer and inner products share
the same eigenvalues. The consequence is that instead of decomposing the
covariance matrix of any given feature map, we can decompose the associated
Gram matrix, _i.e._ the kernel matrix. The former is considered as the
_primal_ formulation and the latter as the _dual_ formulation and they are
both equivalent. Extending Prob. PCA to a dual formulation is however not
straightforward: if all feature maps have an associated kernel, the converse
is trickier. Some kernels correspond to feature maps in infinite dimensional
spaces, where probability distributions cannot be properly defined. We
therefore need to choose well defined finite subspaces to work in and consider
linear operators instead of matrices. All formal definitions, propositions and
proofs are provided in Appendix A.

###  2.1 Primal Spaces

Feature Space $\mathcal{H}$. Given an input space $\mathcal{X}$, we first
consider any feature map $\varphi:\mathcal{X}\rightarrow\mathcal{H}$.
Following Alaíz et al. (2018), we will consider a separable, possibly infinite
dimensional, Hilbert space
$\left(\mathcal{H},\langle\cdot,\cdot\rangle_{\mathcal{H}}\right)$. By
$\bm{\varphi}$, we denote an element of $\mathcal{H}$ and its adjoint by
$\bm{\varphi}^{\ast}=\langle\bm{\varphi},\cdot\rangle\in\mathcal{H}^{\ast}$,
with $\mathcal{H}^{\ast}\sim\mathcal{H}$ its Fréchet-Riesz dual space.
Essentially, it corresponds to the transpose $\bm{\varphi}^{\top}$ in real,
finite dimensional spaces as
$\bm{\varphi}_{1}^{\top}\bm{\varphi}_{2}=\langle\bm{\varphi}_{1},\bm{\varphi}_{2}\rangle_{\mathcal{H}}$,
but generalizes it for the possibly infinite dimensional spaces that will be
necessary for the introduction of kernels. Furthermore, we assume our space to
be defined over the reals such that
$\langle\cdot,\cdot\rangle_{\mathcal{H}}:\mathcal{H}\times\mathcal{H}\rightarrow\mathbb{R}$
and its inner product is symmetric
$\langle\bm{\varphi}_{1},\bm{\varphi}_{2}\rangle_{\mathcal{H}}=\langle\bm{\varphi}_{2},\bm{\varphi}_{1}\rangle_{\mathcal{H}}$.
If $\mathcal{H}$ is of finite dimension $d$, we can therefore identify its
canonical basis $\bm{u}_{1},\ldots,\bm{u}_{d}$ with the canonical basis of
$\mathbb{R}^{d}$.

Finite Feature Space $\mathcal{H}_{\mathcal{E}}$. Considering a set of $N$
observations $\left\\{\bm{x}_{i}\in\mathcal{X}\right\\}_{i=1}^{N}$, the idea
is to work directly in $\mathcal{H}$ by considering instead the feature map of
the datapoints $\bm{\varphi}_{i}=\varphi\left(\bm{x}_{i}\right)$. We can
however not define a normal distribution onto the full $\mathcal{H}$ yet as it
is possibly infinite dimensional. We therefore have to consider a finite
subspace $\mathcal{H}_{\mathcal{E}}\subset\mathcal{H}$. A natural choice would
be
$\mathcal{H}_{\mathcal{E}}=\mathrm{span}\left\\{\bm{\varphi}_{1},\ldots,\bm{\varphi}_{N}\right\\}$.
We now first have to find an orthonormal basis for
$\mathcal{H}_{\mathcal{E}}$.

###  2.2 Dual Spaces

Kernels. For each feature map, there is an induced positive semi-definite
kernel
$k:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}:k\left(\bm{x},\bm{y}\right)=\langle\varphi(\bm{x}),\varphi(\bm{y})\rangle_{\mathcal{H}}=\varphi(\bm{x})^{\ast}\varphi(\bm{y})$.
Inversely, to each positive semi-definite kernel corresponds a, possibly
infinite dimensional, feature map, even if not explicitly defined. This
follows from the theory of _Reproducing Kernel Hilbert Spaces_. We refer to
Schölkopf & Smola (2001) for further info.

Kernel Space $\mathcal{E}$. We now consider a finite dimensional Hilbert space
$\left(\mathcal{E},\langle\cdot,\cdot\rangle_{\mathcal{E}}\right)$ of
dimension $N$, the number of observations. It is defined similarly as above,
with orthonormal basis $\bm{e}_{1},\ldots,\bm{e}_{N}$. The basis also defines
the identity over $\mathcal{E}$ as
$\bm{I}_{\mathcal{E}}=\sum_{i=1}^{N}\bm{e}_{i}\bm{e}_{i}^{\ast}$. The goal for
$\mathcal{E}$ is to represent the space of the kernel representations. We
therefore define the linear operator
$\bm{\Phi}:\mathcal{E}\rightarrow\mathcal{H}:\sum_{i=1}\bm{\varphi}_{i}\bm{e}_{i}^{\ast}$
and its adjoint
$\bm{\Phi}^{\ast}:\mathcal{H}\rightarrow\mathcal{E}:\sum_{i=1}^{N}\bm{e}_{i}\bm{\varphi}_{i}^{\ast}$.
Essentially, $\bm{\Phi}^{\ast}$ returns the kernel value with each datapoint:
$\bm{\Phi}^{\ast}\varphi(\bm{x})=\sum_{i=1}^{N}\bm{e}_{i}\left(\bm{\varphi}_{i}^{\ast}\varphi(\bm{x})\right)=\sum_{i=1}^{N}\bm{e}_{i}k\left(\bm{x}_{i},\bm{x}\right)$
for any $\bm{x}\in\mathcal{X}$. Similarly, $\bm{\Phi}$ projects this value
back as a linear combination of the different $\bm{\varphi}_{i}$’s, thus
mapping back to $\mathcal{H}_{\mathcal{E}}\subset\mathcal{H}$. For this
reason, the covariance
$\bm{\Phi}\circ\bm{\Phi}^{\ast}=\sum_{i=1}^{N}\bm{\varphi}_{i}\bm{\varphi}_{i}^{\ast}$
acts as a projector from $\mathcal{H}\rightarrow\mathcal{H}_{\mathcal{E}}$.
Its eigenvectors therefore form an orthonormal basis of the finite feature
space $\mathcal{H}_{\mathcal{E}}$, which acts as the primal equivalent of the
kernel space $\mathcal{E}$.

Centered Kernels. In most applications however, we prefer to work with the
centered feature map, which we define as
$\varphi_{c}(\cdot)=\varphi(\cdot)-\bm{\varphi}_{c}$ with
$\bm{\varphi}_{c}=\frac{1}{N}\sum_{i=1}^{N}\bm{\varphi}_{i}$. We denote the
associated kernel associated centered kernel
$k_{c}:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}:k_{c}(\bm{x}_{1},\bm{x}_{2})=\varphi_{c}(\bm{x}_{1})^{\ast}\varphi_{c}(\bm{x}_{2})$.
This leads to the definition of a new centered operator
$\bm{\Phi}_{c}=\sum_{i=1}(\bm{\varphi}_{i}-\bm{\varphi}_{c})\bm{e}_{i}^{\ast}=\bm{\Phi}\left(\bm{I}_{\mathcal{E}}-\frac{1}{N}\bm{1}_{\mathcal{E}\times\mathcal{E}}\right)$,
with
$\bm{1}_{\mathcal{E}\times\mathcal{E}}=\sum_{i,j=1}^{N}\bm{e}_{i}\bm{e}_{j}^{\ast}$.
As always, we also consider its adjoint $\bm{\Phi}_{c}^{\ast}$. Considering
the dual operator, we have
$\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}=\sum_{i=1}^{N}(\bm{\varphi}_{i}-\bm{\varphi}_{c})^{\ast}(\bm{\varphi}_{i}-\bm{\varphi}_{c})\bm{e}_{i}\bm{e}_{j}^{\ast}=\sum_{i=1}^{N}k_{c}(\bm{x}_{i},\bm{x}_{j})\bm{e}_{i}\bm{e}_{j}^{\ast}$.
We notice now that
$\mathcal{H}_{\mathcal{E}}=\mathrm{span}\\{\bm{\varphi}_{1},\ldots,\bm{\varphi}_{N}\\}=\mathrm{span}\\{\bm{\varphi}_{1}-\bm{\varphi}_{c},\ldots,\bm{\varphi}_{N}-\bm{\varphi}_{c}\\}$
because $\bm{\varphi}_{c}$ is a linear combination of the elements of the
basis. Therefore, the primal operator
$\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}=\sum_{i=1}^{N}(\bm{\varphi}_{i}-\bm{\varphi}_{c})(\bm{\varphi}_{i}-\bm{\varphi}_{c})^{\ast}$
also acts as a projector from
$\mathcal{H}\rightarrow\mathcal{H}_{\mathcal{E}}$ and we can choose its
eigenvectors instead as an orthonormal basis of $\mathcal{H}_{\mathcal{E}}$.

Covariance and Kernels. We now consider the key idea behind the duality in
PCA: the operators $\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$ and
$\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}$ are self-adjoint, positive semi-
definite and share the same non-zero eigenvalues. We have
$\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}=\sum_{i=1}^{N}\lambda_{i}\bm{v}_{i}\bm{v}_{i}^{\ast}$
and
$\mathcal{H}_{\mathcal{E}}=\mathrm{span}\\{\bm{v}_{1},\ldots,\bm{v}_{N}\\}$.
Similarly, we have
$\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}=\sum_{i=1}^{N}\lambda_{i}\bm{\epsilon}_{i}\bm{\epsilon}_{i}^{\ast}$
and
$\mathcal{E}=\mathrm{span}\\{\bm{\epsilon}_{1},\ldots,\bm{\epsilon}_{N}\\}$.
The identity over the (primal) finite feature space
$\mathcal{H}_{\mathcal{E}}$ can now be defined as
$\bm{I}_{\mathcal{H}_{\mathcal{E}}}=\sum_{i=1}^{N}\bm{v}_{i}\bm{v}_{i}^{\ast}$
and the identity over the (dual) kernel space $\mathcal{E}$ as
$\bm{I}_{\mathcal{E}}=\sum_{i=1}^{N}\bm{\epsilon}_{i}\bm{\epsilon}_{i}^{\ast}$.
This is synthetized in the two first columns of Table 2. The identity over
$\mathcal{H}$ reads
$\bm{I}_{\mathcal{H}}=\bm{I}_{\mathcal{H}_{\mathcal{E}}}+\mathbb{P}_{\mathcal{H}_{\mathcal{E}}^{\perp}}$,
with $\mathbb{P}_{\mathcal{H}_{\mathcal{E}}^{\perp}}$ a projector over the
null space of $\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$. It most be noted that
it may happen that these basis may contain too much basis vectors if the two
operators $\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}$ and
$\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$ are not of full rank. In particular,
this is the case when $\mathrm{dim}(\mathcal{H})=d$ is finite and $d<N$. In
this particular case, we would also have
$\mathrm{dim}(\mathcal{H}_{\mathcal{E}})=\mathrm{dim}(\mathcal{E})=d$. Without
loss of generality, we will assume that this is not the case. Similarly, we
will neglect the case $N>d$ as we could just neglect the null space of
$\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}$.

Notations. We can now define our probabilistic model over
$\mathcal{H}_{\mathcal{E}}$. We will therefore use the notation $\phi$ instead
of $\varphi$ to consider the feature map in our finite dimensional subspace
$\mathcal{H}_{\mathcal{E}}$. More formally, we have
$\phi:\mathcal{X}\rightarrow\mathcal{H}_{\mathcal{E}}:\bm{I}_{\mathcal{H}_{\mathcal{E}}}\circ\varphi$
and following from that
$\phi_{c}:\mathcal{X}\rightarrow\mathcal{H}_{\mathcal{E}}:\bm{I}_{\mathcal{H}_{\mathcal{E}}}\circ\varphi_{c}$.
In particular, we have the observations
$\bm{\phi}_{i}=\phi(\bm{x}_{i})=\bm{\varphi}_{i}$ and
$\bm{\phi}_{c}=\bm{\varphi}_{c}$, as those are linear combinations of the
basis. For the sake of readability, we will write $\bm{\phi}=\phi(\bm{x})$,
the image of a random variable $\bm{x}\in\mathcal{X}$ and refer to it as a
_feature_ observation or representation. Given any Hilbert space, $\bm{a}$ an
element of it and a linear operator $\bm{\Sigma}$ from and to that space, we
consider the _multivariate normal distribution_
$\bm{a}\sim\mathcal{N}\bigl{(}\bm{b},\bm{\Sigma}\bigr{)}$ as the distribution
with density
$\frac{1}{Z}\exp\bigl{(}-\frac{1}{2}(\bm{a}-\bm{b})^{\ast}\bm{\Sigma}^{-1}(\bm{a}-\bm{b})\bigr{)}$.
It is well defined if $Z$ is non-zero and finite.

Dimension | $d$ | $N$ | $q$
---|---|---|---
Primal | Space | $\mathcal{H}(=\mathbb{R}^{d})$ | $\mathcal{H}_{\mathcal{E}}\subset\mathcal{H}$ | $\mathcal{H}_{\mathcal{L}}\subset\mathcal{H}_{\mathcal{E}}$
Canon. Basis | $\\{\bm{u}_{i}\\}_{i=1}^{d}$ | N.A. | N.A.
Other Basis | N.A. | $\\{\bm{v}_{i}\\}_{i=1}^{N}$ | $\\{\bm{\varrho}_{p}\\}_{p=1}^{q}$
Dual | Space | N.A. | $\mathcal{E}=\mathbb{R}^{N}$ | $\mathcal{L}\subset\mathcal{E}$
Canon. Basis | N.A. | $\\{\bm{e}_{i}\\}_{i=1}^{N}$ | N.A.
Other Basis | N.A. | $\\{\bm{\epsilon}_{i}\\}_{i=1}^{N}$ | $\\{\bm{r}_{p}\\}_{p=1}^{q}$

Table 2: Different spaces with their dimension and the canonical orthonormal
basis if it applies, as well as another (non-canonical) basis when used
throughout this paper. The equality $\mathcal{H}=\mathbb{R}^{d}$ only makes
sense if $\mathrm{dim}(\mathcal{H})=d$ is finite.

##  3 Primal Model

We will now essentially follow the work of Tipping & Bishop (1999) and
redefine the model distributions. This section corresponds to the primal
formulation and we only consider the feature representations. It does not yet
introduce the kernel representations, which will appear in the dual
formulation (Section 4).

###  3.1 Model and Latent Space

Factor Analysis. The starting point is to consider a _factor analysis_
relationship Bartholomew et al. (2011); Basilevsky (2009) between the feature
observations $\bm{\phi}$ and the latent variables $\bm{h}$. In particular, we
consider

$\bm{\phi}=\bm{W}\bm{h}+\bm{\mu}+\bm{\zeta}.$ (1)

The observations $\bm{\phi}$ live in the primal space
$\mathcal{H}_{\mathcal{E}}$ of dimension $N$. We consider an isotropic normal
noise
$\bm{\zeta}\sim\mathcal{N}\bigl{(}\bm{0},\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}\bigr{)}$
of variance $\sigma^{2}\in\mathbb{R}_{>0}$ and a mean
$\bm{\mu}\in\mathcal{H}_{\mathcal{E}}$.

Latent Space $\mathcal{L}$. The latent variables $\bm{h}$ on the other hand
live in a latent dual space $\mathcal{L}\subset\mathcal{E}$ of dimension
$q\leq N$. They are related by a primal _interconnection linear operator_
$\bm{W}$. As it was the case before with $\bm{\Phi}$, the interconnection
operator does not project to the full space $\mathcal{H}_{\mathcal{E}}$
because of its reduced dimensionality. It therefore projects to yet another
feature space $\mathcal{H}_{\mathcal{L}}\subset\mathcal{H}_{\mathcal{E}}$,
which acts as the primal equivalent of the latent space $\mathcal{L}$. The
equality of these two spaces only holds if $q=N$. We will therefore consider
the mappings $\bm{W}^{\ast}:\mathcal{H}_{\mathcal{E}}\rightarrow\mathcal{L}$
and $\bm{W}:\mathcal{L}\rightarrow\mathcal{H}_{\mathcal{L}}$. The identity
over $\mathcal{L}$ can be written as
$\bm{I}_{\mathcal{L}}=\sum_{p=1}^{q}\bm{r}_{p}\bm{r}_{p}^{\ast}$, over
$\mathcal{H}_{\mathcal{L}}$ as
$\bm{I}_{\mathcal{H}_{\mathcal{L}}}=\sum_{p=1}^{q}\bm{\varrho}_{p}\bm{\varrho}_{p}^{\ast}$
and finally the identity over $\mathcal{H}_{\mathcal{E}}$ rewritten as
$\bm{I}_{\mathcal{H}_{\mathcal{E}}}=\bm{I}_{\mathcal{H}_{\mathcal{L}}}+\mathbb{P}_{\mathcal{H}_{\mathcal{L}}^{\perp}}$,
with $\mathbb{P}_{\mathcal{H}_{\mathcal{L}}^{\perp}}$ as a projector over the
null space of $\bm{W}^{\ast}\circ\bm{W}$. This is summarized in the last
column of Table 2.

###  3.2 Feature Distributions

Latent-Based Generation. The relation between the feature observations and the
latent variables being set up (Eq. (1)), we can derive the conditional
probability of the feature observations given a latent variable:

$\bm{\phi}|\bm{h}\sim\mathcal{N}\left(\bm{W}\bm{h}-\bm{\mu},\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}\right).$
(2)

As discussed earlier, we see that the latent variables do not participate to
the full scope of the observations in $\mathcal{H}_{\mathcal{E}}$, but only to
their component in $\mathcal{H}_{\mathcal{L}}$. The rest is only constituted
from the isotropic normal noisy mean. This distribution can be interpreted as
a generative one: given a latent variable, we can sample a variable in feature
space.

Absolute Generation. Considering the latent prior
$\bm{h}\sim\mathcal{N}\bigl{(}\bm{0},\bm{I}_{\mathcal{L}}\bigr{)}$, we can
derive the marginal distribution of the observations in feature space:

$\bm{\phi}\sim\mathcal{N}\left(\bm{\mu},\bm{W}\circ\bm{W}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}\right).$
(3)

It can be considered as the data distribution of the model. Sampling from it
also means generating feature representations in a more absolute way, _i.e._ ,
without considering any latent variable, or more precisely considering a
random latent variable according to its prior. As a consequence of Eq. (2) and
the isotropic aspect of the latent prior, we see that the observations are
only non-isotropically distributed in $\mathcal{H}_{\mathcal{L}}$. Again, the
rest is only the isotropically normally noisy mean. In other words, this means
that the model parameter $\bm{W}$ only influences $\bm{\phi}$ for its
components in $\mathcal{H}_{\mathcal{L}}$.

###  3.3 Training the Model

Maximum Likelihood. As we now have the marginal distribution of the model (Eq.
(3)), the goal is to find the optimal hyperparameters $\bm{W}$ and $\bm{\mu}$
to match the set of observations $\\{\bm{\phi}_{i}\\}_{i=1}^{N}$. One way to
determine them is by maximizing the likelihood of our observations. The
_Maximum Likelihood_ (ML) estimator for the hyperparameters is given by:

$\displaystyle\bm{\mu}_{\mathrm{ML}}$ $\displaystyle=$
$\displaystyle\bm{\phi}_{c},$ (4) $\displaystyle\bm{W}_{\mathrm{ML}}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{q}\sqrt{\lambda_{p}/N-\sigma^{2}}\bm{v}_{p}\bm{r}_{p}^{\ast},$
(5)

with $\left\\{\left(\lambda_{p},\bm{v}_{p}\right)\right\\}_{p=1}^{q}$ the $q$
dominant eigenpairs of $\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$
($\lambda_{1}\geq\cdots\geq\lambda_{q}\geq\cdots\lambda_{N}$), and
$\left\\{\bm{r}_{p}\right\\}_{p=1}^{q}$ and arbitrary orthonormal basis of the
latent space $\mathcal{L}$. The choice for the latter basis is arbitrary and
makes the model rotational invariant in latent space. An additional condition
is that $\sigma^{2}\leq\lambda_{q}/N$. It is not surprising to see that the
optimal mean $\bm{\mu}_{\mathrm{ML}}$ corresponds to the mean of the
observations $\bm{\phi}_{c}$. We observe that $\bm{W}_{\mathrm{ML}}$
corresponds to the eigendecomposition of the centered covariance, at the
exception that the noise assumption is substracted from its spectrum. By
looking back at Eq. (1), it makes sense to avoid the noise in
$\bm{W}_{\mathrm{ML}}$ as it is still going to be added by the term
$\bm{\zeta}$.

Noise Variance. Maximizing the likelihood as a function of $\sigma^{2}$ leads
to

$\sigma^{2}_{\mathrm{ML}}=\frac{1}{N(N-q)}\sum_{p=q+1}^{N}\lambda_{p}.$ (6)

The eigenvalue $\lambda_{p}$ corresponds to the variance for each component
$\bm{v}_{p}$ of the covariance $\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$. The
total variance of the data, noise included, is equal to
$\frac{1}{N}\sum_{p=1}^{N}\lambda_{p}$ and the variance learned by the model
through the primal interconnection operator to
$\frac{1}{N}\sum_{p=1}^{q}\lambda_{p}$. Hence, the maximum likelihood
estimator for the noise variance $\sigma^{2}_{\mathrm{ML}}$ can be interpreted
as the mean of the variance that is discarded by the model. It also verifies
the earlier condition that $\sigma^{2}\leq\lambda_{q}/N$, as the eigenvalues
are taken in descending order. It can be interpreted as the normalized mean
variance of the left over eigendirections, _i.e._ the orthogonal space of the
latent space: $\mathcal{L}^{\perp}=\mathcal{E}\backslash\mathcal{L}$. By
consequence, we may decide to choose the latent dimension
$q=\mathrm{dim}(\mathcal{L})$ and deduct $\sigma^{2}_{\mathrm{ML}}$. In the
opposite, we may also decide to set an arbitrary $\sigma^{2}$ and deduct the
latent dimension $q$ instead. We therefore can consider either $\sigma^{2}$ or
$q$ as an additional hyperparameter. We must however keep in mind that this is
strongly going to be influenced by the distribution of the eigenvalues and
that the latent dimension $q$ for the same $\sigma^{2}$ may heavily vary from
application to application.

Uncentered Features. We may also consider not to consider the mean as an
optimizable hyperparameter and set it arbitrarily to $\bm{\mu}=\bm{0}$. In
this case, Eq. (5) would be the same at the difference that the
$\bm{W}_{\mathrm{ML}}$ would be constructed from the dominant eigenpairs of
the uncentered covariance $\bm{\Phi}\circ\bm{\Phi}^{\ast}$ instead of its
centered counterpart $\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$.

###  3.4 Dimensionality Reduction in Feature Space

Latent Projection. Up to now, we only considered the distribution of the
feature variables $\bm{\phi}$. We can also calculate the posterior
distribution of the latent variable $\bm{h}$ given the primal feature variable
$\bm{\bm{\phi}}$:

$\bm{h}|\bm{\phi}\sim\mathcal{N}\left(\bm{\Sigma}_{\bm{h}|\bm{\phi}}^{-1}\circ\bm{W}^{\ast}(\bm{\phi}-\bm{\mu}),\sigma^{2}\bm{\Sigma}_{\bm{h}|\bm{\phi}}^{-1}\right),$
(7)

with
$\bm{\Sigma}_{\bm{h}|\bm{\phi}}=\left(\bm{W}^{\ast}\circ\bm{W}+\sigma^{2}\bm{I}_{\mathcal{\mathcal{L}}}\right)^{-1}$.
The mean of the distribution can be considered as a pseudo-inverse of the
observation $\bm{\phi}$, but regularized by $\sigma^{2}$. This regularization
ensures to avoid the noise. If the prior of the latent variables was
isotropic, this is not the case anymore for the posterior. If we consider the
maximum likelihood estimator for the primal interconnection operator
$\bm{W}_{\mathrm{ML}}$, the variance becomes
$\sigma^{2}\bm{\Sigma}_{\bm{h}|\bm{\phi}}^{-1}=N\sigma^{2}\sum_{p=1}^{q}\lambda_{p}^{-1}\bm{r}_{p}\bm{r}_{p}^{\ast}$.
It can be interpreted as the uncertainty for each component of the latent
variable $\bm{h}$ (w.r.t. the eigendirection $\bm{r}_{p}$), due to the noise
assumption. By consequence, the greater the explained variance $\lambda_{p}$
for the eigendirection $\bm{v}_{p}$ of the covariance
$\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$, the smaller the corresponding
uncertainty on the component $\bm{r}_{p}$ of the latent vairable $\bm{h}$. For
each observation in feature space $\bm{\phi}$, this returns a distribution for
the latent variable $\bm{\phi}$ and can therefore be considered as a sort of
probabilistic projection in latent space $\mathcal{L}$.

Maximum A Posteriori. Up to now, we were only considering distributions. The
only way to go from a feature representation to a latent variable or the
opposite was probabilistic. In order to have a deterministic approach, we need
proper mappings. One way is to consider the _Maximum A Posteriori_ (MAP) of
$\bm{h}$ given $\bm{\phi}$. It maps the feature observation
$\bm{\phi}\in\mathcal{H}_{\mathcal{E}}$ to latent variable
$\bm{h}_{\mathrm{MAP}}\in\mathcal{L}$, hence reducing the dimensionality of
any input to that of the latent space. To allow it to work for any input
$\bm{\varphi}\in\mathcal{H}$, we may again consider the projection
$\bm{\phi}=\bm{I}_{\mathcal{H}_{\mathcal{E}}}\bm{\varphi}$. As
$\bm{W}_{\mathrm{ML}}^{\ast}\circ\bm{I}_{\mathcal{H}_{\mathcal{E}}}=\bm{W}_{\mathrm{ML}}^{\ast}$:

$\begin{split}\bm{h}_{\mathrm{MAP}}=&\left(\bm{W}_{\mathrm{ML}}^{\ast}\circ\bm{W}_{\mathrm{ML}}+\sigma^{2}\bm{I}_{\mathcal{L}}\right)^{-1}\\\
&\circ\bm{W}_{\mathrm{ML}}^{\ast}\left(\bm{\varphi}-\bm{\varphi}_{c}\right).\end{split}$
(8)

To map back to the feature space $\mathcal{H}_{\mathcal{L}}$, we may consider
the _maximum a posteriori_ of $\bm{\phi}$ given $\bm{h}$ (Eq. (3)). This gives

$\bm{\phi}_{\mathrm{MAP}}=\bm{W}_{\mathrm{MAP}}\bm{h}+\bm{\phi}_{c}.$ (9)

The final projection reads

$\begin{split}\bm{\phi}_{\mathrm{MAP}}=&\bm{W}_{\mathrm{ML}}\circ\left(\bm{W}_{\mathrm{ML}}^{\ast}\circ\bm{W}_{\mathrm{ML}}+\sigma^{2}\bm{I}_{\mathcal{L}}\right)^{-1}\\\
&\circ\bm{W}_{\mathrm{ML}}^{\ast}\left(\bm{\varphi}-\bm{\varphi}_{c}\right)+\bm{\phi}_{c}.\end{split}$
(10)

No Noise. We may also decide not to consider $\sigma^{2}$ as a parameter to
optimize and set it to an arbitrary value. The latent dimensions $q$ could
also be set an arbitrary value, without it to be related to the latent
dimension $q$ according to Eq. (6). We notice that in the limit of
$\sigma^{2}\rightarrow 0$, we recover the classical Principal Component
Analysis reconstruction scheme. Indeed the conditional probability
distributions become exact relations. We also notice that the condition
$\sigma^{2}\leq\lambda_{q}/N$ (Prop. 3) is then always satisfied. Furthermore,
when $q=\mathrm{dim}(\mathcal{H}_{\mathcal{E}})$, the reconstruction is
perfect in $\mathcal{H}_{\mathcal{E}}$ and in particular for our original
observations $\\{\bm{\varphi}_{i}\\}_{i=1}^{N}$ and $\bm{\varphi}_{c}$ (as we
have $\bm{\phi}_{i}=\bm{\varphi}_{i}$). Indeed, we would have

$\bm{h}_{\mathrm{MAP}}=\bm{W}_{\mathrm{ML}}^{+}\left(\bm{\varphi}-\bm{\varphi}_{c}\right),$
(11)

with $\bm{W}_{\mathrm{ML}}^{+}$ the Moore-Penrose pseudo-inverse of
$\bm{W}_{\mathrm{ML}}$. . We note here the symmetry with Eq. (9). If the
maximum likelihood estimator for $\sigma^{2}$ is to be respected (Eq. (6)),
this would mean that all components are kept ($\mathcal{L}=\mathcal{E}$) and
the model reconstructs the full feature variance. In this case, the primal
interconnection operator would become
$\bm{W}_{\mathrm{ML}}=\sum_{p=1}^{N}\sqrt{\lambda_{p}/N}\bm{v}_{p}\bm{r}_{p}^{\ast}$
and be invertible. Its Moore-Penrose pseudo-inverse would become an exact
inverse. Eqs. (9) and (11) would become exact opposites and there would be no
loss due to the dimensionality reduction as there would be no noise to
discard. By consequence, the reduction would become an identity over
$\mathcal{H}_{\mathcal{E}}$:
$\bm{\phi}_{\mathrm{MAP}}-\bm{\phi}_{c}=\bm{I}_{\mathcal{H}_{\mathcal{L}}}\left(\bm{\varphi}-\bm{\varphi}_{c}\right)$.

##  4 Dual Model

Kernels without Dual. In Zhang et al. (2004), the authors made the kernel
matrix appear by considering the new observations
$\bigl{\\{}\sum_{i=1}^{d}\bm{u}_{i}\bm{u}_{j}^{\ast}\phi(\bm{x}_{i})\bigr{\\}}_{j=1}^{N}$.
In other words, each new datapoint consists in one particular feature of the
feature map, for each original datapoint. If the original datapoints were
organized as a matrix in $\mathbb{R}^{N\times d}$, this would correspond to
taking its transpose as new datapoints. The outer product of the covariance
matrix is transformed to the inner product of the kernel matrix. If indeed
this formulation makes the kernel appear, it is not a dual formulation of the
original problem, but another problem. In this section, we show how the spaces
defined hereabove help us build an equivalent dual formulation of the problem.

Dual Formulation. While keeping an equivalence with the primal model, we will
now see that we can directly work in dual spaces $\mathcal{E}$ and
$\mathcal{L}$ without considering the feature spaces at all, _i.e._ resorting
to the primal space $\mathcal{H}$ and its subsets. As we did for the primal
feature variable $\bm{\phi}$, we will consider
$\bm{k}_{c}=\bm{\Phi}_{c}^{\ast}(\bm{\phi}-\bm{\phi}_{c})=\sum_{i=1}^{N}k_{c}(\bm{x},\bm{x}_{i})\bm{e}_{i}$
to represent the image in $\mathcal{E}$, of a random variable
$\bm{x}\in\mathcal{X}$. We will refer to it as a _dual feature variable_.

###  4.1 Representation

Considering the dual spaces, we can always express the interconnection
operator $\bm{W}$ in the (non-orthonormal) basis
$\left\\{\bm{\phi}_{1}-\bm{\phi}_{c},\ldots,\bm{\phi}_{N}-\bm{\phi}_{c}\right\\}$.
As a consequence, we can always write

$\bm{W}=\bm{\Phi}_{c}\circ\bm{A},$ (12)

with $\bm{A}:\mathcal{L}\rightarrow\mathcal{L}$, the dual interconnection
operator. Given the maximum likelihood estimator for the primal
interconnection operator $\bm{W}_{\mathrm{ML}}$, we can directly deduce the
dual one:

$\bm{A}_{\mathrm{ML}}=\sum_{p=1}^{q}\sqrt{1/N-\sigma^{2}\lambda_{p}^{-1}}\bm{\epsilon}_{p}\bm{r}_{p}^{\ast},$
(13)

with $\left\\{\left(\lambda_{p},\bm{\epsilon}_{p}\right)\right\\}_{p=1}^{q}$
the $q$ dominant eigenpairs of $\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}$ and
$\left\\{\bm{r}_{p}\right\\}_{p=1}^{q}$ an arbitrary orthonormal basis of the
latent space $\mathcal{L}$. The rotational invariance of the dual
interconnection operator $\bm{A}_{\mathrm{ML}}$ is inherited from its primal
counterpart $\bm{W}_{\mathrm{ML}}$. Again, if we consider an optimized mean
$\bm{\mu}=\bm{0}$, we would have the relation
$\bm{W}_{\mathrm{ML}}=\bm{\Phi}\circ\bm{A}_{\mathrm{ML}}$ with
$\bm{A}_{\mathrm{ML}}$ then based on the eigenpairs of the non-centered
$\bm{\Phi}^{\ast}\circ\bm{\Phi}$ instead. Using the same structure for
$\bm{A}_{\mathrm{ML}}$, the optimal (primal) interconnection operator
$\bm{W}_{\mathrm{ML}}$ could be expressed in the (non-ortonormal) basis
$\\{\bm{\phi}_{1},\ldots,\bm{\phi}_{N}\\}$.

###  4.2 Kernel Distributions

Projection and Generation. We can also consider the dual counterparts of the
distributions of the primal model (Eqs. (2) and (7)). For the sake of
simplicity and to avoid heavier equations with non-centered kernels, we will
only consider here the equations of the trained model, in particular with
$\bm{\mu}_{\mathrm{ML}}=\bm{\phi}_{c}$ leading to centered kernels:

$\displaystyle\bm{k}_{c}|\bm{h}$ $\displaystyle\sim$
$\displaystyle\mathcal{N}\bigl{(}(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c})\circ\bm{A}_{\mathrm{ML}}\bm{h},\sigma^{2}\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\bigr{)},$
(14) $\displaystyle\bm{h}|\bm{k}_{c}$ $\displaystyle\sim$
$\displaystyle\mathcal{N}\left(\bm{\Sigma}_{\bm{h}|\bm{k}_{c}}^{-1}\circ\bm{A}_{\mathrm{ML}}\bm{k}_{c},\bm{\Sigma}^{-1}_{\bm{h}|\bm{k}_{c}}\right),$
(15)

with
$\bm{\Sigma}_{\bm{h}|\bm{k}_{c}}=\left(\bm{A}_{\mathrm{ML}}^{\ast}\circ\bigl{(}\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\bigr{)}\circ\bm{A}_{\mathrm{ML}}+\sigma^{2}\bm{I}_{\mathcal{L}}\right)^{-1}$.

###  4.3 Dimensionality Reduction in Kernel Space

Maximum A Posteriori. This now allows us to consider the dimensionality
reduction in kernel space in a similar fashion as in Section 3.4. Again we
consider the MAP of the latent variable $\bm{h}$ given the kernel
representation $\bm{k}_{c}$:

$\begin{split}\bm{h}_{\mathrm{MAP}}=&\left(\bm{A}_{\mathrm{ML}}^{\ast}\circ\bigl{(}\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\bigr{)}\circ\bm{A}_{\mathrm{ML}}+\sigma^{2}\bm{I}_{\mathcal{L}}\right)^{-1}\\\
&\circ\bm{A}_{\mathrm{ML}}\bm{k}_{c},\end{split}$ (16)

and similarly with the MAP of the kernel representation $\bm{k}_{c}$ given the
latent variable $\bm{h}$:

$\left(\bm{k}_{c}\right)_{\mathrm{MAP}}=\left(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\right)\circ\bm{A}_{\mathrm{ML}}\bm{h}.$
(17)

As for the primal model, the dimensionality reduction in dual is computed as
$\left(\bm{k}_{c}\right)_{\mathrm{MAP}}=\left(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\right)\circ\bm{A}_{\mathrm{ML}}\bm{h}_{\mathrm{MAP}}$.

No Noise. Again, considering $\sigma^{2}\rightarrow 0$ makes both dual
conditional distributions become exact relations. In a ML context for
$\sigma^{2}$ (Eq. (6)), this would imply that $q=\mathrm{dim}(\mathcal{E})$
and we would recover an identity
$\left(\bm{k}_{c}\right)_{\mathrm{MAP}}=\bm{k}_{c}$, _i.e._ no reduction.
Without considering a ML context for $\sigma^{2}\rightarrow 0$ and choosing an
arbitrary $q\leq\mathrm{dim}(\mathcal{E})$, the reduction become exactly the
reconstruction done in KPCA.

###  4.4 Kernel Sampling

Probabilistic Sampling. The dual counterpart of Eq. (3) after training is
given by

$\bm{k}_{c}\sim\mathcal{N}\left(\bm{0},\bm{A}_{\mathrm{ML}}^{\ast}\circ\bm{A}_{\mathrm{ML}}+\sigma^{2}\left(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\right)^{-1}\right).$
(18)

The covariance
$\bm{A}_{\mathrm{ML}}^{\ast}\circ\bm{A}_{\mathrm{ML}}+\sigma^{2}\left(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\right)^{-1}$
can be decomposed as $\bm{B}\circ\bm{B}^{\ast}$, with
$\bm{B}:\mathcal{E}\rightarrow\mathcal{E}:N^{-1/2}\sum_{p=1}^{q}\lambda_{p}\bm{\epsilon}_{p}\bm{\varepsilon}_{p}^{\ast}+\sum_{p=q+1}^{N}\sigma\lambda_{p}^{1/2}\bm{\epsilon}_{p}\bm{\varepsilon}_{p}^{\ast}$
and $\left\\{\bm{\varepsilon}_{i}\right\\}_{i=1}^{N}$ any arbitrary
orthonormal basis of the latent space $\mathcal{E}$. This decomposition allows
us to sample $\bm{k}_{c}$ on the trained model with
$\bm{k}_{c}=\bm{B}\bm{\xi}$ with
$\bm{\xi}\sim\mathcal{N}(\bm{0},\bm{I}_{\mathcal{E}})$. We see that $\bm{B}$
is rotational invariant, which is not surprising as this is also the case for
the distribution from which $\bm{\xi}$ is sampled. In practice and for
simplicity, we may decide too choose the canonical basis for
$\left\\{\bm{\varepsilon}_{i}\right\\}_{i=1}^{N}$ as any choice would be
identified to the same covariance and to the same sampling of $\bm{k}_{c}$. We
will therefore assume that $\bm{\varepsilon}_{i}=\bm{e}_{i}$ for all
$i=1,\ldots,N$. In that particular case, $\bm{B}$ is self-adjoint and by
consequence corresponds to the matrix square root of
$\bm{A}_{\mathrm{ML}}^{\ast}\circ\bm{A}_{\mathrm{ML}}+\sigma^{2}\left(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\right)^{-1}$.

KPCA Sampling The classical sampling done by KPCA Schreurs & Suykens (2018)
corresponds to the limit of $\sigma^{2}\rightarrow 0$ for an arbitrary latent
dimension $q$. Unless the latent dimension is chosen as
$q=\mathrm{dim}(\mathcal{E})$, the sampling in that case can never cover
$\mathcal{E}$ fully, but rather $\mathcal{L}$, as $\bm{B}$ is not a bijection.
The second term of $\bm{B}$
($\sum_{p=q+1}^{N}\sigma\lambda_{p}^{1/2}\bm{\epsilon}_{p}\bm{\varepsilon}_{p}^{\ast}$)
allows $\bm{B}$ to be a bijection no matter what is the choice of the latent
dimension $q$, as long as $\sigma^{2}>0$. We thus always sample in the full
$\mathcal{E}$. This can be observed at Fig. 2.

$\bm{I}_{\mathcal{L}}$
---
$\mathcal{E}$
---
$\mathcal{E}$
---
$\mathcal{L}$
---
$\bm{u}\sim\mathcal{N}\left(\bm{0},\bm{I}_{\mathcal{E}}\right)$
---
$\left(\bm{k}_{c}\right)_{\text{Prob. PCA}}$
---
$\left(\bm{k}_{c}\right)_{\text{KPCA}}$
---
KPCA Generation
---
$\bm{B}$
---
Figure 2: Schematic overview of the dual sampling in Prob. PCA compared to the
generation in KPCA.

##  5 Experiments

-2
---
0
---
2
---
4
---
-1
---
0
---
1
---
2
---
3
---
4
---
Generated
---
Dataset
---
KPCA
---
Prob. PCA
---
(a) With $q=1$ component, the explained variance is $31.23\%$ and
$\sigma^{2}=1.40\%$.

-2
---
0
---
2
---
4
---
-3
---
-2
---
-1
---
0
---
1
---
2
---
3
---
Generated
---
Dataset
---
KPCA
---
Prob. PCA
---
(b) With $q=3$ components, the explained variance is $54.03\%$ and
$\sigma^{2}=0.98\%$.

-2
---
0
---
2
---
4
---
6
---
-2
---
-1
---
0
---
1
---
2
---
3
---
Generated
---
Dataset
---
KPCA
---
Prob. PCA
---
(c) With $q=10$ components, the explained variance is $91.93\%$ and
$\sigma^{2}=0.20\%$.

Figure 3: Visualisation of the Probabilistic PCA reconstruction (in blue) the
classical KPCA (in red). Samples generated by are also given (in grey). The
dataset contains $N=20$ points (in black).

(a) Sample of original datapoints.

(b) Datapoints of Fig. 4(a) after dimensionality reduction.

(c) Generated datapoints. The sample $\tilde{\bm{u}}$ is uniform on
$\left[-1,1\right]$ for the two first components and zero for the others. The
horizontal axis varies in the first component and the vertical one in the
second component.

Figure 4: The Probabilistic PCA dual formulation on the MNIST dataset
restricted to 0’s and 1’s, with $N=500$ datapoints, with $q=2$ components. The
explained variance is $27.97\%$ and $\sigma^{2}=0.14\%$.

Hilbert Spaces to Matrices. Working in Hilbert spaces is helpful to treat
possibly infinite dimensional feature maps, but not very useful for practical
applications. Matrix representations are possible in primal if $d$ is finite
and in dual if $N$ is finite. It suffices to consider the different canonical
basis. For the latent space $\mathcal{L}$, this enforces a unique
representation for $\bm{W}_{\mathrm{ML}}$ and $\bm{A}_{\mathrm{ML}}$, but we
must keep in mind that they are rotational invariant. All the operators and
elements described before are then represented in matrix or vector format
(Table 3). We will use the tilde to denote these matrices and use software-
like notation by denoting with $(\cdot)_{i_{1}:i_{2},j_{1}:j_{2}}$ the matrix
truncated to its $i_{1}$ to $i_{2}$ rows and $j_{1}$ to $j_{2}$ columns.

Preimage. Given a dual representation, we will also consider the _kernel
smoother_ preimage method, as suggested by Schreurs & Suykens (2018):

$\hat{\bm{x}}=\frac{\textstyle\sum_{i=1}^{N}(\tilde{\bm{k}})_{i}\bm{x}_{i}}{{\textstyle\sum_{i=1}^{N}(\tilde{\bm{k}})_{i}}}.$
(19)

In practice, as we work with centered feature maps and kernels, it may be that
the kernel smoother may be unstable due to its normalization term. We
therefore may consider to add a stabilization term.

| Name | Space | Values
---|---|---|---
Dual | $\tilde{\bm{K}}_{c}$ | $\mathbb{R}^{N\times N}$ | $(\tilde{\bm{k}}_{c})_{i,j}=k_{c}(\bm{x}_{i},\bm{x}_{j})$
$\tilde{\bm{E}}$ | $\mathbb{R}^{N\times N}$ | $\bigl{(}\tilde{\bm{E}}\bigr{)}_{i,j}=\bm{e}_{i}^{\ast}\bm{\epsilon}_{j}$
$\tilde{\bm{R}}$ | $\mathbb{R}^{q\times q}$ | $\tilde{\bm{R}}=\bm{I}_{q}$
$\tilde{\bm{h}}$ | $\mathbb{R}^{q}$ | $\bigl{(}\tilde{\bm{h}}\bigr{)}_{p}=\bm{e}_{p}^{\ast}\bm{h}$
$\tilde{\bm{k}}_{c}$ | $\mathbb{R}^{N}$ | $\bigl{(}\tilde{\bm{k}}_{c}\bigr{)}_{i}=\bm{e}_{i}^{\ast}\bm{k}_{c}$
Both | $\tilde{\bm{\Lambda}}$ | $\mathbb{R}^{N\times N}_{\geq 0}$ | $\tilde{\bm{\Lambda}}=\mathrm{diag}(\lambda_{1},\ldots,\lambda_{N})$
$\tilde{\bm{S}}$ | $\mathbb{R}_{\geq 0}^{q\times q}$ | $\tilde{\bm{S}}=\mathrm{diag}(s_{1},\ldots,s_{q})$
Primal | $\tilde{\bm{C}}_{c}$ | $\mathbb{R}^{d\times d}$ | $\bigl{(}\tilde{\bm{C}}_{c}\bigr{)}_{i,j}=\bigl{(}\bm{u}_{i}^{\ast}\bm{\Phi}_{c}\bigr{)}\circ\bigl{(}\bm{u}_{j}^{\ast}\bm{\Phi}_{c}\bigr{)}^{\ast}$
$\tilde{\bm{\Phi}}_{c}$ | $\mathbb{R}^{d\times N}$ | $\bigl{(}\tilde{\bm{\Phi}}_{c}\bigr{)}_{i,j}=\bm{u}_{i}^{\ast}\bm{\Phi}_{c}\bm{e}_{j}$
$\tilde{\bm{V}}$ | $\mathbb{R}^{d\times N}$ | $\bigl{(}\tilde{\bm{V}}\bigr{)}_{i,j}=\bm{u}_{i}^{\ast}\bm{v}_{j}$
$\tilde{\bm{P}}$ | $\mathbb{R}^{d\times q}$ | $\bigl{(}\tilde{\bm{P}}\bigr{)}_{i,p}=\bm{v}_{i}^{\ast}\bm{\varrho}_{p}$
$\tilde{\bm{\phi}}$ | $\mathbb{R}^{d}$ | $\bigl{(}\bm{\phi})_{i}=\bm{v}_{i}^{\ast}\bm{\phi}$
$\tilde{\bm{\phi}}_{c}$ | $\mathbb{R}^{d}$ | $\bigl{(}\bm{\phi}_{c})_{i}=\bm{v}_{i}^{\ast}\bm{\phi}_{c}$
Table 3: Representation of the various operators and elements in their
respective canonical basis, as matrices and vectors. The primal
representations exist if and only if $\mathrm{dim}(\mathcal{H})=d$ is finite.

Name | Space | Trained
---|---|---
$\tilde{\bm{W}}$ | $\mathbb{R}^{d\times q}$ | $\tilde{\bm{V}}_{1:N,1:q}\bigl{(}\tilde{\bm{\Lambda}}_{1:q,1:q}/N-\sigma^{2}\bm{I}_{q}\bigr{)}^{1/2}$
$\tilde{\bm{A}}$ | $\mathbb{R}^{N\times q}$ | $\tilde{\bm{E}}_{1:N,1:q}\bigl{(}\bm{I}_{q}/N-\sigma^{2}\bigl{(}\tilde{\bm{\Lambda}}_{1:q,1:q}\bigr{)}^{-1}\bigr{)}^{1/2}$
$\tilde{\bm{B}}$ | $\mathbb{R}^{N\times q}$ | $\tilde{\bm{E}}\tilde{\bm{\Lambda}}^{1/2}\left[\begin{array}[]{cc}(N)^{-1/2}\tilde{\bm{\Lambda}}_{1:q,1:q}^{1/2}&\bm{0}\\\ \bm{0}&\sigma\bm{I}_{N-q}\end{array}\right]$

Table 4: Value of the different operators in the canonical basis, after
training.

###  5.1 Model

The direct application of the theoretical discussions of the previous sections
leads to the decompositions
$\tilde{\bm{K}}_{c}=\tilde{\bm{E}}\tilde{\bm{\Lambda}}\tilde{\bm{E}}^{\top}$,
$\tilde{\bm{C}}_{c}=\tilde{\bm{V}}\tilde{\bm{\Lambda}}\tilde{\bm{V}}^{\top}$,
$\tilde{\bm{\Phi}}_{c}=\tilde{\bm{V}}\tilde{\bm{\Lambda}}^{1/2}\tilde{\bm{E}}^{\top}$.
The value of the operators after training are given in Table 4. Once the model
is trained, we can verify that
$\tilde{\bm{W}}=\tilde{\bm{\Phi}}_{c}\tilde{\bm{A}}$.We can also have a look
at the hidden variables. A way to do it is to consider the MAP of $\bm{h}$
given $\bm{\phi}$ or $\bm{k}$. We have

$\displaystyle\bm{h}_{\mathrm{MAP}}$
$\displaystyle=N\tilde{\bm{\Lambda}}^{-1}_{1:q,1:q}\tilde{\bm{A}}^{\top}\tilde{\bm{k}}_{c}\,\;\quad\quad\text{(if
$\mathrm{rank}(\tilde{\bm{K}}_{c})\geq q$)},$ (20)
$\displaystyle=N\tilde{\bm{\Lambda}}^{-1}_{1:q,1:q}\tilde{\bm{W}}^{\top}\bigl{(}\tilde{\bm{\phi}}-\tilde{\bm{\phi}}_{c}\bigr{)}\quad\text{(if
$\mathcal{H}$ is finite)},$ (21)

and

$\displaystyle\bigl{(}\bm{k}_{c}\bigr{)}_{\mathrm{MAP}}$
$\displaystyle=\tilde{\bm{K}}_{c}\tilde{\bm{A}}\tilde{\bm{h}}\qquad\quad\;\text{(if
$\mathrm{rank}(\tilde{\bm{K}}_{c})\geq q$)},$ (22)
$\displaystyle\bm{\phi}_{\mathrm{MAP}}$
$\displaystyle=\tilde{{\bm{W}}}\tilde{\bm{h}}+\tilde{\bm{\phi}}_{c}\qquad\qquad\text{(if
$\mathcal{H}$ is finite)}.$ (23)

As developed in Section 4, we can easily generate samples in both feature and
kernel representations. For the latter and in canonical basis, it becomes

$\displaystyle\tilde{\bm{k}}_{c}=\tilde{\bm{B}}\tilde{\bm{u}},\qquad\text{with
$\tilde{\bm{u}}\sim\mathcal{N}(\bm{0},\bm{I}_{N})$}.$ (24)

###  5.2 Examples

As the primal case is already treated by Tipping & Bishop (1999), we consider
here the model in its dual formulation. A toy example can by found in Fig. 3.
We use an RBF kernel
$k(\bm{x},\bm{y})=\exp\bigl{(}-\lVert\bm{x}-\bm{y}\rVert_{2}^{2}/(2\gamma^{2})\bigr{)}$
with bandwidth $\gamma=2$. As the number of components increases, the mean
variance of the $N-q$ unused components $\sigma^{2}$ becomes smaller and the
model tends to the classical KPCA model. Another way the reduce $\sigma^{2}$
is to increase the number of components $q$, with $\sigma^{2}\rightarrow 0$
when $q\rightarrow N$. This can be observed in Fig. 3(c): the Probabilistic
PCA model resembles closely the KPCA model, whereas more variance is left
over, _i.e._ not projected back, in Fig.s 3(a) and 3(b). The results of the
generation is Gaussian, which is a consequence of the linearity of the
preimage method chosen (Eq. (19)). Here again, as the number of components
increases and $\sigma^{2}$ decreases, the model is allowed to project back
more variance and the distribution becomes wider. Another example on the MNIST
dataset LeCun & Cortes (2010) with the RBF kernel with $\gamma=4$ is given at
Fig. 4.

##  6 Conclusion

Probabilistic Interpretation. By reformulating the Prob. PCA model in Hilbert
space, we were able to define a formulation of it. Likewise Prob. PCA in
primal was englobing classical PCA (with $\sigma^{2}\rightarrow 0$), Prob. PCA
in dual is also englobing KPCA in the same limit. Furthermore, we are now able
to sample in dual space, enhancing the understanding of the generation done
with KPCA.

Limitations. As most kernel methods, the model is still limited by the need of
a preimage method to go back to the input space once a sample is projected or
generated. Furthermore, training the model in dual required to find the $q$
first eigenvalues of the kernel matrix, which may become expensive as the
number of datapoints $N$ increases. Generating renders the problem even worse
as it requires the computation of all eigenvalues. The model also requires to
determine a $\sigma^{2}$ or alternatively a latent dimension $q$.

## Acknowledgements

EU: The research leading to these results has received funding from the
European Research Council under the European Union’s Horizon 2020 research and
innovation program / ERC Advanced Grant E-DUALITY (787960). This paper
reflects only the authors’ views and the Union is not liable for any use that
may be made of the contained information. Research Council KUL: Tensor Tools
for Taming the Curse iBOF/23/064, Optimization frameworks for deep kernel
machines C14/18/068. Flemish Government: FWO projects: GOA4917N (Deep
Restricted Kernel Machines: Methods and Foundations), PhD/Postdoc grant. This
research received funding from the Flemish Government (AI Research Program).
Henri De Plaen and Johan A. K. Suykens are also affiliated to Leuven.AI – KU
Leuven institute for AI, B-3000, Leuven, Belgium.

## References

  * Alaíz et al. (2018) Alaíz, C. M., Fanuel, M., and Suykens, J. A. K.  Convex formulation for kernel PCA and its use in semisupervised learning.  _IEEE Transactions on Neural Networks and Learning Systems_ , 29(8):3863–3869, 2018.  doi: 10.1109/TNNLS.2017.2709838.
  * Bartholomew et al. (2011) Bartholomew, D. J., Knott, M., and Moustaki, I.  _Latent variable models and factor analysis: A unified approach_ , volume 904.  John Wiley & Sons, 2011.
  * Basilevsky (2009) Basilevsky, A. T.  _Statistical factor analysis and related methods: theory and applications_.  John Wiley & Sons, 2009.
  * LeCun & Cortes (2010) LeCun, Y. and Cortes, C.  MNIST handwritten digit database.  2010\.  URL http://yann.lecun.com/exdb/mnist/.
  * Mika et al. (1998) Mika, S., Schölkopf, B., Smola, A., Müller, K.-R., Scholz, M., and Rätsch, G.  Kernel PCA and de-noising in feature spaces.  _Advances in neural information processing systems_ , 11, 1998.
  * Pandey et al. (2020) Pandey, A., Schreurs, J., and Suykens, J. A. K.  Robust generative restricted kernel machines using weighted conjugate feature duality.  In _Machine Learning, Optimization, and Data Science: 6th International Conference, LOD 2020, Siena, Italy, July 19–23, 2020, Revised Selected Papers, Part I 6_ , pp. 613–624. Springer, 2020.
  * Pandey et al. (2021) Pandey, A., Schreurs, J., and Suykens, J. A. K.  Generative restricted kernel machines: A framework for multi-view generation and disentangled feature learning.  _Neural Networks_ , 135:177–191, 2021.
  * Pandey et al. (2022a) Pandey, A., De Meulemeester, H., De Plaen, H., De Moor, B., and Suykens, J. A. K.  Recurrent restricted kernel machines for time-series forecasting.  _Proceedings of ESANN 2022_ , 2022a.
  * Pandey et al. (2022b) Pandey, A., Fanuel, M., Schreurs, J., and Suykens, J. A. K.  Disentangled representation learning and generation with manifold optimization.  _Neural Computation_ , 34(10):2009–2036, 2022b.
  * Schölkopf & Smola (2001) Schölkopf, B. and Smola, A. J.  _Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond_.  MIT Press, Cambridge, MA, USA, 2001.  ISBN 0262194759.
  * Schölkopf et al. (1998) Schölkopf, B., Smola, A., and Müller, K.-R.  Nonlinear component analysis as a kernel eigenvalue problem.  _Neural computation_ , 10(5):1299–1319, 1998\.
  * Schreurs & Suykens (2018) Schreurs, J. and Suykens, J. A. K.  Generative kernel PCA.  _26th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN)_ , pp. 129–134, 2018\.
  * Suykens (2017) Suykens, J. A. K.  Deep Restricted Kernel Machines Using Conjugate Feature Duality.  _Neural Computation_ , 29(8):2123–2163, 08 2017\.  ISSN 0899-7667.  doi: 10.1162/neco˙a˙00984.
  * Suykens et al. (2003) Suykens, J. A. K., Van Gestel, T., Vandewalle, J., and De Moor, B.  A support vector machine formulation to PCA analysis and its kernel version.  _IEEE Transactions on Neural Networks_ , 14(2):447–450, 2003.  doi: 10.1109/TNN.2003.809414.
  * Tipping & Bishop (1999) Tipping, M. E. and Bishop, C. M.  Probabilistic principal component analysis.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 61(3):611–622, 1999.
  * Winant et al. (2020) Winant, D., Schreurs, J., and Suykens, J. A. K.  Latent space exploration using generative kernel PCA.  In _Artificial Intelligence and Machine Learning: 31st Benelux AI Conference, BNAIC 2019, and 28th Belgian-Dutch Machine Learning Conference, BENELEARN 2019, Brussels, Belgium, November 6-8, 2019, Revised Selected Papers 28_ , pp. 70–82. Springer, 2020.
  * Zhang et al. (2004) Zhang, Z., Wang, G., Yeung, D.-Y., and Kwok, J. T.  Probabilistic kernel principal component analysis.  _Department of Compputer Science, The Hong Kong University of Science and Technology, Tech. Rep_ , 2004.

##  Appendix A Theoretical Development

For brevity of notations, we will define the _expectation value_ as
$\lVert\bm{a}\rVert_{\bm{\Sigma}}^{2}=\bm{a}^{\ast}\bm{\Sigma}\bm{a}\in\mathbb{R}_{\geq
0}$, with $\bm{a}$ an element of a Hilbert space and $\bm{\Sigma}$ a linear
operator from and to that space. The norm is a particular case with the
identity as operator $\bm{\Sigma}=\bm{I}$. The density function of the
multivariate normal distribution $\bm{a}\sim\mathcal{N}(\bm{b},\bm{\Sigma})$
can be rewritten as
$\frac{1}{Z}\exp\bigl{(}-\frac{1}{2}\lVert\bm{a}-\bm{b}\rVert^{2}_{\bm{\Sigma^{-1}}}\bigr{)}$.

###  A.1 Primal and Dual Spaces

######  Lemma 1.

The operators $\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$ and
$\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}$ are self-adjoint, positive semi-
definite and share the same non-zero eigenvalues. In particular, we have the
eigenvector relations
$\bm{v}_{i}=(\lambda_{i})^{-1/2}\bm{\Phi}_{c}\bm{\epsilon}_{i}$ and
$\bm{\epsilon}_{i}=(\lambda_{i})^{-1/2}\bm{\Phi}_{c}^{\ast}\bm{v}_{i}$.

###### Proof.

The result stated here is inspired from Alaíz et al. (2018). Self-adjointness
is a consequence of the definition of the inner product, which also guarantees
the positive semi-definiteness. $(\Longrightarrow)$ Let us suppose that
$(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c})\bm{e}_{i}=\lambda_{i}\bm{\epsilon}_{i}$
with $\lambda_{i}\neq 0$. We then have
$\bm{\Phi}_{c}\circ(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c})\bm{\epsilon}_{i}=(\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast})\circ\bm{\Phi}_{c}\bm{\epsilon}_{i}=\lambda_{i}\bm{\Phi}_{c}\bm{\epsilon}_{i}$.
Hence, we have that $\bm{\Phi}_{c}\bm{\epsilon}_{i}$ is eigenvector, but not
necessarily normalized. In fact, its norm is given by
$\bm{\epsilon}_{i}^{\ast}\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\bm{\epsilon}_{i}=\lambda_{i}$.
We thus have the relation
$\bm{u}_{i}=(\lambda_{i})^{-1/2}\bm{\Phi}\bm{\epsilon}_{i}$.
$(\Longleftarrow)$ We suppose now that
$(\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast})\bm{u}_{i}$, which leads to
$\bm{\Phi}_{c}^{\ast}\circ(\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast})\bm{u}_{i}=(\bm{\Phi}\circ\bm{\Phi}^{\ast})\circ\bm{\Phi}_{c}^{\ast}\bm{u}_{i}=\lambda_{i}\bm{u}^{\ast}$.
Again, we have the relation
$\bm{\epsilon}_{i}=(\lambda_{i})^{-1/2}\bm{\Phi}_{c}^{\ast}\bm{u}_{i}$. ∎

###  A.2 Primal Model

####  A.2.1 Feature Distribution

######  Definition 1 (Conditional Feature Distribution).

Considering $\bm{\phi},\bm{\mu}\in\mathcal{H}_{\mathcal{E}}$,
$\bm{h}\in\mathcal{L}$, a linear operator
$\bm{W}:\mathcal{L}\rightarrow\mathcal{H}_{\mathcal{L}}$ and its adjoint
$\bm{W}^{\ast}:\mathcal{H}_{\mathcal{E}}\rightarrow\mathcal{L}$. We define the
conditional probability distribution of the primal feature variable
$\bm{\phi}$ with variance $\sigma^{2}\in\mathbb{R}_{>0}$ as

$p\left(\bm{\phi}\left|\bm{h}\right.\right)=\frac{1}{\left(\sigma\sqrt{2\pi}\right)^{N}}\exp\left(\frac{-1}{2\sigma^{2}}\left\lVert\bm{\phi}-\bm{W}\bm{h}-\bm{\mu}\right\rVert_{\bm{I}_{\mathcal{H}_{\mathcal{E}}}}^{2}\right).$
(25)

The following proposition verifies that the conditional distribution of
$\bm{\phi}$ given $\bm{h}$ (Def. 1) is well defined. To ease the readability,
we first consider a Lemma.

######  Lemma 2.

Given values $a\in\mathbb{R}_{>0}$ and $b,c\in\mathbb{R}$, we have the
following integral

$\int_{\mathbb{R}}\exp\Biggl{(}-\frac{1}{2}ax^{2}+bx+c\Biggr{)}\mathrm{d}x=\sqrt{\frac{2\pi}{a}}\exp\left(\frac{b^{2}}{2a}+c\right).$
(26)

###### Proof.

We first find the primitive
$\int\exp\bigl{(}-\frac{1}{2}ax^{2}+bx+c\bigr{)}\mathrm{d}x=\sqrt{\frac{\pi}{2a}}\exp\bigl{(}\frac{b^{2}}{2a}+c\bigr{)}\mathrm{erf}\bigl{(}\frac{ax-b}{\sqrt{2a}}\bigr{)}$,
with the error function defined as
$\mathrm{erf}(x)=\frac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^{2}}\,\mathrm{d}t$.
Indeed, we have
$\frac{\mathrm{d}}{\mathrm{d}x}\mathrm{erf}(x)=\frac{2}{\sqrt{\pi}}e^{-x^{2}}$.
It suffices then to derivate the primitive to verify them. To conclude the
proof, it suffices to notice that the error function is symmetric and that
$\lim_{x\rightarrow+\infty}\mathrm{erf}(x)=1$. ∎

From now on, we will consider the singular value decomposition of the primal
interconnection linear operator
$\bm{W}=\sum_{p=1}^{q}s_{p}\bm{\varrho}_{p}\bm{r}_{p}^{\ast}$, with
$\bm{\varrho}_{p}\in\mathcal{H}_{\mathcal{E}}$ and
$\bm{r}_{p}^{\ast}\in\mathcal{L}^{\ast}$ two sets of orthonormal variables and
$s_{p}\in\mathbb{R}_{>0}$ the singular values.

######  Proposition 1.

Def. 1 is a well-defined distribution. More specifically, its measure is
normalized:

$\int_{\mathcal{L}}p\left(\bm{\phi}\left|\bm{h}\right.\right)\mathrm{d}\bm{\phi}=1.$
(27)

###### Proof.

By considering the singular value decomposition of $\bm{W}$, the term inside
the exponential becomes

$-\frac{1}{2\sigma^{2}}\sum_{i=1}^{N}\Biggl{\\{}\left(\bm{v}_{i}^{\ast}\bm{\phi}\right)^{2}-\left(\bm{v}_{i}^{\ast}\bm{\phi}\right)\Bigl{(}2\sum_{p=1}^{q}\big{(}\left(\bm{r}_{p}^{\ast}\bm{h}\right)\left(\bm{\varrho}_{p}^{\ast}\bm{v}_{i}\right)s_{p}\big{)}+\left(\bm{v}_{i}^{\ast}\bm{\mu}\right)\Bigr{)}\Biggr{\\}}+C,$
(28)

with
$C=-\frac{1}{2\sigma^{2}}\left(\bm{W}\bm{h}+\bm{\mu}\right)^{\ast}\left(\bm{W}\bm{h}+\bm{\mu}\right)=-\frac{1}{2\sigma^{2}}\sum_{i=1}^{N}\sum_{p=1}^{q}\big{(}(\bm{r}_{p}^{\ast}\bm{h})(\bm{\varrho}_{p}^{\ast}\bm{v}_{i})s_{p}+(\bm{v}_{i}^{\ast}\bm{\mu}\big{)}^{2}$.
Integrating over $\mathcal{H}_{\mathcal{E}}$ means integrating over
$\mathrm{span}\left\\{\bm{v}_{i},\ldots,\bm{v}_{N}\right\\}$, thus for all
$\left(\bm{v}_{i}^{\ast}\bm{\phi}\right)\in\mathbb{R}$. Using Lemma 2 and by
Fubini’s theorem, we have

$\displaystyle\int_{\mathcal{F}}p\left(\bm{\phi}\left|\bm{h},\bm{\varrho}\right.\right)\mathrm{d}\bm{\phi}$
$\displaystyle=$
$\displaystyle\left(\sigma\sqrt{2\pi}\right)^{-N}\exp(-C)\int_{\mathbb{R}^{N}}\exp\left(\sum_{i=1}^{N}\left(-\frac{1}{2\sigma^{2}}x_{i}^{2}+b_{i}x_{i}\right)\right)\mathrm{d}\bm{x},$
(29) $\displaystyle=$
$\displaystyle\left(\sigma\sqrt{2\pi}\right)^{-N}\exp(-C)\prod_{i=1}^{N}\int_{\mathbb{R}}\exp\left(-\frac{1}{2\sigma^{2}}x_{i}^{2}+b_{i}x_{i}\right)\mathrm{d}x_{i},$
(30) $\displaystyle=$
$\displaystyle\left(\sigma\sqrt{2\pi}\right)^{-N}\exp(-C)\prod_{i=1}^{N}\sigma\sqrt{2\pi}\exp\left(\frac{1}{2}\sigma^{2}b_{i}^{2}\right),$
(31) $\displaystyle=$
$\displaystyle\exp\left(\frac{1}{2}\sigma^{2}\sum_{i=1}^{N}b_{i}^{2}-C\right),$
(32)

with
$b_{i}=\frac{1}{\sigma^{2}}\big{(}(\bm{r}_{p}^{\ast}\bm{h})(\bm{\varrho}_{p}^{\ast}\bm{v}_{i})s_{p}+(\bm{v}_{i}^{\ast}\bm{\mu}\big{)}$.
The proof is concluded by observing that
$C=\frac{1}{2}\sigma^{2}\sum_{i=1}^{N}b_{i}^{2}$. ∎

We can now consider the marginal distribution of the feature representation
$\bm{\phi}$.

######  Lemma 3.

Both linear operators
$\bm{W}\circ\bm{W}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}$ and
$\bm{W}^{\ast}\circ\bm{W}+\sigma^{2}\bm{I}_{\mathcal{L}}$ are positive
definite, of full rank and invertible. The linear operator
$\bm{W}\circ\bm{W}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}$ shares
the $q$ non-zero eigenvalues of
$\bm{W}^{\ast}\circ\bm{W}+\sigma^{2}\bm{I}_{\mathcal{L}}$, with the remaining
$N-q$ eigenvalues being equal to $\sigma^{2}$. In particular, we have

$\displaystyle\mathrm{eig}\left(\bm{W}^{\ast}\circ\bm{W}+\sigma^{2}\bm{I}_{\mathcal{L}}\right)=\left\\{s^{2}_{p}+\sigma^{2}\right\\}_{p=1}^{q},$
(33)
$\displaystyle\mathrm{eig}\left(\bm{W}\circ\bm{W}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}\right)=\left\\{s^{2}_{p}+\sigma^{2}\right\\}_{p=1}^{q}\cup\left\\{\sigma^{2}\right\\}^{N-q}.$
(34)

###### Proof.

We consider the eigenvalues of
$\bm{W}^{\ast}\circ\bm{W}+\bm{I}_{\mathcal{L}}=\sum_{p=1}^{q}(s_{p}^{2}+\sigma^{2})\bm{\varrho}_{p}\bm{\varrho}_{p}^{\ast}$.
Its eigenvalues are thus given by $s_{p}^{2}+\sigma^{2}$ for $p=1,\ldots,q$
and we directly conclude that they are all strictly positive. In a similar
fashion, the eigenvalues of
$\bm{W}\circ\bm{W}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}=\sum_{p=1}^{q}(s_{p}^{2}+\sigma^{2})\bm{r}_{p}\bm{r}_{p}^{\ast}+\sigma^{2}\mathbb{P}_{\mathcal{H}_{\mathcal{E}}}$
are given by $s_{p}^{2}+\sigma^{2}$ for $p=1,\ldots,q$ and $\sigma^{2}$ for
$k=q+1,\ldots,N$. All the eigenvalues are strictly positive. Hence are the
operators positive definite, full rank and invertible. ∎

######  Proposition 2 (Marginal Feature Distribution).

Assuming the conditional distribution of $\bm{\phi}$ given $\bm{h}$ (Def. 1),
assumed normally distributed, _i.e._ ,
$p(\bm{h})=\left(2\pi\right)^{-q/2}\exp\left(-\frac{1}{2}\bm{h}^{\ast}\bm{h}\right)$,
the posterior distribution of the primal feature vector $\bm{\phi}$ is given
by

$p(\bm{\phi})=\frac{1}{Z_{\bm{\phi}}}\exp\left(-\frac{1}{2}\left\lVert\bm{\phi}-\bm{\mu}\right\rVert^{2}_{\bm{\Sigma}_{\bm{\phi}|\bm{W}}^{-1}}\right),$
(35)

with
$Z_{\bm{\phi}}=(2\pi)^{N/2}\bigl{(}(\sigma^{2})^{N-q}\prod_{p=1}^{q}(s_{p}^{2}+\sigma^{2})\bigr{)}^{1/2}$
and
$\bm{\Sigma}_{\bm{\phi}|\bm{W}}=\bm{W}\circ\bm{W}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}$

###### Proof.

The joint probability distribution is given by Bayes’ theorem as
$p(\bm{\phi},\bm{h})=p(\bm{\phi}|\bm{h})p(\bm{h})$. The integration proceeds
in a very similar fashion as the proof of Prop. 1. Similarly, the term inside
the exponential becomes

$-\frac{1}{2\sigma^{2}}\sum_{p=1}^{q}\Biggl{\\{}\left(\bm{r}_{p}^{\ast}\bm{h}\right)^{2}\left(s_{p}^{2}+\sigma^{2}\right)-\left(\bm{r}_{p}^{\ast}\bm{h}\right)\Bigl{(}2s_{p}\bigl{(}\bm{\varrho}_{p}^{\ast}(\bm{\phi}-\bm{\mu})\bigl{)}\Bigr{)}\Biggr{\\}}+D,$
(36)

with

$\displaystyle D$ $\displaystyle=$
$\displaystyle-\frac{1}{2\sigma^{2}}\left(\bm{\phi}-\bm{\mu}\right)^{\ast}\left(\bm{\phi}-\bm{\mu}\right),$
(37) $\displaystyle=$
$\displaystyle-\frac{1}{2\sigma^{2}}\sum_{p=1}^{q}(\bm{\varrho}_{p}^{\ast}(\bm{\phi}-\bm{\mu}))^{2}-\frac{1}{2\sigma^{2}}\left(\bm{\phi}-\bm{\mu}\right)^{\ast}\mathbb{P}_{\mathcal{H}_{\mathcal{E}}}\left(\bm{\phi}-\bm{\mu}\right).$
(38)

Here again, integrating on the whole latent space $\mathcal{L}$ means
integrating for all $\left(\bm{r}_{p}^{\ast}\bm{h}\right)\in\mathbb{R}$.
Again, we use Lemma 2 and Fubini’s theorem. After some simplification inside
the exponential, we can use the development made in Lemma 3 to find the
following:

$-\frac{1}{2}\left\\{\sum_{p=1}^{q}\frac{(\bm{\varrho}_{p}^{\ast}(\bm{\phi}-\bm{\mu}))^{2}}{s_{p}^{2}+\sigma^{2}}+\frac{1}{\sigma^{2}}\left(\bm{\phi}-\bm{\mu}\right)^{\ast}\mathbb{P}_{\mathcal{H}_{\mathcal{E}}}\left(\bm{\phi}-\bm{\mu}\right)\right\\}=-\frac{1}{2}\left(\bm{\phi}-\bm{\mu}\right)^{\ast}\left(\bm{W}\circ\bm{W}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{\mathcal{H}_{\mathcal{E}}}}\right)^{-1}\left(\bm{\phi}-\bm{\mu}\right).$
(39)

The normalization term follows similarly to Proposition 1 and we can verify
the consistency of the obtained distribution by looking at the spectrum using
Lemma 3. ∎

####  A.2.2 Maximum Likelihood

Training the model in primal corresponds to maximizing the likelihood of the
observations in the finite feature space $\mathcal{H}_{\mathcal{E}}$.

######  Proposition 3 (Primal Maximum Likelihood).

Provided $\sigma^{2}\leq\lambda_{q}/N$, where $\lambda_{q}$ is the $q$th
greatest eigenvalue of $\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$, the _Maximum
Likelihood (ML)_ of $\bm{W}$ and $\bm{\mu}$ given the observations
$\\{\bm{\phi}_{i}\\}_{i=1}^{N}$ is given by

$\displaystyle\bm{\mu}_{\mathrm{ML}}$ $\displaystyle=$
$\displaystyle\bm{\phi}_{c},$ (40) $\displaystyle\bm{W}_{\mathrm{ML}}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{q}\sqrt{\lambda_{p}/N-\sigma^{2}}\bm{v}_{p}\bm{r}_{p}^{\ast},$
(41)

with $\left\\{\left(\lambda_{p},\bm{v}_{p}\right)\right\\}_{p=1}^{q}$ the
greatest eigenpairs of $\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$ (w.r.t. the
eigenvalues).

###### Proof.

The maximum likelihood of the observations $\\{\bm{\phi}_{i}\\}_{i=1}^{N}$ is
computed as
$\mathrm{argmax}_{\bm{W},\bm{\mu}}\log\bigl{(}\prod_{i=1}^{N}p(\bm{\phi}_{i}|\bm{W},\bm{\mu})\bigr{)}=\mathrm{argmax}_{\bm{W},\bm{\mu}}L_{\bm{\phi}}$,
with $L_{\bm{\phi}}$ the likelihood function, which can be written as

$\begin{split}L_{\bm{\phi}}=-\frac{N}{2}\Biggl{\\{}&N\log(2\pi)+(N-q)\log\left(\sigma^{2}\right)+\sum_{p=1}^{q}\log\left(s_{p}^{2}+\sigma^{2}\right)+\frac{1}{N\sigma^{2}}\sum_{i=1}^{N}(\bm{\phi}_{i}-\bm{\mu})^{\ast}\mathbb{P}_{\mathcal{H}_{\mathcal{E}}}(\bm{\phi}_{i}-\bm{\mu})\\\
&+\frac{1}{N}\sum_{p=1}^{q}\frac{1}{s_{p}^{2}+\sigma^{2}}\left(\sum_{i=1}^{N}\left(\bm{\varrho}_{p}^{\ast}(\bm{\phi}_{i}-\bm{\mu})\right)\right)^{2}\Bigg{\\}}.\end{split}$
(42)

The optimization of the mean $\bm{\mu}$ is trivial and we have
$\bm{\mu}_{\mathrm{ML}}=\bm{\phi}_{c}$. The optimization of
$\bm{W}=\sum_{p=1}^{q}s_{p}\bm{\varrho}_{p}\bm{r}_{p}^{\ast}$ is less trivial.
We first note that optimizing for $\\{\bm{\varrho}_{p}\\}_{p=1}^{q}$,
$\\{s_{p}^{2}\\}_{p=1}^{q}$ and $\\{\bm{r}_{p}\\}_{p=1}^{q}$ is not
identifiable: indeed, two singular values may be equal. Furthermore, the
likelihood function $L_{\bm{\phi}}$ is independent of the basis
$\\{\bm{r}_{p}\\}_{p=1}^{q}$ and for any solution of $\bm{\varrho}_{p}$, we
must also admit its opposite $-\bm{\varrho}_{p}$ too. In addition to that,
computing the saddle points of the likelihood is not straightforward as we
cannot consider the vectors $\\{\bm{\varrho}_{p}\\}_{p=1}^{q}$ to be
independent variables as they must remain orthonormal: the variation of one
basis vector $\bm{\varrho}_{p}$ must keep it normalized and has an influence
on the other basis vectors. For $s_{p}$ however, the variations may happen
with the only constraint of strict positivity. We may thus consider

$\frac{\partial L_{\bm{\phi}}}{\partial s_{p}}=0\Longleftrightarrow
s_{p}^{2}+\sigma^{2}=\frac{1}{N}\left(\bm{\varrho}_{p}^{\ast}\Bigl{(}\sum_{n=1}^{N}\bm{\phi}_{n}-\bm{\mu}\Bigr{)}\right)^{2}.$
(43)

We now consider the fact that
$\mathbb{P}_{\mathcal{H}_{\mathcal{E}}}=\bm{I}_{\mathcal{\mathcal{H}_{\mathcal{E}}}}-\sum_{p=1}^{q}\bm{\varrho}_{p}\bm{\varrho}_{p}^{\ast}$.
The likelihood function restricted to the terms dependent on
$\bm{\varrho}_{p}$ reduces to

$\frac{1}{2}\left\\{\sum_{p=1}^{q}\left(\frac{1}{s_{p}^{2}+\sigma^{2}}-\frac{1}{\sigma^{2}}\right)\bm{\Phi}_{c}^{\ast}\circ\left(\bm{\varrho}_{p}\bm{\varrho}_{p}^{\ast}\right)\circ\bm{\Phi}_{c}\right\\}.$
(44)

From there, we can deduce that the likelihood is maximized when
$\\{\bm{\varrho}_{p}\\}_{p=1}^{q}$ forms a basis of
$\mathrm{span}(\bm{v}_{1},\ldots,\bm{v}_{p})$, with $\\{v_{p}\\}_{p=1}^{q}$
the $q$ greatest eigenvectors of $\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}$
(w.r.t. the eigenvalues $\lambda_{p}$’s). We may thus identify both basis:
$\bm{\varrho}_{p}=\bm{v}_{p}$ and by consequence
$s_{p}^{2}+\sigma^{2}=\lambda_{p}/N$ (from Eq. (43)), for all $p=1,\ldots,q$.

What about the other choices of basis? At the end, it would not change
anything as it would be identified to the same solution. Indeed, as the
$\bm{\varrho}_{p}$’s could always be written as a linear combination of
$\\{\bm{v}_{p}\\}_{p=1}^{q}$, we could always write the interconnection
operator as
$\bm{W}_{\mathrm{ML}}=\sum_{p=1}^{q}(\lambda_{p}/N-\sigma^{2})\bm{v}_{p}\bm{r}_{p}^{\ast}$
as the choice of the orthonormal basis $\\{\bm{r}_{p}\\}_{p=1}^{q}$ is
arbitrary. ∎

####  A.2.3 Dimensionality Reduction in Feature Space

######  Proposition 4 (Primal Conditional Latent Distribution).

The conditional distribution of $\bm{h}$ given $\bm{\phi}$ is given by

$p\left(\bm{h}\left|\bm{\phi}\right.\right)=\frac{1}{Z_{\bm{h}|\bm{\phi}}}\exp\left(-\frac{1}{2\sigma^{2}}\left\lVert\bm{h}-\bm{m}\right\rVert^{2}_{\bm{\Sigma}^{-1}_{\bm{h}|\bm{\phi}}}\right),$
(45)

with
$\bm{\Sigma}_{\bm{h}|\bm{\phi}}=\left(\bm{W}^{\ast}\circ\bm{W}+\sigma^{2}\bm{I}_{\mathcal{\mathcal{L}}}\right)^{-1}$,
$\bm{m}=(\bm{\Sigma}_{\bm{h}|\bm{\phi}})^{-1}\bm{W}^{\ast}(\bm{\phi}-\bm{\mu})$
and
$Z_{\bm{h}|\bm{\phi}}=(\sigma\sqrt{2\pi})^{q}\bigl{(}\prod_{p=1}^{q}(s_{p}^{2}+\sigma^{2})\bigr{)}^{-1/2}$.

###### Proof.

The methodology is analogous to Props. 1 and 2. ∎

###  A.3 Dual Model

####  A.3.1 Representation

######  Proposition 5 (Dual Representation).

Given any interconnection operator
$\bm{W}:\mathcal{L}\rightarrow\mathcal{H}_{\mathcal{L}}$, we have the
following representation:

$\bm{W}=\bm{\Phi}_{c}\circ\bm{A},$ (46)

with $\bm{A}:\mathcal{L}\rightarrow\mathcal{L}$, the dual interconnection
operator.

###### Proof.

Following our definitions, we know that
$\mathrm{\bm{\Phi}_{c}\circ\bm{\Phi}_{c}^{\ast}}\geq q$. By consequence, the
linear operator $\bm{\Phi}_{c}:\mathcal{H}\rightarrow\mathcal{E}$ has at least
$q$ non-zero singular values. As we have
$\mathrm{dim}(\mathcal{L})=\mathrm{dim}(\mathcal{H}_{\mathcal{L}})$, the proof
is concluded by recalling that the primal interconnection operator
$\bm{W}:\mathcal{L}\rightarrow\mathcal{H}_{\mathcal{L}}$ is also linear. ∎

######  Proposition 6 (Dual Maximum Likelihood).

Given the operator $\bm{W}_{\mathrm{ML}}$ (Prop. 3) and provided
$\sigma^{2}\leq\lambda_{q}/N$, where $\lambda_{q}$ is the $q$th greatest
eigenvalue of $\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}$, the dual
interconnection operator
$\bm{A}_{\mathrm{ML}}:\mathcal{L}\rightarrow\mathcal{L}$ is given by:

$\bm{A}_{\mathrm{ML}}=\sum_{p=1}^{q}\sqrt{1/N-\sigma^{2}\lambda_{p}^{-1}}\bm{\epsilon}_{p}\bm{r}_{p}^{\ast},$
(47)

with $\left\\{\left(\lambda_{p},\bm{\epsilon}_{p}\right)\right\\}_{p=1}^{q}$
the greatest eigenpairs of $\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}$ (w.r.t.
the eigenvalues) and $\left\\{\bm{r}_{p}\right\\}_{p=1}^{q}$ an arbitrary
orthonormal basis of the latent space $\mathcal{L}$.

###### Proof.

Using the relation
$\bm{v}_{p}=(\lambda_{p})^{-1/2}\bm{\Phi}_{c}\bm{\epsilon}_{p}$, we have

$\displaystyle\bm{W}_{\mathrm{ML}}$ $\displaystyle=$
$\displaystyle\sum_{p=1}^{q}\sqrt{\lambda_{p}/N-\sigma^{2}}\bm{v}_{p}\bm{r}_{p}^{\ast},$
(48) $\displaystyle=$
$\displaystyle\sum_{p=1}^{q}(\lambda_{p})^{-1/2}\sqrt{\lambda_{p}/N-\sigma^{2}}\bm{\Phi}_{c}\bm{\epsilon}_{p}\bm{r}_{p}^{\ast},$
(49) $\displaystyle=$
$\displaystyle\bm{\Phi}_{c}\circ\left(\sum_{p=1}^{q}\sqrt{1/N-\sigma^{2}\lambda_{p}^{-1}}\bm{\epsilon}_{p}\bm{r}_{p}^{\ast}\right).$
(50)

The proof is concluded by insuring that $1/N-\sigma^{2}\lambda_{p}^{-1}$ is
never negative. ∎

####  A.3.2 Dimensionality Reduction in Kernel Space

######  Proposition 7 (Conditional Kernel Distribution).

Under the same assumption as Prop. 6 and provided that
$\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}$ is of full rank, the posterior
distribution of the dual feature variable $\bm{k}_{c}$ given the latent
variable $h$ is given by

$p\left(\bm{k}_{c}|\bm{h}\right)=\frac{1}{Z_{\bm{k}_{c}|\bm{h}}}\exp\left(-\frac{1}{2\sigma^{2}}\left\lVert\bm{k}_{c}-\left(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\right)\circ\bm{A}_{\mathrm{ML}}\bm{h}\right\rVert^{2}_{\bm{\Sigma}^{-1}_{\bm{k}_{c}|\bm{h}}}\right),$
(51)

with
$Z_{\bm{k}_{c}|\bm{h}}=\left(\sigma\sqrt{2\pi}\right)^{N}\prod_{i=1}^{N}\lambda_{i}$
and $\bm{\Sigma}_{\bm{k}_{c}|\bm{h}}=\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}$.

###### Proof.

We consider the optimal primal interconnection operator
$\bm{W}_{\mathrm{ML}}=\bm{\Phi}_{c}\circ\bm{A}_{\mathrm{ML}}$ together with
the optimal mean $\bm{\mu}_{\mathrm{ML}}=\bm{\phi}_{c}$ and fill it in Def. 1.
The covariance is deducted by considering
$\bm{\phi}-\bm{\phi}_{c}=\bigl{(}\bm{\Phi}^{\ast}_{c}\bigr{)}^{-1}\circ\bm{\Phi}^{\ast}_{c}(\bm{\phi}-\bm{\phi}_{c})=\bigl{(}\bm{\Phi}^{\ast}_{c}\bigr{)}^{-1}\bm{k}_{c}$.
∎

######  Proposition 8 (Dual Conditional Latent Distribution).

Under the same assumption as Prop. 6, the posterior distribution of the latent
variable $\bm{h}$ given the dual feature variable $\bm{k}_{c}$ is given by

$\begin{split}p(\bm{h}|\bm{k}_{c})=\frac{1}{Z_{\bm{h}|\bm{k}_{c}}}\exp\left(-\frac{1}{2\sigma^{2}}\left\lVert\bm{h}-\bm{m}^{\prime}\right\rVert^{2}_{\bm{\Sigma}^{-1}_{\bm{h}|\bm{k}_{c}}}\right),\end{split}$
(52)

with
$\bm{\Sigma}_{\bm{h}|\bm{k}_{c}}=\left(\bm{A}_{\mathrm{ML}}^{\ast}\circ\bigl{(}\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\bigr{)}\circ\bm{A}_{\mathrm{ML}}+\sigma^{2}\bm{I}_{\mathcal{L}}\right)^{-1}$,
$Z_{\bm{h}|\bm{k}_{c}}=Z_{\bm{h}|\bm{\phi}}$ (Prop. 4) and
$\bm{m}^{\prime}=\bigl{(}\bm{\Sigma}_{\bm{h}|\bm{k}_{c}}^{-1}\circ\bm{A}_{\mathrm{ML}}^{\ast}\bigr{)}\bm{k}_{c}$.

###### Proof.

It suffices to consider the optimal primal interconnection operator
$\bm{W}_{\mathrm{ML}}=\bm{\Phi}_{c}\circ\bm{A}_{\mathrm{ML}}$, as well as the
optimal mean $\bm{\mu}_{\mathrm{ML}}=\bm{\phi}_{c}$ and fill both in Prop. 4.
∎

####  A.3.3 Kernel Sampling

######  Proposition 9 (Marginal Kernel Distribution).

Provided that $\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}$ is of full rank and
given the assumptions of Prop. 6, the posterior distribution of the the dual
feature $\bm{k}_{c}$ after optimization is given by

$p(\bm{k}_{c})=\frac{1}{Z_{\bm{k}_{c}}}\exp\left(-\frac{1}{2}\left\lVert\bm{k}_{c}\right\rVert^{2}_{\bm{\Sigma}_{\bm{k}_{c}}^{-1}}\right),$
(53)

with
$\bm{\Sigma}_{\bm{k}_{c}}=\bm{A}_{\mathrm{ML}}^{\ast}\circ\bm{A}_{\mathrm{ML}}+\sigma^{2}\left(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\right)^{-1}$
and
$Z_{\bm{k}_{c}}=(2\pi)^{N/2}N^{-q/2}\prod_{p=1}^{q}\lambda_{p}\prod_{p=q+1}^{N}(\sigma^{2}\lambda_{p})^{1/2}$.

###### Proof.

We start from Prop. 2 and develop the covariance:

$\displaystyle\bm{W}_{\mathrm{ML}}\bm{W}_{\mathrm{ML}}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}$
$\displaystyle=$
$\displaystyle\bm{\Phi}_{c}\circ\left(\bm{A}^{\ast}\circ\bm{A}+\sigma^{2}\left(\bm{\Phi}_{c}^{\ast}\circ\bm{\Phi}_{c}\right)^{-1}\right)\circ\bm{\Phi}_{c}^{\ast},$
(54) $\displaystyle=$
$\displaystyle\bm{\Phi}_{c}\circ\left(\frac{1}{N}\sum_{p=1}^{q}\bm{\epsilon}_{p}\bm{\epsilon}_{p}^{\ast}+\sigma^{2}\\!\\!\sum_{p=q+1}^{N}\\!\lambda_{p}^{-1}\bm{\epsilon}_{p}\bm{\epsilon}_{p}^{\ast}\right)\circ\bm{\Phi}_{c}^{\ast}.$
(55)

We set $\bm{\mu}_{\mathrm{ML}}=\bm{\phi}_{c}$ and observe that

$\displaystyle\left(\bm{\phi}-\bm{\phi}_{c}\right)^{\ast}\left(\bm{W}_{\mathrm{ML}}\bm{W}_{\mathrm{ML}}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}\right)^{-1}\left(\bm{\phi}-\bm{\phi}_{c}\right),$
$\displaystyle=$
$\displaystyle\bm{k}_{c}^{\ast}\left(\bm{\Phi}_{c}^{\ast}\circ\left(\bm{W}_{\mathrm{ML}}\bm{W}_{\mathrm{ML}}^{\ast}+\sigma^{2}\bm{I}_{\mathcal{H}_{\mathcal{E}}}\right)\circ\bm{\Phi}_{c}\right)^{-1}\bm{k}_{c},$
(56) $\displaystyle=$
$\displaystyle\bm{k}_{c}^{\ast}\left(\frac{1}{N}\sum_{p=1}^{q}\lambda_{p}^{2}\bm{\epsilon}_{p}\bm{\epsilon}_{p}^{\ast}+\sigma^{2}\\!\\!\sum_{p=q+1}^{N}\\!\lambda_{p}\bm{\epsilon}_{p}\bm{\epsilon}_{p}^{\ast}\right)\bm{k}_{c}.$
(57)

∎


associated to the mixed-derivative action demands immediate attention. As this
is a manifestly chiral theory on shell, it begs the question: whether it
corresponds to a chiral half of a $2d$ CFT, or is there even more to it? To
start with, determination of central charges for the chiral algebra will be an
important step here. It will also be instructive to find quantized Carroll
theories beyond covariant phase space methods, perhaps using a path integral
formalism. However the degenerate nature inherent to these theories will
require a major overhaul for the known techniques.

One of our principal motivations for the investigation of Carrollian
structures is to establish some version of Flat Holography in different
dimensions. To this end, we would like to construct full supersymmetric
Carroll CFTs in future endeavours. Supersymmetric Carrollian theories have
been explored in various contexts, like supersymmetric null strings
Bagchi:2016yyf ; Bagchi:2017cte ; Bagchi:2018wsn , Flat Supergravities
Lodato:2016alv ; Bagchi:2018ryy , and very recently in the form of Carrollian
$\mathcal{N}=1$ superconformal theory Bagchi:2022owq . In all of these cases
it has been firmly established that adding Carrollian spinors make the
structure of these theories extremely rich. Just to take the example in $2d$,
it has been shown that Super-BMS3 algebra has two distinct avatars, called the
homogeneous and inhomogeneous (or democratic and despotic) supersymmetric
theories, based on different inequivalent representation of the degenerate
Clifford algebra. A thorough analysis of the Spin Group defined on the
intrinsically Carrollian metric data is still an open problem, and a genuinely
intricate one at that. We should be able to clarify these structures further
in the near future.

Before we end, a final word about our new mixed derivative theory and possible
connections to null strings. One of the problematic issues in recent attempts
to construct a theory of tensionless strings is the lack of understanding of
open strings. There have been attempts at this earlier Bonelli:2003kh ;
Lindstrom:2003mg , but the manifest null structures that should be associated
with such a construction are far from obvious. To emphasise this point, we
note that Casali:2017zkz  claims the open null string algebras found by
contraction (e.g. in Bonelli:2003kh ) cannot be found by imposing appropriate
boundary conditions on the null string. The $2d$ open string worldsheet has to
become null and that would mean the emergence of a BMS type algebra. The fact
that our mixed derivative theory starts off having BMS invariance and then
reduces to a single copy of Virasoro onshell is rather tantalising and perhaps
indicative of tensionless open strings. It would be instructive to follow this
lead further and also figure out whether this can have further implications to
the very interesting closed-to-open transitions discovered in Bagchi:2019cay .

## Acknowledgements

AB is partially supported by a Swarnajayanti fellowship of the Department of
Science and Technology, India and by the following grants from the Science and
Engineering Research Board: SB/SJF/2019-20/08, MTR/2017/000740,
CGR/2020/002035. SD is supported by grant number 09/092(0971)/2017-EMR-I from
Council of Scientific and Industrial Research (CSIR). The work of ArB is
supported by the Quantum Gravity Unit of the Okinawa Institute of Science and
Technology Graduate University (OIST). He would also like to thank Physics
Department, Kyoto University for hospitality during the final part of this
work.

## APPENDICES

##  Appendix A ILST action vs. vielbein formalism

Inspired by the degenerate metric that appears in (170) at the tensionless
point $T=0$, we consider the degenerate spacetime metric
$h_{\mu\nu}=e_{\mu}e_{\nu}$ with $e_{t}$ arbitrary:

$h_{\mu\nu}=\Big{(}\begin{matrix}e_{\tau}^{2}&e_{\tau}\\\
e_{\tau}&1\end{matrix}\Big{)};\quad\det(h_{\mu\nu})=0.$ (203)

The projective inverse relations (32) give

$\displaystyle e^{\tau}e_{\tau}+e^{\sigma}=1,\quad
e_{\tau}\tau^{\tau}+\tau^{\sigma}=0,\quad
e^{\tau}\tau_{\tau}+e^{\sigma}\tau_{\sigma}=0,\quad\tau^{\tau}\tau_{\tau}+\tau^{\sigma}\tau_{\sigma}=1,$
$\displaystyle e_{\tau}e^{\tau}+\tau_{\tau}\tau^{\tau}=1,\quad
e^{\sigma}+\tau_{\sigma}\tau^{\sigma}=1,\quad
e_{\tau}e^{\sigma}+\tau_{\tau}\tau^{\sigma}=0,\quad
e^{\tau}+\tau_{\sigma}\tau^{\tau}=0.$ (204)

Note that $e_{\tau}=0$ corresponds to the standard flat Carroll spacetime
(39). So here we want to take $e_{\tau}\neq 0$. Then from
$e_{\tau}\tau^{t}+\tau^{\sigma}=0$, we see that either
$\tau^{\sigma}=0=\tau^{\tau}$ or $\tau^{\sigma}\neq 0\neq\tau^{\tau}$. Since
$\tau^{\mu}$ is no-where vanishing, we take $\tau^{\sigma}\neq
0\neq\tau^{\tau}$. Then the ILST parametrization of the null metric with
$V^{\mu}=\frac{1}{2\sqrt{\lambda}}(1,-\rho)$, where $\lambda$ and $\rho$ are
arbitrary functions on the Carroll spacetime, corresponds to

$V^{\mu}=\sqrt{e}\tau^{\mu}\qquad\implies\quad
e=4\lambda,\quad\tau^{\tau}=\frac{1}{4\lambda},\quad\tau^{\sigma}=-\frac{\rho}{4\lambda}.$
(205)

We note that the determinant $e\neq 0$ and a smooth $\tau^{\mu}$ requires
$\lambda\neq 0,\infty$. Then from the projective inverse relations above, we
have the components of zweibeins as

$\displaystyle\tau^{\tau}=\frac{1}{4\lambda},\quad\tau^{\sigma}=-\frac{\rho}{4\lambda},\quad
e^{\tau}=-\frac{\tau_{\sigma}}{4\lambda},\quad
e^{\sigma}=1+\frac{\rho\tau_{\sigma}}{4\lambda},$
$\displaystyle\tau_{\tau}=4\lambda+\rho\tau_{\sigma},\quad\tau_{\sigma}\sim\text{arbitrary},\quad
e_{\tau}=\rho,\quad e_{\sigma}=1,$ (206)

where $\rho$, $\lambda$ and $\tau_{\sigma}$ are arbitrary functions as of now.

For (206) to describe a strong flat Carroll spacetime, we require the
spacetime connection (41) to vanish. Let us write the components of the
spacetime connection for (206):

$\displaystyle\Gamma^{\tau}_{\tau\tau}=\frac{\partial_{\tau}\lambda}{\lambda},\quad\Gamma^{\tau}_{\sigma\tau}=\frac{\partial_{\sigma}\lambda}{\lambda},\quad\Gamma^{\sigma}_{\tau\tau}=\partial_{\tau}\rho-\frac{\rho}{\lambda}\partial_{\tau}\lambda,\quad\Gamma^{\sigma}_{\sigma\tau}=\partial_{\sigma}\rho-\frac{\rho}{\lambda}\partial_{\sigma}\lambda,$
$\displaystyle\Gamma^{\tau}_{\tau\sigma}=\frac{1}{4\lambda}(\partial_{\tau}\tau_{\sigma}-\Omega_{\tau}),\quad\Gamma^{\tau}_{\sigma\sigma}=\frac{1}{4\lambda}(\partial_{\sigma}\tau_{\sigma}-\Omega_{\sigma}),$
$\displaystyle\Gamma^{\sigma}_{\tau\sigma}=-\frac{\rho}{4\lambda}(\partial_{\tau}\tau_{\sigma}-\Omega_{\tau}),\quad\Gamma^{\sigma}_{\sigma\sigma}=-\frac{\rho}{4\lambda}(\partial_{\sigma}\tau_{\sigma}-\Omega_{\sigma}).$
(207)

We see that $\Gamma^{\rho}_{\mu\nu}=0$ restricts the arbitrary functions in
(206) and determines the boost-spin connection as

$\lambda=\text{constant},\quad\rho=\text{constant},\quad\tau_{\sigma}\sim\text{arbitrary};\quad\Omega_{\mu}=\partial_{\mu}\tau_{\sigma}.$
(208)

We can easily see that the boost curvature and the Riemann tensor vanishes
with these choices. We conclude that the ILST parametrization $V^{\mu}$
describes a strong flat Carroll spacetime only for constant $\lambda$ and
$\rho$. In that case, the (BMS) conformal Killing equations become

$\partial_{t}\xi^{\tau}-\partial_{\sigma}\xi^{\sigma}-2\rho\partial_{\sigma}\xi^{\tau}=0,\quad\partial_{\tau}\xi^{\sigma}+\rho^{2}\partial_{\sigma}\xi^{\tau}=0.$
(209)

Putting $\rho=-1$ one can generate the killing equations mentioned in (99).

##  Appendix B Parameter space for ILST actions

As we discussed earlier, there are two generic constraints for a null string,
both coming from dynamics of the scalar field $\Phi$, and can be written as

$P^{2}=0,~{}~{}P\cdot\partial_{\sigma}\Phi=0,$ (210)

$P$ being the canonical momentum of the system. These are the constraints
which are supposed to generate BMS3 symmetries, with canonical brackets of
$\\{\Phi,\Pi\\}$ as input. A generic action for the system is then

$S=\int d^{2}\sigma(P\dot{\Phi}+\lambda P^{2}+\rho P\Phi^{\prime}),$ (211)

where $\lambda$ and $\rho$ are lagrange multipliers (compare with (183)). For
a second order dynamical system, we can fix a gauge where $P$ is generic
linear in the derivatives of the field $\Phi$,

$P=\alpha\dot{\Phi}+\beta\Phi^{\prime}$ (212)

Putting this in the action and using $P=\frac{\partial
L}{\partial\dot{\Phi}}$, we can solve for $(\lambda,\rho)$ in terms of the
gauge parameters $(\alpha,\beta)$, which could in principle be general
functions of $\phi$, however not of its derivatives. This leads us to,

$\lambda=-\frac{1}{2\alpha},~{}~{}\rho=\frac{\beta}{\alpha},~{}~{}\alpha\neq
0.$ (213)

So the Lagrangian for the gauge fixed system becomes

$\mathcal{L}=\frac{\alpha}{2}\dot{\Phi}^{2}+\frac{\beta^{2}}{2\alpha}\Phi^{\prime
2}+\beta\dot{\Phi}\cdot\Phi^{\prime}.$ (214)

Now $(\alpha,\beta)$ remains generally undetermined parameters, but as long as
we have the constraints in place, this should be a general BMS3 invariant
theory in two dimensions. Let us consider distinct cases:

##### Case 1: $\beta=0$ and $\alpha=1$

: We have the Timelike action of ${\mathcal{L}}_{1}=\frac{\dot{\Phi}^{2}}{2}$.

##### Case 2: $\beta=\pm 1$ and $\alpha=1$

The action becomes chiral
${\mathcal{L}}=\frac{1}{2}\left(\dot{\Phi}\pm\Phi^{\prime}\right)^{2}$. This
action is nothing but our spacelike action with fixed $e^{\tau}=1$,

${\mathcal{L}}_{2}=\frac{1}{2}(\partial_{z}\Phi)^{2},$ (215)

written in holomorphic coordinates. The Hamiltonian of such a theory takes the
form

$\mathcal{H}=\frac{1}{2}(\dot{\Phi}^{2}-\Phi^{\prime
2})=\frac{1}{2}\partial_{z}\Phi\partial_{\bar{z}}\Phi,$ (216)

which incidentally is the lagrangian for a $2d$ CFT. But there could be more
self consistent branches of solutions for the gauge parameters
$(\alpha,\beta)$. For example if $\alpha=0$ then above solution is not valid
and the unfixed lagrangian takes a form

${\mathcal{L}}=\beta\dot{\Phi}\cdot\Phi^{\prime}+(\lambda\beta^{2}+\rho\beta)\Phi^{\prime
2}.$ (217)

Now there is no way to trade in $(\lambda,\rho)$ in terms of $(\alpha,\beta)$
anymore and $\beta$ remains unfixed. Although the constraints are robust

$\Phi^{\prime 2}=0,~{}~{}\dot{\Phi}\cdot\Phi^{\prime}=0.$ (218)

A naive choice in the parameter space could be that of $\beta=1$ and
correspondingly, $\lambda\beta+\rho=\pm 1$, which leads to:

${\mathcal{L}}_{3}=\Phi^{\prime}(\dot{\Phi}\pm\Phi^{\prime}),$ (219)

which is also the Floreanini-Jackiw action Floreanini:1987as , _i.e._ the
time-space inverted version of (129).

## References

  * (1) J. Levy-Leblond, _Une nouvelle limite non-relativiste du group de Poincare_ , _Ann.Inst.Henri Poincare_ 3 (1965) 1.
  * (2) M. Henneaux, _Geometry of Zero Signature Space-times_ , _Bull. Soc. Math. Belg._ 31 (1979) 47.
  * (3) C. Duval, G. W. Gibbons, P. A. Horvathy and P. M. Zhang, _Carroll versus Newton and Galilei: two dual non-Einsteinian concepts of time_ , _Class. Quant. Grav._ 31 (2014) 085016 [1402.0657].
  * (4) C. Duval, G. W. Gibbons and P. A. Horvathy, _Conformal Carroll groups and BMS symmetry_ , _Class. Quant. Grav._ 31 (2014) 092001 [1402.5894].
  * (5) C. Duval, G. W. Gibbons and P. A. Horvathy, _Conformal Carroll groups_ , _J. Phys._ A47 (2014) 335204 [1403.4213].
  * (6) H. Bondi, M. G. J. van der Burg and A. W. K. Metzner, _Gravitational waves in general relativity. 7. Waves from axisymmetric isolated systems_ , _Proc. Roy. Soc. Lond._ A269 (1962) 21.
  * (7) R. Sachs, _Asymptotic symmetries in gravitational theory_ , _Phys. Rev._ 128 (1962) 2851.
  * (8) G. Barnich and G. Compere, _Classical central extension for asymptotic symmetries at null infinity in three spacetime dimensions_ , _Class. Quant. Grav._ 24 (2007) F15 [gr-qc/0610130].
  * (9) A. Bagchi and R. Fareghbal, _BMS/GCA Redux: Towards Flatspace Holography from Non-Relativistic Symmetries_ , _JHEP_ 10 (2012) 092 [1203.5795].
  * (10) A. Bagchi, R. Gopakumar, I. Mandal and A. Miwa, _GCA in 2d_ , _JHEP_ 08 (2010) 004 [0912.1090].
  * (11) A. Bagchi, _Correspondence between Asymptotically Flat Spacetimes and Nonrelativistic Conformal Field Theories_ , _Phys. Rev. Lett._ 105 (2010) 171601 [1006.3354].
  * (12) J. M. Maldacena, _The Large N limit of superconformal field theories and supergravity_ , _Int. J. Theor. Phys._ 38 (1999) 1113 [hep-th/9711200].
  * (13) G. Barnich and C. Troessaert, _Aspects of the BMS/CFT correspondence_ , _JHEP_ 05 (2010) 062 [1001.1541].
  * (14) A. Bagchi, S. Detournay and D. Grumiller, _Flat-Space Chiral Gravity_ , _Phys. Rev. Lett._ 109 (2012) 151301 [1208.1658].
  * (15) A. Bagchi, S. Detournay, R. Fareghbal and J. Simón, _Holography of 3D Flat Cosmological Horizons_ , _Phys. Rev. Lett._ 110 (2013) 141302 [1208.4372].
  * (16) G. Barnich, _Entropy of three-dimensional asymptotically flat cosmological solutions_ , _JHEP_ 10 (2012) 095 [1208.4371].
  * (17) A. Bagchi and R. Basu, _3D Flat Holography: Entropy and Logarithmic Corrections_ , _JHEP_ 03 (2014) 020 [1312.5748].
  * (18) A. Bagchi, D. Grumiller and W. Merbis, _Stress tensor correlators in three-dimensional gravity_ , _Phys. Rev. D_ 93 (2016) 061502 [1507.05620].
  * (19) A. Bagchi, R. Basu, D. Grumiller and M. Riegler, _Entanglement entropy in Galilean conformal field theories and flat holography_ , _Phys. Rev. Lett._ 114 (2015) 111602 [1410.4089].
  * (20) H. Jiang, W. Song and Q. Wen, _Entanglement Entropy in Flat Holography_ , _JHEP_ 07 (2017) 142 [1706.07552].
  * (21) E. Hijano and C. Rabideau, _Holographic entanglement and Poincaré blocks in three-dimensional flat space_ , _JHEP_ 05 (2018) 068 [1712.07131].
  * (22) A. Bagchi, S. Detournay, D. Grumiller and J. Simon, _Cosmic Evolution from Phase Transition of Three-Dimensional Flat Space_ , _Phys. Rev. Lett._ 111 (2013) 181301 [1305.2919].
  * (23) S. Detournay, D. Grumiller, F. Schöller and J. Simón, _Variational principle and one-point functions in three-dimensional flat space Einstein gravity_ , _Phys. Rev. D_ 89 (2014) 084061 [1402.3687].
  * (24) H. Afshar, A. Bagchi, R. Fareghbal, D. Grumiller and J. Rosseel, _Spin-3 Gravity in Three-Dimensional Flat Space_ , _Phys. Rev. Lett._ 111 (2013) 121603 [1307.4768].
  * (25) H. A. Gonzalez, J. Matulich, M. Pino and R. Troncoso, _Asymptotically flat spacetimes in three-dimensional higher spin gravity_ , _JHEP_ 09 (2013) 016 [1307.5651].
  * (26) J. Hartong, _Holographic Reconstruction of 3D Flat Space-Time_ , _JHEP_ 10 (2016) 104 [1511.01387].
  * (27) J. Hartong, _Gauging the Carroll Algebra and Ultra-Relativistic Gravity_ , _JHEP_ 08 (2015) 069 [1505.05011].
  * (28) A. Bagchi, M. Gary and Zodinmawia, _Bondi-Metzner-Sachs bootstrap_ , _Phys. Rev._ D96 (2017) 025007 [1612.01730].
  * (29) G. Barnich, L. Donnay, J. Matulich and R. Troncoso, _Asymptotic symmetries and dynamics of three-dimensional flat supergravity_ , _JHEP_ 08 (2014) 071 [1407.4275].
  * (30) R. Fareghbal and A. Naseh, _Aspects of Flat/CCFT Correspondence_ , _Class. Quant. Grav._ 32 (2015) 135013 [1408.6932].
  * (31) D. Grumiller, P. Parekh and M. Riegler, _Local quantum energy conditions in non-Lorentz-invariant quantum field theories_ , 1907.06650.
  * (32) L. Ciambelli, C. Marteau, A. C. Petkou, P. M. Petropoulos and K. Siampos, _Flat holography and Carrollian fluids_ , _JHEP_ 07 (2018) 165 [1802.06809].
  * (33) A. Bagchi, R. Basu, A. Kakkar and A. Mehra, _Flat Holography: Aspects of the dual field theory_ , _JHEP_ 12 (2016) 147 [1609.06203].
  * (34) A. Strominger, _Lectures on the Infrared Structure of Gravity and Gauge Theory_ , 1703.05448.
  * (35) S. Pasterski, _Lectures on celestial amplitudes_ , _Eur. Phys. J. C_ 81 (2021) 1062 [2108.04801].
  * (36) A.-M. Raclariu, _Lectures on Celestial Holography_ , 2107.02075.
  * (37) L. Donnay, A. Fiorucci, Y. Herfray and R. Ruzziconi, _A Carrollian Perspective on Celestial Holography_ , 2202.04702.
  * (38) A. Bagchi, S. Banerjee, R. Basu and S. Dutta, _Scattering Amplitudes: Celestial and Carrollian_ , 2202.08438.
  * (39) S. W. Hawking, M. J. Perry and A. Strominger, _Soft Hair on Black Holes_ , _Phys. Rev. Lett._ 116 (2016) 231301 [1601.00921].
  * (40) S. W. Hawking, M. J. Perry and A. Strominger, _Superrotation Charge and Supertranslation Hair on Black Holes_ , _JHEP_ 05 (2017) 161 [1611.09175].
  * (41) L. Donnay, G. Giribet, H. A. Gonzalez and M. Pino, _Supertranslations and Superrotations at the Black Hole Horizon_ , _Phys. Rev. Lett._ 116 (2016) 091101 [1511.08687].
  * (42) H. Afshar, S. Detournay, D. Grumiller, W. Merbis, A. Perez, D. Tempo et al., _Soft Heisenberg hair on black holes in three dimensions_ , _Phys. Rev. D_ 93 (2016) 101503 [1603.04824].
  * (43) R. F. Penna, _Near-horizon BMS symmetries as fluid symmetries_ , _JHEP_ 10 (2017) 049 [1703.07382].
  * (44) L. Donnay and C. Marteau, _Carrollian Physics at the Black Hole Horizon_ , _Class. Quant. Grav._ 36 (2019) 165002 [1903.09654].
  * (45) S. Carlip, _Black Hole Entropy from Bondi-Metzner-Sachs Symmetry at the Horizon_ , _Phys. Rev. Lett._ 120 (2018) 101301 [1702.04439].
  * (46) S. Carlip, _Near-horizon Bondi-Metzner-Sachs symmetry, dimensional reduction, and black hole entropy_ , _Phys. Rev. D_ 101 (2020) 046002 [1910.01762].
  * (47) A. Schild, _Classical Null Strings_ , _Phys. Rev._ D16 (1977) 1722.
  * (48) J. Isberg, U. Lindstrom, B. Sundborg and G. Theodoridis, _Classical and quantized tensionless strings_ , _Nucl. Phys._ B411 (1994) 122 [hep-th/9307108].
  * (49) A. Bagchi, _Tensionless Strings and Galilean Conformal Algebra_ , _JHEP_ 05 (2013) 141 [1303.0291].
  * (50) A. Bagchi, S. Chakrabortty and P. Parekh, _Tensionless Strings from Worldsheet Symmetries_ , _JHEP_ 01 (2016) 158 [1507.04361].
  * (51) A. Bagchi, A. Banerjee and P. Parekh, _Tensionless Path from Closed to Open Strings_ , _Phys. Rev. Lett._ 123 (2019) 111601 [1905.11732].
  * (52) A. Bagchi, A. Banerjee, S. Chakrabortty, S. Dutta and P. Parekh, _A tale of three — tensionless strings and vacuum structure_ , _JHEP_ 04 (2020) 061 [2001.00354].
  * (53) A. Bagchi, M. Mandlik and P. Sharma, _Tensionless tales: vacua and critical dimensions_ , _JHEP_ 08 (2021) 054 [2105.09682].
  * (54) A. Bagchi, A. Banerjee and S. Chakrabortty, _Rindler Physics on the String Worldsheet_ , _Phys. Rev. Lett._ 126 (2021) 031601 [2009.01408].
  * (55) A. Bagchi, A. Banerjee, S. Chakrabortty and R. Chatterjee, _A Rindler Road to Carrollian Worldsheets_ , 2111.01172.
  * (56) A. Bagchi, A. Mehra and P. Nandi, _Field Theories with Conformal Carrollian Symmetry_ , _JHEP_ 05 (2019) 108 [1901.10147].
  * (57) A. Bagchi, R. Basu, A. Mehra and P. Nandi, _Field Theories on Null Manifolds_ , _JHEP_ 02 (2020) 141 [1912.09388].
  * (58) J. de Boer, J. Hartong, N. A. Obers, W. Sybesma and S. Vandoren, _Carroll symmetry, dark energy and inflation_ , 2110.02319.
  * (59) R. Basu and U. N. Chowdhury, _Dynamical structure of Carrollian Electrodynamics_ , _JHEP_ 04 (2018) 111 [1802.09366].
  * (60) M. Henneaux and P. Salgado-Rebolledo, _Carroll contractions of Lorentz-invariant theories_ , _JHEP_ 11 (2021) 180 [2109.06708].
  * (61) P.-x. Hao, W. Song, X. Xie and Y. Zhong, _A BMS-invariant free scalar model_ , 2111.04701.
  * (62) B. Chen, R. Liu and Y.-f. Zheng, _On Higher-dimensional Carrollian and Galilean Conformal Field Theories_ , 2112.10514.
  * (63) L. Bidussi, J. Hartong, E. Have, J. Musaeus and S. Prohazka, _Fractons, dipole symmetries and curved spacetime_ , 2111.03668.
  * (64) L. Ciambelli and C. Marteau, _Carrollian conservation laws and Ricci-flat gravity_ , _Class. Quant. Grav._ 36 (2019) 085004 [1810.11037].
  * (65) N. Gupta and N. V. Suryanarayana, _Constructing Carrollian CFTs_ , _JHEP_ 03 (2021) 194 [2001.03056].
  * (66) E. Bergshoeff, J. Gomis, B. Rollier, J. Rosseel and T. ter Veldhuis, _Carroll versus galilei gravity_ , _Journal of High Energy Physics_ 2017 (2017) .
  * (67) L. Ciambelli, R. G. Leigh, C. Marteau and P. M. Petropoulos, _Carroll Structures, Null Geometry and Conformal Isometries_ , _Phys. Rev. D_ 100 (2019) 046010 [1905.02221].
  * (68) R. Floreanini and R. Jackiw, _Selfdual Fields as Charge Density Solitons_ , _Phys. Rev. Lett._ 59 (1987) 1873.
  * (69) J. Sonnenschein, _CHIRAL BOSONS_ , _Nucl. Phys. B_ 309 (1988) 752.
  * (70) E. Casali and P. Tourkine, _On the null origin of the ambitwistor string_ , _JHEP_ 11 (2016) 036 [1606.05636].
  * (71) E. Casali, Y. Herfray and P. Tourkine, _The complex null string, Galilean conformal algebra and scattering equations_ , _JHEP_ 10 (2017) 164 [1707.09900].
  * (72) D. Grumiller and W. Merbis, _Near horizon dynamics of three dimensional black holes_ , _SciPost Phys._ 8 (2020) 010 [1906.10694].
  * (73) S. Deser, R. Jackiw and S. Templeton, _Three-Dimensional Massive Gauge Theories_ , _Phys. Rev. Lett._ 48 (1982) 975.
  * (74) W. Li, W. Song and A. Strominger, _Chiral Gravity in Three Dimensions_ , _JHEP_ 04 (2008) 082 [0801.4566].
  * (75) D. Grumiller and N. Johansson, _Instability in cosmological topologically massive gravity at the chiral point_ , _JHEP_ 07 (2008) 134 [0805.2610].
  * (76) A. Maloney, W. Song and A. Strominger, _Chiral Gravity, Log Gravity and Extremal CFT_ , _Phys. Rev. D_ 81 (2010) 064007 [0903.4573].
  * (77) A. Bagchi, R. Basu, S. Detournay and P. Parekh, _Flatspace Chiral Supergravity_ , _Phys. Rev._ D97 (2018) 106020 [1801.03245].
  * (78) A. Bagchi, S. Chakrabortty and P. Parekh, _Tensionless Superstrings: View from the Worldsheet_ , _JHEP_ 10 (2016) 113 [1606.09628].
  * (79) A. Bagchi, A. Banerjee, S. Chakrabortty and P. Parekh, _Inhomogeneous Tensionless Superstrings_ , _JHEP_ 02 (2018) 065 [1710.03482].
  * (80) A. Bagchi, A. Banerjee, S. Chakrabortty and P. Parekh, _Exotic Origins of Tensionless Superstrings_ , _Phys. Lett. B_ (2019)  [1811.10877].
  * (81) I. Lodato and W. Merbis, _Super-BMS 3 algebras from $\mathcal{N}=2$ flat supergravities_, _JHEP_ 11 (2016) 150 [1610.07506].
  * (82) A. Bagchi, D. Grumiller and P. Nandi, _Carrollian superconformal theories and super BMS_ , 2202.01172.
  * (83) G. Bonelli, _On the tensionless limit of bosonic strings, infinite symmetries and higher spins_ , _Nucl. Phys._ B669 (2003) 159 [hep-th/0305155].
  * (84) U. Lindstrom and M. Zabzine, _Tensionless strings, WZW models at critical level and massless higher spin fields_ , _Phys. Lett._ B584 (2004) 178 [hep-th/0305098].