$\begin{array}[]{rcl}\mathbfcal{T}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}&=&\displaystyle\sum_{k_{1}=1}^{{K_{1}}}(\mathbf{w}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{w}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{D_{1}})\otimes(\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{O_{1}})\\\
\vskip 3.0pt plus 1.0pt minus
1.0pt\mathbfcal{T}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}&=&\displaystyle\sum_{k_{2}=1}^{{K_{2}}}(\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{D_{2}})\otimes(\mathbf{v}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{O_{2}})\,,\end{array}$

and contiguous partitions113113113Which are already implicitly specified by
writing the multi-index sets this way, but to be unambiguous,
$\\{\\{N_{1}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}},\dots,N_{D_{1}}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}\\},\\{M_{1}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}},\dots,M_{O_{1}}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}\\}\\}$
and
$\\{\\{N_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}},\dots,N_{D_{2}}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}\\},\\{M_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}},\dots,M_{O_{2}}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}\\}\\}$.
of the mode index sets of
$[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]$
and
$[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]$,
then this bilinear function, to be called the Zehfuss product114114114Johann
Georg Zehfuss described the Kronecker product of matrices in 1858, the first
person known to do so., applied to
$\mathbfcal{T}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}$ and
$\mathbfcal{T}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$ has the value:

$\bigotimes_{Z}(\mathbfcal{T}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}},\mathbfcal{T}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}})=\sum_{k_{1}=1}^{{K_{1}}}\sum_{k_{2}=1}^{{K_{2}}}(\mathbf{w}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{w}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{D_{1}})\otimes(\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{D_{2}})\otimes(\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{O_{1}})\otimes(\mathbf{v}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{O_{2}})\,,$

whereas the standard Segre outer product would return the following tensor:

$\mathbfcal{T}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}\otimes\mathbfcal{T}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}=\sum_{k_{1}=1}^{{K_{1}}}\sum_{k_{2}=1}^{{K_{2}}}(\mathbf{w}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{w}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{D_{1}})\otimes(\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{O_{1}})\otimes(\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{D_{2}})\otimes(\mathbf{v}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{O_{2}})\,.$

Although the contiguous partitions of the modes of
$\mathbfcal{T}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}$ and
$\mathbfcal{T}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$ have been
suppressed from the notation for the Zehfuss product given in the above
equations, the contiguous partitions must be specified somehow before the
Zehfuss product can be well-defined.

In the case that both tensors are the standard contravariant representation of
linear functions between coordinate spaces, as in the examples above, the
contiguous partitions of the modes into two blocks can be considered
implicitly specified by the requirements that (i) the first block of modes
corresponds to the function’s codomain and (ii) the second block of modes
corresponds to the function’s domain. Yet even in that case it is necessary to
make explicit which linear function between coordinate spaces the tensor is
intended to represent.

For example, already for an order-$2$ tensor (matrix)
$\mathbf{L}\in\mathbb{R}^{H\times W}$ can be considered, in an equally valid
manner115115115Note that switching between any of these interpretations
corresponds to the operations of “raising indices” and “lowering indices” in
Einstein tensor notation, i.e. that used in e.g. [205] or [7]., the
contravariant representation of any of the following linear functions:

  1. 1.

a linear function $\mathscr{L}_{|12}:\mathbb{R}\to\mathbb{R}^{H\times W}$ such
that $\mathcal{L}_{|12}(1)=\mathbf{L}$,

  2. 2.

a linear function $\mathcal{L}_{1|2}:\mathbb{R}^{W}\to\mathbb{R}^{H}$ such
that $\mathcal{L}_{1|2}(\mathbf{v})=\mathbf{L}\bullet_{2}\mathbf{v}$,

  3. 3.

a linear function116116116This is the unique linear function
$\mathcal{L}:\mathbb{R}^{H\times W}\to\mathbb{R}$ such that
$\mathcal{L}\circ\otimes=\mathscr{M}$, where $\mathscr{M}$ is the bilinear
function(/form) $\mathbb{R}^{H}\times\mathbb{R}^{W}\to\mathbb{R}$ such that
$\mathscr{M}(\mathbf{v},\mathbf{w})=\mathbf{L}\bullet_{1}\mathbf{v}\bullet_{2}\mathbf{w}$.
Cf. the discussion in section A.8.2, i.e. $\mathcal{L}=\Upsilon(\mathscr{M})$.
$\mathcal{L}_{12|}:\mathbb{R}^{H\times W}\to\mathbb{R}$ such that
$\mathcal{L}_{12|}(\mathbf{T})=\langle\mathbf{L},\mathbf{T}\rangle$.

In general, an order $O$ tensor can be interpreted, in an equally valid
manner, as the contravariant representation of $(O+1)$ distinct linear
functions, even after having chosen a single “standard” way of identifying
linear functions with any domain or codomain as tensors. Thus simply stating
that two tensors represent linear functions is by itself insufficient to
implicitly specify the contiguous partitions of modes needed to make the
Zehfuss product well-defined; the domains and codomains of those linear
functions must also be specified.

Observe that the map
$\mathcal{L}(\mathbb{R}^{[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]};\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]})$
to
$\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$
which returns the standard contravariant representation of a linear function
corresponds to the morphism in $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{2}$
which is the tensor product of (i) the standard contravariant representation
of
$\mathcal{L}(\mathbb{R}^{[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]};\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]})$
with tensors in
$\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}$
and (ii) the standard contravariant representation of
$\mathcal{L}(\mathbb{R}^{[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]};\mathbb{R}^{[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]})$
with tensors in
$\mathbb{R}^{[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$,
as long as the chosen tensor product:

$\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}\times\mathbb{R}^{[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}\to\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$

is the corresponding Zehfuss product, and not the standard Segre outer
product. When using $\zeta$ to denote identifications of linear functions with
corresponding contravariant tensors, one has the following identity (cf.
definition of tensor product of linear functions in section A.10.3):

$\bigotimes_{Z}\circ(\zeta_{[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}];[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]}\times\zeta_{[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}];[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]})=\zeta_{[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}];[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]}\circ\bigotimes_{\mathcal{L}(\mathbb{R}^{[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]};\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]})}\,.$

The definition of Zehfuss product can be extended to more than $2$ tensors
(each with a specified contiguous partitions of its modes) in a
straightforward way. The most common example in the practice is the case of
$O$ order $2$ tensors, each being the contravariant representation
$\mathbf{L}_{o}$ of a linear function in
$\mathcal{L}_{o}\in\mathcal{L}(\mathbb{R}^{M_{o}};\mathbb{R}^{N_{o}})$. Then
the $O$-fold Zehfuss product of these tensors is the order $2O$ tensor given
by the outer product decomposition:

$\begin{array}[]{rcl}\displaystyle\bigotimes_{Z}(\mathbf{L}_{1},\dots,\mathbf{L}_{O})&=&\displaystyle\sum_{m_{1}=1}^{{M_{1}}}\cdots\sum_{m_{O}=1}^{{M_{O}}}\mathcal{L}_{1}(\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}})\otimes\cdots\otimes\mathcal{L}_{o}(\mathbf{e}_{m_{O}}^{\scriptsize\scaleto{({M_{O}})\mathstrut}{0.7em}})\otimes\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{O}}^{\scriptsize\scaleto{({M_{O}})\mathstrut}{0.7em}}\\\
&=&\vskip 3.0pt plus 1.0pt minus
1.0pt\displaystyle\sum_{m_{1}=1}^{{M_{1}}}\cdots\sum_{m_{O}=1}^{{M_{O}}}(\mathbf{L}_{1})_{:m_{1}}\otimes\cdots\otimes(\mathbf{L}_{O})_{:m_{O}}\otimes\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{O}}^{\scriptsize\scaleto{({M_{O}})\mathstrut}{0.7em}}\,,\end{array}$

since this is the (standard) contravariant representation of the linear
transformation
$\mathcal{L}\overset{\text{def}}{=}\bigotimes_{o=1}^{O}\mathcal{L}_{o}$.

Defining $[M_{\scaleto{[O]\mathstrut}{0.5em}}]=\bigtimes_{o=1}^{O}[M_{o}]$,
the lexicographical matricization of the above Zehfuss product
$\bigotimes_{Z}(\mathbf{L}_{1},\dots,\mathbf{L}_{O})$ representing the linear
function
$\mathcal{L}\overset{\text{def}}{=}\bigotimes_{o=1}^{O}\mathcal{L}_{o}$ is the
matrix:

$\begin{array}[]{cl}&\displaystyle\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}\operatorname{vec}_{\operatorname{lex}}(\mathcal{L}_{1}(\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}})\otimes\cdots\otimes\mathcal{L}_{o}(\mathbf{e}_{m_{O}}^{\scriptsize\scaleto{({M_{O}})\mathstrut}{0.7em}}))\otimes\operatorname{vec}_{\operatorname{lex}}(\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{O}}^{\scriptsize\scaleto{({M_{O}})\mathstrut}{0.7em}})\\\
=&\vskip 3.0pt plus 1.0pt minus
1.0pt\displaystyle\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}\operatorname{vec}_{\operatorname{lex}}((\mathbf{L}_{1})_{:m_{1}}\otimes\cdots\otimes(\mathbf{L}_{O})_{:m_{O}})\otimes\operatorname{vec}_{\operatorname{lex}}(\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{O}}^{\scriptsize\scaleto{({M_{O}})\mathstrut}{0.7em}})\\\
=&\vskip 3.0pt plus 1.0pt minus
1.0pt\mathbf{L}_{1}\otimes_{\operatorname{Kr}}\mathbf{L}_{2}\otimes_{\operatorname{Kr}}\cdots\otimes_{\operatorname{Kr}}\mathbf{L}_{O}\,,\end{array}$

and the colexicographical matricization is the matrix:

$\begin{array}[]{cl}&\displaystyle\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}\operatorname{vec}_{\operatorname{col}}(\mathcal{L}_{1}(\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}})\otimes\cdots\otimes\mathcal{L}_{o}(\mathbf{e}_{m_{O}}^{\scriptsize\scaleto{({M_{O}})\mathstrut}{0.7em}}))\otimes\operatorname{vec}_{\operatorname{col}}(\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{O}}^{\scriptsize\scaleto{({M_{O}})\mathstrut}{0.7em}})\\\
=&\vskip 3.0pt plus 1.0pt minus
1.0pt\displaystyle\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}\operatorname{vec}_{\operatorname{col}}((\mathbf{L}_{1})_{:m_{1}}\otimes\cdots\otimes(\mathbf{L}_{O})_{:m_{O}})\otimes\operatorname{vec}_{\operatorname{col}}(\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{O}}^{\scriptsize\scaleto{({M_{O}})\mathstrut}{0.7em}})\\\
=&\vskip 3.0pt plus 1.0pt minus
1.0pt\mathbf{L}_{O}\otimes_{\operatorname{Kr}}\mathbf{L}_{O-1}\otimes_{\operatorname{Kr}}\cdots\otimes_{\operatorname{Kr}}\mathbf{L}_{1}\,,\end{array}$

where $\otimes_{\operatorname{Kr}}$ denotes the Kronecker product of matrices.
The above formulae explain the reason why the Kronecker product occurs so
frequently in the discussion of tensors: it can be used to represent
matricized forms of the contravariant representations of linear functions
between spaces of tensors.

In addition to the $O$-fold Zehfuss product of two tensors with contiguous
partitions of modes which have $2$ blocks, one can also define the $2$-fold
Zehfuss product of two tensors,
$\mathbfcal{T}\in\mathbb{R}^{[M_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]}$ and
$\mathbfcal{U}\in\mathbb{R}^{[N_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]}$, each
with contiguous partitions of modes which have $O$ blocks:

$\begin{array}[]{rcl}\displaystyle\bigotimes_{Z}(\mathbfcal{T},\mathbfcal{U})&=&\displaystyle\sum_{k_{1}=1}^{{K_{1}}}\sum_{k_{2}=1}^{{K_{2}}}(\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{B_{1}})\otimes(\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{C_{1}})\otimes\cdots\\\
&&\displaystyle\phantom{\sum_{k_{1}=1}^{{K_{1}}}\sum_{k_{2}=1}^{{K_{2}}}}\otimes(\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{D_{1}-B_{O}+1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{D_{1}})\otimes(\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{D_{2}-C_{O}+1}\otimes\cdots\otimes\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{D_{2}})\,,\end{array}$

where
$\sum_{k_{1}=1}^{{K_{1}}}\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k_{1})\mathstrut}{0.7em}}_{D_{1}}$
and
$\sum_{k_{2}=1}^{{K_{2}}}\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{w}^{\scriptsize\scaleto{(k_{2})\mathstrut}{0.7em}}_{D_{2}}$
are arbitrary tensor product decompositions of $\mathbfcal{T}$ and
$\mathbfcal{U}$ respectively, and $B_{1}$ and $C_{1}$ are the lengths of the
first blocks of the contiguous partitions of the mode index sets $[D_{1}]$ and
$[D_{2}]$ respectively, $B_{2}$ and $C_{2}$ are the lengths of the second
blocks of the contiguous partitions of $[D_{1}]$ and $[D_{2}]$ respectively,
…, and $B_{O}$ and $C_{O}$ are the lengths of the $O$’th blocks of the
contiguous partitions of $[D_{1}]$ and $[D_{2}]$. Compare this with the
definition of covariance tensor from [140] discussed below in appendix A.13.

Obviously the above definitions can be extended even further to the
$O_{1}$-fold Zehfuss product of $O_{1}$ tensors each of which has contiguous
partitions of their modes into $O_{2}$ blocks.

##  Appendix A.10 Categories of Vector Spaces and Tensor Spaces

The category $\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$ of finite-dimensional
real vector spaces has as objects (unsurprisingly) finite-dimensional real
vector spaces117117117The detailed definition of a finite-dimensional real
vector space is outside the scope of this review, and assumed to be
prerequisite knowledge for the review. Consider consulting e.g. [207] for an
introduction before continuing further., and as morphisms all linear functions
between them.

However the category of finite-dimensional real $D$-factorized tensor spaces
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$ is assumed to possibly be
unfamiliar to the reader. Compare what follows with section 3.2.5 of [129].

###  A.10.1 The Tensor Product Functor

Given $D$ finite-dimensional real vector spaces
$\mathbb{V}_{1},\dots,\mathbb{V}_{D}$, the pair consisting of the two data:

  1. (i)

a vector space $\mathbb{T}$,

  2. (ii)

a multilinear map
$\bigotimes_{\mathbb{T}}:\mathbb{V}_{1}\times\dots\times\mathbb{V}_{D}\to\mathbb{T}$,

is said to satisfy the universal property of the $D$-fold tensor
product118118118See section 3, as well as section 14.2 of [67], section 15.2
of [193], Proposition 3.22 of [129], or chapter 2 of [14]. if for every other
pair of a vector space $\mathbb{W}$ and multilinear map
$\Phi:\mathbb{V}_{1}\times\dots\times\mathbb{V}_{D}\to\mathbb{W}$, there
exists a unique linear map $\phi:\mathbb{T}\to\mathbb{W}$ such that
$\Phi=\phi\circ\bigotimes_{\mathbb{T}}$. This property is useful inasmuch as
it describes the desired behavior, but it is inadequate for specifying a
construction which realizes that behavior. In other words, there is
never119119119See endnote LABEL:tensor-universal-property-uniqueness from
section 3 for an explanation of why this is true. Compare also proposition
3.21 of [129]. a unique $(\mathbb{T},\bigotimes_{\mathbb{T}})$ pair satisfying
this universal property.

However, given both a $(\mathbb{T},\bigotimes_{\mathbb{T}})$ and a
$(\tilde{\mathbb{T}},\tilde{\bigotimes}_{\mathbb{T}})$ satisfying the
universal property, a corollary of the universal property is that there is a
unique120120120Specifically, only unique among those isomorphisms which are
compatible with both of the chosen tensor products, there are of course other
isomorphisms between $\mathbb{T}$ and $\tilde{\mathbb{T}}$, those isomorphisms
just aren’t as “good”. linear isomorphism between $\mathbb{T}$ and
$\tilde{\mathbb{T}}$. Thus to some extent there is no problem in making an
arbitrary choice among the infinitely many pairs of
$(\mathbb{T},\bigotimes_{\mathbb{T}})$ satisfying the universal property for
given $\mathbb{V}_{1},\dots,\mathbb{V}_{D}$, since any other choice one might
have made instead can still be recovered uniquely from the initial choice.

Because of the aforementioned indeterminacy when referring to “the” tensor
product of vector spaces, the correct definition in terms of the universal
property is often overlooked in favor of a construction which can be defined
and guaranteed to exist for every possible choice of $\mathbb{V}_{1}$, …,
$\mathbb{V}_{D}$, the idea being to make the same arbitrary choice for all
vector spaces rather than making one arbitrary choice for some class of vector
spaces and making a distinct arbitrary choice for another class of vector
spaces, compare chapter 2 of [14] or section 3.2.1 of [129].

However, in this review the preferred tensor product construction for the
finite-dimensional real vector spaces
$\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$ (for all possible multi-
index sets $[M_{\scaleto{[O]\mathstrut}{0.5em}}]$) will always be the Segre
outer product, even though the Segre outer product is not a construction which
is defined for all possible finite-dimensional real vector spaces. Therefore
this review falls into the group of expositions which wish to make one
arbitrary choice of tensor product for one class of vector spaces (Segre outer
product for the $\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$’s), while
making a distinct arbitrary choice for all other vector spaces.

Thus the approach pursued in chapter 2 of [14] or section 3.2.1 of [129], of
always defining the tensor product space as the suitable quotient of the
(massive) free vector spaceHendnotepage.5linkHendnote.5ee5macro:-¿Actually the
notion of “free vector space” opens its own separate can of worms, since it is
also defined via a universal property which only specifies a vector space up
to isomorphism, and does not give priority to any particular construction over
another. The approach taken in section 3.1.2 of “cite –hackbusch˝ for
constructing the free vector space is very similar to the usual construction
of taking the direct sum of the vector spaces $“R ^–s˝$ for every element $s
“in S$ for an arbitrary set $S$. This would seem to not open yet another can
of worms since the direct sum refers to a specific construction, although that
construction is of interest primarily because it satisfies the universal
property of the coproduct. generated by all of the vectors in the vector space
$\mathbb{V}_{1}\times\cdots\times\mathbb{V}_{D}$, is not suitable here.

This causes some difficulty, however, when trying to define a tensor product
functor:

$\bigotimes:\underbrace{\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}\times\cdots\times\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}}_{D\text{
times}}\to\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}\,,$

since using the Segre outer product leads to this functor not automatically
being defined for all $D$-tuples of finite-dimensional real vector spaces (in
contrast to the situation where the “one arbitrary choice for all vector
spaces” approach had been utilized instead), since the Segre outer product
only defines the functor for those $D$-tuples which consist exclusively of
$\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$’s.

So the concern which needs to be addressed is the following: will different
arbitrary choices of vector space satisfying the universal property of the
tensor product to use as the value of the tensor product functor for all
remaining $D$-tuples lead to (meaningfully) different tensor product functors?
Will the study of one type of tensor product functor defined partially in
terms of the Segre outer product lead to different conclusions compared to the
tensor product functor defined on objects by always using the aforementioned
quotient space construction?

First an interlude which will soon prove to be relevant. In order to define a
tensor product functor, it actually suffices to define it only on objects,
doing so by choosing for each $D$-tuple of finite-dimensional real vector
spaces an element of the equivalence class of vector spaces satisfying the
universal property for them. This is because the universal property can then
be used to automatically define the tensor product functor on morphisms.

More concretely, for linear functions
$\mathbb{V}_{1}\xrightarrow{f_{1}}\mathbb{W}_{1}$, …,
$\mathbb{V}_{D}\xrightarrow{f_{D}}\mathbb{W}_{D}$, what one wants to do is
have a choice of linear function from the chosen value
$\bigotimes_{d=1}^{D}\mathbb{V}_{d}$ of the tensor product functor for
$\mathbb{V}_{1},\dots,\mathbb{V}_{D}$ to the chosen value
$\bigotimes_{d=1}^{D}\mathbb{W}_{d}$ of the tensor product functor for
$\mathbb{W}_{1},\dots,\mathbb{W}_{D}$, so that the chosen linear function can
then represent the chosen value $\bigotimes_{d=1}^{D}f_{d}$ for the tensor
product functor on the morphisms $f_{1},\dots,f_{D}$. The universal property
automatically makes such a choice.

Consider the Cartesian product of linear functions (which itself is linear):

$\mathbb{V}_{1}\times\cdots\times\mathbb{V}_{D}\xrightarrow{f_{1}\times\cdots\times
f_{D}}\mathbb{W}_{1}\times\cdots\times\mathbb{W}_{D}\,,$

given via the rule of assignment $(f_{1}\times\cdots\times
f_{D})(\mathbf{v}_{1},\dots,\mathbf{v}_{D})=(f_{1}(\mathbf{v}_{1}),\dots,f_{D}(\mathbf{v}_{D}))$.
Then

$\bigotimes_{\mathbb{W}}\circ(f_{1}\times\cdots\times
f_{D}):\mathbb{V}_{1}\times\cdots\times\mathbb{V}_{D}\to\bigotimes_{d=1}^{D}\mathbb{W}_{d}$

is multilinear (since the function
$\bigotimes_{\mathbb{W}}:\mathbb{W}_{1}\times\cdots\times\mathbb{W}_{D}\to\bigotimes_{d=1}^{D}\mathbb{W}_{d}$
is multilinear). Therefore the universal property of the tensor product for
$(\bigotimes_{d=1}^{D}\mathbb{V}_{d},\bigotimes_{\mathbb{V}})$ guarantees the
existence of a unique linear function
$h:\bigotimes_{d=1}^{D}\mathbb{V}_{d}\to\bigotimes_{d=1}^{D}\mathbb{W}_{d}$
such that
$h\circ\bigotimes_{\mathbb{V}}=\bigotimes_{\mathbb{W}}\circ(f_{1}\times\cdots\times
f_{D})$.

It turns out that, given the choices of
$(\bigotimes_{d=1}^{D}\mathbb{V}_{d},\bigotimes_{\mathbb{V}})$ for the value
of the tensor product functor on $\mathbb{V}_{1}$, …, $\mathbb{V}_{D}$, and
$(\bigotimes_{d=1}^{D}\mathbb{W}_{d},\bigotimes_{\mathbb{W}})$ for the value
of the tensor product functor on $\mathbb{W}_{1}$, …, $\mathbb{W}_{D}$, then
the linear function
$h:\bigotimes_{d=1}^{D}\mathbb{V}_{d}\to\bigotimes_{d=1}^{D}\mathbb{W}_{d}$
defined uniquely via the universal property of the tensor product is the
natural choice for the value of the tensor product functor on
$f_{1},\dots,f_{D}$, i.e. the natural choice is given by defining
$\bigotimes_{d=1}^{D}f_{d}\overset{\text{def}}{=}h$.

The choice is natural in the following sense: given one tensor product functor
$\bigotimes^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}$ defined on objects by
making one class of arbitrary choices of vector spaces satisfying the
universal property, and defined on morphisms by using the arbitrary choices on
objects and the universal property in the manner explained above, and another
tensor product functor
$\bigotimes^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$ defined on objects by
making another class of arbitrary choices of vector spaces satisfying the
universal property, and also defined on morphisms by using the universal
property as explained above to extend the arbitrary choices on objects, then
the two functors $\bigotimes^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}$ and
$\bigotimes^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$ will always be
naturally isomorphic121121121“Natural isomorphism” is a precisely defined
notion of category theory, cf. footnote 108 of appendix A.7., or in other
words they will be for all intents and purposes entirely equivalent. This is
because of the unique isomorphisms which, due to the universal property, exist
between the arbitrary choices made on objects by the first functor and the
arbitrary choices made on objects by the second functor, and because both
functors were defined on morphisms by using the unique linear functions found
using the universal property in the way which was described above.

Note that the tensor product of linear transformations as it has been defined
in section 6.2 is the same definition one gets by using the Segre outer
product to define the tensor product functor on
$\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$’s and then using the
universal property to define the values of the tensor product functor on
morphisms. So even though the tensor product functor as defined in this review
is slightly different, at least on
$\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$’s and linear functions
between them, then the tensor product functor more commonly defined elsewhere,
both functors are still naturally isomorphic to one another, and this remains
true regardless of which arbitrary choices of vector spaces satisfying the
universal property are used to define the tensor product functor on $D$-tuples
of vector spaces for which not all of the $\mathbb{V}_{d}$’s are equal to
$\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$ for some
$[M_{\scaleto{[O]\mathstrut}{0.5em}}]$. Therefore, for the sake of simplicity,
when at least one of the vector spaces $\mathbb{V}_{d}$ is not equal to some
$\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$, for this review define
the $D$-fold tensor product of those vector spaces (and associated $D$-linear
map) via the usual construction as found e.g. in chapter 2 of [14]. Moreover,
for this review the value of the tensor product functor on morphisms will
always use the linear maps generated by the universal product as described
above.

###  A.10.2 Objects of the Category of Finite-Dimensional Real $D$-Factorized
Tensor Spaces

For a given $D\geq 1$, the objects of
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$ consist of three data:

  1. (i)

a finite-dimensional real vector space $\mathbb{T}$,

  2. (ii)

a $D$-tuple of finite-dimensional real vectors spaces
$(\mathbb{V}_{1},\dots,\mathbb{V}_{D})$,

  3. (iii)

a multilinear map

$\bigotimes:\mathbb{V}_{1}\times\mathbb{V}_{2}\times\cdots\times\mathbb{V}_{D}\to\mathbb{T}$

which satisfies the universal property of the tensor product.

The standard example will always be (where
${[M_{\scaleto{[D]\mathstrut}{0.5em}}]\overset{\text{def}}{=}\bigtimes_{d=1}^{D}[M_{d}]}$)
the special case when
$\mathbb{T}=\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$,
$\mathbb{V}_{1}=\mathbb{R}^{[M_{1}]}$, $\mathbb{V}_{2}=\mathbb{R}^{[M_{2}]}$,
…, $\mathbb{V}_{D}=\mathbb{R}^{[M_{D}]}$, and $\bigotimes$ denotes the Segre
outer product.

###  A.10.3 Morphisms of the Category of Finite-Dimensional Real
$D$-Factorized Tensor Spaces

Given two objects $\mathbb{V}$ and $\mathbb{W}$ of
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$, a morphism
$\mathbb{V}\to\mathbb{W}$ is any linear function of the form:

$\bigotimes_{d=1}^{D}f_{d}:\mathbb{V}\to\mathbb{W}\,,\quad\quad
f_{1}:\mathbb{V}_{1}\to\mathbb{W}_{1}\,,\dots\,,f_{d}:\mathbb{V}_{d}\to\mathbb{W}_{d}\,,\dots\,,f_{D}:\mathbb{V}_{D}\to\mathbb{W}_{D}\,.$

Here $\bigotimes_{d=1}^{D}f_{d}$ refers to the unique linear function
$\mathbb{V}\to\mathbb{W}$ such that

$\left(\bigotimes_{d=1}^{D}f_{d}\right)\circ\bigotimes_{\mathbb{V}}=\bigotimes_{\mathbb{W}}\circ(f_{1}\times\cdots\times
f_{D})$

which exists since both $\mathbb{V}$ and $\mathbb{W}$ satisfy the universal
property of the $D$-fold tensor product, cf. the discussion section A.10.1
above. It can be thought of as the unique linear function
$\mathbb{V}\to\mathbb{W}$ corresponding to the multilinear function
$\bigotimes_{\mathbb{W}}\circ(f_{1}\times\cdots\times f_{D})$ which is
compatible with the tensor product structure on $\mathbb{V}$ (which is
encapsulated via $\bigotimes_{\mathbb{V}}$).

In this review the standard example will always be functions of the form:

$\bigotimes_{d=1}^{D}f_{d}:\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\to\mathbb{R}^{[N_{\scaleto{[D]\mathstrut}{0.5em}}]}\,,\quad\bigotimes_{d=1}^{D}f_{d}:\mathbf{v}_{1}\otimes\cdots\otimes\mathbf{v}_{D}\mapsto
f_{1}(\mathbf{v}_{1})\otimes\cdots\otimes f_{D}(\mathbf{v}_{D})\,,$

where it suffices to define the function only for elementary tensors due to
the universal property of the tensor product, and where $\otimes$ refers to
the Segre outer product; compare this with section 6.2.

Tensor spaces $\mathbb{V}$, $\mathbb{W}$ are isomorphic (as $D$-factorized
tensor spaces) if there are morphisms in
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$:

$f:\mathbb{V}\to\mathbb{W}\,,\quad g:\mathbb{W}\to\mathbb{V}\,,$

such that $g\circ f=\operatorname{Id}_{\mathbb{V}}$ and $f\circ
g=\operatorname{Id}_{\mathbb{W}}$.

###  A.10.4 Remarks about the Category of Finite-Dimensional Real
$D$-Factorized Tensor Spaces

Analogous to $\bm{\mathsf{MltIdx}}$ and $\bm{\mathsf{MltIdx}}_{1}$, both the
objects and morphisms of $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{1}$ and
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$ are essentially the same, making both
categories for all intents and purposes essentially the same. Thus, from now
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{1}$ will be redefined so that
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{1}=\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$.

Also analogous to the multi-index categories, for any $O>D$ one also always
has $\binom{O-1}{D-1}$ distinct forgetful functors from
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{O}$ to
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$, with each distinct forgetful
functor corresponding to a different contiguous partition of the $O$ vector
spaces $\mathbb{V}_{1},\dots,\mathbb{V}_{O}$.

As an example, consider the case when $O=3$, $D=2$,
$\mathbb{V}=\mathbb{R}^{[24]\times[24]}$,
$\bigotimes_{\mathbb{V}}=\otimes_{\operatorname{Kr}}$ (the Kronecker matrix
product), $\mathbb{V}_{1}=\mathbb{R}^{[4]\times[2]}$,
$\mathbb{V}_{2}=\mathbb{R}^{[3]\times[3]}$, and
$\mathbb{V}_{3}=\mathbb{R}^{[2]\times[4]}$. Then
$\mathscr{F}_{12|3}(\mathbb{V})$ corresponds to
$\mathbb{V}_{1}=\mathbb{R}^{[12]\times[6]}$ and
$\mathbb{V}_{2}=\mathbb{R}^{[2]\times[4]}$, while
$\mathscr{F}_{1|23}(\mathbb{V})$ corresponds to
$\mathbb{V}_{1}=\mathbb{R}^{[4]\times[2]}$ and
$\mathbb{V}_{2}=\mathbb{R}^{[6]\times[12]}$.

Also, as before, any morphism in $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{O}$
between $\mathbb{V}$ and $\mathbb{W}$ will also be a morphism in
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$ between
$\mathscr{F}_{Q}(\mathbb{V})$ and $\mathscr{F}_{Q}(\mathbb{W})$ for any
contiguous partition $Q$ of $[O]$ into $D$ blocks, but there will also always
be morphisms between $\mathscr{F}_{Q}(\mathbb{V})$ and
$\mathscr{F}_{Q}(\mathbb{W})$ in $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$
which are not morphisms in $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{O}$.
Preserving $O$-fold tensor product structure is sufficient to preserve any
coarser $D$-fold tensor product structure, but preserving $D$-fold tensor
product structure is not enough to guarantee also preserving $O$-fold tensor
product structure for $O>D$.

Note that, for each $D\geq 1$, at least as long as the $D$-fold tensor product
functor is defined in any of the naturally isomorphic ways described above in
section A.10.1, the category $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$ is
equivalent to the image of the $D$-fold tensor product functor:

$\bigotimes_{d=1}^{D}:\underbrace{\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}\times\cdots\times\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}}_{D\text{
times}}\to\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}\,,$

in the sense that every object of $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$
is isomorphic as a $D$-factorized tensor space (isomorphic in
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$) to an object
$\bigotimes_{d=1}^{D}\mathbb{V}_{d}$ in the image of the $D$-fold tensor
product functor $\bigotimes_{d=1}^{D}$.

Note also that, by choosing coordinates, every object of
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$ is isomorphic as a $D$-factorized
tensor space to $\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$ for some
multi-index set $[M_{\scaleto{[D]\mathstrut}{0.5em}}]$ of order $D$. More
specifically, given an object $\mathbb{T}$ of
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$ which has the $D$-tuple of vector
spaces $(\mathbb{V}_{1},\dots,\mathbb{V}_{D})$ as its “tensor factors”, and if
$M_{d}\overset{\text{def}}{=}\dim(\mathbb{V}_{d})$ for all $d\in[D]$, then the
aforementioned $[M_{\scaleto{[D]\mathstrut}{0.5em}}]$ equals
$\bigtimes_{d=1}^{D}[M_{d}]$. One construction of such an isomorphism is
described in appendix A.6.2.

Thus, even when considering $D$-factorized tensor spaces and $D$-fold tensor
product structure, there is still is no loss of generality in restricting
attention only to coordinate spaces
$\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$ with the Segre outer
product as their tensor product, analogous to the situation for vector spaces.

##  Appendix A.11 Tensor Shape Functors

The following sections describe the interpretation of the dual tensor shape
functor as well as how to relate general linear functions to those in the
image of $\mathfrak{F}$.

###  A.11.1 Interpretation of Dual Tensor Shape Functor

By construction, the tensor shape functor
$\bm{\mathsf{MltIdx}}\to\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$ can be
considered the restriction of a free functor
$\mathfrak{F}:\mathsf{Fin}\mathsf{Set}\to\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$,
since
$\mathfrak{F}([M_{\scaleto{[O]\mathstrut}{0.5em}}])=\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$
satisfies the universal property of the free real vector space generated by
$[M_{\scaleto{[O]\mathstrut}{0.5em}}]$, cf. endnote LABEL:free-functor-endnote
of section 4 or section 3.1.2 of [129].

Any functor has a corresponding opposite functor which behaves the same on
objects and morphisms as the original functor. The tensor shape functor in
particular has an opposite functor:

$\mathfrak{F}^{\operatorname{op}}:\bm{\mathsf{MltIdx}}^{\operatorname{op}}\to\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}^{\operatorname{op}}\,.$

The universal property of the free real vector space implies that, for any
finite-dimensional real vector space $\mathbb{V}$, the following functors
$\bm{\mathsf{MltIdx}}^{\operatorname{op}}\to\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$
are naturally equivalent:

$\operatorname{Hom}_{\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}}(\mathfrak{F}^{op}(\cdot),\mathbb{V})\cong\operatorname{Hom}_{\mathsf{Set}}(\cdot,\mathscr{F}(\mathbb{V}))=\operatorname{Hom}_{\mathsf{Set}}(\cdot,\mathbb{V})\,,$

where $\mathscr{F}$ denotes the forgetful functor
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}\to\mathsf{Set}$. Note that
$\operatorname{Hom}_{\mathsf{Set}}([M_{\scaleto{[O]\mathstrut}{0.5em}}],\mathbb{V})\overset{\text{def}}{=}\mathbb{V}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$
is a finite-dimensional vector space (for any multi-index set
$[M_{\scaleto{[O]\mathstrut}{0.5em}}]$ and finite-dimensional real vector
space $\mathbb{V}$, where addition and scalar multiplication are defined
“pointwise” for the functions, and the dimension as a real vector space is
equal to the product of the cardinality of
$[M_{\scaleto{[O]\mathstrut}{0.5em}}]$ and the dimension of $\mathbb{V}$ as a
real vector space, i.e.
$\dim(\mathbb{V}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]})=|[M_{\scaleto{[O]\mathstrut}{0.5em}}]|\dim(\mathbb{V})$).

When $\mathbb{V}=\mathbb{R}$, this gives the following natural equivalence of
functors
$\bm{\mathsf{MltIdx}}^{\operatorname{op}}\to\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$:

$\operatorname{Hom}_{\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}}(\mathfrak{F}^{op}(\cdot),\mathbb{R})\cong\operatorname{Hom}_{\mathsf{Set}}(\cdot,\mathbb{R})\,.$

By definition, the dual tensor shape functor
$\mathfrak{F}^{\ast}:\bm{\mathsf{MltIdx}}^{\operatorname{op}}\to\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$
is the same as the functor in the right-hand side of the natural equivalence
written above:

$\mathfrak{F}^{\ast}\overset{\text{def}}{=}\operatorname{Hom}_{\mathsf{Set}}(\cdot,\mathbb{R})\,,$

so the above equivalence of functors also implies the following:

$\operatorname{Hom}_{\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}}(\mathfrak{F}^{op}(\cdot),\mathbb{R})\cong\mathfrak{F}^{\ast}\,.$

Similarly, definitionally for the dual functor
$\mathfrak{D}:\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}^{op}\to\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$
one has that:

$\mathfrak{D}\overset{\text{def}}{=}\operatorname{Hom}_{\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}}(\cdot,\mathbb{R})\,.$

Therefore one can rewrite left-hand side of the above natural equivalence:

$\operatorname{Hom}_{\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}}(\mathfrak{F}^{op}(\cdot),\mathbb{R})=\operatorname{Hom}_{\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}}(\cdot,\mathbb{R})\circ\mathfrak{F}^{\operatorname{op}}=\mathfrak{D}\circ\mathfrak{F}^{\operatorname{op}}\,.$

Combing the above, the universal property of the free vector space finally
leds to the equivalence:

$\mathfrak{D}\circ\mathfrak{F}^{\operatorname{op}}\cong\mathfrak{F}^{\ast}\,,$

i.e. the dual tensor shape functor $\mathfrak{F}^{\ast}$ is a naturally
equivalent functor to the functor resulting from composing the opposite
functor $\mathfrak{F}^{\operatorname{op}}$ of the tensor shape functor
$\mathfrak{F}$ with the dual functor $\mathfrak{D}$. This is the justification
for the name “dual tensor shape functor” and also explains why
$\mathfrak{F}^{\ast}(f)$ always corresponds to the adjoint linear
transformation of $\mathfrak{F}(f)$ for any morphism $f$ in
$\bm{\mathsf{MltIdx}}$.

###  A.11.2 Combinatorial Linear Algebra

The sub-category of $\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$ which is the
image122122122 That the image of a functor in this case is a bona fide
subcategory, i.e. closed under composition of morphisms, can be derived as a
corollary of the fact that the functor is injective on objects.  of the tensor
shape functor $\mathfrak{F}$ can be thought of as a subset of (multi)linear
algebra which is secretly entirely combinatorial in nature, in the sense that
all problems can be reduced to problems about multi-index sets and functions
between them via the tensor shape functor. Such problems include (but are not
limited to) permuting modes of a tensor and reshaping a tensor. The objects of
this category are all of the coordinate spaces, and the morphisms are a
heavily restricted subset of the possible linear functions between them,
namely linear functions of the form $\mathfrak{F}(f)$ for some function
between multi-index sets $f$.

However, it is possible to use this very limited subset of linear functions to
recover all non-zero linear functions123123123Which is the same thing as all
linear functions with rank greater than or equal to one. by combining them
with changes of basis124124124In more algebraic terms: linear automorphisms,
or linear self-isomorphisms, of vector spaces. of the coordinate spaces.
Specifically, every non-zero linear function $\mathcal{L}$ between coordinate
spaces may be factorized as:

$\left(\mathbb{R}^{[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}\xrightarrow{\mathcal{L}}\mathbb{R}^{[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}\right)=\left(\mathbb{R}^{[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}\xrightarrow{\mathscr{A}_{S}}\mathbb{R}^{[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}\xrightarrow{\mathscr{C}}\mathbb{R}^{[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}\xrightarrow{\mathscr{A}_{T}}\mathbb{R}^{[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}\right)\,,$

where $\mathscr{A}_{S}$ is a change of basis for
$\mathbb{R}^{[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}$ (i.e. an automorphism
of the source vector space), $\mathscr{C}$ is equal to $\mathfrak{F}(f)$ for
some function
$f:[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\to[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]$
(i.e. a “combinatorial linear map”), and $\mathscr{A}_{T}$ is a change of
basis for $\mathbb{R}^{[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$ (i.e. an
automorphism of the target space).

The general idea is always to choose a basis $\mathscr{B}_{S}$ for
$\mathbb{R}^{[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}$ and a basis
$\mathscr{B}_{T}$ for $\mathbb{R}^{[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$
such that the linear function $\mathcal{L}$ sends every element of
$\mathscr{B}_{S}$ either to an element of $\mathscr{B}_{T}$ or to zero. (While
for any basis of $\mathbb{R}^{[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}$ it is
true that $\mathcal{L}$ will send it to a finite subset of
$\mathbb{R}^{[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$, for a general basis
of $\mathbb{R}^{[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}$ it need not be the
case that this subset of vectors is linearly independent excluding the zero
vector.) That such a choice can always be made can be seen, for example, as
consequence of the existence of a singular value
decompositionHendnotepage.6linkHendnote.6ff6macro:-¿“label –vectorize-endnote˝
In the case that any of $“M [O˙1]$ or $“N $ are “proper multi-index sets”,
i.e. of order $2$ or greater, then one has to apply the SVD to the linear
function $“L ˙–“operatorname –vec˝˝ : “R ^“mathbf –M˝ “to “R ^“mathbf –N˝$,
where $“mathbf –M˝ “defequals “prod ˙–o˙1=1˝^–O˙1˝ M˙–o˙1˝$, $“mathbf –N˝
“defequals “prod ˙–o˙2=1˝^–O˙2˝ N˙–o˙2˝$, and “[ “L ˙–“operatorname –vec˝˝ =
“operatorname –vec˝˙N “circ “L “circ “operatorname –vec˝˙M^–-1˝ “, , “quad “L
= “operatorname –vec˝˙N^–-1˝ “circ “L ˙–“operatorname –vec˝˝ “circ
“operatorname –vec˝˙M “, , “] with $“operatorname –vec˝˙M$ and $“operatorname
–vec˝˙N$ any arbitrary isomorphisms $“R ^–“M [O˙1]˝ “to “R ^“mathbf –M˝$ and
$“R ^–“N ˝ “to “R ^“mathbf –N˝$ respectively.““ “par From the left singular
vectors $“mathbf –u˝$ of $“L ˙–“operatorname –vec˝˝$ one then gets “left
singular tensors” $“mathbfcal –U˝$ via $“mathbfcal –U˝ = “operatorname
–vec˝˙N^–-1˝(“mathbf –u˝)$, and from the right singular vectors $“mathbf –v˝$
of $“L ˙–“operatorname –vec˝˝$ one gets “right singular tensors” $“mathbfcal
–V˝$ via $“mathbfcal –V˝ = “operatorname –vec˝˙M^–-1˝(“mathbf –v˝)$. Then one
has $“L (“mathbfcal –U˝) = “sigma “mathbfcal –V˝$ for every pair of “singular
tensors” and their corresponding singular value, allowing one to factorize $“L
$ as a combinatorial linear map preceded and followed by changes of basis in
the manner described above. ““ “par Of course this will in general destroy
whatever additional structure $“L $ might have as a tensor product of other
linear functions $“L = “bigotimes ˙–d=1˝^D “L ˙d$ (clearly $D “le “min “–
O˙1,O˙2“˝ $). In that case it would be best to first separately perform SVD’s
in the manner described above for each of the linear functions $“L ˙d$ in the
tensor product factorization, and then define “singular tensor” basises for
each of $“R ^–“M [O˙1]˝$ and $“R ^–“N ˝$ via the tensor products of the
respective “singular tensors” of the linear functions $“L ˙d$, i.e.
$“mathbfcal –U˝ = “mathbfcal –U˝˙1 “otimes “cdots “otimes “mathbfcal –U˝˙D$
and $“mathbfcal –V˝ = “mathbfcal –V˝˙1 “otimes “cdots “otimes “mathbfcal
–V˝˙D$. Then it can be verified that “[ “L (“mathbfcal –V˝) = “L ˙1(“mathbfcal
–V˝˙1) “otimes “cdots “otimes “L ˙D(“mathbfcal –V˝˙D) = “left ( “prod ˙–d=1˝^D
“sigma ˙d “right ) “mathbfcal –U˝˙1 “otimes “cdots “otimes “mathbfcal –U˝˙D =
“left ( “prod ˙–d=1˝^D “sigma ˙d “right ) “mathbfcal –U˝ “, , “] meaning that
this choice of basises for $“R ^–“M [O˙1]˝$ and $“R ^–“N ˝$ satisfies the
conditions above while also better reflecting the tensor product structure of
$“L $ than the result one would get by vectorizing $“R ^–“M ˝$ and $“R ^–“N
˝$. (Note that in general it will still be necessary to vectorize the domains
and codomains of the respective $“L ˙d$’s however.) . If the rank is at least
one, then a trick can be employed to make everything work when the null space
of $\mathcal{L}$ is non-zero (see section A.11.3).

However, the fact that the zero linear map can not factorize this way, and
that an ugly trick is required whenever the null space is non-zero, correctly
suggests that $\bm{\mathsf{MltIdx}}$ is not the “correct” category to which
one should attempt to reduce all multilinear algebra. The problem is that
$\bm{\mathsf{MltIdx}}$ does not have any “zero object” and consequently no
notions of “zero morphisms” or “sending an element to zero”, whereas
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$ does. Thus these aspects of
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$ cannot be completely or directly
captured using $\bm{\mathsf{MltIdx}}$ and the tensor shape functor
$\mathfrak{F}$, even after changing basises.

This can be fairly easily rectified however by replacing
$\bm{\mathsf{MltIdx}}$ with the category of pointed multi-index
sets125125125The idea is analogous to the “four subspace” diagrams of Gilbert
Strang (when the multi-index sets are index sets).
$\bm{\mathsf{MltIdx}}^{\star}$, i.e. multi-index sets with an extra
“point”/element $\star$ appended to represent a notion of “zero object”, and
functions
$f:[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\to[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]$
limited to those which send the zero points $\star_{M}$ of their domains to
the zero points $\star_{N}$ of their codomains. This requires modifying the
Cartesian product operation to be the so-called “smash product” which ensures
that the product of pointed multi-index sets is still a pointed multi-index
sets (i.e. only has one zero point, not multiple), and some other relatively
straightforward changes. In particular, the category $\bm{\mathsf{MltIdx}}$
can be “embedded” as a subcategory of $\bm{\mathsf{MltIdx}}^{\star}$ in a
fairly obvious way, so there is no “loss of information”. The details of these
constructions are tedious and outside the scope of this review.

###  A.11.3 Non-Zero Null Space Trick

As explained in endnote A.14, without loss of generality one may assume that
the multi-index sets $[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]$ and
$[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]$ are index sets $[M]$ and $[N]$
respectively. Then one possibly way of choosing $\mathscr{B}_{S}$ and
$\mathscr{B}_{T}$ consists of making the right singular vectors the elements
of $\mathscr{B}_{S}$ and

$\tau_{i}\mathbf{u}_{i}\quad\text{where }\mathbf{u}_{i}\text{ is a left
singular vector and }\quad\tau_{i}=\begin{cases}\sigma_{i}&\sigma_{i}>0\\\
1&\sigma_{i}=0\end{cases}$

($\sigma_{i}$ denote singular values) the first elements of $\mathscr{B}_{T}$,
with $\mathscr{B}_{T}$ itself being an arbitrary completion of the
$\tau_{i}\mathbf{u}_{i}$ to a basis of $\mathbb{R}^{N}$. Then clearly
$\mathcal{L}$ sends every element of $\mathscr{B}_{S}$ either to an element of
$\mathscr{B}_{T}$ (when the corresponding $\sigma_{i}$ is non-zero) or to
$\mathbf{0}$.

Starting from this, the first attempt at choosing $\mathscr{A}_{S}$ sends the
elements of $\mathscr{B}_{S}$ to the standard basis of $\mathbb{R}^{M}$, and
the first attempt at choosing $\mathscr{A}_{T}$ sends the elements of
$\mathscr{B}_{T}$ to the standard basis of $\mathbb{R}^{M}$.

This works without any problems in the case that $\mathcal{L}$ is injective
(i.e. all of the singular values are non-zero, or equivalently the null space
is zero), since in this case $\mathscr{C}$ can be chosen to be
$\mathfrak{F}(f)$ for $f:[M]\to[N]$ which embeds $[M]$ into $[N]$. (Note that
$\mathcal{L}:\mathbb{R}^{M}\to\mathbb{R}^{N}$ being injective necessarily
implies that $M<N$.) However $\mathscr{A}_{S}$ must be modified when
$\mathcal{L}$ is not injective.

The idea of the trick is that “sending $k$ dimensions to zero” is the same as
“sending $k+1$ dimensions to the same (one) dimension”, and thus requires the
assumption that there is at least one dimension in the image of $\mathcal{L}$,
i.e. that $\mathcal{L}$ has rank of at least one.

Assume without loss of generality that
$\mathscr{A}_{S}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}$ sends a right
singular vector corresponding to a non-zero singular value to the first
standard basis vector, and that
$\mathscr{A}_{S}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}$ sends all of the
right singular vectors corresponding to zero singular values to the $2$nd
through $(k+1)$’st standard basis vectors. (If there are any remaining right
singular vectors corresponding to non-zero singular values,
$\mathscr{A}_{S}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}$ sends them to
the remaining standard basis vectors in a one-to-one manner126126126Basically
$\mathscr{A}_{S}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}$ is the same as
the function $\mathscr{A}_{S}$ described before up to reordering of the
elements of $\mathscr{B}_{S}$..) Then the goal is to find a second change of
basis $\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$ such that
$\mathscr{C}\circ\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$
will send the $2$nd through $(k+1)$’st standard basis vectors of
$\mathbb{R}^{M}$ to zero and the first standard basis vector of
$\mathbb{R}^{M}$ to the first standard basis vector of $\mathbb{R}^{N}$, when
$\mathscr{C}=\mathfrak{F}(f)$ for an $f:[M]\to[N]$ such that $f(m)=1$ for all
$1\leq m\leq k+1$, with the result that $\mathscr{C}$ sends all of the $k+1$
first standard basis vectors of $\mathbb{R}^{M}$ to the first standard basis
vector of $\mathbb{R}^{N}$. (Also it is assumed that $f(m)=m-k$ for all
$k+2\leq m\leq M$, so that $\mathscr{C}$ sends all of the $m$’th standard
basis vectors of $\mathbb{R}^{M}$ to the $(m-k)$’th standard basis vectors of
$\mathbb{R}^{N}$ for $k+2\leq m\leq M$, assuming that
$\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$ sends the $m$’th
standard basis vector of $\mathbb{R}^{M}$ to itself for all $k+2\leq m\leq M$.
This is much easier to understand by drawing a picture, e.g. when $k=1$.)

One possible choice of
$\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$ which
accomplishes this goal:

$\begin{array}[]{rcll}\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}:&\mathbf{e}_{1}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}&\mapsto&\displaystyle\frac{1}{k+1}\sum_{i=1}^{k+1}\mathbf{e}_{i}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\\\
\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}:&\mathbf{e}_{2}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}&\mapsto&\displaystyle\frac{1}{2}\mathbf{e}_{2}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}-\frac{1}{2k}\left(\sum_{i\not=2,1\leq
i\leq k+1}\mathbf{e}_{i}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\right)\\\
&\vdots&&\\\
\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}:&\mathbf{e}_{j}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}&\mapsto&\displaystyle\frac{1}{2}\mathbf{e}_{j}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}-\frac{1}{2k}\left(\sum_{i\not=j,1\leq
i\leq k+1}\mathbf{e}_{i}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\right)\\\
&\vdots&&\\\
\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}:&\mathbf{e}_{k+1}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}&\mapsto&\displaystyle\frac{1}{2}\mathbf{e}_{k+1}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}-\frac{1}{2k}\left(\sum_{i=1}^{k}\mathbf{e}_{i}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\right)\\\
\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}:&\mathbf{e}_{k+2}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}&\mapsto&\mathbf{e}_{k+2}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\\\
&\vdots&&\\\
\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}:&\mathbf{e}_{M}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}}&\mapsto&\mathbf{e}_{M}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}}\end{array}$

With such a choice of
$\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$, one then has
that $\mathcal{L}=\mathscr{A}_{T}\circ\mathscr{C}\circ\mathscr{A}_{S}$ where
$\mathscr{A}_{S}\overset{\text{def}}{=}\mathscr{A}_{S}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}\circ\mathscr{A}_{S}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}$.

##  Appendix A.12 Non-Commutative Polynomials

Informally speaking, the equivalence between tensors and non-commutative
polynomials can be understood by applying the following rules.

Rule 1: Only tensors of the exact same shape can be added. All other addition
is formal addition.

(One way the notion of formal addition can be made rigorous is by taking the
direct sum of vector spaces, in this case all of the
$\mathbb{R}^{[I_{\scaleto{[O]\mathstrut}{0.5em}}]}$ for all possible shapes
$[I_{\scaleto{[O]\mathstrut}{0.5em}}]$. Compare with e.g. [294].)

Example of non-formal addition:
$\mathbf{x}_{1}\otimes(\sqrt{2}\mathbf{y}_{3})\otimes\mathbf{z}_{2}+\pi\mathbf{x}_{1}\otimes\mathbf{y}_{3}\otimes\mathbf{z}_{5}=\mathbf{x}_{1}\otimes\mathbf{y}_{3}\otimes(\sqrt{2}\mathbf{z}_{2}+\pi\mathbf{z}_{5})$.

Example of formal addition:
$3\mathbf{x}_{2}\otimes\mathbf{y}_{4}\otimes\mathbf{z}_{1}+5\mathbf{v}_{8}\otimes\mathbf{w}_{3}$
can never be simplified further.

Rule 2: For each given shape
$[I_{\scaleto{[O]\mathstrut}{0.5em}}]=[I_{1}]\times[I_{2}]\times\dots\times[I_{O}]$,
choose a basis for $\mathbb{R}^{I_{o}}$ for every ${o\in[O]}$.

This leads to $O$ basises,
$\mathscr{B}_{1}=\\{\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{1},\dots,\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{I_{1}}\\}\subset\mathbb{R}^{I_{1}},\dots,\mathscr{B}_{O}=\\{\mathbf{x}^{\scriptsize\scaleto{(O)\mathstrut}{0.7em}}_{1},\dots,\mathbf{x}^{\scriptsize\scaleto{(O)\mathstrut}{0.7em}}_{I_{O}}\\}\subset\mathbb{R}^{I_{O}}$.

Example: For the given shape
$[I_{\scaleto{[O]\mathstrut}{0.5em}}]=[3]\times[2]$, one must choose a basis
$\mathscr{B}_{1}$ for $\mathbb{R}^{3}$ and a basis $\mathscr{B}_{2}$ for
$\mathbb{R}^{2}$. For example, one might choose
$\mathscr{B}_{1}=\\{\mathbf{e}_{1}^{\scriptsize\scaleto{(3)\mathstrut}{0.7em}},\mathbf{e}_{2}^{\scriptsize\scaleto{(3)\mathstrut}{0.7em}},\mathbf{e}_{3}^{\scriptsize\scaleto{({3})\mathstrut}{0.7em}}\\}$
and
$\mathscr{B}_{2}=\\{\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}},\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}}\\}$.

Rule 3: Every tensor of a given shape $[I_{\scaleto{[O]\mathstrut}{0.5em}}]$
must be written as a sum of elementary (i.e. rank-one) tensors such that each
factor belongs to the corresponding basis chosen in the step above.

It is always possible to write a tensor this way, and to do so uniquely, since
given basises
${\mathscr{B}_{1}=\\{\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{1},\dots,\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{I_{1}}\\}\subset\mathbb{R}^{I_{1}}}$,
$\dots,\mathscr{B}_{O}=\\{\mathbf{x}^{\scriptsize\scaleto{(O)\mathstrut}{0.7em}}_{1},\dots,\mathbf{x}^{\scriptsize\scaleto{(O)\mathstrut}{0.7em}}_{I_{O}}\\}\subset\mathbb{R}^{I_{O}}$,
the set:

$\mathscr{B}\overset{\text{def}}{=}\\{\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{i_{1}}\otimes\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{i_{2}}\otimes\cdots\otimes\mathbf{x}^{\scriptsize\scaleto{(O)\mathstrut}{0.7em}}_{i_{O}}:\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{i_{1}}\in\mathscr{B}_{1},\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{i_{2}}\in\mathscr{B}_{2},\dots,\mathbf{x}^{\scriptsize\scaleto{(O)\mathstrut}{0.7em}}_{i_{O}}\in\mathscr{B}_{O}\\}$

is always a basis for
$\mathbb{R}^{[I_{\scaleto{[O]\mathstrut}{0.5em}}]}$[129]127127127Specifically,
cf. Lemma 3.11 of [129].. (As a corollary, the unit tensors of
$\mathbb{R}^{[I_{\scaleto{[O]\mathstrut}{0.5em}}]}$ are always a basis.)

After applying rule 3, the chosen vectors
$\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{1},\dots,\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{I_{1}},\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{1},\dots,\mathbf{x}^{\scriptsize\scaleto{(O)\mathstrut}{0.7em}}_{I_{O}}$
are the variables of the non-commutative polynomials corresponding to tensors
of shape $[I_{\scaleto{[O]\mathstrut}{0.5em}}]$.

Example: repeating from the last example, for the given shape
$[I_{\scaleto{[O]\mathstrut}{0.5em}}]=[3]\times[2]$, choose
$\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{1}=\mathbf{e}_{1}^{\scriptsize\scaleto{(3)\mathstrut}{0.7em}}$,
$\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{2}=\mathbf{e}_{2}^{\scriptsize\scaleto{(3)\mathstrut}{0.7em}}$
,
$\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{3}=\mathbf{e}_{3}^{\scriptsize\scaleto{({3})\mathstrut}{0.7em}}$
,
$\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{1}=\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$
, and
$\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{2}=\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}}$.
Then the tensor

$\begin{bmatrix}3&2\\\ 6&0\\\ 0&4\end{bmatrix}$

is written as the non-commutative polynomial
$3\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{1}\otimes\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{1}+2\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{1}\otimes\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{2}+6\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{2}\otimes\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{1}+4\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{3}\otimes\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{2}$
or even more suggestively as
$3\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{1}\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{1}+2\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{1}\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{2}+6\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{2}\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{1}+4\mathbf{x}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{3}\mathbf{x}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{2}$.

The total number of variables can be reduced by applying:

Rule 4 (Optional): Always choose the same set to be the basis for
$\mathbb{R}^{I}$ every time $[I]$ occurs as one of the modes of a tensor of a
given shape.

Example: For $[I_{\scaleto{[O]\mathstrut}{0.5em}}]=[3]\times[3]\times[2]$, one
needs to choose a basis $\mathscr{B}_{1}$ for $\mathbb{R}^{3}$, another basis
$\mathscr{B}_{2}$ for $\mathbb{R}^{3}$, and a basis $\mathscr{B}_{3}$ for
$\mathbb{R}^{2}$. In general, $\mathscr{B}_{1}\not=\mathscr{B}_{2}$, and the
non-commutative polynomials of shape $[I_{\scaleto{[O]\mathstrut}{0.5em}}]$
are written in terms of $3+3+2=8$ non-commuting variables (which are the
distinct elements of $\mathscr{B}_{1},\mathscr{B}_{2},\mathscr{B}_{3}$). For
example, one could choose

$\mathscr{B}_{1}=\\{\mathbf{e}_{1}^{\scriptsize\scaleto{(3)\mathstrut}{0.7em}},\mathbf{e}_{2}^{\scriptsize\scaleto{(3)\mathstrut}{0.7em}},\mathbf{e}_{3}^{\scriptsize\scaleto{({3})\mathstrut}{0.7em}}\\}\,,\quad\mathscr{B}_{2}=\\{(1,1,1),(1,1,0),(0,1,1)\\}\,,\quad\mathscr{B}_{3}=\\{\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}},\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}}\\}\,.$

Then the $8$ non-commuting variables are
$\mathbf{x}_{1}=(1,0,0),\mathbf{x}_{2}=(0,1,0),\mathbf{x}_{3}=(0,0,1)$,
${\mathbf{y}_{1}=(1,1,1)}$, $\mathbf{y}_{2}=(1,1,0),\mathbf{y}_{3}=(0,1,1)$,
$\mathbf{z}_{1}=(1,0)$, and $\mathbf{z}_{2}=(0,1)$.

However, if one chooses the same basis for $\mathbb{R}^{3}$ twice, i.e.
$\mathscr{B}_{1}=\mathscr{B}_{2}$, then the number of non-commuting variables
becomes $3+2=5$ (the distinct elements of $\mathscr{B}_{1}=\mathscr{B}_{2}$
and $\mathscr{B}_{3}$). For example, one could choose
$\mathscr{B}_{1}=\\{\mathbf{e}_{1}^{\scriptsize\scaleto{(3)\mathstrut}{0.7em}},\mathbf{e}_{2}^{\scriptsize\scaleto{(3)\mathstrut}{0.7em}},\mathbf{e}_{3}^{\scriptsize\scaleto{({3})\mathstrut}{0.7em}}\\}=\mathscr{B}_{2}$
and
$\mathscr{B}_{3}=\\{\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}},\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}}\\}$.
Then the $5$ non-commuting variables are
$\mathbf{v}_{1}=(1,0,0),\mathbf{v}_{2}=(0,1,0),\mathbf{v}_{3}=(0,0,1)$,$\mathbf{w}_{1}=(1,0)$,
and $\mathbf{w}_{2}=(0,1)$.

In particular, when rule 4 is observed, the same variables are used for every
mode of cubical tensors, which makes them easier to work with as non-
commutative polynomials.

Example: Consider when
$[I_{\scaleto{[O]\mathstrut}{0.5em}}]=[2]\times[2]\times[2]$. When rule 4 is
not observed, there could potentially be up to $6$ distinct non-commuting
variables just for tensors of shape $[I_{\scaleto{[O]\mathstrut}{0.5em}}]$
alone, not to mention tensors of other shapes. But when rule $4$ is observed,
one only has $2$ non-commuting variables. Moreover, those $2$ non-commuting
variables would also be the same as the $2$ non-commuting variables used for
tensors of shape $[2]\times[2]$ or of shape $[2]\times[2]\times[2]\times[2]$,
as well as the same as $2$ of the $5$ non-commuting variables used to describe
tensors of shape $[3]\times[3]\times[2]$ and $[3]\times[2]$.

Rule 5 (Optional): Only consider cubical tensors, and of those only cubical
tensors whose modes all have the same dimension, regardless of their order.

Example: If one restricts to tensors of shape $[2]$, $[2]\times[2]$,
$[2]\times[2]\times[2]$, $[2]\times[2]\times[2]\times[2]$, $\dots$, and always
uses the same basis for $\mathbb{R}^{2}$, then regardless of the order of the
tensor, the number of non-commuting variables is always $2$. Thus these
tensors correspond to bivariate non-commutative polynomials in a very
straightforward way128128128Strictly speaking one also needs to take care to
include formal sums with “tensors or order zero”, i.e. scalars or constants,
in order to be fully analogous with (commutative) polynomials.. Similarly,
tensors of shape
$[3],[3]\times[3],[3]\times[3]\times[3]$,$[3]\times[3]\times[3]\times[3]$,
$\dots$ can be made to correspond to trivariate non-commutative polynomials.

To go from the result of applying rule 5 to something isomorphic to regular,
commutative polynomials, one needs to do at least both of the following:

  * •

Restrict the subset of cubical tensors under consideration to only those which
are symmetric,

  * •

Use a different product which preserves the symmetric property, i.e. a
property under which symmetric tensors are closed.

This is because the regular tensor product does not preserve the property of
being a symmetric tensor, i.e. given two symmetric tensors, the value of their
tensor product need not be symmetric.

Example: Letting as always
$\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}},\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}}$
denote the standard basis of $\mathbb{R}^{2}$, then while
$\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}\otimes\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}$
and
$\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}}\otimes\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}}$
are both symmetric tensors, one also has that
$(\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}\otimes\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}})\otimes(\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}}\otimes\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}})=\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}\otimes\mathbf{e}_{1}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}\otimes\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}}\otimes\mathbf{e}_{2}^{\scriptsize\scaleto{({2})\mathstrut}{0.7em}}$
is not a symmetric tensor. Thus $\otimes$ did not preserve the property of
being a symmetric tensor.

For details about how to rectify this and go from the symmetric but non-
commutative polynomials above to regular, commutative polynomials129129129Or
in other words how to go from the tensor algebra generated by $\mathbb{R}^{I}$
for some fixed integer $I$ to the symmetric algebra generated by
$\mathbb{R}^{I}$ (which is isomorphic to the set of $I$-variate real-valued
polynomials)., consult for example [67].

Thus, in conclusion the dictionary between tensors and polynomials can be
extended roughly as:

Polynomials | Tensors
---|---
degree | order
monomial | elementary tensor
commutative | non-commutative
variable | basis element
number of variables | dimension
which and how many variables are available does not depend on degree or position within product of factors |  which and how many variables are available could depend on order or position within product of factors

##  Appendix A.13 Separable Kronecker Covariance Structure

Having a separable Kronecker covariance structure means that the covariance
matrix of the vectorized130130130The paper [190] uses the same unfolding
conventions as [168], which strongly suggests that the vectorization here is
the colexicographical one. However, the type of vectorization used is never
specified explicitly. error tensors can be written as a sequence of Kronecker
product factors:

$\begin{array}[]{rcl}\bm{\Sigma}_{O}\otimes_{\operatorname{Kr}}\bm{\Sigma}_{O-1}\otimes_{\operatorname{Kr}}\cdots\otimes_{\operatorname{Kr}}\bm{\Sigma}_{1}&=&\operatorname{Cov}(\operatorname{vec}_{\operatorname{col}}(\mathbfcal{E}_{i}))\\\
&=&\displaystyle\mathbb{E}\left[(\operatorname{vec}_{\operatorname{col}}(\mathbfcal{E}_{i})-\mathbb{E}[\operatorname{vec}_{\operatorname{col}}(\mathbfcal{E}_{i})])\otimes(\operatorname{vec}_{\operatorname{col}}(\mathbfcal{E}_{i})-\mathbb{E}[\operatorname{vec}_{\operatorname{col}}(\mathbfcal{E}_{i})])\right]\\\
&=&\mathbb{E}[\operatorname{vec}_{\operatorname{col}}(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])\otimes\operatorname{vec}_{\operatorname{col}}(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])]\,.\end{array}$

The last equation follows from the linearity of the colexicographical
vectorization function.

Represent each covariance matrix $\bm{\Sigma}_{o}$ in terms of its eigenvalue
decomposition:

$\bm{\Sigma}_{o}=\sum_{m_{o}=1}^{M_{o}}(\sqrt{\lambda_{o}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}}\mathbf{u}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}_{o})\otimes(\sqrt{\lambda_{o}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}}\mathbf{u}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}_{o})\overset{\text{def}}{=}\sum_{m_{o}=1}^{M_{o}}\mathbf{v}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}_{o}\otimes\mathbf{v}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}_{o}\,,$

where for each $m_{o}\in[M_{o}]$, one has that
$\lambda_{o}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}$ is the
$m_{o}$’th eigenvalue of $\bm{\Sigma}_{o}$ and is non-negative since each
$\bm{\Sigma}_{o}$ is PSD, that
$\mathbf{u}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}_{o}$ is the
$m_{o}$’th eigenvector of $\bm{\Sigma}_{o}$, and by definition
${\mathbf{v}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}_{o}=\sqrt{\lambda_{o}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}}\mathbf{u}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}_{o}.}$
Then the Kronecker product of the $\bm{\Sigma}_{o}$’s is equal to the
following matricization:

$\bm{\Sigma}_{O}\otimes_{\operatorname{Kr}}\bm{\Sigma}_{O-1}\otimes_{\operatorname{Kr}}\cdots\otimes_{\operatorname{Kr}}\bm{\Sigma}_{1}=\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}\operatorname{vec}_{\operatorname{col}}(\mathbf{v}^{\scriptsize\scaleto{(m_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(m_{O})\mathstrut}{0.7em}}_{O})\otimes\operatorname{vec}_{\operatorname{col}}(\mathbf{v}^{\scriptsize\scaleto{(m_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(m_{O})\mathstrut}{0.7em}}_{O})\,.$

Applying the linear transformation inverse to the matricization, the above
means that for $\mathbfcal{E}_{i}$:

$\mathbb{E}[(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])\otimes(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])]=\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}(\mathbf{v}^{\scriptsize\scaleto{(m_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(m_{O})\mathstrut}{0.7em}}_{O})\otimes(\mathbf{v}^{\scriptsize\scaleto{(m_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(m_{O})\mathstrut}{0.7em}}_{O})\,,$

whenever the residuals $\mathbfcal{E}_{i}$ have a separable Kronecker
covariance structure. In particular, if one writes
$\bm{\Sigma}=\bm{\Sigma}_{O}\otimes_{\operatorname{Kr}}\cdots\otimes_{\operatorname{Kr}}\bm{\Sigma}_{1}$,
then the above equations imply131131131Since the product of two matrices is
the sum of the outer products of the columns of the first matrix and the rows
of the second matrix, and the equation below gives a standard column-wise
outer product decomposition. the following:

$\begin{array}[]{rcl}\bm{\Sigma}^{1/2}&=&\displaystyle\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}\operatorname{vec}_{\operatorname{col}}(\mathbf{v}^{\scriptsize\scaleto{(m_{1})\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(m_{O})\mathstrut}{0.7em}}_{O})\otimes\operatorname{vec}_{\operatorname{col}}(\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{O}}^{\scriptsize\scaleto{({M_{O}})\mathstrut}{0.7em}})\vskip
3.0pt plus 1.0pt minus 1.0pt\\\ &=&\vskip 3.0pt plus 1.0pt minus
1.0pt\bm{\Sigma}_{O}^{1/2}\otimes_{\operatorname{Kr}}\bm{\Sigma}_{O-1}^{1/2}\otimes_{\operatorname{Kr}}\cdots\otimes_{\operatorname{Kr}}\bm{\Sigma}_{1}^{1/2}\,,\end{array}$

where for each $o\in[O]$, the square root has been defined using the
eigenvalue decomposition as:

$\bm{\Sigma}_{o}^{1/2}=\sum_{m_{o}=1}^{M_{o}}\mathbf{v}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}_{o}\otimes\mathbf{e}_{m_{o}}^{\scriptsize\scaleto{(M_{o})\mathstrut}{0.7em}}\overset{\text{def}}{=}\sum_{m_{o}=1}^{M_{o}}\sqrt{\lambda_{o}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}}\mathbf{u}^{\scriptsize\scaleto{(m_{o})\mathstrut}{0.7em}}_{o}\otimes\mathbf{e}_{m_{o}}^{\scriptsize\scaleto{(M_{o})\mathstrut}{0.7em}}\,.$

Thus one finds the following relationship implied by separable Kronecker
covariance structure:

$\mathbb{E}[(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])\otimes(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])]=\displaystyle\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}\left(\mathbfcal{I}_{m_{\scaleto{[O]\mathstrut}{0.5em}}}\bullet_{1}\bm{\Sigma}_{1}^{1/2}\cdots\bullet_{O}\bm{\Sigma}_{O}^{1/2}\right)\otimes\left(\mathbfcal{I}_{m_{\scaleto{[O]\mathstrut}{0.5em}}}\bullet_{1}\bm{\Sigma}_{1}^{1/2}\cdots\bullet_{O}\bm{\Sigma}_{O}^{1/2}\right)\,.$

Define temporarily for convenience
$\mathcal{S}:\mathbfcal{T}\mapsto\mathbfcal{T}\bullet_{1}\bm{\Sigma}_{1}^{1/2}\cdots\bullet_{O}\bm{\Sigma}_{O}^{1/2}$,
which is a linear function. Therefore its tensor product with itself, namely
${\mathcal{S}\otimes\mathcal{S}:\mathbfcal{T}_{1}\otimes\mathbfcal{T}_{2}\mapsto\mathcal{S}(\mathbfcal{T}_{1})\otimes\mathcal{S}(\mathbfcal{T}_{2})}$,
is also a linear function. The terms in the above summation are all of the
form
${\mathcal{S}(\mathbfcal{I}_{m_{\scaleto{[O]\mathstrut}{0.5em}}})\otimes\mathcal{S}(\mathbfcal{I}_{m_{\scaleto{[O]\mathstrut}{0.5em}}})}=(\mathcal{S}\otimes\mathcal{S})(\mathbfcal{I}_{m_{\scaleto{[O]\mathstrut}{0.5em}}}\otimes\mathbfcal{I}_{m_{\scaleto{[O]\mathstrut}{0.5em}}})$.
Therefore the right hand side of the above equation also equals:

$(\mathcal{S}\otimes\mathcal{S})\left(\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}\mathbfcal{I}_{m_{\scaleto{[O]\mathstrut}{0.5em}}}\otimes\mathbfcal{I}_{m_{\scaleto{[O]\mathstrut}{0.5em}}}\right)\,.$

Note that the tensor
$\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}\mathbfcal{I}_{m_{\scaleto{[O]\mathstrut}{0.5em}}}\otimes\mathbfcal{I}_{m_{\scaleto{[O]\mathstrut}{0.5em}}}$
corresponds exactly to the covariance tensor of a random tensor
$\mathbfcal{U}_{i}$ whose entries are all uncorrelated and which all have unit
variance.

Thus the covariance tensor of $\mathbfcal{E}_{i}$ is the same as the
covariance tensor of the tensor:

$\mathbfcal{U}_{i}\bullet_{1}\bm{\Sigma}_{1}^{1/2}\cdots\bullet_{O}\bm{\Sigma}_{O}^{1/2}=\mathcal{S}(\mathbfcal{U}_{i})\,,$

where again $\mathbfcal{U}_{i}$ is any random tensor whose entries are
uncorrelated and have unit variance. The above calculations should be compared
with section 2.2 of [140], noting that $k,K$ in [140] correspond to $o,O$
here, and in particular that the $\mathbf{A}_{k}$ of [140] essentially
correspond to the $\bm{\Sigma}_{o}^{1/2}$ here so that the notation
$\mathbf{Z}\times\mathbf{A}$ corresponds to $\mathcal{S}(\mathbfcal{U}_{i})$.
The definition of covariance tensor of the random tensor $\mathbfcal{E}_{i}$
(corresponding to $\mathbf{Y}$ of [140]) in [140] is a permutation of the
modes of
$\mathbb{E}[(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])\otimes(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])]$,
i.e. essentially
$\mathbb{E}[(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])\otimes_{Z}(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])]$
with $\otimes_{Z}$ denoting a Zehfuss product for the only possible contiguous
partition of the modes of ${\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}]}$
into $O$ blocks, see appendix A.9.

The definition of covariance tensor from [140] is also such that the following
is true:

$\begin{array}[]{rcl}\mathbb{E}[(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])\otimes_{Z}(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])]&=&\displaystyle\sum_{m_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]}(\mathbf{v}^{\scriptsize\scaleto{(m_{1})\mathstrut}{0.7em}}_{1}\otimes\mathbf{v}^{\scriptsize\scaleto{(m_{1})\mathstrut}{0.7em}}_{1})\otimes\cdots\otimes(\mathbf{v}^{\scriptsize\scaleto{(m_{O})\mathstrut}{0.7em}}_{O}\otimes\mathbf{v}^{\scriptsize\scaleto{(m_{O})\mathstrut}{0.7em}}_{O})\vskip
3.0pt plus 1.0pt minus 1.0pt\\\
&=&\displaystyle\left(\sum_{m_{1}=1}^{{M_{1}}}\mathbf{v}^{\scriptsize\scaleto{(m_{1})\mathstrut}{0.7em}}_{1}\otimes\mathbf{v}^{\scriptsize\scaleto{(m_{1})\mathstrut}{0.7em}}_{1}\right)\otimes\cdots\otimes\left(\sum_{m_{O}=1}^{{M_{O}}}\mathbf{v}^{\scriptsize\scaleto{(m_{O})\mathstrut}{0.7em}}_{O}\otimes\mathbf{v}^{\scriptsize\scaleto{(m_{O})\mathstrut}{0.7em}}_{O}\right)\vskip
3.0pt plus 1.0pt minus 1.0pt\\\
&=&\bm{\Sigma}_{1}\otimes\bm{\Sigma}_{2}\otimes\cdots\otimes\bm{\Sigma}_{O}\,.\end{array}$

Thus the intuition is that, for a separable Kronecker covariance structure,
$\bm{\Sigma}_{1}$ is the “covariance matrix for the first mode”,
$\bm{\Sigma}_{2}$ is the “covariance matrix for the second mode”, and so on.

In the case that all of the $\bm{\Sigma}_{o}$’s are positive definite, or
equivalently in the case that all of the $\bm{\Sigma}^{1/2}$’s are invertible,
one can also say that the random tensor with the following definition:

$\mathbfcal{U}_{i}\overset{\text{def}}{=}\mathbfcal{E}_{i}\bullet_{1}\bm{\Sigma}^{-1/2}\cdots\bullet_{O}\bm{\Sigma}^{-1/2}=\mathcal{S}^{-1}(\mathbfcal{E}_{i})$

has all of its entries uncorrelated from one another and with unit variance.

When the residuals $\mathbfcal{E}_{i}$ have what is called a multilinear
normal distribution[225], one can go further in making statements about the
distribution of $\mathbfcal{E}_{i}$ based on its covariance tensor alone. More
specifically, as is partially explained in [225], under such an assumption one
can decompose the vectorized form of the tensor,
$\operatorname{vec}_{\operatorname{col}}(\mathbfcal{E}_{i})$, via the
following relationship:

$\begin{array}[]{rcl}\operatorname{vec}_{\operatorname{col}}(\mathbfcal{E}_{i}-\mathbb{E}[\mathbfcal{E}_{i}])&=&\\\
\operatorname{vec}_{\operatorname{col}}(\mathbfcal{E}_{i})-\operatorname{vec}_{\operatorname{col}}(\mathbb{E}[\mathbfcal{E}_{i}])&=&\bm{\Sigma}^{1/2}\operatorname{vec}_{\operatorname{col}}(\mathbfcal{U}_{i})\\\
&=&(\bm{\Sigma}_{O}^{1/2}\otimes_{\operatorname{Kr}}\bm{\Sigma}_{O-1}^{1/2}\otimes_{\operatorname{Kr}}\cdots\otimes_{\operatorname{Kr}}\bm{\Sigma}_{1}^{1/2})\operatorname{vec}_{\operatorname{col}}(\mathbfcal{U}_{i})\\\
&=&\operatorname{vec}_{\operatorname{col}}(\mathbfcal{U}_{i}\bullet_{1}\bm{\Sigma}_{1}^{1/2}\cdots\bullet_{O}\bm{\Sigma}_{O}^{1/2})\,,\end{array}$

where by definition $\mathbfcal{U}_{i}$ is a tensor whose entries consist of
independent and identically distributed ${\mathscr{N}(0,1)}$ random variables,
and where the identity [126][166] from 6.5.1 giving the colexicographically
vectorized form of the Tucker decomposition was used. Since
$\operatorname{vec}_{\operatorname{col}}$ is an invertible and linear
function, the above relationship means that assuming the residuals have a
multilinear normal distribution is equivalent to implicitly defining them in
terms of a Tucker decomposition:

$\mathbfcal{E}_{i}=\mathbb{E}[\mathbfcal{E}_{i}]+\mathbfcal{U}_{i}\bullet_{1}\bm{\Sigma}^{1/2}_{1}\cdots\bullet_{O}\bm{\Sigma}_{O}^{1/2}\,.$

Note in particular that a commonly assumed case, where the entries of
$\mathbfcal{E}_{i}$ are all i.i.d. standard normal random variables, is (a
rather uninteresting) special case of the above. The above gives a sense in
which the statement that the matrix $\bm{\Sigma}_{o}$ is “the covariance
matrix for the $o$-th mode of $\mathbfcal{E}_{i}$” can be made precise for a
multilinear normal distribution.

##  Appendix A.14 Relationships Between Eigen- and Singular Vectors and Values
of Tensors

For a general non-cubical tensor, only singular values and singular vectors
exist, so there is no need to wonder about their possible relationships with
eigenpairs. However, starting already with arbitrary cubical tensors, one
might begin to wonder whether certain relationships are guaranteed to exist.
For example, the definition of eigenpair seems similar to the definition of
singular value and vector tuple, and so one might wonder whether, given any
eigenvector of a cubical tensor, the corresponding tuple of $O$ copies of that
eigenvector is also a singular vector tuple. However, one needs to keep in
mind that the condition of being a mode-$o$ eigenvector for some $o\in[O]$ for
a general cubical tensor corresponds to nothing stronger than satisfying at
least one of the $O$ necessary conditions for the corresponding tuple of $O$
copies of that vector to be a singular vector tuple, much weaker than
satisfying all of them simultaneously with the same value of $\lambda$ or
$\sigma$. In order to have its $O$ copies be a singular vector tuple, the
given vector would need to not only be a mode-$o$ eigenvector simultaneously
for all $O$ modes of the tensor, it would also need to correspond to the same
eigenvalue for each mode. Since, as discussed previously, higher-order tensors
need not have the same eigenvalues for all modes, those tensors for which this
is the case cannot have any singular vector tuples consisting of $O$ copies
(up to signs) of the same vector, and thus their eigenpairs and singular
values and singular vector tuples will be unrelated.

Even when the various modes of a cubical tensor happen to have the same
eigenvalues (e.g. as in the matrix case), they need not necessarily have the
same eigenvectors, and even in this case any eigenvector which does not
correspond to all modes of the tensor has no chance of corresponding to a
singular vector tuple of $O$ copies (up to signs) of itself. It seems a direct
consequence from the definitions that a vector has $O$ copies of itself (up to
signs) as a singular vector tuple if and only if it is an eigenvector
simultaneously for all modes corresponding to the same eigenvalue. In
particular a tensor has a singular vector tuple consisting of $O$ copies (up
to signs) of the same vector if and only if it has such an eigenvector. Thus
the definitions of eigenpair and singular value and vector tuple for a cubical
tensor are not as similar as lazily glancing would suggest. One might still
wonder whether the remaining singular vector tuples with at least two distinct
(up to signs) vectors have any weaker relationships with the eigenvectors for
any of the modes, but already the case of matrices suggests that this is not
the case. However, when the cubical tensor is symmetric, every eigenvector
automatically satisfies all of the $O$ necessary conditions with the same
$\lambda$. So one can conclude that that every eigenpair of a symmetric tensor
corresponds (up to signs) to a singular value and a singular vector tuple
consisting of $O$ copies (up to signs) of the same vector. This leads to the
questions of whether all singular values and singular vector tuples of a
higher-order symmetric tensor can be so characterized, and if not, what can be
said about those singular vector tuples of a higher-order symmetric tensor
consisting of distinct (even up to signs) vectors.

Like the case of order-two tensors (matrices), the largest $\ell^{2}$-singular
value of a symmetric higher-order tensor equals the absolute value of its
largest-in-magnitude $\ell^{2}$-eigenvalue, and (up to signs) the
corresponding tuple of $\ell^{2}$-singular vectors is the same as the
corresponding $\ell^{2}$-eigenvector[136]. For symmetric matrices, the
analogous statements are also true for the smaller singular values and
smaller-in-magnitude eigenvalues. It is not immediately obvious whether this
is also true for higher-order tensors. This is because the variational problem
defining $\ell^{2}$-singular values for arbitrary tensors (not necessarily
symmetric) is related to the optimization problem defining the best rank-one
approximation in such a way that, if
$\mathbf{v}_{1}\otimes\cdots\otimes\mathbf{v}_{O}$ is the best rank-one
approximation, then the largest $\ell^{2}$-singular value
$\sigma_{1}=\prod_{o=1}^{O}||\mathbf{v}_{o}||_{2}$ and for each of the
corresponding $\ell^{2}$-singular vectors
$\mathbf{x}_{o}=\mathbf{v}_{o}/||\mathbf{v}_{o}||_{2}$[136][192]. (In fact
this relationship allows one to prove the NP-hardness of finding a best rank-
one approximation starting from the fact that finding the largest
$\ell^{2}$-singular value is NP-hard[136]. Moreover just like for matrices the
largest $\ell^{2}$-singular value equals, basically by definition, the
spectral norm of the tensor, a fact utilized in [297].) In the case that the
tensor is symmetric, a theorem by Banach guarantees that the corresponding
optimization problem for the largest-in-magnitude $\ell^{2}$-eigenvalue
coincides with the problem for finding the largest $\ell^{2}$-singular value
(cf. the statement about best rank-one Waring decompositions made in section
9.6).

One might therefore suppose that the analogous statements for smaller
$\ell^{2}$-singular values and smaller-in-magnitude $\ell^{2}$-eigenvalues are
not true for higher-order tensors. The method of proof used for the largest
$\ell^{2}$-singular value and largest-in-magnitude $\ell^{2}$-eigenvalue seems
to break down since, for higher-order tensors, best rank-$r$ and rank-$(r+1)$
approximations need not exist, and even if they do exist they are generally
unrelated to each other, unlike the matrix case. For the matrix case, one
would use the fact that, given a matrix $\mathbf{T}$ and its best rank-one
approximation $\mathbf{T}_{(1)}$, its best rank-two approximation would be
$\mathbf{T}_{(1)}+\mathbf{T}_{(2)}$, where $\mathbf{T}_{(2)}$ denotes the best
rank one approximation of $\mathbf{T}-\mathbf{T}_{(1)}$. For a given higher
order tensor $\mathbfcal{T}$, even though a best rank-one approximation
$\mathbfcal{T}_{(1)}$ to $\mathbfcal{T}$ and a best rank-one approximation
$\mathbfcal{T}_{(2)}$ to $\mathbfcal{T}-\mathbfcal{T}_{(1)}$ will exist, in
general $\mathbfcal{T}_{(1)}+\mathbfcal{T}_{(2)}$ is not the best rank-two
approximation to $\mathbfcal{T}$ and in fact such an approximation need not
even exist. On the other hand, even though
$\mathbfcal{T}_{(1)}+\mathbfcal{T}_{(2)}$ is not the best rank-two
approximation to $\mathbfcal{T}$, one might still hope that
$||\mathbfcal{T}_{(2)}||_{F}$ is the second largest $\ell^{2}$-singular value
and that the corresponding $\ell^{2}$-singular vector tuple is the same as the
tensor product factors in $\mathbfcal{T}_{(2)}$ up to scaling. As mentioned
before, in the case that $\mathbfcal{T}$ is symmetric, a theorem from Banach
guarantees that $\mathbfcal{T}_{(1)}$ will be symmetric, and thus that also
$\mathbfcal{T}-\mathbfcal{T}_{(1)}$ will be symmetric, which in turn by the
theorem implies that $\mathbfcal{T}_{(2)}$ will be symmetric, and so one might
hope in that case that $||\mathbfcal{T}_{(2)}||_{F}$ is also the absolute
value of the second largest-in-magnitude $\ell^{2}$-eigenvalue of
$\mathbfcal{T}$ and that the single repeated tensor product factor defining
$\mathbfcal{T}_{(2)}$ will be (up to scaling and signs) the corresponding
$\ell^{2}$-eigenvector.

However, if none of this is true, then one might also suppose that the
$\ell^{2}$-singular vector tuple corresponding to the second largest
$\ell^{2}$-singular value of a higher-order symmetric tensor could possibly
(even up to signs) consist of distinct vectors, rather than $O$ copies of a
related $\ell^{2}$-eigenvector. Even though it is fairly clear that the
eigenpairs of a higher-order symmetric tensor have to correspond to singular
tuples with $O$ copies (up to signs) of the eigenvector and the eigenvalue the
same (up to signs) as the singular value, the converse relationship is much
less clear except in the case of the $\ell^{2}$ definitions for the largest
$\ell^{2}$-singular value only. It is less straightforward to discern whether
every singular vector tuple of a higher-order symmetric tensor must consist of
$O$ copies of the same (up to signs) vector (recall that when a singular
vector tuple does, the repeated vector is automatically an eigenvector and the
singular value is up to signs automatically an eigenvalue).

f. This conflicts unfortunately with the common convention that random scalars
and random vectors be denoted with upper-case letters. Thus in this review the
reader must memorize whether the symbol denotes a random or a deterministic
quantity. One could be perverse and argue that this is a non-issue, inasmuch
as technically deterministic quantities are just degenerate random quantities
(i.e. corresponding to Dirac measures).

f. A full subcategory contains only a subset of the objects of its parent
category, yet all of the morphisms between those objects which exist in the
parent category. For example, the category of real vector spaces
$\mathsf{Vect}_{\mathbb{R}}$ is not a full subcategory of the category of sets
$\mathsf{Set}$, because the morphisms between vector spaces are restricted to
be linear functions in the category of real vector spaces
$\mathsf{Vect}_{\mathbb{R}}$, whereas in the parent category of sets
$\mathsf{Set}$ all possible arbitrary set functions between any two vector
spaces are also present. So while the function
${f:\mathbb{R}^{2}\to\mathbb{R}^{2}}$ given by $(x,y)\mapsto(x^{2},y^{2})$ is
in $\mathsf{Mor}_{\mathsf{Set}}(\mathbb{R}^{2},\mathbb{R}^{2})$, it is not in
$\mathsf{Mor}_{\mathsf{Vect}_{\mathbb{R}}}(\mathbb{R}^{2},\mathbb{R}^{2})$
since it is not a linear function.

f. Or technically speaking rather usually as the object of
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D+1}$ satisfying the universal
property of the tensor product for the $(D+1)$-tuple of vector spaces
${(\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]},\left(\mathbb{R}^{M_{1}}\right)^{\ast},\left(\mathbb{R}^{M_{2}}\right)^{\ast},\dots,\left(\mathbb{R}^{M_{D}}\right)^{\ast})}$,
or as the object of $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{O+D}$ satisfying
the universal property of the tensor product for the $(O+D)$-tuple of vector
spaces
${(\mathbb{R}^{N_{1}},\dots,\mathbb{R}^{N_{O}},\left(\mathbb{R}^{M_{1}}\right)^{\ast},\dots,\left(\mathbb{R}^{M_{D}}\right)^{\ast})}$.

f. Recall that for any two vector spaces $\mathbb{T}_{1}$, $\mathbb{T}_{2}$
which are isomorphic (as vector spaces) via the isomorphism
$\Upsilon:\mathbb{T}_{2}\to\mathbb{T}_{1}$, and for which $\mathbb{T}_{1}$
satisfies the universal property of the $D$-fold tensor product for vector
spaces $(\mathbb{V}_{1},\dots,\mathbb{V}_{D})$ using the tensor product
$\bigotimes_{\mathbb{T}_{1}}$, it is always the case that $\mathbb{T}_{2}$
then also satisfies the universal property of the $D$-fold tensor product for
$(\mathbb{V}_{1},\dots,\mathbb{V}_{D})$ using the tensor product
$\bigotimes_{\mathbb{T}_{2}}\overset{\text{def}}{=}\Upsilon^{-1}\circ\bigotimes_{\mathbb{T}_{1}}$.
Cf. endnote LABEL:tensor-universal-property-uniqueness of section 3.

f. Actually the notion of “free vector space” opens its own separate can of
worms, since it is also defined via a universal property which only specifies
a vector space up to isomorphism, and does not give priority to any particular
construction over another. The approach taken in section 3.1.2 of [129] for
constructing the free vector space is very similar to the usual construction
of taking the direct sum of the vector spaces $\mathbb{R}^{s}$ for every
element $s\in S$ for an arbitrary set $S$. This would seem to not open yet
another can of worms since the direct sum refers to a specific construction,
although that construction is of interest primarily because it satisfies the
universal property of the coproduct.

f. In the case that any of $[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]$ or
$[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]$ are “proper multi-index sets”, i.e.
of order $2$ or greater, then one has to apply the SVD to the linear function
$\mathcal{L}_{\operatorname{vec}}:\mathbb{R}^{\mathbf{M}}\to\mathbb{R}^{\mathbf{N}}$,
where $\mathbf{M}\overset{\text{def}}{=}\prod_{o_{1}=1}^{O_{1}}M_{o_{1}}$,
$\mathbf{N}\overset{\text{def}}{=}\prod_{o_{2}=1}^{O_{2}}N_{o_{2}}$, and

$\mathcal{L}_{\operatorname{vec}}=\operatorname{vec}_{N}\circ\mathcal{L}\circ\operatorname{vec}_{M}^{-1}\,,\quad\mathcal{L}=\operatorname{vec}_{N}^{-1}\circ\mathcal{L}_{\operatorname{vec}}\circ\operatorname{vec}_{M}\,,$

with $\operatorname{vec}_{M}$ and $\operatorname{vec}_{N}$ any arbitrary
isomorphisms
$\mathbb{R}^{[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}\to\mathbb{R}^{\mathbf{M}}$
and
$\mathbb{R}^{[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}\to\mathbb{R}^{\mathbf{N}}$
respectively.

From the left singular vectors $\mathbf{u}$ of
$\mathcal{L}_{\operatorname{vec}}$ one then gets “left singular tensors”
$\mathbfcal{U}$ via $\mathbfcal{U}=\operatorname{vec}_{N}^{-1}(\mathbf{u})$,
and from the right singular vectors $\mathbf{v}$ of
$\mathcal{L}_{\operatorname{vec}}$ one gets “right singular tensors”
$\mathbfcal{V}$ via $\mathbfcal{V}=\operatorname{vec}_{M}^{-1}(\mathbf{v})$.
Then one has $\mathcal{L}(\mathbfcal{U})=\sigma\mathbfcal{V}$ for every pair
of “singular tensors” and their corresponding singular value, allowing one to
factorize $\mathcal{L}$ as a combinatorial linear map preceded and followed by
changes of basis in the manner described above.

Of course this will in general destroy whatever additional structure
$\mathcal{L}$ might have as a tensor product of other linear functions
$\mathcal{L}=\bigotimes_{d=1}^{D}\mathcal{L}_{d}$ (clearly
$D\leq\min\\{O_{1},O_{2}\\}$). In that case it would be best to first
separately perform SVD’s in the manner described above for each of the linear
functions $\mathcal{L}_{d}$ in the tensor product factorization, and then
define “singular tensor” basises for each of
$\mathbb{R}^{[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}$ and
$\mathbb{R}^{[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$ via the tensor
products of the respective “singular tensors” of the linear functions
$\mathcal{L}_{d}$, i.e.
$\mathbfcal{U}=\mathbfcal{U}_{1}\otimes\cdots\otimes\mathbfcal{U}_{D}$ and
$\mathbfcal{V}=\mathbfcal{V}_{1}\otimes\cdots\otimes\mathbfcal{V}_{D}$. Then
it can be verified that

$\mathcal{L}(\mathbfcal{V})=\mathcal{L}_{1}(\mathbfcal{V}_{1})\otimes\cdots\otimes\mathcal{L}_{D}(\mathbfcal{V}_{D})=\left(\prod_{d=1}^{D}\sigma_{d}\right)\mathbfcal{U}_{1}\otimes\cdots\otimes\mathbfcal{U}_{D}=\left(\prod_{d=1}^{D}\sigma_{d}\right)\mathbfcal{U}\,,$

meaning that this choice of basises for
$\mathbb{R}^{[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}$ and
$\mathbb{R}^{[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$ satisfies the
conditions above while also better reflecting the tensor product structure of
$\mathcal{L}$ than the result one would get by vectorizing
$\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$ and
$\mathbb{R}^{[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$. (Note that in general
it will still be necessary to vectorize the domains and codomains of the
respective $\mathcal{L}_{d}$’s however.)