$\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$ is always chosen to be the
unit tensors, which are the tensor products of the standard basis vectors for
each mode. This has the effect that the standard isomorphism generated by this
basis is always the identity, rather than some other automorphism of
$\mathbb{R}^{M}$ or $\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$. (This
is perhaps somewhat tautological or circular inasmuch as the standard
isomorphism is defined in terms of the standard basis, but in any case it is
definitely easier for humans to reason with.) There is also no loss of
generality here, since to use another basis, one just needs to first apply the
automorphism sending that basis to the standard basis (this is usually
described as “using the change of basis formula”), and then follow the
discussion in this review as before. The specifics of this assumption are
discussed in much greater detail in appendix A.6.1 and appendix A.6.2.

  5. 5.

The chosen identification of/linear isomorphism between $\mathbb{R}^{M}$ and
its dual $(\mathbb{R}^{M})^{\ast}$, or between
$\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$ and its dual
$(\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]})^{\ast}$, will always be
the unique isomorphism which sends the standard basis to the dual basis of the
standard basis. Identifications of a vector space with its dual amount to
defining a non-degenerate bilinear product on the vector space. In this case,
the chosen identification amounts to the standard inner product (i.e. the
operation which returns the sum all of the entries of the entrywise a.k.a.
Hadamard product).

  6. 6.

Only contiguous set partitions will be used, rather than arbitrary ordered set
partitions. This is because all ordered set partitions (even when
distinguishing between partitions based on the order of the elements within
each block) can be generated from a contiguous set partition and a suitable
permutation. It is more straightforward to create arbitrary ordered set
partitions from the simpler building blocks of contiguous set partitions and
permutations. (Compare this convention with that used in [138] with regards to
“multipartite indices”.)

##  Appendix A.2 Partitions of Finite Sets

A set partition of $[N]$ is a set of subsets of $[N]$ such that (i) their
union is equal to $[N]$ and (ii) they are all disjoint, i.e. all of their
intersections with each other are empty[297]. Furthermore, a contiguous set
partition of $[N]$ is a set partition such that if $n_{1},n_{2}$ are in the
same “block”, i.e. element of the partition (which is a subset of $[N]$), then
$n_{1}\leq n_{3}\leq n_{2}$ implies that $n_{3}$ is also in the same block. An
ordered set partition is a set partition where the order of the blocks is
considered important, i.e. a “tuple” of subsets of $[N]$ rather than just a
set of subsets of $[N]$.

There is a bijection between surjective functions $[N]\to[B]$ and ordered set
partitions of $[N]$ into $B$ blocks. There is also a bijection between
monotone and surjective functions $[N]\to[B]$ and contiguous set
partitions898989In turn there is a bijection between contiguous set partitions
of $[N]$ into $B$ blocks and ordered (integer) partitions, also known as
compositions, of $N$ of length $B$, see e.g. [277] for definitions of these
terms. of $[N]$ into $B$ blocks.

Given contiguous set partitions $P_{1}$ of $[N]$ into $B_{1}$ blocks and $Q$
of $[B_{1}]$ into $B_{2}$ blocks, the coarsening/fattening/merging of $P_{1}$
along $Q$, denoted $P_{1}\blacktriangleright Q$, will be defined as the unique
contiguous set partition of $[N]$ into $B_{2}$ blocks corresponding to the
surjective and monotone function:

$[N]\xrightarrow{P_{1}\blacktriangleright
Q}[B_{2}]\overset{\text{def}}{=}[N]\xrightarrow{Q\circ P_{1}}[B_{2}]\,.$

The same operation is also defined for ordered set partitions by dropping the
assumption that $P_{1}$ and $Q$ correspond to monotone functions and only
requiring them to be surjective functions.

As an example, let $N=6$, and consider the contiguous set partition:

$P_{1}\overset{\text{def}}{=}\\{\\{1,2,3\\},\\{4\\},\\{5,6\\}\\}\,,$

denoted using “stars and bars” notation (without the stars) as

$P_{1}=123|4|56\,.$

This corresponds to the function $[6]\xrightarrow{P_{1}}[3]$ given by
$P_{1}(1)=1$, $P_{1}(2)=1$, $P_{1}(3)=1$, $P_{1}(4)=2$, $P_{1}(5)=3$,
$P_{1}(6)=3$. This function is both monotone and surjective. Given an element
of $[6]$ the function states which of the three blocks the element has been
placed into by the partition $P_{1}$. Note that here $B_{1}=3$. Consider also
the contiguous set partition $Q$ of $[3]$ into $B_{2}=2$ blocks given by
$12|3$, which corresponds to the function $[3]\xrightarrow{Q}[2]$ given by
$Q(1)=1$, $Q(2)=1$, $Q(3)=2$, which again is surjective and monotone.

Then in this case $P_{1}\blacktriangleright Q$ is the contiguous set partition
of $[6]$ into $2$ blocks given by

$1234|56\,,$

since $Q\circ P_{1}(1)=1$, $Q\circ P_{1}(2)=1$, $Q\circ P_{1}(3)=1$, $Q\circ
P_{1}(4)=1$, $Q\circ P_{1}(5)=2$, $Q\circ P_{1}(6)=2$. Notice how $Q\circ
P_{1}$ itself is a surjective and monotone function, since surjective
functions are closed under composition and monotone functions are closed under
composition, so functions which are both surjective and monotone are also
closed under composition. The operation $P_{1}\blacktriangleright Q$ basically
says to merge the first two blocks of $P_{1}$ (since the first block of $Q$ is
$\\{1,2\\}$) and to leave the $3$rd block unchanged (since $\\{3\\}$ is its
own block in $Q$).

A contiguous set partition $P_{2}$ of $[N]$ into $B_{2}$ blocks is said to be
coarser than the contiguous set parition $P_{1}$ of $[N]$ into $B_{1}$ blocks
(necessarily $B_{2}\leq B_{1}$) when for any $n_{1},n_{2}\in[N]$ one has that
$P_{1}(n_{1})=P_{1}(n_{2})$ implies that $P_{2}(n_{1})=P_{2}(n_{2})$ (even
though the converse need not be true). The same definition also works for
general ordered set partitions. If $P_{2}$ is coarser than $P_{1}$, then
$P_{1}$ is said to be finer than $P_{2}$, for both contiguous set partitions
and for ordered set partitions[297].

Given any contiguous set partition $P_{2}$ of $[N]$ into $B_{2}$ blocks which
is coarser than the contiguous set partition $P_{1}$ of $[N]$ into $B_{1}$
blocks, a unique contiguous set partition $Q$ of $[B_{1}]$ into $B_{2}$ blocks
such that $P_{2}=P_{1}\blacktriangleright Q$ exists, denoted
$\frac{P_{2}}{P_{1}}\overset{\text{def}}{=}Q$, and which will be called the
refinement quotient of $P_{2}$ by $P_{1}$. The analogous notion exists and is
well-defined for general ordered set partitions also. All of these notions
exist and are well-defined because of the following elementary lemmas:

  * •

For surjective functions $f:X\to Y$, $h:X\to Z$, if $h=g\circ f$, then $g$ is
surjective, since for each $z\\!\in\\!Z$ there is an $x\\!\in\\!X$ with
$g(f(x))=h(x)=z$, so $g(y)=z$ for $y\overset{\text{def}}{=}f(x)$.

  * •

For a monotone909090The assumption is that there are total orders on $X$, $Y$,
and $Z$, not just partial orders. Then the contrapositive of the claimed
statement $f(x_{1})\prec f(x_{2})\implies x_{1}\preceq x_{2}$ can be seen to
be valid for a monotonic function $f$. function $h:X\to Z$, and a monotone and
surjective function $f:X\to Y$, if $h=g\circ f$, then $g$ is also monotone.
Given $y_{1},y_{2}\in Y$ with $y_{1}\prec y_{2}$, there exist $x_{1},x_{2}\in
X$ with $x_{1}\preceq x_{2}$ with $f(x_{1})=y_{1}$ and $f(x_{2})=y_{2}$, since
$f$ is monotone and surjective. Since $h$ is monotone,
$g(y_{1})=g(f(x_{1}))=h(x_{1})\preceq h(x_{2})=g(f(x_{2}))=g(y_{2})$. If
$y_{1}=y_{2}$, then clearly $g(y_{1})=g(y_{2})\preceq g(y_{2})$, so in both
cases $y_{1}\preceq y_{2}$ means that $g(y_{1})\preceq g(y_{2})$.

  * •

For a surjective function $f:X\to Y$, and any function $h:X\to Z$ such that
$f(x_{1})=f(x_{2})$ implies $h(x_{1})=h(x_{2})$, there exists a function $g$
such that $h=g\circ f$. Namely, for any $y\in Y$, choose any $x\in X$ such
that $f(x)=y$, and then define $g(y)=h(x)$. This is well-defined, since for
any $x_{1},x_{2}\in X$ with $f(x_{1})=y=f(x_{2})$, by the assumptions on $h$
one also that $h(x_{1})=h(x_{2})$. Clearly one has that $g(f(x))=h(x)$ for all
$x\in X$.

  * •

For a surjective $f:X\to Y$, and any $h:X\to Z$, if $g:Y\to Z$ is such that
$h=g\circ f$, then $g$ is unique with this property, since given any
$\tilde{g}$ such that $\tilde{g}\circ f=h$ and any $y\in Y$, one has for any
$x\in X$ with $f(x)=y$ that $\tilde{g}(y)=\tilde{g}(f(x))=h(x)=g(f(x))=g(y)$,
i.e. $\tilde{g}=g$.

As discussed implicitly in appendix A.1.2, any ordered partition of the set
$[N]$ can be derived from a unique pair consisting of (i) a permutation of the
elements of $[N]$ and (ii) a contiguous set partition of $[N]$ (corresponding
to a composition a.k.a. ordered integer partition [277] of the integer $N$).
It is straightforward to see this in the case where both the order of the
blocks of the partition of $[N]$ as well as the order of the elements inside
of each block are considered important. In the case where only the order of
the blocks of the partition of $[N]$ is considered important, but not the
order of the elements inside of each block, one can without loss of generality
assume that the elements inside of each block are in ascending order, and in
that way still associate every such ordered partition with a unique pair of
permutation of $[N]$ and contiguous partition of $[N]$ – see section 2.2 of
[145] for a more formal explanation of this argument. Therefore, when also
allowing for permutations of the elements of $[N]$, there is no loss of
generality in restricting attention only to contiguous partitions, rather than
directly considering all possible ordered set partitions. (Compare the above
convention with the convention given for “multipartite indices” in [138].)

##  Appendix A.3 Orderings of Multi-Index Sets

Here some basic facts allowing one to better understand vectorization of
tensors are presented.

###  A.3.1 Equivalent Characterization of Total Orders for Finite Sets

Let $X$ be a finite set, and define $N\overset{\text{def}}{=}|X|$, the number
of elements in $X$.

Then there is a one-to-one correspondence between total orders (orderings) on
the set $X$ and bijections $[N]\overset{\phi}{\to}X$, allowing one to consider
the two notions equivalent.

Given a bijection $[N]\overset{\phi}{\to}X$, one can define a total order
$\preceq$ on $X$ via

$x_{1}\preceq x_{2}\iff\phi^{-1}(x_{1})\leq\phi^{-1}(x_{2})\text{ for all
}x_{1},x_{2}\in X\,.$

This is the total order generated by requiring the function $\phi$ to be
monotone, i.e. order-preserving.

Conversely, given a total order $\preceq$ on $X$, one can define a bijection
with $[N]$ via the rule:

$\begin{array}[]{rcl}\phi(1)&\overset{\text{def}}{=}&\text{smallest element of
}X\text{ with respect to }\preceq\\\
\phi(2)&\overset{\text{def}}{=}&\text{second smallest element of }X\text{ with
respect to }\preceq\\\ &\vdots&\\\
\phi(N-1)&\overset{\text{def}}{=}&\text{second largest element of }X\text{
with respect to }\preceq\\\ \phi(N)&\overset{\text{def}}{=}&\text{largest
element of }X\text{ with respect to }\preceq\end{array}$

Given this equivalence, elsewhere in the review there will in general be no
explicit distinction made between a total order on a finite set and a
bijection between that set and the index set corresponding to its cardinality.
In particular, given a multi-index set
$[M_{\scaleto{[O]\mathstrut}{0.5em}}]\overset{\text{def}}{=}\bigtimes_{o=1}^{O}[M_{o}]$,
a total order on $[M_{\scaleto{[O]\mathstrut}{0.5em}}]$ can equivalently be
characterized via a choice of bijection $[M_{1}\dots M_{o}\dots
M_{O}]\to[M_{\scaleto{[O]\mathstrut}{0.5em}}]$.

###  A.3.2 Equivalent Characterizations of Permutations of Finite Sets

There are two equivalent ways of thinking of permutations of a finite set $X$:
(1) as orderings of the set $X$, i.e. bijections $[N]\xrightarrow{\phi}X$
(where $|X|=N$), or (2) as (when $X$ already has a specified ordering) re-
orderings of the set $X$, i.e. (self-)bijections $X\xrightarrow{\pi}X$.

The elements of the set $X$ can be interpreted as “labels”, and the elements
of the set $[N]$ can be thought of as “rankings”. So a bijection
$[N]\xrightarrow{\phi}X$ can be interpreted as a way of assigning each ranking
(i.e. position) a unique label, such that each label is also assigned a unique
ranking.

One often sees the notation e.g. of the form $(312)$ for the permutation of
the elements of a $3$-element set. This can be interpreted as specifying a
bijection $[3]\xrightarrow{\phi}X$ which specifies that the ranking of one,
i.e. the first position, belongs to the element of the set $X$ with label
“$3$”, the ranking two belongs to the element of the set $X$ with label “$1$”,
and the ranking three belongs to the element of the set $X$ with label “$2$”.
(In other words, it does not say the opposite, e.g. that the element of the
set $X$ with label $1$ gets sent to position $3$; in this case that element
actually gets sent to position $2$, not position $3$.) The above notation for
a permutation can also be interpreted as specifying a (self-)bijection
$[3]\xrightarrow{\pi}[3]$ such that $\pi(1)=3$, $\pi(2)=1$, $\pi(3)=2$.

Again, above $\pi(o_{1})=o_{2}$ says that (after the permutation) the
$o_{1}$’th position now belongs to $o_{2}$, and it does not say that the
position of $o_{1}$ after the permutation is $o_{2}$ (which would be the
opposite statement, i.e. correspond to $\pi^{-1}$). In other words, this
review uses the convention that permutations, in both of their equivalent
formulations, are specified via bijections:

$\text{rankings}\quad\longrightarrow\quad\text{labels}\,,$

even though it is also possible to specify orderings using the inverse
functions ${\text{labels}\to\text{rankings}}$.

Given a specified ordering $[N]\xrightarrow{\psi}X$ of the finite set $X$,
e.g. the elements of $X$ have been named such that
$x_{1}\overset{\text{def}}{=}\psi(1)$, …,
$x_{N}\overset{\text{def}}{=}\psi(N)$, one would like to consider how to re-
arrange the elements of $X$ so that they have the ordering corresponding to
the bijection $[N]\xrightarrow{\pi}X$. In other words, given an element $x\in
X$, which has rank/position $\psi^{-1}(x)$ under the old ordering, one wants
to know which element has that position under the new ordering,
$\phi(\psi^{-1}(x))=(\phi\circ\psi^{-1})(x)$. In other words, the new ordering
$[N]\xrightarrow{\phi}X$ corresponds to the (self-)bijection
$\phi\circ\psi^{-1}$ of $X$:

$X\xrightarrow{\pi}X\overset{\text{def}}{=}X\xrightarrow{\psi^{-1}}[N]\xrightarrow{\phi}X\,.$

Conversely, starting with a self-bijection $X\xrightarrow{\pi}X$, and some
specified (“standard”) ordering of $X$ specified via the bijection
$[N]\xrightarrow{\psi}X$, one gets a new ordering $\phi$ of $X$ via the rule:

$[N]\xrightarrow{\phi}X\overset{\text{def}}{=}[N]\xrightarrow{\psi}X\xrightarrow{\pi}X\,.$

In other words, if the element $x$ is in the $n$’th position under the old
order ($\psi(n)=x$), after the re-ordering specified by $\pi$, the element
$\pi(x)$ is now in the $n$’th position, i.e. $\phi(n)=\pi(x)$.

Thus these two distinct ways of thinking about permutations can be seen to be
equivalent, since (given a “standard ordering” on the set $X$) one permutation
of the first type specifies a unique permutation of the second type, and vice
versa. According to [35], the more straightforward interpretation
as919191Technically this is being sloppy and implicitly using the
identification of orderings of $X$ with bijections $[N]\to X$ described in
section A.3.1 above. Strictly speaking the elementary interpretation
corresponds to total orders of $X$. bijections $[N]\to X$ is older (and can be
referred to for the sake of unambiguity as “passive permutations”), while the
more abstract and modern interpretation as bijections $X\to X$ used to be
referred to as “substitutions” (and can be referred to for the sake of
unambiguity as “active permutations”). Of course, in the case that the finite
set $X$ is itself an index set $[N]$ there is mathematically speaking no
distinction between the two. Since that is the most common scenario occurring
in this review (usually the index set is the mode index set), no explicit
distinction will be made between “passive permutations” and “active
permutations” in this review. (The “standard ordering” in this case can be
assumed to be the identity ${[N]\xrightarrow{\operatorname{Id}_{[N]}}[N]}$.)

###  A.3.3 Lexicographical and Colexicographical Order

Given a multi-index set
$[M_{\scaleto{[O]\mathstrut}{0.5em}}]=\bigtimes_{o=1}^{O}[M_{o}]$, for each
$o\in[O]$ the corresponding mode $[M_{o}]$ has a canonical order, namely the
numerical order of the first $M_{o}$ integers $1,2,\dots,M_{o}$. Given $O$
totally ordered sets, there is a standard partial order $\preceq$ defined on
their Cartesian product, called the product order. For the multi-index set
$[M_{\scaleto{[O]\mathstrut}{0.5em}}]$, the product order $\preceq$ amounts to
the following:

$\forall
m_{\scaleto{[O]\mathstrut}{0.5em}},n_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]\,,\quad
m_{\scaleto{[O]\mathstrut}{0.5em}}\preceq
n_{\scaleto{[O]\mathstrut}{0.5em}}\iff m_{1}\leq n_{1}\,,m_{2}\leq
n_{2}\,,\dots\,,m_{o}\leq n_{o}\,,\dots\,,m_{O}\leq n_{O}\,.$

This is not a total order unless $O=1$ because it leaves many, and usually
most, pairs of multi-indices incomparable. For example, when
$[M_{\scaleto{[O]\mathstrut}{0.5em}}]=[2]\times[2]$, it is neither the case
that $(1,2)\preceq(2,1)$ nor the case that $(2,1)\preceq(1,2)$. All that can
be said is that with respect to the product order $\preceq$, there is no way
to compare $(1,2)$ and $(2,1)$. Thus more needs to be done to find a total
order.

A total order $\leq$ is said to be a linear extension929292This definition
generalizes to the linear extension of any partial order, and the name comes
from the fact that an alternative name for total order is linear order, since
the numbers on the number line are totally ordered. of the product order
$\preceq$ if $\leq$ is compatible with $\preceq$ in the sense that
$m_{\scaleto{[O]\mathstrut}{0.5em}}\preceq n_{\scaleto{[O]\mathstrut}{0.5em}}$
always implies that $m_{\scaleto{[O]\mathstrut}{0.5em}}\leq
n_{\scaleto{[O]\mathstrut}{0.5em}}$. Of course the converse,
$m_{\scaleto{[O]\mathstrut}{0.5em}}\leq n_{\scaleto{[O]\mathstrut}{0.5em}}$
implies $m_{\scaleto{[O]\mathstrut}{0.5em}}\preceq
n_{\scaleto{[O]\mathstrut}{0.5em}}$, is usually false, since again the partial
order leaves most pairs of multi-indices incomparable. Thus the task of a
linear extension is to compare those pairs left untouched by the product order
$\preceq$ in a way that preserves transitivity ($a\leq b$ and $b\leq c$
implies $a\leq c$).

The requirement that the total order for the multi-index set be a linear
extension of the product order is a sensible one, since any order in which,
for example, the following was possible:

$(2081,496,1287)<(1,2,3)\,,$

would seem and feel bizarre to most people. However, for most multi-index sets
it still leaves far too many options available. For instance, for multi-index
sets of the form either $[2]\times[M]$ or $[M]\times[2]$ with $M\geq 2$, the
number of linear extensions of the product order is equal to the $M$’th
Catalan number, which grows super-exponentially with $M$. The number is of
course even higher for multi-index sets with either a larger number of modes
or modes of greater sizes.

Thus to limit the options available to a reasonable amount, a second
restriction on the total order can be imposed, namely that it be defined via a
process939393Of course this is a sufficiently imprecise statement as to be
non-falsifiable, i.e. “nicht einmal falsch”, and therefore at best somewhat
misleading. What is an “ordering process” or “ordering scheme” which “works
for multiple sets”? which works for all non-degenerate multi-index sets with
more than one mode of any size. The smallest such multi-index set is
$[2]\times[2]$, and thankfully there are only two linear extensions of its
product order, each corresponding to one of the two possible ways to choose
how to resolve the question $(1,2)\overset{?}{\lessgtr}(2,1)$ mentioned above:

$(1,1)<(1,2)<(2,1)<(2,2)\quad\text{and}\quad(1,1)<(2,1)<(1,2)<(2,2)\,.$

The first generalizes949494Both probably also generalize to other ordering
schemes too. So more precise restrictions seem to be required to conclude that
these are the “only possible” generalizations of the total orders for
$[2]\times[2]$. Compare footnote 93. to a process definable on all multi-index
sets called lexicographical order; the second generalizes to a process
definable on all multi-index sets called colexicographical order. Since
lexicographical order and colexicographical order are the “only” general
“ordering schemes” which satisfy both of these sensible requirements, they are
the only ones which will be used in this review. Not coincidentally, they are
also the most widely used in practice.

Lexicographical ordering is defined such that for any
$m_{\scaleto{[O]\mathstrut}{0.5em}},n_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]$,
$m_{\scaleto{[O]\mathstrut}{0.5em}}\not=n_{\scaleto{[O]\mathstrut}{0.5em}}$,
one has:

  * •

if $m_{1}<n_{1}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$, if
$m_{1}>n_{1}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise

  * •

if $m_{2}<n_{2}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$, if
$m_{2}>n_{2}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise

  * •

$\vdots$

  * •

if $m_{O-1}<n_{O-1}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$, if
$m_{O-1}>n_{O-1}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise

  * •

if $m_{O}<n_{O}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$.

Colexicographical ordering is defined such that for any
$m,n_{\scaleto{[O]\mathstrut}{0.5em}}\in[M_{\scaleto{[O]\mathstrut}{0.5em}}]$,
$m_{\scaleto{[O]\mathstrut}{0.5em}}\not=n_{\scaleto{[O]\mathstrut}{0.5em}}$,
one has:

  * •

if $m_{O}<n_{O}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$, if
$m_{O}>n_{O}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise,

  * •

if $m_{O-1}<n_{O-1}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$, if
$m_{O-1}>n_{O-1}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise,

  * •

$\vdots$

  * •

if $m_{2}<n_{2}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$, if
$m_{2}>n_{2}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise

  * •

if $m_{1}<n_{1}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$.

Notice the recursive nature of both definitions, specifically that for
lexicographical ordering:

  * •

if $m_{1}<n_{1}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$, if
$m_{1}>n_{1}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise

  * •

if $m_{\scaleto{[O]\setminus 1\mathstrut}{0.5em}}<n_{\scaleto{[O]\setminus
1\mathstrut}{0.5em}}$ in lexicographical order then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$.

Similarly for colexicographical ordering:

  * •

if $m_{O}<n_{O}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$, if
$m_{O}>n_{O}$ then
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise

  * •

if $m_{\scaleto{[O]\setminus O\mathstrut}{0.5em}}<n_{\scaleto{[O]\setminus
O\mathstrut}{0.5em}}$ in colexicographical order then
$m_{\scaleto{[O]\mathstrut}{0.5em}}<n_{\scaleto{[O]\mathstrut}{0.5em}}$,
otherwise
$m_{\scaleto{[O]\mathstrut}{0.5em}}>n_{\scaleto{[O]\mathstrut}{0.5em}}$.

##  Appendix A.4 The Categories of Multi-Index Sets

Explicitly defining categories is helpful for identifying the specific
mathematical structure of interest. Working generally with the category of
(unstructured) multi-index sets, or in special cases with the categories of
structured multi-index sets, enables one to identify those issues in tensor
algebra which are inherently combinatorial in nature and isolate them. The
tensor shape functor then enables the transfer of combinatorial observations
from the multi-index categories back to tensors, where they can subsequently
be composed with algebraic observations.

###  A.4.1 The (Unfactorized) Category of Multi-Index Sets

The category of multi-index sets $\bm{\mathsf{MltIdx}}$ is a full
subcategoryHendnotepage.2linkHendnote.2bb2macro:-¿A full subcategory contains
only a subset of the objects of its parent category, yet all of the morphisms
between those objects which exist in the parent category. For example, the
category of real vector spaces $“mathsf –Vect˝˙“R $ is “textit –not˝ a full
subcategory of the category of sets $“sets $, because the morphisms between
vector spaces are restricted to be linear functions in the category of real
vector spaces $“mathsf –Vect˝˙“R $, whereas in the parent category of sets
$“sets $ all possible arbitrary set functions between any two vector spaces
are also present. So while the function $–f: “R ^2 “to “R ^2˝$ given by $(x,y)
“mapsto (x^2, y^2)$ is in $“morphisms ˙“sets (“R ^2, “R ^2)$, it is “textit
–not˝ in $“morphisms ˙–“mathsf –Vect˝˙“R ˝(“R ^2, “R ^2)$ since it is not a
linear function. of the category of finite sets $\mathsf{Fin}\mathsf{Set}$,
which in turn is a full subcategory of the category of sets $\mathsf{Set}$.
The category $\mathsf{Set}$ has all sets as its objects and all functions
between sets as its morphisms, while the category of finite sets
$\mathsf{Fin}\mathsf{Set}$ has all finite sets as its objects and all
functions between those sets as its morphisms.

Thus the objects of $\bm{\mathsf{MltIdx}}$ are all multi-index sets, i.e. all
finite sets of the form:

$[M_{1}]\times[M_{2}]\times[M_{3}]\times\dots\times[M_{O}]\,,\quad\quad
O,M_{1},M_{2},\dots,M_{O}\in\mathbb{N}\,,$

and the morphisms of $\bm{\mathsf{MltIdx}}$ are all functions between multi-
index sets.

###  A.4.2 Cartesian Product in the Category of Multi-Index Sets

The Cartesian product defines a functor
$\bm{\mathsf{MltIdx}}\times\bm{\mathsf{MltIdx}}\to\bm{\mathsf{MltIdx}}$.

####  A.4.2.1 Definition of the Cartesian Product on Objects

The product operation defined for $\bm{\mathsf{MltIdx}}$ is different from the
Cartesian product construction that is typically the product in subcategories
of $\mathsf{Set}$. The idea is to exploit the highly specific structure of
multi-index sets to define a modification of the typical Cartesian product
which:

  * •

is associative up to equality, instead of only being associative up to
isomorphism,

  * •

still satisfies the universal property of the (categorical) product,

  * •

returns another multi-index as output when given two multi-index as input.

For conceptual simplicity, since both product operations lead to sets which
are uniquely isomorphic to each other in $\mathsf{Fin}\mathsf{Set}$ and
$\mathsf{Set}$959595Note that it would not be correct to say that they are
also isomorphic in $\bm{\mathsf{MltIdx}}$, even though it is a full
subcategory of $\mathsf{Fin}\mathsf{Set}$, since the typical Cartesian product
of two multi-index sets is usually not even an object of
$\bm{\mathsf{MltIdx}}$., and they correspond to functors which are naturally
equivalent969696For insight about why the distinct products are nevertheless
naturally equivalent functors, compare the discussion in A.10.1. Here the
universal property satisfied by both functors is the universal property of the
(categorical) product., and since many might not even notice the distinction
at all without it being pointed to them explicitly, the modified product
operation in $\bm{\mathsf{MltIdx}}$ will also be referred to as the Cartesian
product, even though it is distinct from the construction which is more
typically called “Cartesian product”.

Given two objects of $\bm{\mathsf{MltIdx}}$ (where below $\bigtimes$ refers to
the Cartesian product of $\mathsf{Set}$):

$[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]=\bigtimes_{o=1}^{O_{1}}[M_{o}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}]\,,\quad[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]=\bigtimes_{o=1}^{O_{2}}[M_{o}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}]\,,$

their product $\times_{\bm{\mathsf{MltIdx}}}$ in $\bm{\mathsf{MltIdx}}$ is
defined to be the set:

$[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\overset{\text{def}}{=}\bigtimes_{o=1}^{O_{1}+O_{2}}[M_{o}]\,,\quad
M_{o}=\begin{cases}M_{o}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}&\text{when
}1\leq o\leq O_{1}\\\
M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{o-O_{1}}&\text{when
}O_{1}+1\leq o\leq O_{1}+O_{2}\end{cases}\,.$

For example, if $[M_{\scaleto{[2]\mathstrut}{0.5em}}]=[2]\times[2]$, and
$[N_{\scaleto{[1]\mathstrut}{0.5em}}]=[2]$, then one has that:

$\begin{array}[]{rcl}[M_{\scaleto{[2]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[N_{\scaleto{[1]\mathstrut}{0.5em}}]&=&[2]\times_{\mathsf{Set}}[2]\times_{\mathsf{Set}}[2]\\\
&=&\\{(1,1,1),(1,1,2),(1,2,1),(1,2,2),\\\ &&\phantom{\ \
}(2,1,1),(2,1,2),(2,2,1),(2,2,2)\\}\,.\end{array}$

Note the single level of parentheses, as well as that this is a multi-index
set. In contrast:

$\begin{array}[]{rcl}[M_{\scaleto{[2]\mathstrut}{0.5em}}]\times_{\mathsf{Set}}[N_{\scaleto{[1]\mathstrut}{0.5em}}]&=&([2]\times[2])\times[2]\\\
&=&\\{((1,1),1),((1,1),2),((1,2),1),((1,2),2),\\\ &&\phantom{\ \
}((2,1),1),((2,1),2),((2,2),1),((2,2),2)\\}\,.\end{array}$

Note the two levels of parentheses, and that this is not a Cartesian product
of three index sets, but a Cartesian product of one multi-index set and one
index set, and thus not a multi-index set itself.

Observe that by design these two constructions coincide for index sets:

$[M]\times_{\mathsf{Set}}[N]=[M]\times_{\bm{\mathsf{MltIdx}}}[N]\,,\quad\forall
M,N\in\mathbb{N}\,.$

Observe also that the product in $\bm{\mathsf{MltIdx}}$ is associative up to
equality, whereas the typical Cartesian product is only associative up to
isomorphism. For example, for $\times_{\mathsf{Set}}$ one has that:

$\begin{array}[]{rcl}([2]\times_{\mathsf{Set}}[2])\times_{\mathsf{Set}}[2]&=&\\{((1,1),1),((1,1),2),((1,2),1),((1,2),2),\\\
&&\phantom{\ \ }((2,1),1),((2,1),2),((2,2),1),((2,2),2)\\}\\\
&\not=&\\{(1,(1,1)),(1,(1,2)),(1,(2,1)),(1,(2,2)),\\\ &&\phantom{\ \
}(2,(1,1)),(2,(1,2)),(2,(2,1)),(2,(2,2))\\}\\\
&=&[2]\times_{\mathsf{Set}}([2]\times_{\mathsf{Set}}[2])\,.\end{array}$

Even though there is an obvious bijection between the two sets above, such
that they are isomorphic, they are nevertheless not equal. In contrast, for
$\times_{\bm{\mathsf{MltIdx}}}$ one has that:

$([2]\times_{\bm{\mathsf{MltIdx}}}[2])\times_{\bm{\mathsf{MltIdx}}}[2]=[2]\times_{\bm{\mathsf{MltIdx}}}[2]\times_{\bm{\mathsf{MltIdx}}}[2]=[2]\times_{\bm{\mathsf{MltIdx}}}([2]\times_{\bm{\mathsf{MltIdx}}}[2])\,,$

where by definition
$[2]\times_{\bm{\mathsf{MltIdx}}}[2]\times_{\bm{\mathsf{MltIdx}}}[2]=[2]\times_{\mathsf{Set}}[2]\times_{\mathsf{Set}}[2]$.
No isomorphisms are required to identify any of the three sets, instead all
three of them are already equal.

More generally, one only has for the Cartesian product in $\mathsf{Set}$ that:

$\begin{array}[]{rcccl}(X\times_{\mathsf{Set}}Y)\times_{\mathsf{Set}}Z&\not=&X\times_{\mathsf{Set}}Y\times_{\mathsf{Set}}Z&\not=&X\times_{\mathsf{Set}}(Y\times_{\mathsf{Set}})Z\,,\\\
(X\times_{\mathsf{Set}}Y)\times_{\mathsf{Set}}Z&\cong_{\mathsf{Set}}&X\times_{\mathsf{Set}}Y\times_{\mathsf{Set}}Z&\cong_{\mathsf{Set}}&X\times_{\mathsf{Set}}(Y\times_{\mathsf{Set}})Z\,,\end{array}$

since for arbitrary elements $x\in X$, $y\in Y$, $z\in Z$, elements of
$(X\times_{\mathsf{Set}}Y)\times_{\mathsf{Set}}Z$ are of the form $((x,y),z)$,
elements of $X\times_{\mathsf{Set}}Y\times_{\mathsf{Set}}Z$ are of the form
$(x,y,z)$, and elements of $X\times_{\mathsf{Set}}(Y\times_{\mathsf{Set}})Z$
are of the form $(x,(y,z))$, none of which are equal. Here
$\cong_{\mathsf{Set}}$ means equivalent up to isomorphism in the category
$\mathsf{Set}$, i.e. that there exists a bijection between the sets979797The
relation $\cong_{\mathsf{Set}}$ of equivalence up to isomorphism is a
transitive relation, while technically $\not=$ is not. Nevertheless it also
true that
$(X\times_{\mathsf{Set}}Y)\times_{\mathsf{Set}}Z\not=X\times_{\mathsf{Set}}(Y\times_{\mathsf{Set}})Z$,
even if it did not say so explicitly above..

In contrast, for the product operation $\times_{\bm{\mathsf{MltIdx}}}$ in the
category $\bm{\mathsf{MltIdx}}$, one has that:

$([M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}])\times_{\bm{\mathsf{MltIdx}}}[P_{\scaleto{[O_{3}]\mathstrut}{0.5em}}]=[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[P_{\scaleto{[O_{3}]\mathstrut}{0.5em}}]=[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}([N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[P_{\scaleto{[O_{3}]\mathstrut}{0.5em}}])\,,$

for all multi-index sets $[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]$,
$[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]$, and
$[P_{\scaleto{[O_{3}]\mathstrut}{0.5em}}]$ in $\bm{\mathsf{MltIdx}}$. This is
associativity up to equality.

For two given multi-index sets
${[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]=\bigtimes_{o=1}^{O_{1}}[M_{o}]}$
and
${[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]=\bigtimes_{o=1}^{O_{2}}[N_{o}]}$,
there always exists a canonical989898It is unique among all bijections in
transforming the projection maps of one set to the projection maps of the
other, i.e. in being compatible with the universal property of the product
satisfied by both sets. bijection
$\phi:[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\mathsf{Set}}[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\to[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]$.
Here is its rule of assignment:

$\begin{array}[]{rlcrl}&(m_{\scaleto{[O_{1}]\mathstrut}{0.5em}},n_{\scaleto{[O_{2}]\mathstrut}{0.5em}})=&&&\\\
\phi:&((m_{1},\dotsc,m_{O_{1}}),(n_{1},\dotsc,n_{O_{2}}))&\mapsto&(m_{1},\dotsc,m_{O_{1}},n_{1},\dotsc,n_{O_{2}})&\overset{\text{def}}{=}\\\
&&&m_{\scaleto{[O_{1}]\mathstrut}{0.5em}}\times_{\bm{\mathsf{MltIdx}}}n_{\scaleto{[O_{2}]\mathstrut}{0.5em}}\phantom{\,.}&\overset{\text{def}}{=}\\\
&&&m_{\scaleto{[O_{1}]\mathstrut}{0.5em}}n_{\scaleto{[O_{2}]\mathstrut}{0.5em}}\,.&\end{array}$

For example, taking $[M_{\scaleto{[2]\mathstrut}{0.5em}}]=[2]\times[2]$ and
$[N_{\scaleto{[1]\mathstrut}{0.5em}}]=[2]$ as in the example above, one has
that:

$\phi:((1,1),2)\mapsto(1,1,2)\,,\quad\phi:((2,2),1))\mapsto(2,2,1)\,,\quad\phi:((1,2),1)\mapsto(1,2,1)\,,\quad\text{etc.}$

This definition of product in $\bm{\mathsf{MltIdx}}$ also makes the definition
of the Segre outer product $\otimes$ much less unwieldy compared to a
comparable definition using the tensor product of functions $\otimes_{func}$
defined in section 3. In particular, one has for tensors
$\mathbfcal{T}:[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\to\mathbb{R}$ and
$\mathbfcal{U}:[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\to\mathbb{R}$ that:

$\mathbfcal{T}\otimes\mathbfcal{U}\overset{\text{def}}{=}(\mathbfcal{T}\otimes_{func}\mathbfcal{U})\circ\phi^{-1}\,,$

since, comparing domains and codomains, one sees that:

$\mathbfcal{T}\otimes\mathbfcal{U}:[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\to\mathbb{R}\,,\quad\mathbfcal{T}\otimes_{func}\mathbfcal{U}:[M_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\mathsf{Set}}[N_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\to\mathbb{R}\,.$

One example of why this is useful is because it allows one to say for any
three tensors $\mathbfcal{S},\mathbfcal{T},\mathbfcal{U}$:

${(\mathbfcal{S}\otimes\mathbfcal{T})\otimes\mathbfcal{U}}=\mathbfcal{S}\otimes\mathbfcal{T}\otimes\mathbfcal{U}=\mathbfcal{S}\otimes(\mathbfcal{T}\otimes\mathbfcal{U})\,,$

i.e. that all three tensors are the same up to equality, not merely up to
isomorphism999999Which strictly speaking is all one can do using the typical
tensor product construction found e.g. in Chapter 2 of [14].. However, the
analogous statement would be false using the definition $\otimes_{func}$: the
three functions
${(\mathbfcal{S}\otimes_{func}\mathbfcal{T})\otimes_{func}\mathbfcal{U}}$,
${\mathbfcal{S}\otimes_{func}\mathbfcal{T}\otimes_{func}\mathbfcal{U}}$, and
$\mathbfcal{S}\otimes_{func}(\mathbfcal{T}\otimes_{func}\mathbfcal{U})$ don’t
even have the same domain and therefore certainly aren’t equal. Moreover, in
general none of the functions
${(\mathbfcal{S}\otimes_{func}\mathbfcal{T})\otimes_{func}\mathbfcal{U}}$,
${\mathbfcal{S}\otimes_{func}\mathbfcal{T}\otimes_{func}\mathbfcal{U}}$, and
$\mathbfcal{S}\otimes_{func}(\mathbfcal{T}\otimes_{func}\mathbfcal{U})$ are
tensors, since their domains are not multi-index sets, only Cartesian products
of multi-index sets. Thus the definition for the product in
$\bm{\mathsf{MltIdx}}$ ensures that the Segre outer product returns a tensor
as output when given tensors as input.

Throughout this review, whenever $\bigtimes$ or $\times$ is written to denote
some product of multi-index sets, one can safely assume that the product in
question is the product operation $\times_{\bm{\mathsf{MltIdx}}}$ and not
$\times_{\mathsf{Set}}$.

####  A.4.2.2 Definition of the Cartesian Product on Morphisms

One has the same issues for functions that one has for sets with
$\times_{\mathsf{Set}}$. Specifically, given functions:

$f_{1}:X_{1}\to Y_{1}\,,\quad f_{2}:X_{2}\to Y_{2}\,,\quad f_{3}:X_{3}\to
Y_{3}\,,$

none of the following three functions are equal:

$\begin{array}[]{rccl}(f_{1}\times_{\mathsf{Set}}f_{2})\times_{\mathsf{Set}}f_{3}:&(X_{1}\times_{\mathsf{Set}}X_{2})\times_{\mathsf{Set}}X_{3}&\to&(Y_{1}\times_{\mathsf{Set}}Y_{2})\times_{\mathsf{Set}}Y_{3}\,,\\\
f_{1}\times_{\mathsf{Set}}f_{2}\times_{\mathsf{Set}}f_{3}:&X_{1}\times_{\mathsf{Set}}X_{2}\times_{\mathsf{Set}}X_{3}&\to&Y_{1}\times_{\mathsf{Set}}Y_{2}\times_{\mathsf{Set}}Y_{3}\,,\\\
f_{1}\times_{\mathsf{Set}}(f_{2}\times_{\mathsf{Set}}f_{3}):&X_{1}\times_{\mathsf{Set}}(X_{2}\times_{\mathsf{Set}}X_{3})&\to&Y_{1}\times_{\mathsf{Set}}(Y_{2}\times_{\mathsf{Set}}Y_{3})\,.\end{array}$

This is clearly true since none of their domains are the same, and none of
their codomains are the same. They all take different types of inputs and
return different types of outputs:

$\begin{array}[]{rccl}(f_{1}\times_{\mathsf{Set}}f_{2})\times_{\mathsf{Set}}f_{3}:&((x_{1},x_{2}),x_{3})&\mapsto&((f_{1}(x_{1}),f_{2}(x_{2})),f_{3}(x_{3}))\,,\\\
f_{1}\times_{\mathsf{Set}}f_{2}\times_{\mathsf{Set}}f_{3}:&(x_{1},x_{2},x_{3})&\mapsto&(f_{1}(x_{1}),f_{2}(x_{2}),f_{3}(x_{3}))\,,\\\
f_{1}\times_{\mathsf{Set}}(f_{2}\times_{\mathsf{Set}}f_{3}):&(x_{1},(x_{2},x_{3}))&\mapsto&(f_{1}(x_{1}),(f_{2}(x_{2}),f_{3}(x_{3})))\,.\end{array}$

To avoid problems like this, the Cartesian product of functions in
$\bm{\mathsf{MltIdx}}$ is defined differently. (It needs to be defined
differently in any case, since the Cartesian product in $\mathsf{Set}$ of two
functions
$f_{1}:[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\to[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]$
and
$f_{2}:[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\to[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]$
is the function:

$f_{1}\times_{\mathsf{Set}}f_{2}:[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\mathsf{Set}}[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\to[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times_{\mathsf{Set}}[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\,,$

for which neither the domain nor codomain are multi-index sets in general.
Moreover, the Cartesian product functor in $\bm{\mathsf{MltIdx}}$ applied to
the domains of $f_{1}$ and $f_{2}$ yields
${[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$,
and applied to their codomains yields
${[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]}$,
so those must be the domain and codomain respectively for any definition of
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}$.) Specifically, given functions
$f_{1}:[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\to[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]$
and
$f_{2}:[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\to[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]$,
their Cartesian product in $\bm{\mathsf{MltIdx}}$ set is

$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\to[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]$

which has the rule of assignment:

$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}\times_{\bm{\mathsf{MltIdx}}}m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}\mapsto
f_{1}(m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}})\times_{\bm{\mathsf{MltIdx}}}f_{2}(m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}})\,.$

More precisely, if $\phi$ is the canonical100100100See footnote 98. bijection
$\phi:[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\mathsf{Set}}[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\to[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]$
and $\psi$ is the canonical bijection
$\psi:[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times_{\mathsf{Set}}[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\to[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times_{\bm{\mathsf{MltIdx}}}[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]$,
then one has:

$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}\overset{\text{def}}{=}\psi\circ(f_{1}\times_{\mathsf{Set}}f_{2})\circ\phi^{-1}\,.$

The bijections $\phi,\psi$ basically “remove parentheses”, and the bijections
$\phi^{-1},\psi^{-1}$ “add parentheses”.

Note that this definition of Cartesian product, in addition to being the
“correct choice” by virtue of being the morphism induced by the universal
property of the product, also has the virtue of being associative up to
equality, not just isomorphism. In other words, given any morphisms of
$\bm{\mathsf{MltIdx}}$:

$f_{1}:[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\to[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\,,\quad
f_{2}:[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]\to[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\,,\quad
f_{3}:[M^{\scriptsize\scaleto{(3)\mathstrut}{0.7em}}_{\scaleto{[O_{3}]\mathstrut}{0.5em}}]\to[N^{\scriptsize\scaleto{(3)\mathstrut}{0.7em}}_{\scaleto{[D_{3}]\mathstrut}{0.5em}}]\,,$

one always has the following relationships:

$(f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2})\times_{\bm{\mathsf{MltIdx}}}f_{3}=f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}\times_{\bm{\mathsf{MltIdx}}}f_{3}=f_{1}\times_{\bm{\mathsf{MltIdx}}}(f_{2}\times_{\bm{\mathsf{MltIdx}}}f_{3})\,.$

Therefore the order and number of parentheses do not affect the result.

For an example of the definition of Cartesian product for functions in
$\bm{\mathsf{MltIdx}}$, consider

$f_{1}:[4]\to[2]\times[2]\,,\quad
f_{1}(1)=(1,1)\,,f_{1}(2)=(2,1)\,,f_{1}(3)=(1,2)\,,f_{1}(4)=(2,2)\,,$

and

$f_{2}:[2]\times[2]\to[4]\,,\quad
f_{2}(1,1)=1\,,f_{2}(1,2)=2\,,f_{2}(2,1)=3\,,f_{2}(2,2)=4\,.$

Then for the values of the Cartesian product function of
$\bm{\mathsf{MltIdx}}$:

$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:[4]\times_{\bm{\mathsf{MltIdx}}}([2]\times[2])=[4]\times[2]\times[2]\to[2]\times[2]\times[4]=([2]\times[2])\times_{\bm{\mathsf{MltIdx}}}[4]$

one has the following:

$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(1,1,1)$ | $\mapsto$ | $(1,1,1)$
---|---|---
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(1,1,2)$ | $\mapsto$ | $(1,1,2)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(1,2,1)$ | $\mapsto$ | $(1,1,3)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(1,2,2)$ | $\mapsto$ | $(1,1,4)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(2,1,1)$ | $\mapsto$ | $(2,1,1)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(2,1,2)$ | $\mapsto$ | $(2,1,2)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(2,2,1)$ | $\mapsto$ | $(2,1,3)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(2,2,2)$ | $\mapsto$ | $(2,1,4)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(3,1,1)$ | $\mapsto$ | $(1,2,1)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(3,1,2)$ | $\mapsto$ | $(1,2,2)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(3,2,1)$ | $\mapsto$ | $(1,2,3)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(3,2,2)$ | $\mapsto$ | $(1,2,4)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(4,1,1)$ | $\mapsto$ | $(2,2,1)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(4,1,2)$ | $\mapsto$ | $(2,2,2)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(4,2,1)$ | $\mapsto$ | $(2,2,3)$
$f_{1}\times_{\bm{\mathsf{MltIdx}}}f_{2}:(4,2,2)$ | $\mapsto$ | $(2,2,4)\,.$

In contrast, for the values of the Cartesian product function of
$\mathsf{Set}$:

$f_{1}\times_{\mathsf{Set}}f_{2}:[4]\times_{\mathsf{Set}}([2]\times[2])\to([2]\times[2])\times_{\mathsf{Set}}[4]\,,$

one has the following instead:

$f_{1}\times_{\mathsf{Set}}f_{2}:(1,(1,1))$ | $\mapsto$ | $((1,1),1)$
---|---|---
$f_{1}\times_{\mathsf{Set}}f_{2}:(1,(1,2))$ | $\mapsto$ | $((1,1),2)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(1,(2,1))$ | $\mapsto$ | $((1,1),3)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(1,(2,2))$ | $\mapsto$ | $((1,1),4)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(2,(1,1))$ | $\mapsto$ | $((2,1),1)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(2,(1,2))$ | $\mapsto$ | $((2,1),2)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(2,(2,1))$ | $\mapsto$ | $((2,1),3)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(2,(2,2))$ | $\mapsto$ | $((2,1),4)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(3,(1,1))$ | $\mapsto$ | $((1,2),1)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(3,(1,2))$ | $\mapsto$ | $((1,2),2)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(3,(2,1))$ | $\mapsto$ | $((1,2),3)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(3,(2,2))$ | $\mapsto$ | $((1,2),4)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(4,(1,1))$ | $\mapsto$ | $((2,2),1)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(4,(1,2))$ | $\mapsto$ | $((2,2),2)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(4,(2,1))$ | $\mapsto$ | $((2,2),3)$
$f_{1}\times_{\mathsf{Set}}f_{2}:(4,(2,2))$ | $\mapsto$ | $((2,2),4)\,.$

Clearly these functions are very similar, and in fact they are related via
canonical bijections induced by the universal property of the product, but
only the first one is a morphism in $\bm{\mathsf{MltIdx}}$.

###  A.4.3 The $D$-Factorized Categories of Multi-Index Sets

The definition of the $D$-factorized categories of multi-index sets
$\bm{\mathsf{MltIdx}}_{D}$ initially requires some explanation, but becomes
straightforward to understand after thinking through some examples.

####  A.4.3.1 Objects of the $D$-Factorized Category of Multi-Index Sets

For a given $D\geq 1$, the objects of $\bm{\mathsf{MltIdx}}_{D}$ each consist
of two data: (i) a multi-index set with $O\geq D$ modes, and (ii) a chosen
contiguous101101101As explained elsewhere, only considering contiguous
partitions does not actually come with any loss of generality, due to the
possibility of permuting modes beforehand to ensure that one’s desired
partition becomes contiguous. partition of those $O$ modes into $D$ blocks.

For an example of what this means, consider the $2$-factorized category of
multi-index sets, $\bm{\mathsf{MltIdx}}_{2}$. Then for the following multi-
index set with $3\geq 2$ modes:

$[M_{\scaleto{[3]\mathstrut}{0.5em}}]\overset{\text{def}}{=}[7]\times[5]\times[14]$

there are $\binom{3-1}{2-1}=\binom{2}{1}=2$ objects of
$\bm{\mathsf{MltIdx}}_{2}$ associated with it, since $\binom{O-1}{D-1}$ counts
the number of distinct ways to partition a set with $O$ elements into $D$
contiguous blocks.

Specifically, the two objects of $\bm{\mathsf{MltIdx}}_{2}$ associated with
$[M_{\scaleto{[3]\mathstrut}{0.5em}}]=[7]\times[5]\times[4]$ are:

$[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{12|3}\overset{\text{def}}{=}([7]\times[5]\times[14],\\{\\{1,2\\},\\{3\\}\\}),\quad[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|23}\overset{\text{def}}{=}([7]\times[5]\times[14],\\{\\{1\\},\\{2,3\\}\\})\,.$

In contrast, given a multi-index set with $2$ modes, there is exactly
$\binom{2-1}{1-1}=\binom{1}{0}=1$ object of $\bm{\mathsf{MltIdx}}_{2}$ which
is associated with it. For instance, the multi-index set:

$[23]\times[17]$

is associated uniquely with the following object of
$\bm{\mathsf{MltIdx}}_{2}$:

$([23]\times[17],\\{\\{1\\},\\{2\\}\\})\,.$

The objects of $\bm{\mathsf{MltIdx}}_{D}$ can be thought of as consisting of
two distinct strata: (i) all multi-index sets with exactly $D$ modes, which
are endowed with no extra structure, and (ii) multi-index sets with $O>D$
modes endowed with a given contiguous partition of their $O$ modes into $D$
blocks (with the result that each such multi-index set is associated with
$\binom{O-1}{D-1}$ distinct objects of $\bm{\mathsf{MltIdx}}_{D}$).

The idea is that the objects of $\bm{\mathsf{MltIdx}}_{D}$ should have
“$D$-fold Cartesian product structure”.

While there is clearly only one way that a multi-index set with exactly $D$
modes can be considered to have such $D$-fold Cartesian product structure, for
multi-index sets with $O>D$ modes one needs to be specific about how the
multi-index set is being considered as having $D$-fold Cartesian product
structure, since there are $\binom{O-1}{D-1}$ distinct possible groupings of
the $O$ modes which would enable such an interpretation, each corresponding to
a different contiguous partition of $[O]$.

Also notice that, since for every $O\geq 1$ there is exactly
$\binom{O-1}{1-1}=\binom{O-1}{0}=1$ way to partition $O$ modes into one
block102102102Since there is only one block, the restriction that the
partition be contiguous is redundant., the objects of
$\bm{\mathsf{MltIdx}}_{1}$ are essentially the same as those of
$\bm{\mathsf{MltIdx}}$.

####  A.4.3.2 Morphisms of the $D$-Factorized Category of Multi-Index Sets

Given two objects of $\bm{\mathsf{MltIdx}}_{D}$, the morphisms between them
consist of all $D$-fold Cartesian products of functions between their $D$
corresponding blocks of modes.

For an example of what this definition means when $D=2$, consider the objects
$[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{12|3}$ and $[23]\times[17]$ of
$\bm{\mathsf{MltIdx}}_{2}$. Given any two functions (equivalently morphisms in
$\bm{\mathsf{MltIdx}}$):

$f_{1}:[7]\times[5]\to[23]\,,\quad f_{2}:[14]\to[17]\,,$

then the Cartesian product of those two functions:

$f_{1}\times f_{2}:[7]\times[5]\times[14]\to[23]\times[17]$

is a morphism in $\bm{\mathsf{MltIdx}}_{2}$ from
$[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{12|3}$ to $[23]\times[17]$.

Conversely, by definition every morphism in $\bm{\mathsf{MltIdx}}_{2}$
starting from $[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{12|3}$ and going to
$[23]\times[17]$ is of the form $f_{1}\times f_{2}$ for some functions
$f_{1}:[7]\times[5]\to[23]$ and $f_{2}:[14]\to[17]$.

Similarly, the morphisms in $\bm{\mathsf{MltIdx}}_{2}$ starting from
$[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|23}$ (note the different partition)
and going to $[23]\times[17]$ are the set of all functions which are of the
form:

$g_{1}\times g_{2}:[7]\times[5]\times[14]\to[23]\times[17]\,,$

for any functions $g_{1}$ and $g_{2}$ such that:

$g_{1}:[7]\to[23]\,,\quad g_{2}:[5]\times[14]\to[17]\,.$

The idea is that the morphisms in $\bm{\mathsf{MltIdx}}_{D}$ should preserve
the specified $D$-fold Cartesian product structures of their domain and
codomain, and thus have $D$-fold Cartesian product structure as well.

Note that for functions $g_{1}:[7]\to[23]$ and $g_{2}:[5]\times[14]\to[17]$,
the function $g_{1}\times g_{2}$ is not a morphism in
$\bm{\mathsf{MltIdx}}_{2}$ from $[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{12|3}$
to $[23]\times[17]$, even though the function is a morphism from
$[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|23}$ to $[23]\times[17]$ in
$\bm{\mathsf{MltIdx}}_{2}$, as well as a morphism from
$[M_{\scaleto{[3]\mathstrut}{0.5em}}]$ to $[23]\times[17]$ in
$\bm{\mathsf{MltIdx}}$.

Similarly, for functions $f_{1}:[7]\times[5]\to[23]$ and $f_{2}:[14]\to[17]$,
the function $f_{1}\times f_{2}$ is not a morphism in
$\bm{\mathsf{MltIdx}}_{2}$ from $[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|23}$
to $[23]\times[17]$, even though the function is a morphism from
$[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{12|3}$ to $[23]\times[17]$ in
$\bm{\mathsf{MltIdx}}_{2}$, as well as a morphism from
$[M_{\scaleto{[3]\mathstrut}{0.5em}}]$ to $[23]\times[17]$ in
$\bm{\mathsf{MltIdx}}$.

Thus the principal effect of attaching the extra structure of a partition of
modes to the objects in $\bm{\mathsf{MltIdx}}_{D}$ is to limit the number of
functions between those objects which are considered valid morphisms.
Moreover, when different partitions of modes into $D$ blocks are attached to
the same multi-index set, the answer to which functions are valid morphisms in
$\bm{\mathsf{MltIdx}}_{D}$ changes.

Also notice that, since the $1$-fold Cartesian product of a function $f$ is
just the function $f$ itself, the morphisms of $\bm{\mathsf{MltIdx}}_{1}$ are
essentially the same as those of $\bm{\mathsf{MltIdx}}$.

####  A.4.3.3 Remarks about the $D$-Factorized Categories of Multi-Index Sets

Since both the objects of $\bm{\mathsf{MltIdx}}_{1}$ and the morphisms of
$\bm{\mathsf{MltIdx}}_{1}$ are essentially the same as their counterparts in
$\bm{\mathsf{MltIdx}}$, the two categories $\bm{\mathsf{MltIdx}}_{1}$ and
$\bm{\mathsf{MltIdx}}$ are for all intents and purposes the same category.
Therefore, from now on $\bm{\mathsf{MltIdx}}_{1}$ will be redefined so that
$\bm{\mathsf{MltIdx}}_{1}=\bm{\mathsf{MltIdx}}$.

Notice that for any $O>D$, the objects of $\bm{\mathsf{MltIdx}}$ which are
associated with the objects of $\bm{\mathsf{MltIdx}}_{O}$ (namely those multi-
index sets with $O$ or more modes) are a strict subset of the objects of
$\bm{\mathsf{MltIdx}}$ which are associated with objects of
$\bm{\mathsf{MltIdx}}_{D}$ (those multi-index sets with $D$ or more modes).

Moreover, for $O>D$, one can define $\binom{O-1}{D-1}$ distinct forgetful
functors from $\bm{\mathsf{MltIdx}}_{O}$ to $\bm{\mathsf{MltIdx}}_{D}$. I.e.
there are $\binom{O-1}{D-1}$ distinct ways for a multi-index set to “forget”
its $O$-fold Cartesian product structure, and to remember only a coarser
$D$-fold Cartesian product structure instead. Each given contiguous partition
of $[O]$ into $D$ blocks provides a pattern for merging the $O$ blocks of
modes of the objects in $\bm{\mathsf{MltIdx}}_{O}$ into $D$ blocks of modes,
and thus selects associated objects of $\bm{\mathsf{MltIdx}}_{D}$.

For example, there are $\binom{3-1}{2-1}=2$ distinct forgetful functors from
$\bm{\mathsf{MltIdx}}_{3}$ to $\bm{\mathsf{MltIdx}}_{2}$.

The first such functor, denoted $\mathscr{F}_{1|23}$, corresponding to the
contiguous partition $\\{\\{1\\},\\{2,3\\}\\}$, allows an object of
$\bm{\mathsf{MltIdx}}_{3}$ to remember its first block of modes, but makes
that object of $\bm{\mathsf{MltIdx}}_{3}$ forget that its second and third
blocks of modes were distinct and thus merges them into one block of modes.
Here are some examples of the behavior of $\mathscr{F}_{1|23}$, where
$[N_{\scaleto{[4]\mathstrut}{0.5em}}]\overset{\text{def}}{=}[5]\times[37]\times[29]\times[41]$:

$\begin{array}[]{lcl}\mathscr{F}_{1|23}([M_{\scaleto{[3]\mathstrut}{0.5em}}])&=&[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|23}\\\
\mathscr{F}_{1|23}([N_{\scaleto{[4]\mathstrut}{0.5em}}]_{12|3|4})&=&[N_{\scaleto{[4]\mathstrut}{0.5em}}]_{12|34}\\\
\end{array}$

The second such functor, denoted $\mathscr{F}_{12|3}$, corresponding to the
contiguous partition $\\{\\{1,2\\},\\{3\\}\\}$, makes an object of
$\bm{\mathsf{MltIdx}}_{3}$ forget that its first and second of modes were
distinct and thus merges them into one block of modes, but allows that object
of $\bm{\mathsf{MltIdx}}_{3}$ to remember its third block of modes. Here are
some examples of the behavior of $\mathscr{F}_{12|3}$:

$\begin{array}[]{lcl}\mathscr{F}_{12|3}([M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|2|3})&=&[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{12|3}\\\
\mathscr{F}_{12|3}([N_{\scaleto{[4]\mathstrut}{0.5em}}]_{12|3|4})&=&[N_{\scaleto{[4]\mathstrut}{0.5em}}]_{123|4}\\\
\end{array}$

Observe that these forgetful functors do not modify any of the morphisms in
$\bm{\mathsf{MltIdx}}_{O}$. If a function (i.e. morphism in
$\bm{\mathsf{MltIdx}}$) $f$ is a valid morphism in $\bm{\mathsf{MltIdx}}_{O}$
from some
$[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]_{P_{1}}$
to
$[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]_{P_{2}}$,
then for any contiguous partition $Q$ of $[O]$ into $D$ blocks, $f$ will also
be a valid morphism in $\bm{\mathsf{MltIdx}}_{D}$ from103103103Here $P_{1}$
denotes a contiguous partition of $[O_{1}]$ into $O$ blocks, $P_{2}$ denotes a
contiguous partition of $[O_{2}]$ into $O$ blocks. See appendix A.2 for an
explanation of the notation $\blacktriangleright$.
$\mathscr{F}_{Q}([M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]_{P_{1}})=[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]_{P_{1}\blacktriangleright
Q}$ to
$\mathscr{F}_{Q}([M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]_{P_{2}})=[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]_{P_{2}\blacktriangleright
Q}$. In other words, for any $O>D$, any function preserving $O$-fold Cartesian
product structure will also preserve any coarser $D$-fold Cartesian product
structure (and thus also be a morphism in $\bm{\mathsf{MltIdx}}_{D}$).

For instance, consider the function:

$g=g_{1}\times g_{2}\times
g_{3}:[M_{\scaleto{[3]\mathstrut}{0.5em}}]\to[N_{\scaleto{[4]\mathstrut}{0.5em}}]\,,$

where the functions $g_{1}$, $g_{2}$, and $g_{3}$ have the following domains
and codomains:

$g_{1}:[7]\to[5]\times[37]\,,\quad g_{2}:[5]\to[29]\,,\quad
g_{3}:[14]\to[41]\,.$

Then $g$ is a morphism in $\bm{\mathsf{MltIdx}}_{3}$:

$[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|2|3}\quad\to\quad[N_{\scaleto{[4]\mathstrut}{0.5em}}]_{12|3|4}\,,$

a morphism in $\bm{\mathsf{MltIdx}}_{2}$ after applying the second forgetful
functor
$\mathscr{F}_{12|3}:\bm{\mathsf{MltIdx}}_{3}\to\bm{\mathsf{MltIdx}}_{2}$:

$\mathscr{F}_{12|3}([M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|2|3})=[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{12|3}\quad\to\quad[N_{\scaleto{[4]\mathstrut}{0.5em}}]_{123|4}=\mathscr{F}_{12|3}([N_{\scaleto{[4]\mathstrut}{0.5em}}]_{12|3|4})\,,$

and a morphism in $\bm{\mathsf{MltIdx}}_{2}$ after applying the first
forgetful functor
$\mathscr{F}_{1|23}:\bm{\mathsf{MltIdx}}_{3}\to\bm{\mathsf{MltIdx}}_{2}$:

$\mathscr{F}_{1|23}([M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|2|3})=[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|23}\quad\to\quad[N_{\scaleto{[4]\mathstrut}{0.5em}}]_{12|34}=\mathscr{F}_{1|23}([N_{\scaleto{[4]\mathstrut}{0.5em}}]_{12|3|4})\,.$

However, the converse is false. In other words, there will always be morphisms
in $\bm{\mathsf{MltIdx}}_{D}$ from
$\mathscr{F}_{Q}([M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]_{P_{1}})$
to
$\mathscr{F}_{Q}([M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]_{P_{2}})$
which do not preserve $O$-fold Cartesian product structure, i.e. which are not
morphisms in $\bm{\mathsf{MltIdx}}_{O}$ from
$[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]_{P_{1}}$
to
$[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]_{P_{2}}$.
Such morphisms in $\bm{\mathsf{MltIdx}}_{D}$ are not the image of any morphism
in $\bm{\mathsf{MltIdx}}_{O}$ under any forgetful functor
$\bm{\mathsf{MltIdx}}_{O}\to\bm{\mathsf{MltIdx}}_{D}$. Thus the image of any
of these forgetful functors always fails to be a full subcategory of
$\bm{\mathsf{MltIdx}}_{D}$.

As an example, while the function

$f=f_{1}\times
f_{2}:[M_{\scaleto{[3]\mathstrut}{0.5em}}]\to[N_{\scaleto{[4]\mathstrut}{0.5em}}]\,,$

where the functions $f_{1}$ and $f_{2}$ have the following domains and
codomains:

$f_{1}:[7]\times[5]\to[5]\times[37]\times[29]\,,\quad f_{2}:[14]\to[41]\,,$

is a morphism in $\bm{\mathsf{MltIdx}}_{2}$:

$\mathscr{F}_{12|3}([M_{\scaleto{[3]\mathstrut}{0.5em}}])=[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{12|3}\quad\to\quad[N_{\scaleto{[4]\mathstrut}{0.5em}}]_{123|4}=\mathscr{F}_{12|3}([N_{\scaleto{[4]\mathstrut}{0.5em}}]_{12|3|4})\,,$

in general it is not a morphism:

$[M_{\scaleto{[3]\mathstrut}{0.5em}}]_{1|2|3}\quad\to\quad[N_{\scaleto{[4]\mathstrut}{0.5em}}]_{12|3|4}$

in $\bm{\mathsf{MltIdx}}_{3}$ (except whenever $f_{1}=h_{1}\times h_{2}$ for
some $h_{1}:[7]\to[5]\times[37]$ and $h_{2}:[5]\to[29]$).

Note finally that, for each $D\geq 1$, the category $\bm{\mathsf{MltIdx}}_{D}$
is equivalent to the image of the $D$-fold Cartesian product functor defined
above for the category $\bm{\mathsf{MltIdx}}$ in section A.4.2:

$\bigtimes_{d=1}^{D}\underbrace{\bm{\mathsf{MltIdx}}\times\cdots\times\bm{\mathsf{MltIdx}}}_{D\text{
times}}\to\bm{\mathsf{MltIdx}}\,.$

In other words, every object of $\bm{\mathsf{MltIdx}}_{D}$ has isomorphic
$D$-fold Cartesian product structure to some object
$\bigtimes_{d=1}^{D}[M^{\scriptsize\scaleto{(d)\mathstrut}{0.7em}}_{\scaleto{[O_{d}]\mathstrut}{0.5em}}]$
in the image of the $D$-fold Cartesian product functor.

##  Appendix A.5 Identification of $\mathbb{R}^{L}$ with $\mathbb{R}^{[L]}$

It can be verified for each $\ell\in[L]$, the projection functions
$\pi_{\ell}:\mathbb{R}^{[L]}\to\mathbb{R}$ given by
$\mathbf{v}\mapsto\mathbf{v}(\ell)\overset{\text{def}}{=}v_{\ell}$ are linear,
and that $\mathbb{R}^{[L]}$ together with the $L$ projection functions
$\pi_{\ell}$ satisfy the universal property of the $L$-fold product for
$\mathbb{R}$ in the category of finite-dimensional real vector spaces.

One may also verify that the $L$-fold Cartesian product of copies of
$\mathbb{R}$, $\bigtimes_{\ell=1}^{L}\mathbb{R}$, whose elements correspond to
ordered lists, or tuples, of real numbers of length $L$,
$(v_{1},\dots,v_{\ell},\dots,v_{L})$, is also a real vector space, and that
$\bigtimes_{\ell=1}^{L}\mathbb{R}$ together with the $L$ linear projection
functions $p_{\ell}:\bigtimes_{\ell=1}^{L}\mathbb{R}\to\mathbb{R}$ (which,
given such a tuple $(v_{1},\dots,v_{\ell},\dots,v_{L})$ return the $\ell$’th
element $v_{\ell}$) also satisfy the universal property of the $L$-fold
product of $\mathbb{R}$ in the category of finite-dimensional real vector
spaces.

Therefore, there is a canonical linear isomorphism104104104It is the unique
isomorphism compatible with the specified projection functions $\pi_{\ell}$
and $p_{\ell}$. between $\mathbb{R}^{[L]}$ and
$\bigtimes_{\ell=1}^{L}\mathbb{R}$, such that they are “essentially the same”
real vector space. In light of this, in what follows $\mathbb{R}^{[L]}$ and
$\bigtimes_{\ell=1}^{L}\mathbb{R}$ will generally not be distinguished between
explicitly. Instead the notation105105105Which is more commonly reserved for
$\bigtimes_{\ell=1}^{L}\mathbb{R}$ only. $\mathbb{R}^{L}$ will be used to
denote both spaces and the notation $(v_{1},\dots,v_{\ell},\dots,v_{L})$ to
denote elements of either space.

##  Appendix A.6 Coordinate Isomorphisms

Every object $\mathbb{V}$ in $\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$ is
isomorphic as a vector space to $\mathbb{R}^{M}$ for $M=\dim(V)$. Similarly,
every object $\mathbb{T}$ (with “tensor factors” $\mathbb{V}_{1}$,
$\mathbb{V}_{2}$, …, $\mathbb{V}_{D}$) of
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$ is isomorphic as a $D$-fold
tensor space to $\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$ where
$[M_{\scaleto{[D]\mathstrut}{0.5em}}]=\bigtimes_{d=1}^{D}[M_{d}]$ with
$M_{d}=\dim(\mathbb{V}_{d})$ for all $d\in\mathbb{V}_{d}$. Combining these two
statements, one could reach the conclusion that (for finite-dimensional spaces
with real scalars) all problems in linear algebra and all problems in
multilinear algebra can be reduced to the study of coordinate spaces
$\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$. This is the reason why
such spaces are the focus of this review.

###  A.6.1 Coordinate Isomorphisms for Vector Spaces

Given a vector space $\mathbb{V}$ with $\dim(\mathbb{V})=M$, choose some basis
$\mathscr{B}=\\{\mathbf{b}_{1},\dots,\mathbf{b}_{M}\\}$, which then uniquely
defines $M$ linear functions in
$\mathcal{L}(\mathbb{V};\mathbb{R})\overset{\text{def}}{=}\left(\mathbb{V}\right)^{\ast}$
via the formulae:

$\varphi_{m}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}:\mathbf{b}_{\mu}\mapsto\delta_{m\mu}\,,$

where $\delta_{m\mu}$ denotes the Kronecker delta ($\delta_{m\mu}=1$ if
$m=\mu$ and $0$ otherwise). Then with respect to the chosen basis
$\mathscr{B}$, any $\mathbf{v}\in\mathbb{V}$ may be written in the form:

$\mathbf{v}=\varphi_{1}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})\mathbf{b}_{1}+\varphi_{2}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})\mathbf{b}_{2}+\dots+\varphi_{M}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})\mathbf{b}_{M}\,.$

For any vector $\mathbf{v}\in V$ the $M$ linear functions
$\varphi_{m}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}$ give the
scalar coefficients in the unique linear combination writing $\mathbf{v}$ in
terms of the elements of the basis $\mathscr{B}$. In other words, the $M$
linear functions
$\varphi_{m}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}$ define,
for every vector $\mathbf{v}\in\mathbb{V}$, the coordinates of $\mathbf{v}$
with respect to the basis $\mathscr{B}$.

For any chosen basis $\mathscr{B}$, one can verify that all of the coordinate
functions $\varphi_{1}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}$
together collectively form a basis for $\left(\mathbb{V}\right)^{\ast}$,
denoted $\left(\mathscr{B}\right)^{\ast}$. The basis
$\left(\mathscr{B}\right)^{\ast}$ is called the dual basis of $\mathscr{B}$.

As an example, if $\mathbb{V}=\\{a\sin(x)+b\sin(2x):a,b\in\mathbb{R}\\}$, then
for $\mathbf{v}=\sin(x)+\sin(2x)$, for

$\mathscr{B}=\\{\sin(x),\sin(2x)\\}\,,$

then the values of the coordinate functions are
$\varphi_{1}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})=1$
and
$\varphi_{2}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})=1$,
and for

$\mathscr{C}=\left\\{\frac{1}{2}\sin(x)+\frac{1}{2}\sin(2x),\frac{1}{2}\sin(x)-\frac{1}{2}\sin(2x)\right\\}\,,$

the values of the coordinate functions are
$\varphi_{1}^{\scriptsize\scaleto{(\mathscr{C})\mathstrut}{0.7em}}(\mathbf{v})=2$
and
$\varphi_{2}^{\scriptsize\scaleto{(\mathscr{C})\mathstrut}{0.7em}}(\mathbf{v})=0$.

As another example, the dual basis corresponding to the standard basis of
$\mathbb{R}^{L}$ consists of the projection functions $\pi_{\ell}$ or
equivalently $p_{\ell}$ defined in section A.5. For each $\ell\in[L]$, one has
that
$\varphi_{\ell}^{\scriptsize\scaleto{(L)\mathstrut}{0.7em}}(\mathbf{v})=\pi_{\ell}(\mathbf{v})=v_{\ell}$,
i.e. it just returns the $\ell$’th entry of the vector $\mathbf{v}$. It is
also true for higher-order tensors that the dual basis of the standard basis
is very simple, consisting of coordinate functions which return the
corresponding entry of the tensor.

For any basis $\mathscr{B}$ of any $M$-dimensional real vector space
$\mathbb{V}$, the dual basis $\left(\mathscr{B}\right)^{\ast}$ construction
allows one to define an isomorphism identifying $\mathbb{V}$ with
$\mathbb{R}^{M}$:

$\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}:\mathbb{V}\to\mathbb{R}^{M}$

called the coordinate isomorphism with respect to $\mathscr{B}$, using the
rule of assignment:

$\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}:\mathbf{v}\mapsto\varphi_{1}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})\mathbf{e}_{1}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}+\varphi_{2}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})\mathbf{e}_{2}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}+\cdots+\varphi_{M}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})\mathbf{e}_{M}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}}\,.$

Therefore everything that can be said about $\mathbb{V}$ as a finite-
dimensional real vector space can be explored by choosing a basis for
$\mathscr{B}$ and subsequently working in the coordinate space
$\mathbb{R}^{M}$.

###  A.6.2 Coordinate Isomorphisms for Tensor Spaces

This is largely a straightforward extension106106106Compare what follows with
section 2 of [16] as well as Facts 12 and 13 from section 15-8 of [193]. of
the coordinate isomorphism construction in
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Vect}$, just modified to explicitly capture
the $D$-fold tensor product structure for an object $\mathbb{T}$ in
$\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{D}$. Specifically, given such a
$\mathbb{T}$ (cf. section A.10.2), consider its associated $D$-tuple of vector
spaces, $(\mathbb{V}_{1},\dots,\mathbb{V}_{D})$, and for each $d\in[D]$ define
$M_{d}=\dim(\mathbb{V}_{d})$. Choose, for each $d\in[D]$, a basis
$\mathscr{B}_{d}$ for $\mathbb{V}_{d}$; this in turn defines $D$ dual basises
$\left(\mathscr{B}_{d}\right)^{\ast}=\\{\varphi_{1}^{\scriptsize\scaleto{(\mathscr{B}_{d})\mathstrut}{0.7em}},\dots,\varphi_{M_{d}}^{\scriptsize\scaleto{(\mathscr{B}_{d})\mathstrut}{0.7em}}\\}$,
one for each $d\in[D]$.

Note that, since each $\mathscr{B}_{d}$ is a basis for $\mathbb{V}_{d}$, the
set:

$\mathscr{B}\overset{\text{def}}{=}\\{\mathbfcal{B}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}\overset{\text{def}}{=}\mathbf{b}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{m_{1}}\otimes_{\mathbb{T}}\mathbf{b}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{m_{2}}\otimes_{\mathbb{T}}\cdots\otimes_{\mathbb{T}}\mathbf{b}^{\scriptsize\scaleto{(D)\mathstrut}{0.7em}}_{m_{D}}:\mathbf{b}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{m_{1}}\in\mathscr{B}_{1},\dots,\mathbf{b}^{\scriptsize\scaleto{(D)\mathstrut}{0.7em}}_{m_{D}}\in\mathscr{B}_{d}\\}$

is always a basis for $\mathbb{T}$ [129]107107107Specifically, cf. Lemma 3.11
of [129].. In the above expression, observe that $\otimes_{\mathbb{T}}$ need
not denote the Segre outer product, and refers to whatever the multilinear
map:

$\mathbb{V}_{1}\times\mathbb{V}_{2}\times\cdots\times\mathbb{V}_{D}\overset{\otimes_{\mathbb{T}}}{\longrightarrow}\mathbb{T}$

is which endows $\mathbb{T}$ with its $D$-fold tensor product structure, cf.
section A.10.2.

Therefore, one may define a dual basis for $\left(\mathbb{T}\right)^{\ast}$ as

$\left(\mathscr{B}\right)^{\ast}\overset{\text{def}}{=}\\{\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}:\mathbb{T}\to\mathbb{R}\,,m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]\\}$

where, to clarify,
$[M_{\scaleto{[D]\mathstrut}{0.5em}}]\overset{\text{def}}{=}\bigtimes_{d=1}^{D}[M_{d}]$,
and for each
$m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]$,
the corresponding element of $\left(\mathscr{B}\right)^{\ast}$ is defined via
the formula
$\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbfcal{B}_{\mu_{\scaleto{[D]\mathstrut}{0.5em}}})=\delta_{m_{\scaleto{[D]\mathstrut}{0.5em}}\mu_{\scaleto{[D]\mathstrut}{0.5em}}}$,
which again denotes the Kronecker delta, which equals $1$ if and only if
$m_{\scaleto{[D]\mathstrut}{0.5em}}=\mu_{\scaleto{[D]\mathstrut}{0.5em}}$ and
$0$ otherwise.

Observe that, if $\otimes_{func}$ denotes the tensor product of functions
defined in section 3, then for each multi-index
$m_{\scaleto{[D]\mathstrut}{0.5em}}\overset{\text{def}}{=}(m_{1},\dots,m_{D})$,
the corresponding dual basis element
$\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}$
is the unique linear function $\mathbb{T}\to\mathbb{R}$ guaranteed to exist by
the universal property of the tensor product such that

$\varphi_{m_{1}}^{\scriptsize\scaleto{(\mathscr{B}_{1})\mathstrut}{0.7em}}\otimes_{func}\varphi_{m_{2}}^{\scriptsize\scaleto{(\mathscr{B}_{2})\mathstrut}{0.7em}}\otimes_{func}\cdots\otimes_{func}\varphi_{m_{D}}^{\scriptsize\scaleto{(\mathscr{B}_{D})\mathstrut}{0.7em}}=\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\circ\otimes_{\mathbb{T}}\,.$

Note that as before this dual basis gives one the coordinates of any tensor
$\mathbfcal{T}\in\mathbb{T}$ with respect to the basis $\mathscr{B}$.
Specifically, one has for every $\mathbfcal{T}\in\mathbb{T}$ the following:

$\mathbfcal{T}=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbfcal{T})\mathbfcal{B}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbfcal{T})\mathbf{b}_{m_{1}}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}\otimes_{\mathbb{T}}\cdots\otimes_{\mathbb{T}}\mathbf{b}_{m_{D}}^{\scriptsize\scaleto{(D)\mathstrut}{0.7em}}\,.$

Note further that one can exploit multilinearity such that for any elementary
tensor:

$\mathbfcal{T}=\mathbf{v}_{1}\otimes_{\mathbb{T}}\mathbf{v}_{2}\otimes_{\mathbb{T}}\cdots\otimes_{\mathbb{T}}\mathbf{v}_{D}\,,$

one has the following relationship:

$\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbfcal{T})=\varphi_{m_{1}}^{\scriptsize\scaleto{(\mathscr{B}_{1})\mathstrut}{0.7em}}(\mathbf{v}_{1})\varphi_{m_{2}}^{\scriptsize\scaleto{(\mathscr{B}_{2})\mathstrut}{0.7em}}(\mathbf{v}_{2})\cdots\varphi_{m_{D}}^{\scriptsize\scaleto{(\mathscr{B}_{D})\mathstrut}{0.7em}}(\mathbf{v}_{D})\,,$

and therefore:

$\mathbfcal{T}=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbfcal{T})\mathbfcal{B}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\left(\prod_{d=1}^{D}\varphi_{m_{d}}^{\scriptsize\scaleto{(\mathscr{B}_{d})\mathstrut}{0.7em}}(\mathbf{v}_{d})\right)\mathbfcal{B}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}\,.$

Thus, for any tensor $\mathbfcal{T}\in\mathbb{T}$, given any outer product
decomposition of $\mathbfcal{T}$:

$\mathbfcal{T}=\sum_{k=1}^{K}\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{1}\otimes_{\mathbb{T}}\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{2}\otimes_{\mathbb{T}}\cdots\otimes_{\mathbb{T}}\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{D}\,,$

one has the relationship:

$\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbfcal{T})=\sum_{k=1}^{K}\left(\prod_{d=1}^{D}\varphi_{m_{d}}^{\scriptsize\scaleto{(\mathscr{B}_{d})\mathstrut}{0.7em}}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{d})\right)\,,$

so that finally:

$\mathbfcal{T}=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\left(\sum_{k=1}^{K}\left(\prod_{d=1}^{D}\varphi_{m_{d}}^{\scriptsize\scaleto{(\mathscr{B}_{d})\mathstrut}{0.7em}}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{d})\right)\right)\mathbf{b}_{m_{1}}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}\otimes_{\mathbb{T}}\mathbf{b}_{m_{2}}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}\otimes_{\mathbb{T}}\cdots\otimes_{\mathbb{T}}\mathbf{b}_{m_{D}}^{\scriptsize\scaleto{(D)\mathstrut}{0.7em}}\,.$

The result does not depend on the particular outer product decomposition of
$\mathbfcal{T}$ chosen due to the universal property of the $D$-fold tensor
product satisfied (by assumption) by $\mathbb{T}$ using
$\otimes_{\mathbb{T}}$.

Analogous to before, the existence of a dual basis $\left(B\right)^{\ast}$ for
the chosen basis $\mathscr{B}$ of $\mathbb{T}$, which allows one to write the
coordinates of any $\mathbfcal{T}\in\mathbb{T}$ with respect to $\mathscr{B}$
using the elements of $\left(B\right)^{\ast}$, also enables one to define a
$D$-fold tensor space isomorphism:

$\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}:\mathbb{T}\to\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\,,$

called again the coordinate isomorphism, via the rule of assignment:

$\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}:\mathbfcal{T}\mapsto\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbfcal{T})\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbfcal{T})\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{D}}^{\scriptsize\scaleto{({M_{D}})\mathstrut}{0.7em}}\,,$

where as usual $\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}$ denotes
the unit tensor (see section 2) of
$\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$ corresponding to the
multi-index $m_{\scaleto{[D]\mathstrut}{0.5em}}$, and $\otimes$ denotes the
Segre outer product.

Observe that this $D$-fold tensor space coordinate isomorphism
$\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}$ is already
completely determined by the $D$ respective vector space coordinate
isomorphisms defined in section A.6.1

$\Phi^{\scriptsize\scaleto{(\mathscr{B}_{1})\mathstrut}{0.7em}}:\mathbb{V}_{1}\to\mathbb{R}^{M_{1}}\,,\dots\,,\Phi^{\scriptsize\scaleto{(\mathscr{B}_{d})\mathstrut}{0.7em}}:\mathbb{V}_{d}\to\mathbb{R}^{M_{d}}\,,\dots,\,,\Phi^{\scriptsize\scaleto{(\mathscr{B}_{D})\mathstrut}{0.7em}}:\mathbb{V}_{D}\to\mathbb{R}^{M_{D}}\,,$

since given any outer product decomposition of any tensor
$\mathbfcal{T}\in\mathbb{T}$, one has that:

$\begin{array}[]{rcl}\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbfcal{T})&=&\displaystyle\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\left(\sum_{k=1}^{K}\left(\prod_{d=1}^{D}\varphi_{m_{d}}^{\scriptsize\scaleto{(\mathscr{B}_{d})\mathstrut}{0.7em}}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{d})\right)\right)\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{D}}^{\scriptsize\scaleto{({M_{D}})\mathstrut}{0.7em}}\\\
&=&\displaystyle\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\left(\sum_{k=1}^{K}\left(\prod_{d=1}^{D}\varphi_{m_{d}}^{\scriptsize\scaleto{(\mathscr{B}_{d})\mathstrut}{0.7em}}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{d})\right)\right)\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}\\\
&=&\displaystyle\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\left(\sum_{k=1}^{K}\left(\prod_{d=1}^{D}\left(\Phi^{\scriptsize\scaleto{(\mathscr{B}_{d})\mathstrut}{0.7em}}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{d})\right)_{m_{d}}\right)\right)\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}\,.\end{array}$

In fact, one even has that
$\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}$ is the unique
linear function
$\mathbb{T}\to\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$ such that

$\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\circ\otimes_{\mathbb{T}}=\otimes\circ\left(\Phi^{\scriptsize\scaleto{(\mathscr{B}_{1})\mathstrut}{0.7em}}\times\cdots\times\Phi^{\scriptsize\scaleto{(\mathscr{B}_{D})\mathstrut}{0.7em}}\right)$

which is guaranteed to exist by the universal property of the $D$-fold tensor
product. Thus everything that can ever be said about $\mathbb{T}$ as a
$D$-fold tensor space can be explored by choosing basises for each of the
$\mathbb{V}_{1}$, …, $\mathbb{V}_{D}$, and subsequently working in the
coordinate space $\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$.

##  Appendix A.7 Representing Dual Spaces with Coordinates

Given a finite-dimensional real vector space $\mathbb{V}$ with
$\dim(\mathbb{V})=M$, one has that $\dim(\left(\mathbb{V}\right)^{\ast})=M$
(recall that
$\left(\mathbb{V}\right)^{\ast}\overset{\text{def}}{=}\mathcal{L}(\mathbb{V};\mathbb{R})$),
thus $\mathbb{V}$ and $\left(\mathbb{V}\right)^{\ast}$ are isomorphic as real
vector spaces. The most common choices of isomorphism used to identify them
choose a basis $\mathscr{B}=\\{\mathbf{b}_{1},\dots,\mathbf{b}_{M}\\}$ for
$\mathbb{V}$ and then associate each element $\mathbf{b}_{m}$ of $\mathscr{B}$
with the corresponding element
$\varphi_{m}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}$ of the
dual basis $\left(\mathscr{B}\right)^{\ast}$ of
$\left(\mathbb{V}\right)^{\ast}$. Specifically, the isomorphism
$\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}:\left(\mathbb{V}\right)^{\ast}\to\mathbb{V}$
has the rule of assignment:

$\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}:\phi\mapsto\phi(\mathbf{b}_{1})\mathbf{b}_{1}+\phi(\mathbf{b}_{2})\mathbf{b}_{2}+\cdots+\phi(\mathbf{b}_{M})\mathbf{b}_{M}$

and has inverse
$(\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}})^{-1}:\mathbb{V}\to\left(\mathbb{V}\right)^{\ast}$
with rule of assignment:

$(\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}})^{-1}:\mathbf{v}\mapsto\varphi_{1}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})\varphi_{1}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}+\varphi_{2}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})\varphi_{2}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}+\cdots+\varphi_{M}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}(\mathbf{v})\varphi_{M}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\,.$

The fact that $\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}$ and
$(\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}})^{-1}$ as defined
above are inverses can be seen using the fact that $\mathbb{V}$ is naturally
isomorphic to its double dual
$\left(\left(\mathbb{V}\right)^{\ast}\right)^{\ast}$ via the isomorphism
$\mathbf{v}\mapsto(\phi\mapsto\phi(\mathbf{v}))$. (This isomorphism does not
depend on any choice of basis for $\mathbb{V}$, which is why it is possible
for it to be natural108108108This is precisely defined using the concept of
natural transformation, cf. footnote A.7 of appendix A.10.1. in $\mathbb{V}$.)
The basis $\mathscr{B}$ is effectively also a dual basis for
$\left(\mathscr{B}\right)^{\ast}$, since for any
$\phi\in\left(\mathbb{V}\right)^{\ast}$:

$\phi=\phi(\mathbf{b}_{1})\varphi_{1}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}+\phi(\mathbf{b}_{2})\varphi_{2}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}+\cdots+\phi(\mathbf{b}_{M})\varphi_{M}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\,,$

the functions $\phi\mapsto\phi(\mathbf{b}_{m})$ give the coordinates of $\phi$
with respect to the basis $\left(\mathscr{B}\right)^{\ast}$ for any
$\phi\in\left(\mathbb{V}\right)^{\ast}$.

Observe that the given isomorphism
$\left(\mathbb{V}\right)^{\ast}\overset{\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}}{\underset{(\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}})^{-1}}{\rightleftarrows}}\mathbb{V}$
depends on a choice of basis for $\mathbb{V}$ (or equivalently a choice of
basis for $\left(\mathbb{V}\right)^{\ast}$), and thus is not
natural109109109In fact there is no natural isomorphism between $\mathbb{V}$
and $\left(\mathbb{V}\right)^{\ast}$, although discussing that observation in
the detail it deserves would derail the discussion too far afield. in
$\mathbb{V}$.

Observe also that a choice of isomorphism
$\left(\mathbb{V}\right)^{\ast}\overset{\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}}{\underset{(\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}})^{-1}}{\rightleftarrows}}\mathbb{V}$
is equivalent to choosing a bilinear function
$\mathbb{V}\times\mathbb{V}\to\mathbb{R}$, in the sense that there is a one-
to-one correspondence between identifications $\psi$ of
$\left(\mathbb{V}\right)^{\ast}$ with $\mathbb{V}$ and bilinear functions
$\langle\cdot,\cdot\rangle_{\psi}:\mathbb{V}\times\mathbb{V}\to\mathbb{R}$
given by

$\psi^{-1}:\mathbf{v}\mapsto\langle\mathbf{v},\cdot\rangle_{\psi}$

and

$\langle\cdot,\cdot\rangle_{\psi}:(\mathbf{v}_{1},\mathbf{v}_{2})\mapsto\left(\psi^{-1}(\mathbf{v}_{1})\right)(\mathbf{v}_{2})\,.$

This fact is related to the Riesz Representation theorem and the statement
that Hilbert spaces are isomorphic to their duals, although discussing this in
the detail it deserves would also derail the discussion too far afield. Note
at the very least that the identification between
$\mathcal{L}(\mathbb{V};\left(V\right)^{\ast})$ and
$\mathcal{L}(\mathbb{V},\mathbb{V};\mathbb{R})$ is not just a bijection, but
also a linear isomorphism.

Usually the vector space $\mathbb{V}$ under discussion will be
$\mathbb{R}^{M}$, with the chosen basis the standard basis
$\mathscr{B}=\\{\mathbf{e}_{1}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}},\dots,\mathbf{e}_{M}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}}\\}$.
Note that the dual basis to the standard basis consists of the coordinate
projection functions $\pi_{m}$ as defined in section A.5. For each $m\in[M]$,
the element $\varphi_{m}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}$ of the
dual basis to the standard basis corresponding to
$\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}}$ gives the
$m$’th entry of the vector,
$\varphi_{m}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}(\mathbf{v})=v_{m}$.
The identification of $\mathbb{R}^{M}$ with
$\left(\mathbb{R}^{M}\right)^{\ast}$ via the identification of the standard
basis $\mathscr{B}$ of $\mathbb{R}^{M}$ with its dual basis
$\left(\mathscr{B}\right)^{\ast}$ in $\left(\mathbb{R}^{M}\right)^{\ast}$ also
corresponds to the standard dot product $\bullet$ on $\mathbb{R}^{M}$.

Let
$\varphi_{1}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}},\dots,\varphi_{M}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}$
denote the dual basis $\left(\mathscr{B}\right)^{\ast}$ of
$\left(\mathbb{R}^{M}\right)^{\ast}$ corresponding to the standard basis
${\mathscr{B}=\\{\mathbf{e}_{1}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}},\dots,\mathbf{e}_{M}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}}\\}}$
of $\mathbb{R}^{M}$, i.e. for all $m\in[M]$:

$\varphi_{m}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}(\mathbf{e}_{\mu}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}})=\delta_{m\mu}\,.$

Denote the isomorphism $\left(\mathbb{R}^{M}\right)^{\ast}\to\mathbb{R}^{M}$
defined via sending the standard dual basis $\left(\mathscr{B}\right)^{\ast}$
of $\left(\mathbb{R}^{M}\right)^{\ast}$ to the standard basis $\mathscr{B}$ of
$\mathbb{R}^{M}$ by $\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}$, so
that for all $m\in[M]$:

$\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}:\varphi_{m}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\mapsto\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}}\,.$

Then one has that, for every $\mathbf{v}_{1},\mathbf{v}_{2}\in\mathbb{R}^{M}$:

$\mathbf{v}_{1}\bullet\mathbf{v}_{2}=\left(\left(\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\right)^{-1}(\mathbf{v}_{1})\right)(\mathbf{v}_{2})\,,\quad\left(\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\right)^{-1}:\mathbf{v}\mapsto(\mathbf{w}\mapsto\mathbf{v}\bullet\mathbf{w})\,.$

Given a vector space $\mathbb{V}$ and a choice of basis $\mathscr{B}$ for
$\mathbb{V}$, thus also a choice of basis $\left(\mathscr{B}\right)^{\ast}$
for $\left(\mathbb{V}\right)^{\ast}$, one can define a “dual coordinate
isomorphism”
$\left(\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\right)^{\ast}:\left(\mathbb{V}\right)^{\ast}\to\left(\mathbb{R}^{M}\right)^{\ast}$
such that for all $m\in[M]$:

$\left(\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\right)^{\ast}:\varphi_{m}^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\mapsto\varphi_{m}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\,.$

For any basis $\mathscr{B}$ of any $M$-dimensional real vector space
$\mathbb{V}$ the following is also true:

$\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\circ\left(\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\right)^{\ast}=\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\circ\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\,,$

and since all functions in question are invertible:

$\begin{array}[]{rcl}\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}&=&\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\circ\left(\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\right)^{\ast}\circ\left(\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\right)^{-1}\,,\\\
\left(\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\right)^{\ast}&=&\left(\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\right)^{-1}\circ\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\circ\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\,,\\\
\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}&=&\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\circ\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\circ\left(\left(\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\right)^{\ast}\right)^{-1}\,,\\\
\psi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}&=&\left(\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\right)^{-1}\circ\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}\circ\left(\Phi^{\scriptsize\scaleto{(\mathscr{B})\mathstrut}{0.7em}}\right)^{\ast}\,.\end{array}$

Therefore all questions regarding the identification of any $M$-dimensional
vector space $\mathbb{V}$ with its dual $\left(\mathbb{V}\right)^{\ast}$,
using any choice of basis $\mathscr{B}$ for $\mathbb{V}$, can always be
reduced to the study of the identification of $\mathbb{R}^{M}$ with its dual
$\left(\mathbb{R}^{M}\right)^{\ast}$ using its standard basis. Thus, in what
follows the discussion will be restricted exclusively to the identification of
$\mathbb{R}^{M}$ with $\left(\mathbb{R}^{M}\right)^{\ast}$ via the standard
basis of $\mathbb{R}^{M}$ and its dual basis, with it implicit that the
restricted discussion can readily be generalized from coordinate spaces
$\mathbb{R}^{M}$ with the standard basises to arbitrary finite-dimensional
real vector spaces with arbitrary basises via the use of the coordinate
isomorphisms and dual coordinate isomorphisms.

##  Appendix A.8 Representing Linear Functions with Coordinates

Given the standard identification
$\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}:\left(\mathbb{R}^{M}\right)^{\ast}\to\mathbb{R}^{M}$
from appendix A.7, it is possible not only to identify
$\mathcal{L}(\mathbb{R}^{M};\mathbb{R})=\left(\mathbb{R}^{M}\right)^{\ast}$
with $\mathbb{R}^{M}$, it is also possible to identify the space of all linear
functions $\mathbb{R}^{M}\to\mathbb{R}^{N}$, denoted
$\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{N})$, with
$\mathbb{R}^{[N]\times[M]}$. Specifically, given a linear function
$\mathcal{L}:\mathbb{R}^{M}\to\mathbb{R}^{N}$, one may focus attention onto
the values it takes for the standard basis of $\mathbb{R}^{M}$, i.e.
$\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})\in\mathbb{R}^{N}$
for all $m\in[M]$. Doing this one then discovers the following for any
$\mathbf{v}\in\mathbb{R}^{M}$:

$\mathcal{L}:\mathbf{v}\mapsto\sum_{m=1}^{M}\varphi_{m}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}(\mathbf{v})\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})=\sum_{m=1}^{M}v_{m}\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})\,.$

One can then show that $\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{N})$ satisfies
the universal property of the tensor product for
$\left(\mathbb{R}^{N},\left(\mathbb{R}^{M}\right)^{\ast}\right)$ by using the
tensor product defined for each $\mathbf{w}\in\mathbb{R}^{N}$,
$\phi\in\left(\mathbb{R}^{M}\right)^{\ast}$ via the rule:

$\bigotimes_{\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{N})}:(\mathbf{w},\phi)\mapsto(\mathbf{v}\mapsto\phi(\mathbf{v})\mathbf{w})\,.$

Hence one usually chooses to identify
$\mathcal{L}\in\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{N})$ with the following
tensor $\mathbf{L}\in\mathbb{R}^{N\times M}$:

$\mathbf{L}=\sum_{m=1}^{M}\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})\otimes\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}(\varphi_{m}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}})=\sum_{m=1}^{M}\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})\otimes\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}}\,,$

where in the above equation $\otimes$ denotes the Segre outer product.

The above identification is a tensor product of linear maps as defined in
appendix A.10.3:

$\operatorname{Id}_{\mathbb{R}^{N}}\otimes\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}:\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{N})=\mathbb{R}^{N}\underset{{\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{N})}}{\otimes}\left(\mathbb{R}^{M}\right)^{\ast}\to\mathbb{R}^{N}\otimes\mathbb{R}^{M}=\mathbb{R}^{N\times
M}\,,$

which is an isomorphism in $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{2}$ as it
is the tensor product of two vector space isomorphisms.

$\mathbf{L}$ is the matrix whose $m$’th column $\mathbf{L}_{:m}$ is
$\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})$
represented with respect to the standard basis of $\mathbb{R}^{N}$. This leads
to the typical110110110E.g. as opposed to say something unconventional like
Cracovian notation. For more about Cracovians, see [164]. definition of matrix
multiplication: for any $\mathbf{v}\in\mathbb{R}^{M}$,

$\sum_{m=1}^{M}v_{m}\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})=\sum_{m=1}^{M}v_{m}\mathbf{L}_{:m}=\mathbf{L}\bullet_{2}\mathbf{v}=\mathcal{L}(\mathbf{v})\,.$

###  A.8.1 Linear Functions with Vector Inputs and Tensor Outputs

Clearly one can extend this identification further to linear functions
$\mathcal{L}\in\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$,
since analogous to the relationship above one has for every
$\mathbf{v}\in\mathbb{R}^{M}$ that

$\mathcal{L}:\mathbf{v}\mapsto\sum_{m=1}^{M}\varphi_{m}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}(\mathbf{v})\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})=\sum_{m=1}^{M}v_{m}\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})\,,$

Using $\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}$ one may identify
$\mathcal{L}\in\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$
with an order $(O+1)$ tensor:

$\mathbfcal{L}=\sum_{m=1}^{M}\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})\otimes\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}(\varphi_{m}^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}})=\sum_{m=1}^{M}\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})\otimes\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}}\,,$

where as before $\otimes$ denotes the Segre outer product in the above
expression. $\mathbfcal{L}$ is a tensor whose $m$’th hyperslice of shape
$[N_{\scaleto{[O]\mathstrut}{0.5em}}]$ (corresponding to the first $O$ modes)
$\mathbfcal{L}_{:_{[O]}m}$ is the value of
$\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})$
represented with respect to the standard basis of unit tensors
$\mathbfcal{I}_{n_{\scaleto{[O]\mathstrut}{0.5em}}}$ of
$\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}$.

Once more analogous to the relationship above, the value of $\mathcal{L}$
applied to any vector $\mathbf{v}\in\mathbb{R}^{M}$ can be expressed via
contraction with the final (in this case $(O+1)$’th) mode of the tensor
$\mathbfcal{L}$:

$\sum_{m=1}^{{M}}v_{m}\mathcal{L}(\mathbf{e}_{m}^{\scriptsize\scaleto{({M})\mathstrut}{0.7em}})=\sum_{m=1}^{{M}}v_{m}\mathbfcal{L}_{:_{[O]}m}=\mathbfcal{L}\bullet_{O+1}\mathbf{v}=\mathcal{L}(\mathbf{v})\,.$

And also again analogous to the above,
$\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$
satisfies the universal property of the tensor product for the pair of vector
spaces
$\left(\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]},\left(\mathbb{R}^{M}\right)^{\ast}\right)$,
when using the tensor product defined for each
$\mathbfcal{U}\in\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}$ and
$\phi\in\left(\mathbb{R}^{M}\right)^{\ast}$ via the rule of assignment:

$\bigotimes_{\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})}:(\mathbfcal{U},\phi)\mapsto(\mathbf{v}\mapsto\phi(\mathbf{v})\mathbfcal{U})\,.$

The above identification of
$\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$
with $\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]\times[M]}$ is a tensor
product of linear maps:

$\operatorname{Id}_{\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}}\otimes\psi^{\scriptsize\scaleto{(M)\mathstrut}{0.7em}}:\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})=\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}\underset{\mathcal{L}(\mathbb{R}^{M};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})}{\otimes}\left(\mathbb{R}^{M}\right)^{\ast}\to\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}\otimes\mathbb{R}^{M}=\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]\times[M]}\,,$

which again is an isomorphism in $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{2}$
as a result of being the tensor product of two vector space isomorphisms as
defined in appendix A.10.3.

###  A.8.2 Linear Functions with Tensor Inputs and Tensor Outputs

Moreover, this procedure extends yet further to identify linear functions
$\mathcal{L}(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$
with tensors in
$\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]\times[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$
once one establishes a standard way of identifying
$\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}$ with
$\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$, since
$\mathcal{L}(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$
satisfies the universal property of the tensor product (and thus is an object
of $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{2}$) for the pair of vector spaces
$\left(\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]},\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}\right)$
when using the tensor product defined for every
$\mathbfcal{U}\in\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}$ and
$\phi\in\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}$
via the relationship:

$\bigotimes_{\mathcal{L}(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})}:(\mathbfcal{U},\phi)\mapsto(\mathbfcal{T}\mapsto\phi(\mathbfcal{T})\mathbfcal{U})\,.$

Note that the object of $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{2}$ which is
most commonly mentioned as the representative vector space of those which
satisfy the universal property of the tensor product for the
pairHendnotepage.3linkHendnote.3cc3macro:-¿Or technically speaking rather
usually as the object of $“fintensor –D+1˝$ satisfying the universal property
of the tensor product for the $(D+1)$-tuple of vector spaces $–(“R ^–“N [O]˝,
“dual –“R ^–M˙1˝˝, “dual –“R ^–M˙2˝˝, “dots , “dual –“R ^–M˙D˝˝ )˝$, or as the
object of $“fintensor –O+D˝$ satisfying the universal property of the tensor
product for the $(O+D)$-tuple of vector spaces““ $– ( “R ^–N˙1˝, “dots , “R
^–N˙O˝, “dual –“R ^–M˙1˝˝, “dots , “dual –“R ^–M˙D˝ ˝ )˝$.
$\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}$ and
$\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}$ is
usually
$\mathcal{L}(\mathbb{R}^{M_{1}},\dots,\mathbb{R}^{M_{D}};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$,
the space of multilinear functions:

$\mathbb{R}^{M_{1}}\times\cdots\times\mathbb{R}^{M_{D}}\to\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}\,,$

cf. Chapter 8 of [294]. However, the space of linear functions
$\mathcal{L}(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$
and the space of multilinear functions
$\mathcal{L}(\mathbb{R}^{M_{1}},\dots,\mathbb{R}^{M_{D}};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$
are isomorphic vector spaces, with isomorphism:

$\Upsilon:\mathcal{L}(\mathbb{R}^{M_{1}},\dots,\mathbb{R}^{M_{D}};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})\to\mathcal{L}(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})\,,\quad\Upsilon:\mathscr{M}\mapsto\mathcal{L}\,,$

where for a given multilinear function
$\mathscr{M}\in\mathcal{L}(\mathbb{R}^{M_{1}},\dots,\mathbb{R}^{M_{D}};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$,
$\mathcal{L}$ denotes the unique linear function
$\mathcal{L}:\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\to\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}$
such that $\mathscr{M}=\mathcal{L}\circ\bigotimes$ (where $\bigotimes$ denotes
the Segre outer product
$\mathbb{R}^{M_{1}}\times\cdots\times\mathbb{R}^{M_{D}}\to\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$)
which is guaranteed to exist due to the fact that
$\mathbb{R}^{[M_{\scaleto{[O]\mathstrut}{0.5em}}]}$, together with the Segre
outer product, satisfies the universal property of the tensor product for the
$D$-tuple of vector spaces $(\mathbb{R}^{M_{1}},\dots,\mathbb{R}^{M_{D}})$.

It is therefore just as
legitimateHendnotepage.4linkHendnote.4dd4macro:-¿Recall that for any two
vector spaces $“bT ˙1$, $“bT ˙2$ which are isomorphic (as vector spaces) via
the isomorphism $“Upsilon : “bT ˙2 “to “bT ˙1$, and for which $“bT ˙1$
satisfies the universal property of the $D$-fold tensor product for vector
spaces $(“V ˙1, “dots , “V ˙D)$ using the tensor product $“bigotimes ˙–“bT
˙1˝$, it is always the case that $“bT ˙2$ then also satisfies the universal
property of the $D$-fold tensor product for $(“V ˙1, “dots , “V ˙D)$ using the
tensor product $“bigotimes ˙–“bT ˙2˝ “defequals “Upsilon ^–-1˝ “circ
“bigotimes ˙–“bT ˙1˝$. Cf. endnote “ref –tensor-universal-property-uniqueness˝
of section “ref –sec:tensor-product˝. to define tensors of mixed variance
$(O,D)$ as being linear functions
$\mathcal{L}:\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\to\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}$
as it is to use the more common definition of tensors of mixed variance
$(O,D)$, namely as being multilinear functions
$\mathscr{M}\in\mathcal{L}(\mathbb{R}^{M_{1}},\dots,\mathbb{R}^{M_{D}};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$.

Recall from section A.6.2 that111111111Technically that section discussed a
more general situation. What is discussed here is the special case where one
has that $\mathbb{T}=\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$,
$\mathbb{V}_{1}=\mathbb{R}^{M_{1}}$, …, $\mathbb{V}_{D}=\mathbb{R}^{M_{D}}$.
that the standard basis of unit tensors $\mathscr{B}$ for
$\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$ is

$\mathscr{B}=\\{\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}=\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{D}}^{\scriptsize\scaleto{({M_{D}})\mathstrut}{0.7em}}:m_{1}\in[M_{1}],\dots,m_{D}\in[M_{D}]\\}\,,$

in other words the standard basis of unit tensors for
$\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$ consists of all possible
combinations of the $D$-fold Segre outer product applied to the standard basis
vectors of $\mathbb{R}^{M_{1}}$, …, $\mathbb{R}^{M_{D}}$, respectively.

Note that for the dual basis
$\left(\mathscr{B}\right)^{\ast}\subset\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}$
of $\mathscr{B}$, each element is simply the linear function
$\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\to\mathbb{R}$ which returns
the corresponding entry of the tensor: for each
$m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]$,
for every $\mathbfcal{T}\in\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$
one has that
$\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{([M_{\scaleto{[D]\mathstrut}{0.5em}}])\mathstrut}{0.7em}}(\mathbfcal{T})=\mathcal{T}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}$,
the $m_{\scaleto{[D]\mathstrut}{0.5em}}$’th entry of $\mathbfcal{T}$.
Moreover, the following is also true for the dual basis
$\left(\mathscr{B}\right)^{\ast}$ of
$\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}$
corresponding to the standard basis $\mathscr{B}$ of
$\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$: for any tensor
$\mathbfcal{T}\in\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$, and any
outer product decomposition

$\mathbfcal{T}=\sum_{k=1}^{{K}}\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{D}\,,\quad\text{where
for all
}k\in[K],\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{1}\in\mathbb{R}^{M_{1}}\,,\dots\,,\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{D}\in\mathbb{R}^{M_{D}}\,,$

then for each multi-index
$m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]$
the following is true of
$\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{([M_{\scaleto{[D]\mathstrut}{0.5em}}])\mathstrut}{0.7em}}$:

$\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{([M_{\scaleto{[D]\mathstrut}{0.5em}}])\mathstrut}{0.7em}}(\mathbfcal{T})=\mathcal{T}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}=\sum_{k=1}^{{K}}\left(\prod_{d=1}^{D}\varphi_{m_{d}}^{\scriptsize\scaleto{(M_{d})\mathstrut}{0.7em}}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{d})\right)=\sum_{k=1}^{{K}}\left(\prod_{d=1}^{D}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{d})_{m_{d}}\right)\,.$

As before,
$\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{([M_{\scaleto{[D]\mathstrut}{0.5em}}])\mathstrut}{0.7em}}$
denotes the element of the dual basis for
$\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}$
corresponding to the unit tensor
$\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}\in\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}=\mathbf{e}_{m_{1}}^{\scriptsize\scaleto{({M_{1}})\mathstrut}{0.7em}}\otimes\cdots\otimes\mathbf{e}_{m_{D}}^{\scriptsize\scaleto{({M_{D}})\mathstrut}{0.7em}}$
in the standard basis for $\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$,
and $\varphi_{m_{d}}^{\scriptsize\scaleto{(M_{d})\mathstrut}{0.7em}}$ denotes
the element of the standard dual basis for
$\left(\mathbb{R}^{M_{d}}\right)^{\ast}$ corresponding to the unit vector
$\mathbf{e}_{m_{d}}^{\scriptsize\scaleto{(M_{d})\mathstrut}{0.7em}}$ in the
standard basis for $\mathbb{R}^{M_{d}}$. The above result does not depend on
the chosen outer product decomposition for $\mathbfcal{T}$ due to the
universal property of the $D$-fold tensor product.

Thus, given any linear function
$\mathcal{L}\in\mathcal{L}(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$,
one has that for every
$\mathbfcal{T}\in\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$:

$\mathcal{L}:\mathbfcal{T}\mapsto\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{([M_{\scaleto{[D]\mathstrut}{0.5em}}])\mathstrut}{0.7em}}(\mathbfcal{T})\mathcal{L}(\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}})=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\mathcal{T}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}\mathcal{L}(\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}})\,.$

Moreover, for every outer product decomposition of $\mathbfcal{T}$:

$\mathbfcal{T}=\sum_{k=1}^{{K}}\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{D}\,,$

one can write additionally that:

$\mathcal{L}:\mathbfcal{T}\mapsto\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\left(\sum_{k=1}^{{K}}\left(\prod_{d=1}^{D}\varphi_{m_{d}}^{\scriptsize\scaleto{(M_{d})\mathstrut}{0.7em}}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{d})\right)\right)\mathcal{L}(\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}})=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\left(\sum_{k=1}^{{K}}\left(\prod_{d=1}^{D}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{d})_{m_{d}}\right)\right)\mathcal{L}(\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}})\,.$

As a result, one then generally chooses to identify $\mathcal{L}$ with the
following tensor
$\mathbfcal{L}\in\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]\times[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$:

$\mathbfcal{L}=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\mathcal{L}(\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}})\otimes\psi^{\scriptsize\scaleto{([M_{\scaleto{[D]\mathstrut}{0.5em}}])\mathstrut}{0.7em}}(\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{([M_{\scaleto{[D]\mathstrut}{0.5em}}])\mathstrut}{0.7em}})=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\mathcal{L}(\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}})\otimes\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}\,,$

where
$\psi^{\scriptsize\scaleto{([M_{\scaleto{[D]\mathstrut}{0.5em}}])\mathstrut}{0.7em}}:\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}\to\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$
denotes the identification of
$\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$ with its dual
$\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}$ which
sends each
$\varphi_{m_{\scaleto{[D]\mathstrut}{0.5em}}}^{\scriptsize\scaleto{([M_{\scaleto{[D]\mathstrut}{0.5em}}])\mathstrut}{0.7em}}\in\left(\mathscr{B}\right)^{\ast}$
of the standard dual basis of
$\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}$ to the
corresponding element
$\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}\in\mathscr{B}$ of the
standard basis of unit tensors of
$\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$, analogous to the
identification defined in appendix A.7 above. Here $\otimes$ also denotes the
Segre outer product.

$\mathbfcal{L}$ is the order $(O+D)$ tensor whose
$m_{\scaleto{[D]\mathstrut}{0.5em}}$’th hypersplice of shape
$[N_{\scaleto{[O]\mathstrut}{0.5em}}]$,
$\mathbfcal{L}_{:_{[O]}m_{\scaleto{[D]\mathstrut}{0.5em}}}$, is the value of
$\mathcal{L}$ applied to the $m_{\scaleto{[D]\mathstrut}{0.5em}}$’th unit
tensor $\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}$. Therefore one has
that, for any
$\mathbfcal{T}\in\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$:

$\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\mathcal{T}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}\mathcal{L}(\mathbfcal{I}_{m_{\scaleto{[D]\mathstrut}{0.5em}}})=\sum_{m_{\scaleto{[D]\mathstrut}{0.5em}}\in[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\mathcal{T}_{m_{\scaleto{[D]\mathstrut}{0.5em}}}\mathbfcal{L}_{:_{[O]}m_{\scaleto{[D]\mathstrut}{0.5em}}}=\mathbfcal{L}\bullet_{[O+D]\setminus[O]}\mathbfcal{T}=\mathcal{L}(\mathbfcal{T})\,.$

Given any outer product decomposition
$\mathbfcal{T}=\sum_{k=1}^{{K}}\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{D}$
of $\mathbfcal{T}$, the above also equals112112112Compare this calculation
with Fact 12 of section 15-1 in [193].

$\sum_{k=1}^{{K}}\mathbfcal{L}\bullet_{O+1}\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{1}\bullet_{O+2}\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{2}\cdots\bullet_{O+D}\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{D}\,.$

As seen above, tensor contractions amount to applying linear functions between
coordinate spaces. For example, using the notation from [205], which is
usually called Einstein notation, the symbol:

$\mathcal{T}^{m_{1}m_{2}}_{m_{3}m_{4}m_{5}}$

corresponds to the standard contravariant representation (as defined above) of
some linear function
$\mathbb{R}^{[M_{3}]\times[M_{4}]\times[M_{5}]}\to\mathbb{R}^{[M_{1}]\times[M_{2}]}$.
The contraction written in Einstein notation as

$\mathcal{T}^{m_{1}m_{2}}_{m_{3}m_{4}m_{5}}\mathcal{U}^{m_{3}m_{4}m_{5}}$

corresponds to the result of applying the aforementioned linear function to
some $\mathbfcal{U}\in\mathbb{R}^{[M_{3}]\times[M_{4}]\times[M_{5}]}$.
Finally, the contraction written in Einstein notation as:

$\mathcal{T}^{m_{1}m_{2}}_{m_{3}m_{4}m_{5}}\mathcal{S}^{m_{3}m_{4}m_{5}}_{m_{6}m_{7}}$

corresponds to composing another linear function in
$\mathcal{L}(\mathbb{R}^{[M_{6}]\times[M_{7}]};\mathbb{R}^{[M_{3}]\times[M_{4}]\times[M_{5}]})$
with the aforementioned linear function in
$\mathcal{L}(\mathbb{R}^{[M_{3}]\times[M_{4}]\times[M_{5}]};\mathbb{R}^{[M_{1}]\times[M_{2}]})$.
Of course the above three examples don’t provide a direct interpretation for
all possible types of contractions which can be written using Einstein
notation, e.g. the very existence of the notion of tensor network implies the
existence of far more complicated patterns, but it is a start.

Observe that, analogous to previous examples, the above identification of
$\mathcal{L}(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})$
with
$\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]\times[M_{\scaleto{[D]\mathstrut}{0.5em}}]}$
corresponds to the tensor product of linear maps:

$\operatorname{Id}_{\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}}\otimes\psi^{\scriptsize\scaleto{([M_{\scaleto{[D]\mathstrut}{0.5em}}])\mathstrut}{0.7em}}:\mathcal{L}(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})=\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}\underset{\mathcal{L}(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]};\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]})}{\otimes}\left(\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\right)^{\ast}\to\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]}\otimes\mathbb{R}^{[M_{\scaleto{[D]\mathstrut}{0.5em}}]}=\mathbb{R}^{[N_{\scaleto{[O]\mathstrut}{0.5em}}]\times[M_{\scaleto{[D]\mathstrut}{0.5em}}]}\,,$

which again is a morphism in $\mathsf{Fin}_{\mathbb{R}}\mathsf{Tens}_{2}$
whose existence and unique definition can be inferred from the universal
property of the tensor product.

##  Appendix A.9 Kronecker Products and Coordinate Representation of the
Tensor Product of Linear Transformations

Recall from section 6.2 the definition of the tensor product

$\bigotimes:\mathcal{L}(\mathbb{R}^{[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]};\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]})\times\mathcal{L}(\mathbb{R}^{[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]};\mathbb{R}^{[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]})\to\mathcal{L}(\mathbb{R}^{[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]};\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]})$

given the rule of assignment:

$\bigotimes:(\mathcal{L}_{1},\mathcal{L}_{2})\mapsto\begin{array}[]{l}\left(\mathbfcal{T}=\displaystyle\sum_{k=1}^{{K}}\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{O_{1}}\otimes\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{O_{1}+1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{O_{1}+O_{2}}\right.\\\
\left.\phantom{(()\mathbfcal{T}=}\mapsto\displaystyle\sum_{k=1}^{{K}}\mathcal{L}_{1}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{O_{1}})\otimes\mathcal{L}_{2}(\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{O_{1}+1}\otimes\cdots\otimes\mathbf{v}^{\scriptsize\scaleto{(k)\mathstrut}{0.7em}}_{O_{1}+O_{2}})\right)\,.\end{array}$

In section A.8 it was explained that the standard way to represent
${\mathcal{L}_{1}\in\mathcal{L}(\mathbb{R}^{[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]};\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]})}$
as a contravariant tensor is as the following tensor
$\mathbfcal{L}_{1}\in\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}$:

$\mathbfcal{L}_{1}=\sum_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}\in[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}\mathcal{L}_{1}(\mathbfcal{I}_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}})\otimes\mathbfcal{I}_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}}\,,$

and the standard way to represent
$\mathcal{L}_{2}\in\mathcal{L}(\mathbb{R}^{[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]};\mathbb{R}^{[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]})$
as a contravariant tensor is as the following tensor
$\mathbfcal{L}_{2}\in\mathbb{R}^{[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$:

$\mathbfcal{L}_{2}=\sum_{m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}\in[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}\mathcal{L}_{2}(\mathbfcal{I}_{m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}})\otimes\mathbfcal{I}_{m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}}\,.$

Thus one might be tempted to conclude that
$\mathbfcal{L}_{1}\otimes\mathbfcal{L}_{2}$ would be the contravariant
representation of $\mathcal{L}_{1}\otimes\mathcal{L}_{2}$, but that is not
true. Most noticeably, while $\mathbfcal{L}_{1}\otimes\mathbfcal{L}_{2}$ is a
tensor in
$\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$,
the pattern in section A.8 gives the contravariant representation of
$\mathcal{L}_{1}\otimes\mathcal{L}_{2}$ as the following tensor
$\mathbfcal{L}$ in
$\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$:

$\begin{array}[]{rcl}\mathbfcal{L}&=&\displaystyle\sum_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}\in[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}(\mathcal{L}_{1}\otimes\mathcal{L}_{2})(\mathbfcal{I}_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}})\otimes\mathbfcal{I}_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}}\\\
&=&\displaystyle\sum_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}\in[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}(\mathcal{L}_{1}\otimes\mathcal{L}_{2})(\mathbfcal{I}_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}}\otimes\mathbfcal{I}_{m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}})\otimes\mathbfcal{I}_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}}\otimes\mathbfcal{I}_{m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}}\\\
&=&\displaystyle\sum_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}\in[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}\mathcal{L}_{1}(\mathbfcal{I}_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}})\otimes\mathcal{L}_{2}(\mathbfcal{I}_{m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}})\otimes\mathbfcal{I}_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}}\otimes\mathbfcal{I}_{m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}}\,.\end{array}$

In contrast, one has that $\mathbfcal{L}_{1}\otimes\mathbfcal{L}_{2}$ equals

$\mathbfcal{L}_{1}\otimes\mathbfcal{L}_{2}=\sum_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}\in[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}\mathcal{L}_{1}(\mathbfcal{I}_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}})\otimes\mathbfcal{I}_{m^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}}\otimes\mathcal{L}_{2}(\mathbfcal{I}_{m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}})\otimes\mathbfcal{I}_{m^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}}\,.$

Comparing the above two outer product representations, it is clear that,
although $\mathbfcal{L}\not=\mathbfcal{L}_{1}\otimes\mathbfcal{L}_{2}$, all of
the information necessary to compute $\mathbfcal{L}$ is still present in
$\mathbfcal{L}_{1}$ and $\mathbfcal{L}_{2}$. There exists a well-defined
bilinear function which, given $\mathbfcal{L}_{1}$ and $\mathbfcal{L}_{2}$,
will return the corresponding $\mathbfcal{L}$:

$\bigotimes_{Z}:\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}\times\mathbb{R}^{[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}\to\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}\,,$

which can be seen to differ from the usual Segre outer product, and yet also
be recovered from the Segre outer product by applying a suitable permutation
of the modes.

Given two arbitrary outer product decompositions of
$\mathbfcal{T}^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}\in\mathbb{R}^{[N^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[D_{1}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(1)\mathstrut}{0.7em}}_{\scaleto{[O_{1}]\mathstrut}{0.5em}}]}$
and
$\mathbfcal{T}^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}\in\mathbb{R}^{[N^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[D_{2}]\mathstrut}{0.5em}}]\times[M^{\scriptsize\scaleto{(2)\mathstrut}{0.7em}}_{\scaleto{[O_{2}]\mathstrut}{0.5em}}]}$: