# Characteristic and Universal Tensor Product Kernels

Zoltán Szabó<EMAIL_ADDRESS>
CMAP, École Polytechnique
Route de Saclay, 91128 Palaiseau, France Bharath K. Sriperumbudur
<EMAIL_ADDRESS>
Department of Statistics
Pennsylvania State University
314 Thomas Building
University Park, PA 16802

###### Abstract

Maximum mean discrepancy (MMD), also called energy distance or N-distance in
statistics and Hilbert-Schmidt independence criterion (HSIC), specifically
distance covariance in statistics, are among the most popular and successful
approaches to quantify the difference and independence of random variables,
respectively. Thanks to their kernel-based foundations, MMD and HSIC are
applicable on a wide variety of domains. Despite their tremendous success,
quite little is known about when HSIC characterizes independence and when MMD
with tensor product kernel can discriminate probability distributions. In this
paper, we answer these questions by studying various notions of characteristic
property of the tensor product kernel.

Keywords: tensor product kernel, kernel mean embedding, characteristic kernel,
$\mathcal{I}$-characteristic kernel, universality, maximum mean discrepancy,
Hilbert-Schmidt independence criterion

##  1 Introduction

Kernel methods (Schölkopf and Smola, 2002) are among the most flexible and
influential tools in machine learning and statistics, with superior
performance demonstrated in a large number of areas and applications. The key
idea in these methods is to map the data samples into a possibly infinite-
dimensional feature space—precisely, a reproducing kernel Hilbert space (RKHS;
Aronszajn, 1950)—and apply linear methods in the feature space, without the
explicit need to compute the map. A generalization of this idea to probability
measures, i.e., mapping probability measures into an RKHS (Berlinet and
Thomas-Agnan, 2004, Chapter 4; Smola et al., 2007) has found novel
applications in nonparametric statistics and machine learning. Formally, given
a probability measure $\mathbb{P}$ defined on a measurable space $\mathscr{X}$
and an RKHS $\mathscr{H}_{k}$ with
$k:\mathscr{X}\times\mathscr{X}\rightarrow\mathbb{R}$ as the reproducing
kernel (which is symmetric and positive definite), $\mathbb{P}$ is embedded
into $\mathscr{H}_{k}$ as

$\mathbb{P}\mapsto\int_{\mathscr{X}}k(\cdot,x)\,\mathrm{d}\mathbb{P}(x)=:\mu_{k}(\mathbb{P}),$
(1)

where $\mu_{k}(\mathbb{P})$ is called the mean element or kernel mean
embedding of $\mathbb{P}$. The _mean embedding_ of $\mathbb{P}$ has lead to a
new generation of solutions in two-sample testing (Baringhaus and Franz, 2004;
Székely and Rizzo, 2004, 2005; Borgwardt et al., 2006; Harchaoui et al., 2007;
Gretton et al., 2012), goodness-of-fit testing (Chwialkowski et al., 2016; Liu
et al., 2016; Jitkrittum et al., 2017b; Balasubramanian et al., 2017), domain
adaptation (Zhang et al., 2013) and generalization (Blanchard et al., 2017),
kernel belief propagation (Song et al., 2011), kernel Bayes’ rule (Fukumizu et
al., 2013), model criticism (Lloyd et al., 2014; Kim et al., 2016),
approximate Bayesian computation (Park et al., 2016), probabilistic
programming (Schölkopf et al., 2015), distribution classification (Muandet et
al., 2011; Zaheer et al., 2017), distribution regression (Szabó et al., 2016;
Law et al., 2018) and topological data analysis (Kusano et al., 2016). A
recent survey on the topic is provided by Muandet et al. (2017).

Crucial to the success of the mean embedding based representation is whether
it encodes all the information about the distribution, in other words whether
the map in (1) is injective in which case the kernel is referred to as
characteristic (Fukumizu et al., 2008; Sriperumbudur et al., 2010). Various
characterizations for the characteristic property of $k$ is known in the
literature (Fukumizu et al., 2008, 2009; Sriperumbudur et al., 2010; Gretton
et al., 2012) using which the popular kernels on $\mathbb{R}^{d}$ such as
Gaussian, Laplacian, B-spline, inverse multiquadrics, and the Matérn class are
shown to be characteristic. The characteristic property is closely related to
the notion of universality (Steinwart, 2001; Micchelli et al., 2006; Carmeli
et al., 2010; Sriperumbudur et al., 2011)—$k$ is said to be universal if the
corresponding RKHS $\mathscr{H}_{k}$ is dense in a certain target function
class, for example, the class of continuous functions on compact domains—and
the relation between these notions has recently been explored by Sriperumbudur
et al. (2011); Simon-Gabriel and Schölkopf (2016).

Based on the mean embedding in (1), Smola et al. (2007) and Gretton et al.
(2012) defined a semi-metric, called the maximum mean discrepancy (MMD) on the
space of probability measures:

$\text{MMD}_{k}(\mathbb{P},\mathbb{Q}):=\|\mu_{k}(\mathbb{P})-\mu_{k}(\mathbb{Q})\|_{\mathscr{H}_{k}},$

which is a metric iff $k$ is characteristic. A fundamental application of MMD
is in non-parametric hypothesis testing that includes two-sample (Gretton et
al., 2012) and independence tests (Gretton et al., 2008). Particularly in
independence testing, as a measure of independence, MMD measures the distance
between the joint distribution $\mathbb{P}_{XY}$ and the product of marginals
$\mathbb{P}_{X}\otimes\mathbb{P}_{Y}$ of two random variables $X$ and $Y$
which are respectively defined on measurable spaces $\mathscr{X}$ and
$\mathscr{Y}$, with the kernel $k$ being defined on
$\mathscr{X}\times\mathscr{Y}$. As aforementioned, if $k$ is characteristic,
then $\text{MMD}_{k}(\mathbb{P}_{XY},\mathbb{P}_{X}\otimes\mathbb{P}_{Y})=0$
implies $\mathbb{P}_{XY}=\mathbb{P}_{X}\otimes\mathbb{P}_{Y}$, i.e., $X$ and
$Y$ are independent. A simple way to define a kernel on
$\mathscr{X}\times\mathscr{Y}$ is through the tensor product of kernels
$k_{X}$ and $k_{Y}$ defined on $\mathscr{X}$ and $\mathscr{Y}$ respectively:
$k=k_{X}\otimes k_{Y}$, i.e.,
$k\left(\left(x,y\right),\left(x^{\prime},y^{\prime}\right)\right)=k_{X}(x,x^{\prime})k_{Y}(y,y^{\prime}),\,x,x^{\prime}\in\mathscr{X},\,y,y^{\prime}\in\mathscr{Y}$,
with the corresponding RKHS
$\mathscr{H}_{k}=\mathscr{H}_{k_{X}}\otimes\mathscr{H}_{k_{Y}}$ being the
tensor product space generated by $\mathscr{H}_{k_{X}}$ and
$\mathscr{H}_{k_{Y}}$. This means, when $k=k_{X}\otimes k_{Y}$,

$\text{MMD}_{k}(\mathbb{P}_{XY},\mathbb{P}_{X}\otimes\mathbb{P}_{Y})=\left\|\mu_{k_{X}\otimes
k_{Y}}(\mathbb{P}_{XY})-\mu_{k_{X}\otimes
k_{Y}}(\mathbb{P}_{X}\otimes\mathbb{P}_{Y})\right\|_{\mathscr{H}_{k_{X}}\otimes\mathscr{H}_{k_{Y}}}.$
(2)

In addition to the simplicity of defining a joint kernel $k$ on
$\mathscr{X}\times\mathscr{Y}$, the tensor product kernel offers a principled
way of combining inner products ($k_{X}$ and $k_{Y}$) on domains that can
correspond to different modalities (say images, texts, audio). By exploiting
the isomorphism between tensor product Hilbert spaces and the space of
Hilbert-Schmidt operators111In the equivalence one assumes that
$\mathscr{H}_{k_{X}}$, $\mathscr{H}_{k_{Y}}$ are separable; this holds under
mild conditions, for example if $\mathscr{X}$ and $\mathscr{Y}$ are separable
topological domains and $k_{X}$, $k_{Y}$ are continuous (Steinwart and
Christmann, 2008, Lemma 4.33)., it follows from (2) that

$\displaystyle\text{MMD}_{k}(\mathbb{P}_{XY},\mathbb{P}_{X}\otimes\mathbb{P}_{Y})=\|C_{XY}\|_{\text{HS}}=:\text{HSIC}_{k}(\mathbb{P}_{XY}),$
(3)

which is the Hilbert-Schmidt norm of the cross-covariance operator
$C_{XY}:=\mu_{k_{X}\otimes
k_{Y}}(\mathbb{P}_{XY})-\mu_{k_{X}}(\mathbb{P}_{X})\otimes\mu_{k_{Y}}(\mathbb{P}_{Y})$
and is known as the Hilbert-Schmidt independence criterion (HSIC) (Gretton et
al., 2005a). HSIC has enjoyed tremendous success in a variety of applications
such as independent component analysis (Gretton et al., 2005a), feature
selection (Song et al., 2012), independence testing (Gretton et al., 2008;
Jitkrittum et al., 2017a), post selection inference (Yamada et al., 2018) and
causal detection (Mooij et al., 2016; Pfister et al., 2017; Strobl et al.,
2017). Recently, MMD and HSIC (as defined in (3) for two components) have been
shown by Sejdinovic et al. (2013b) to be equivalent to other popular
statistical measures such as the energy distance (Baringhaus and Franz, 2004;
Székely and Rizzo, 2004, 2005)—also known as N-distance (Zinger et al., 1992;
Klebanov, 2005)—and distance covariance (Székely et al., 2007; Székely and
Rizzo, 2009; Lyons, 2013) respectively. HSIC has been generalized to $M\geq$ 2
components (Quadrianto et al., 2009; Sejdinovic et al., 2013a) to measure the
joint independence of $M$ random variables

$\displaystyle\text{HSIC}_{k}\left(\mathbb{P}\right)=\left\|\mu_{\otimes^{M}_{m=1}k_{m}}(\mathbb{P})-\otimes^{M}_{m=1}\mu_{k_{m}}\left(\mathbb{P}_{m}\right)\right\|_{\otimes^{M}_{m=1}\mathscr{H}_{k_{m}}},$
(4)

where $\mathbb{P}$ is a joint measure on the product space
$\mathscr{X}:=\times^{M}_{m=1}\mathscr{X}_{m}$ and
$\left(\mathbb{P}_{m}\right)^{M}_{m=1}$ are the marginal measures of
$\mathbb{P}$ defined on $(\mathscr{X}_{m})^{M}_{m=1}$ respectively. The
extended HSIC measure has recently been analyzed in the context of
independence testing (Pfister et al., 2017). In addition to testing, the
extended HSIC measure is also useful in the problem of independent subspace
analysis (ISA; Cardoso, 1998), wherein the latent sources are separated by
maximizing the degree of independence among them. In all the applications of
HSIC, the key requirement is that $k=\otimes^{M}_{m=1}k_{m}$ captures the
joint independence of $M$ random variables (with joint distribution
$\mathbb{P}$)—we call this property as $\mathcal{I}$-characteristic—, which is
guaranteed if $k$ is characteristic. Since $k$ is defined in terms of
$(k_{m})^{M}_{m=1}$, it is of fundamental importance to understand the
characteristic and $\mathcal{I}$-characteristic properties of $k$ in terms of
the characteristic property of $(k_{m})^{M}_{m=1}$, which is one of the main
goals of this work.

For $M=2$, the characterization of independence, i.e., the
$\mathcal{I}$-characteristic property of $k$, is studied by Blanchard et al.
(2011) and Gretton (2015) where it has been shown that if $k_{1}$ and $k_{2}$
are universal, then $k$ is universal222Blanchard et al. (2011) deal with
$c$-universal kernels while Gretton (2015) deals with $c_{0}$-universal
kernels. A brief description of these notions are given in Section 3. Carmeli
et al. (2010); Sriperumbudur et al. (2010) provide further details on these
notions of universality. and therefore HSIC captures independence. A stronger
version of this result can be obtained by combining (Lyons, 2013, Theorem
3.11) and (Sejdinovic et al., 2013b, Proposition 29): if $k_{1}$ and $k_{2}$
are characteristic, then the HSIC associated with $k=k_{1}\otimes k_{2}$
characterizes independence. Apart from these results, not much is known about
the characteristic/$\mathcal{I}$-characteristic/universality properties of $k$
in terms of the individual kernels. Our goal is to resolve this question and
understand the characteristic, $\mathcal{I}$-characteristic and universal
property of the product kernel ($\otimes_{m=1}^{M}k_{m}$) in terms of the
kernel components ($(k_{m})_{m=1}^{M}$) for $M\geq 2$. Because of the
relatedness of MMD and HSIC to energy distance and distance covariance, our
results also contribute to the better understanding of these other measures
that are popular in the statistical literature.

Specifically, our results shed light on the following surprising phenomena of
the $\mathcal{I}$-characteristic property of $\otimes_{m=1}^{M}k_{m}$ for
$M\geq 3$:

  1. 1.

characteristic property of $(k_{m})_{m=1}^{M}$ is not sufficient but necessary
for $\otimes_{m=1}^{M}k_{m}$ to be $\mathcal{I}$-characteristic;

  2. 2.

universality of $(k_{m})_{m=1}^{M}$ is sufficient for $\otimes_{m=1}^{M}k_{m}$
to be $\mathcal{I}$-characteristic, and

  3. 3.

if at least one of $(k_{m})^{M}_{m=1}$ is only characteristic and not
universal, then $\otimes_{m=1}^{M}k_{m}$ need not be
$\mathcal{I}$-characteristic.

The paper is organized as follows. In Section 3, we conduct a comprehensive
analysis about the above mentioned properties of $k$ and $(k_{m})^{M}_{m=1}$
for any positive integer $M$. To this end, we define various notions of
characteristic property on the product space $\mathscr{X}$ (see Definition 1
and Figure 2(a) in Section 3) and explore the relation between them. In order
to keep our presentation in this section to be non-technical, we relegate the
problem formulation to Section 3, with the main results of the paper being
presented in Section 4. A summary of the results is captured in Figure 1 while
the proofs are provided in Section 5. Various definitions and notation that
are used throughout the paper are collected in Section 2.

${\otimes_{0}\text{-char}}$${\otimes\text{-char}}$char${c_{0}\text{-universal}}$${\mathcal{I}\text{-char}}$${(k_{m})^{M}_{m=1}\,\,\text{char}}$${(k_{m})^{M}_{m=1}\,\,c_{0}\text{-universal}}$$\scriptstyle{\definecolor[named]{.}{rgb}{0,0,1}\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\eqref{eq:char-
relations}}$$\scriptstyle{\definecolor[named]{.}{rgb}{0,0,1}\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\text{Remark}~{}\ref{Rem:remark}(iii)}$$\scriptstyle{\definecolor[named]{.}{rgb}{0,0,1}\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\eqref{eq:char-
relations}}$Remark
7$\scriptstyle{\definecolor[named]{.}{rgb}{0,0,1}\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\eqref{eq:char-
relations}}$$\scriptstyle{\definecolor[named]{.}{rgb}{0,0,1}\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\eqref{eq:char-
relations}}$/Example 1Theorem 5/Example 1/Example
1$\scriptstyle{\definecolor[named]{.}{rgb}{0,0,1}\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\text{Theorem}~{}\ref{thm2:k_m:char<=>xk_m:I-char}\,\,(M=2)}$/$\scriptstyle{\definecolor[named]{.}{rgb}{1,0,0}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\text{Example}~{}\ref{example:k_m:char=does
not=>x_m k_m:I-char}\,\,(M\geq 3)}$/Sriperumbudur et al. (2011)Theorem
3Sriperumbudur et al. (2011) Figure 1: Summary of results: “char” denotes
characteristic. In addition to the usual characteristic property, three new
notions $\otimes_{0}$-characteristic, $\otimes$-characteristic and
$\mathcal{I}$-characteristic are introduced in Definition 1 which along with
$c_{0}$-universal (in the top right corner) correspond to the property of the
tensor product kernel $\otimes^{M}_{m=1}k_{m}$, while the bottom part of the
picture corresponds to the individual kernels $(k_{m})^{M}_{m=1}$ being
characteristic or $c_{0}$-universal. If $(k_{m})^{M}_{m=1}$-s are continuous,
bounded and translation invariant kernels on $\mathbb{R}^{d_{m}},\,m\in[M]$,
all the notions are equivalent (see Theorem 4).

##  2 Definitions and Notation

$\mathbb{N}:=\\{1,2,\ldots\\}$ and $\mathbb{R}$ denotes the set of natural
numbers and real numbers respectively. For $M\in\mathbb{N}$,
$[M]:=\\{1,\ldots,M\\}$. $\mathbf{1}_{d}:=(1,1,\ldots,1)\in\mathbb{R}^{d}$ and
$\mathbf{0}$ denotes the matrix of zeros. For
$a:=(a_{1},\ldots,a_{d})\in\mathbb{R}^{d}$ and
$b:=(b_{1},\ldots,b_{d})\in\mathbb{R}^{d}$, $\langle
a,b\rangle=\sum^{d}_{i=1}a_{i}b_{i}$ is the Euclidean inner product. For sets
$A$ and $B$, $A\backslash B=\\{a\in A:a\notin B\\}$ is their difference, $|A|$
is the cardinality of $A$ and
$\times_{m=1}^{M}A_{m}=\\{\left(a_{1},\ldots,a_{M}\right):a_{m}\in
A_{m}\,m\in[M]\\}$ is the Descartes product of sets $(A_{m})^{M}_{m=1}$.
$\mathscr{P}(\mathscr{X})$ denotes the power set of a set $\mathscr{X}$, i.e.,
all subsets of $\mathscr{X}$ (including the empty set and $\mathscr{X}$). The
Kronecker delta is defined as $\delta_{a,b}=1$ if $a=b$, and zero otherwise.
$\chi_{A}$ is the indicator function of set $A$: $\chi_{A}(x)=1$ if $x\in A$
and $\chi_{A}(x)=0$ otherwise. $\mathbb{R}^{d_{1}\times\ldots\times d_{M}}$ is
the set of $d_{1}\times\ldots\times d_{M}$-sized tensors.

For a topological space $\left(\mathscr{X},\tau_{\mathscr{X}}\right)$,
$\mathscr{B}(\mathscr{X}):=\mathscr{B}(\tau_{\mathscr{X}})$ is the Borel
sigma-algebra on $\mathscr{X}$ induced by the topology $\tau_{\mathscr{X}}$.
Probability and finite signed measures in the paper are meant w.r.t. the
measurable space $(\mathscr{X},\mathscr{B}(\mathscr{X}))$. Given
$\left\\{\left(\mathscr{X}_{i},\tau_{i}\right)\right\\}_{i\in I}$ topological
spaces, their product $\times_{i\in I}\mathscr{X}_{i}$ is enriched with the
product topology; it is the coarsest topology for which the canonical
projections $\pi_{i}:\times_{i\in
I}\mathscr{X}_{i}\rightarrow(\mathscr{X}_{i},\tau_{i})$ are continuous for all
$i\in I$. A topological space $\left(\mathscr{X},\tau_{\mathscr{X}}\right)$ is
called second-countable if $\tau_{\mathscr{X}}$ has a countable
basis.333Second-countability implies separability; in metric spaces the two
notions coincide (Dudley, 2004, Proposition 2.1.4). By the Urysohn’s theorem,
a topological space is separable and metrizable if and only if it is regular,
Hausdorff and second-countable. Any uncountable discrete space is _not_
second-countable. $C(\mathscr{X})$ denotes the space of continuous functions
on $\mathscr{X}$. $C_{0}(\mathscr{X})$ denotes the class of real-valued
functions vanishing at infinity on a locally compact Hausdorff (LCH)
space444LCH spaces include $\mathbb{R}^{d}$, discrete spaces, and topological
manifolds. Open or closed subsets, finite products of LCH spaces are LCH.
Infinite-dimensional Hilbert spaces are _not_ LCH. $\mathscr{X}$, i.e., for
any $\epsilon>0$, the set $\\{x\in\mathscr{X}:|f(x)|\geq\epsilon\\}$ is
compact. $C_{0}(\mathscr{X})$ is endowed with the uniform norm
$\left\|f\right\|_{\infty}=\sup_{x\in\mathscr{X}}|f(x)|$.
$\mathscr{M}_{b}(\mathscr{X})$ and $\mathscr{M}_{1}^{+}(\mathscr{X})$ are the
space of finite signed measures and probability measures on $\mathscr{X}$,
respectively. For $\mathbb{P}_{m}\in\mathscr{M}_{1}^{+}(\mathscr{X}_{m})$,
$\otimes^{M}_{m=1}\mathbb{P}_{m}$ denotes the product probability measure on
the product space $\times^{M}_{m=1}\mathscr{X}_{m}$, i.e.,
$\otimes^{M}_{m=1}\mathbb{P}_{m}\in\mathscr{M}_{1}^{+}(\times^{M}_{m=1}\mathscr{X}_{m})$.
$\delta_{x}$ is the Dirac measure supported on $x\in\mathscr{X}$. For
$\mathbb{F}\in\mathscr{M}_{b}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)$,
the finite signed measure $\mathbb{F}_{m}$ denotes its marginal on
$\mathscr{X}_{m}$. $\mathscr{H}_{k_{m}}$ is the reproducing kernel Hilbert
space (RKHS) associated with the reproducing kernel
$k_{m}:\mathscr{X}_{m}\times\mathscr{X}_{m}\rightarrow\mathbb{R}$, which in
this paper is assumed to be measurable and bounded. The tensor product of
$(k_{m})^{M}_{m=1}$ is a kernel, defined as

$\displaystyle\otimes_{m=1}^{M}k_{m}\left(\left(x_{1},\ldots,x_{M}\right),\left(x_{1}^{\prime},\ldots,x_{M}^{\prime}\right)\right)$
$\displaystyle=\prod_{m=1}^{M}k_{m}\left(x_{m},x_{m}^{\prime}\right),\quad
x_{m},x_{m}^{\prime}\in\mathscr{X}_{m},$

whose associated RKHS is denoted as
$\mathscr{H}_{\otimes_{m=1}^{M}k_{m}}=\otimes_{m=1}^{M}\mathscr{H}_{k_{m}}$
(Berlinet and Thomas-Agnan, 2004, Theorem 13), where the r.h.s. is the tensor
product of RKHSs $(\mathscr{H}_{k_{m}})^{M}_{m=1}$. For
$h_{m}\in\mathscr{H}_{m}$, $m\in[M]$, the multi-linear operator
$\otimes_{m=1}^{M}h_{m}\in\otimes_{m=1}^{M}\mathscr{H}_{m}$ is defined as

$\displaystyle\left(\otimes_{m=1}^{M}h_{m}\right)\left(v_{1},\ldots,v_{M}\right)=\prod_{m=1}^{M}\left<h_{m},v_{m}\right>_{\mathscr{H}_{m}},\quad
v_{m}\in\mathscr{H}_{m}.$

A kernel $k:\mathscr{X}\times\mathscr{X}\rightarrow\mathbb{R}$ defined on a
LCH space $\mathscr{X}$ is called a $c_{0}$-kernel if $k(\cdot,x)\in
C_{0}(\mathscr{X})$ for all $x\in\mathscr{X}$.
$k:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}$ is said to be a
translation invariant kernel on $\mathbb{R}^{d}$ if
$k(x,y)=\psi(x-y),\,x,y\in\mathbb{R}^{d}$ for a positive definite function
$\psi:\mathbb{R}^{d}\rightarrow\mathbb{R}$. $\mu_{k}(\mathbb{F})$ denotes the
kernel mean embedding of $\mathbb{F}\in\mathscr{M}_{b}(\mathscr{X})$ to
$\mathscr{H}_{k}$ which is defined as
$\mu_{k}(\mathbb{F})=\int_{\mathscr{X}}k(\cdot,x)\,\mathrm{d}\mathbb{F}(x)$,
where the integral is meant in the Bochner sense.

##  3 Problem Formulation

In this section, we formally introduce the goal of the paper. To this end, we
start with a definition. For simplicity, throughout the paper, we assume that
all kernels are bounded. The definition is based on the observation
(Sriperumbudur et al., 2010, Lemma 8) that a bounded kernel $k$ on a
topological space $\left(\mathscr{X},\tau_{\mathscr{X}}\right)$ is
characteristic if and only if

$\int_{\mathscr{X}}\int_{\mathscr{X}}k(x,x^{\prime})\,\mathrm{d}\mathbb{F}(x)\,\mathrm{d}\mathbb{F}(x^{\prime})>0,\,\,\forall\,\mathbb{F}\in\mathscr{M}_{b}(\mathscr{X})\backslash\\{0\\}\,\,\text{such
that}\,\,\mathbb{F}(\mathscr{X})=0.$

In other words, characteristic kernels are integrally strictly positive
definite (ispd; see Sriperumbudur et al., 2010, p. 1523) w.r.t. the class of
finite signed measures that assign zero measure to $\mathscr{X}$. The
following definition extends this observation to tensor product kernels on
product spaces.

######  Definition 1 ($\mathscr{F}$-ispd tensor product kernel)

Suppose $k_{m}:\mathscr{X}_{m}\times\mathscr{X}_{m}\rightarrow\mathbb{R}$ is a
bounded kernel on a topological space
$\left(\mathscr{X}_{m},\tau_{\mathscr{X}_{m}}\right),\,m\in[M]$. Let
$\mathscr{F}\subseteq\mathscr{M}_{b}\left(\mathscr{X}\right)$ be such that
$0\in\mathscr{F}$ where $\mathscr{X}:=\times^{M}_{m=1}\mathscr{X}_{m}$.
$k:=\otimes_{m=1}^{M}k_{m}$ is said to be _$\mathscr{F}$ -ispd_ if

$\displaystyle\mu_{k}(\mathbb{F})=0\Rightarrow\mathbb{F}=0\quad(\mathbb{F}\in\mathscr{F})\text{,
or equivalently}$
$\displaystyle\left\|\mu_{k}(\mathbb{F})\right\|_{\mathscr{H}_{k}}^{2}=\int_{\times_{m=1}^{M}\mathscr{X}_{m}}\int_{\times_{m=1}^{M}\mathscr{X}_{m}}\left(\otimes_{m=1}^{M}k_{m}\right)\left(x,x^{\prime}\right)\,\mathrm{d}\mathbb{F}(x)\,\mathrm{d}\mathbb{F}(x^{\prime})>0,\quad\forall\,\mathbb{F}\in\mathscr{F}\backslash\\{0\\}.$
(5)

Specifically,

  * •

if $k_{m}$-s are $c_{0}$-kernels on locally compact Polish (LCP) 555A
topological space is called Polish if it is complete, separable and
metrizable. For example, $\mathbb{R}^{d}$ and countable discrete spaces are
Polish. Open and closed subsets, products and disjoint unions of countably
many Polish spaces are Polish. Every second-countable LCH space is Polish.
spaces $\mathscr{X}_{m}$-s and $\mathscr{F}=\mathscr{M}_{b}(\mathscr{X})$,
then $k$ is called $c_{0}$-universal.

  * •

if

$\displaystyle\mathscr{F}$
$\displaystyle=\left[\mathscr{M}_{b}(\mathscr{X})\right]^{0}\hskip
38.41139pt:=\left\\{\mathbb{F}\in\mathscr{M}_{b}(\mathscr{X}):\mathbb{F}(\mathscr{X})=0\right\\},$
$\displaystyle\mathscr{F}$
$\displaystyle=\left[\otimes_{m=1}^{M}\mathscr{M}_{b}(\mathscr{X}_{m})\right]^{0}:=\left\\{\mathbb{F}\in\otimes_{m=1}^{M}\mathscr{M}_{b}\left(\mathscr{X}_{m}\right),\mathbb{F}(\mathscr{X})=0\right\\},$
$\displaystyle\mathscr{F}$ $\displaystyle=\mathcal{I}\hskip
73.97733pt:=\left\\{\mathbb{P}-\otimes_{m=1}^{M}\mathbb{P}_{m}:\mathbb{P}\in\mathscr{M}_{1}^{+}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)\right\\},\quad(M\geq
2)$ $\displaystyle\mathscr{F}$
$\displaystyle=\otimes_{m=1}^{M}\mathscr{M}_{b}^{0}(\mathscr{X}_{m})\hskip
13.6572pt:=\left\\{\mathbb{F}=\otimes_{m=1}^{M}\mathbb{F}_{m}\,:\,\mathbb{F}_{m}\in\mathscr{M}_{b}\left(\mathscr{X}_{m}\right),\,\mathbb{F}_{m}(\mathscr{X}_{m})=0,\,\,\forall\,m\in[M]\right\\},$

then $k$ is called _characteristic_ , _$\otimes$ -characteristic_,
_$\mathcal{I}$ -characteristic_ and _$\otimes_{0}$ -characteristic_,
respectively.

In Definition 1, $k$ being characteristic matches the usual notion of
characteristic kernels on a product space, i.e., there are no two distinct
probability measures on $\mathscr{X}=\times^{M}_{m=1}\mathscr{X}_{m}$ such
that the MMD between them is zero. The other notions such as
$\otimes$-characteristic, $\mathcal{I}$-characteristic and
$\otimes_{0}$-characteristic are typically weaker than the usual
characteristic property since

$\lx@xy@svg{\hbox{\raise 0.0pt\hbox{\kern
28.80746pt\hbox{\ignorespaces\ignorespaces\ignorespaces\hbox{\vtop{\kern
0.0pt\offinterlineskip\halign{\entry@#!@&&\entry<EMAIL_ADDRESS>0.0pt\hbox{\hbox{\kern 0.0pt\raise 0.0pt\hbox{\hbox{\kern 3.0pt\raise
0.0pt\hbox{$\textstyle{\otimes_{m=1}^{M}\mathscr{M}_{b}^{0}(\mathscr{X}_{m})\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$}}}}}}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces{}\ignorespaces{\hbox{\kern
35.28731pt\raise 0.0pt\hbox{\hbox{\kern
0.0pt\raise-1.75pt\hbox{$\scriptstyle{\subseteq}$}}}}}\ignorespaces{}{\hbox{\kern
43.03383pt\raise 0.0pt\hbox{\hbox{\kern 0.0pt\raise 0.0pt\hbox{\hbox{\kern
3.0pt\raise
0.0pt\hbox{$\textstyle{\left[\otimes_{m=1}^{M}\mathscr{M}_{b}(\mathscr{X}_{m})\right]^{0}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$}}}}}}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces{}\ignorespaces{\hbox{\kern
113.3953pt\raise 0.0pt\hbox{\hbox{\kern
0.0pt\raise-1.75pt\hbox{$\scriptstyle{\subseteq}$}}}}}\ignorespaces{}{\hbox{\kern
123.23071pt\raise 0.0pt\hbox{\hbox{\kern 0.0pt\raise 0.0pt\hbox{\hbox{\kern
3.0pt\raise
0.0pt\hbox{$\textstyle{\left[\mathscr{M}_{b}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)\right]^{0}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$}}}}}}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces{}\ignorespaces{\hbox{\kern
191.50328pt\raise 0.0pt\hbox{\hbox{\kern
0.0pt\raise-1.75pt\hbox{$\scriptstyle{\subseteq}$}}}}}\ignorespaces{}{\hbox{\kern
203.4276pt\raise 0.0pt\hbox{\hbox{\kern 0.0pt\raise 0.0pt\hbox{\hbox{\kern
3.0pt\raise
0.0pt\hbox{$\textstyle{\mathscr{M}_{b}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)}$}}}}}}}{\hbox{\kern-3.0pt\raise-26.6998pt\hbox{\hbox{\kern
0.0pt\raise 0.0pt\hbox{\hbox{\kern 3.0pt\raise
0.0pt\hbox{$\textstyle{}$}}}}}}}{\hbox{\kern
73.01909pt\raise-26.6998pt\hbox{\hbox{\kern 0.0pt\raise 0.0pt\hbox{\hbox{\kern
3.0pt\raise 0.0pt\hbox{$\textstyle{}$}}}}}}}{\hbox{\kern
151.41042pt\raise-26.6998pt\hbox{\hbox{\kern 0.0pt\raise
0.0pt\hbox{\hbox{\kern 3.0pt\raise
0.0pt\hbox{$\textstyle{\mathcal{I}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$}}}}}}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces{}\ignorespaces{\hbox{\kern
153.51418pt\raise-13.3499pt\hbox{\hbox{\kern
0.0pt\raise-2.72223pt\hbox{$\scriptstyle{\rotatebox{90.0}{\text{$\subseteq$}}}$}}}}}\ignorespaces{}{\hbox{\kern
229.23506pt\raise-26.6998pt\hbox{\hbox{\kern 0.0pt\raise
0.0pt\hbox{\hbox{\kern 3.0pt\raise
0.0pt\hbox{$\textstyle{}$}}}}}}}\ignorespaces}}}}\ignorespaces.$ (6)

Below we provide further intuition on the $\mathscr{F}$ measure classes
enlisted in Definition 1.

###### Remark 2

  * (i)

$\bm{\mathscr{F}=\mathscr{M}_{b}(\mathscr{X}):}$ If $k_{m}$-s are
$c_{0}$-kernels on LCH spaces $\mathscr{X}_{m}$ for all $m\in[M]$, then $k$ is
also a $c_{0}$-kernel on LCH space $\mathscr{X}$ implying that if $k$
satisfies (5), then $k$ is $c_{0}$-universal (Sriperumbudur et al., 2010,
Proposition 2). It is well known (Sriperumbudur et al., 2010) that
$c_{0}$-universality reduces to $c$-universality (i.e., the notion of
universality proposed by Steinwart, 2001) if $\mathscr{X}$ is compact which is
guaranteed if and only if each $\mathscr{X}_{m},\,m\in[M]$ is compact.

  * (ii)

$\bm{\mathscr{F}=\mathcal{I}:}$ This family is useful to describe the joint
_independence_ of $M$ random variables—hence the name
$\mathcal{I}$-characteristic—defined on kernel-endowed domains
$(\mathscr{X}_{m})_{m=1}^{M}$: If $\mathbb{P}$ denotes the joint distribution
of random variables $(X_{m})_{m=1}^{M}$ and $(\mathbb{P}_{m})_{m=1}^{M}$ are
the associated marginals on $(\mathscr{X}_{m})_{m=1}^{M}$, then by definition
$k=\otimes_{m=1}^{M}k_{m}$ is $\mathcal{I}$-characteristic iff

$\displaystyle\emph{HSIC}_{k}(\mathbb{P})=0\Longleftrightarrow\mathbb{P}=\otimes^{M}_{m=1}\mathbb{P}_{m}.$

In other words, HSIC captures joint independence exactly with
$\mathcal{I}$-characteristic kernels. Similarly, the
$\mathcal{I}$-characteristic property ensures that COCO (constrained
covariance; Gretton et al., 2005b) is a joint independence measure as COCO is
defined by replacing the Hilbert-Schmidt norm of the cross-covariance operator
(see (3) and (4)) with its spectral norm.

  * (iii)

$\bm{\mathscr{F}=\otimes_{m=1}^{M}\mathscr{M}_{b}^{0}(\mathscr{X}_{m}):}$ In
this case $\mathscr{F}$ is chosen to be the product of finite signed measures
on $\mathscr{X}$ such that each marginal measure $\mathbb{F}_{m}$ assigns zero
to the corresponding space $\mathscr{X}_{m}$. This choice is relevant as the
characteristic property of individual kernels $(k_{m})^{M}_{m=1}$ need _not_
imply the characteristic property of $\otimes_{m=1}^{M}k_{m}$, but is
equivalent to the $\otimes_{0}$-characteristic property of
$\otimes_{m=1}^{M}k_{m}$. The equivalence holds for bounded kernels
$k_{m}:\mathscr{X}_{m}\times\mathscr{X}_{m}\rightarrow\mathbb{R}$ on
topological spaces $\mathscr{X}_{m}$ ($m\in[M]$) since for any
$\mathbb{F}=\otimes_{m=1}^{M}\mathbb{F}_{m}\in\otimes_{m=1}^{M}\mathscr{M}_{b}\left(\mathscr{X}_{m}\right)$,
$\mathbb{F}_{m}(\mathscr{X}_{m})=0$ ($\forall\,m\in[M])$

$\displaystyle\left\|\mu_{k}(\mathbb{F})\right\|_{\mathscr{H}_{\otimes_{m=1}^{M}k_{m}}}^{2}$
$\displaystyle=\prod_{m=1}^{M}\left\|\mu_{k_{m}}(\mathbb{F}_{m})\right\|^{2}_{\mathscr{H}_{k_{m}}},$
(7)

and the l.h.s. is positive iff each term on the r.h.s. is positive.

  * (iv)

$\bm{\mathscr{F}=\left[\otimes_{m=1}^{M}\mathscr{M}_{b}(\mathscr{X}_{m})\right]^{0}:}$
This class is similar to the one discussed in (iii) above—i.e., class of
product measures—with the slight difference that the joint measure
$\mathbb{F}$ is restricted to assign zero measure to $\mathscr{X}$ without
requiring all the marginal measures $\mathbb{F}_{m}$ to assign zero measure to
the corresponding space $\mathscr{X}_{m}$. While the need for considering such
a measure class may not be clear at this juncture, however, based on (7), it
turns out that this choice of $\mathscr{F}$ has quite surprising connections
to the characteristic property and $c_{0}$-universality of the product kernel;
for details see Remark 7.

  * (v)

$\bm{\mathscr{F}}$-ispd relations: Given the relations in (6), it immediately
follows that $k=\otimes^{M}_{m=1}k_{m}$ satisfies

---
$\textstyle{\otimes_{0}\text{-characteristic}}$$\textstyle{\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\otimes\text{-characteristic}}$$\scriptstyle{\Longleftarrow}$characteristic$\scriptstyle{\Longleftarrow}$

$\Leftarrow$

$\textstyle{\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces
c_{0}\text{-universal}}$$\scriptstyle{\Longleftarrow}$$\textstyle{\mathcal{I}\text{-characteristic}}$
(8)

when $\mathscr{X}_{m}$ for all $m\in[M]$ are LCP. A visual illustration of (6)
and (8) is provided in Figure 2.

  * (vi)

$\bm{\left[\otimes_{m=1}^{M}\mathscr{M}_{b}(\mathscr{X}_{m})\right]^{0}\cap\mathcal{I}=\\{0\\}:}$
While it is clear that
$\left[\otimes_{m=1}^{M}\mathscr{M}_{b}(\mathscr{X}_{m})\right]^{0}$ and
$\mathcal{I}$ are subsets of
$\left[\mathscr{M}_{b}(\times^{M}_{m=1}\mathscr{X}_{m})\right]^{0}$, it is
interesting to note that
$\left[\otimes_{m=1}^{M}\mathscr{M}_{b}(\mathscr{X}_{m})\right]^{0}$ and
$\mathcal{I}$ have a trivial intersection with $0$ being the measure common to
each of them, assuming that $\mathscr{X}_{m}$-s are second-countable for all
$m\in[M]$; see Section 5.1.

Having defined the $\mathscr{F}$-ispd property, our goal is to investigate
whether the characteristic or $c_{0}$-universal property of $k_{m}$-s
($m\in[M]$) imply different $\mathscr{F}$-ispd properties of
$\otimes_{m=1}^{M}k_{m}$, and vice versa.

$c_{0}$-univ.characteristic$\mathcal{I}$-char.$\otimes$-char.$\otimes_{0}$-char.(a)Example
1
$\mathscr{M}_{b}(\mathscr{X})$$\left[\mathscr{M}_{b}(\mathscr{X})\right]^{0}$$\mathcal{I}$$\otimes_{m}\mathscr{M}_{b}^{0}(\mathscr{X}_{m})$$[\otimes_{m}\mathscr{M}_{b}(\mathscr{X}_{m})]^{0}$(b)

Figure 2: (a) $\mathscr{F}$-ispd $\otimes_{m=1}^{M}k_{m}$ kernels (see (8));
(b) $\mathscr{F}\subseteq\mathscr{M}_{b}(\mathscr{X})$,
$\mathscr{X}=\times_{m=1}^{M}\mathscr{X}_{m}$. Example 1:
$\otimes_{m=1}^{M}k_{m}$ is $\otimes_{0}$-characteristic but not
$\otimes$-characteristic and therefore not characteristic.

##  4 Main Results

In this section, we present our main results related to the $\mathscr{F}$-ispd
property of tensor product kernels, which are summarized in Figure 1. The
results in this section will deal with various assumptions on
$\mathscr{X}_{m}$, such as second-countability, Hausdorff, locally compact
Hausdorff (LCH) and locally compact Polish (LCP), so that they are presented
in more generality. However, for simplicity, all these assumptions can be
unified by simply assuming a stronger condition that $\mathscr{X}_{m}$’s are
LCP.

Our first example illustrates that the characteristic property of $k_{m}$-s
does not imply the characteristic property of the tensor product kernel. In
light of Remark 2(iv) of Section 3, it follows that the class of
$\otimes_{0}$-characteristic tensor product kernels form a _strictly_ larger
class than characteristic tensor product kernels; see also Figure 2.

###### Example 1

Let $\mathscr{X}_{1}=\mathscr{X}_{2}=\\{1,2\\}$,
$\tau_{\mathscr{X}_{1}}=\tau_{\mathscr{X}_{2}}=\mathscr{P}(\\{1,2\\})$,
$k_{1}(x,x^{\prime})=k_{2}(x,x^{\prime})=2\delta_{x,x^{\prime}}-1$. It is easy
to verify that $k_{1}$ and $k_{2}$ are characteristic. However, it can be
proved that $k_{1}\otimes k_{2}$ is _not_ $\otimes$-characteristic and
therefore not characteristic. On the hand, interestingly, $k_{1}\otimes k_{2}$
is $\mathcal{I}$-characteristic. We refer the reader to Section 5.2 for
details.

In the above example, we showed that the tensor product of $k_{1}$ and $k_{2}$
(which are characteristic kernels) is $\mathcal{I}$-characteristic. The
following result generalizes this behavior for any bounded characteristic
kernels. In addition, under a mild assumption, it shows the converse to be
true for any $M$.

###### Theorem 3

Let $k_{m}:\mathscr{X}_{m}\times\mathscr{X}_{m}\rightarrow\mathbb{R}$ be
bounded kernels on topological spaces $\mathscr{X}_{m}$ for all $m\in[M]$,
$M\geq 2$. Then the following holds.

  * (i)

Suppose $\mathscr{X}_{m}$ is second-countable for all $m\in[M]$ with $M=2$. If
$k_{1}$ and $k_{2}$ are characteristic, then $k_{1}\otimes k_{2}$ is
$\mathcal{I}$-characteristic.

  * (ii)

Suppose $\mathscr{X}_{m}$ is Hausdorff and $|\mathscr{X}_{m}|\geq 2$ for all
$m\in[M]$. If $\otimes_{m=1}^{M}k_{m}$ is $\mathcal{I}$-characteristic, then
$k_{1},\ldots,k_{M}$ are characteristic.

Lyons (2013) has showed an analogous result to Theorem 3(i) for distance
covariances ($M=2$) on metric spaces of negative type (Lyons, 2013, Theorem
3.11), which by Sejdinovic et al. (2013b, Proposition 29) holds for HSIC
yielding the $\mathcal{I}$-characteristic property of $k_{1}\otimes k_{2}$.
Recently, Gretton (2015) presented a direct proof showing that HSIC
corresponding to $k_{1}\otimes k_{2}$ captures independence if $k_{1}$ and
$k_{2}$ are translation invariant characteristic kernels on $\mathbb{R}^{d}$
(which is equivalent to $c_{0}$-universality). Blanchard et al. (2011) proved
a result similar to Theorem 3(i) assuming that $\mathscr{X}_{m}$’s are compact
and $k_{1}$, $k_{2}$ being $c$-universal. In contrast, Theorem 3(i)
establishes the result for bounded kernels on general second-countable
topological spaces. In fact, the results of Gretton (2015); Blanchard et al.
(2011) are special cases of Theorems 4 and 5 below. Theorem 3(i) raises a
pertinent question: whether $\otimes^{M}_{m=1}k_{m}$ is
$\mathcal{I}$-characteristic if $k_{m}$-s are characteristic for all $m\in[M]$
where $M>2$? The following example provides a negative answer to this
question. On a positive side, however, we will see in Theorem 5 that the
$\mathcal{I}$-characteristic property of $\otimes_{m=1}^{M}k_{m}$ can be
guaranteed for any $M\geq 2$ if a stronger condition is imposed on $k_{m}$-s
(and $\mathscr{X}_{m}$-s). Theorem 3(ii) generalizes Proposition 3.15 of Lyons
(2013) for any $M>2$, which states that every kernel $k_{m},\,m\in[M]$ being
characteristic is necessary for the tensor kernel $\otimes^{M}_{m=1}k_{m}$ to
be $\mathcal{I}$-characteristic.

###### Example 2

Let $M=3$ and $\mathscr{X}_{m}:=\\{1,2\\}$,
$\tau_{\mathscr{X}_{m}}=\mathscr{P}(\mathscr{X}_{m})$,
$k_{m}\left(x,x^{\prime}\right)=2\delta_{x,x^{\prime}}-1$ ($m=1,2,3$). As
mentioned in Example 1, $(k_{m})^{3}_{m=1}$ are characteristic. However, it
can be shown that $\otimes_{m=1}^{3}{k_{m}}$ is not
$\mathcal{I}$-characteristic. See Section 5.4 for details.

In Remark 2(iii) and Example 1, we showed that in general, only the
$\otimes_{0}$-characteristic property of $\otimes_{m=1}^{M}k_{m}$ is
equivalent to the characteristic property of $k_{m}$-s. Our next result shows
that all the various notions of characteristic property of
$\otimes_{m=1}^{M}k_{m}$ _coincide_ if $k_{m}$-s are translation-invariant,
continuous bounded kernels on $\mathbb{R}^{d}$.

###### Theorem 4

Suppose
$k_{m}:\mathbb{R}^{d_{m}}\times\mathbb{R}^{d_{m}}\rightarrow\mathbb{R}$ are
continuous, bounded and translation-invariant kernels for all $m\in[M]$. Then
the following statements are equivalent:

  1. (i)

$k_{m}$-s are characteristic for all $m\in[M]$;

  2. (ii)

$\otimes_{m=1}^{M}k_{m}$ is $\otimes_{0}$-characteristic;

  3. (iii)

$\otimes_{m=1}^{M}k_{m}$ is $\otimes$-characteristic;

  4. (iv)

$\otimes_{m=1}^{M}k_{m}$ is $\mathcal{I}$-characteristic;

  5. (v)

$\otimes_{m=1}^{M}k_{m}$ is characteristic.

The following result shows that on LCP spaces, the tensor product of $M\geq 2$
$c_{0}$-universal kernels is also $c_{0}$-universal, and vice versa.

###### Theorem 5

Suppose $k_{m}:\mathscr{X}_{m}\times\mathscr{X}_{m}\rightarrow\mathbb{R}$ are
$c_{0}$-kernels on LCP spaces $\mathscr{X}_{m}$ ($m\in[M]$). Then
$\otimes_{m=1}^{M}k_{m}$ is $c_{0}$-universal iff $k_{m}$-s are
$c_{0}$-universal for all $m\in[M]$.

###### Remark 6

  * (i)

A special case of Theorem 5 for $M=2$ is proved by Lyons (2013, Lemma 3.8) in
the context of distance covariance which reduces to Theorem 5 through the
equivalence established by Sejdinovic et al. (2013b). Another special case of
Theorem 5 is proved by Blanchard et al. (2011, Lemma 5.2) for $c$-universality
with $M=2$ using the Stone-Weierstrass theorem: if $k_{1}$ and $k_{2}$ are
$c$-universal then $k_{1}\otimes k_{2}$ is $c$-universal.

  * (ii)

Since the notions of $c_{0}$-universality and characteristic property are
equivalent for translation invariant $c_{0}$-kernels on $\mathbb{R}^{d}$
(Carmeli et al., 2010, Prop. 5.16, Sriperumbudur et al., 2010, Theorem 9),
Theorem 4 can be considered as a special case of Theorem 5. In other words,
requiring $(k_{m})_{m=1}^{M}$ to be also $c_{0}$-kernels in Theorem 4(i)-(iv)
is equivalent to

    1. (v)

$k_{m}$-s are $c_{0}$-universal for all $m\in[M]$;

    2. (vi)

$\otimes_{m=1}^{M}k_{m}$ is $c_{0}$-universal.

  * (iii)

Since the $c_{0}$-universality of $\otimes_{m=1}^{M}k_{m}$ implies its
$\mathcal{I}$-characteristic property (see (8)), Theorem 5 also provides a
generalization of Theorem 3(i) to $M\geq 2$ under additional assumptions on
$k_{m}$-s, while constraining $\mathscr{X}_{m}$-s to LCP-s instead of second-
countable topological spaces.

In Example 2 and Theorem 5, we showed that for $M\geq 3$ components while the
characteristic property of $(k_{m})_{m=1}^{M}$ is not sufficient, their
universality is enough to guarantee the $\mathcal{I}$-characteristic property
of $\otimes_{m=1}^{M}k_{m}$. The next example demonstrates that these results
are tight: If at least one $k_{m}$ is not universal but only characteristic,
then $\otimes_{m=1}^{M}k_{m}$ might not be $\mathcal{I}$-characteristic.

###### Example 3

Let $M=3$ and $\mathscr{X}_{m}:=\\{1,2\\}$,
$\tau_{\mathscr{X}_{m}}=\mathscr{P}(\mathscr{X}_{m})$, for all $m\in[3]$,
$k_{1}\left(x,x^{\prime}\right)=2\delta_{x,x^{\prime}}-1$, and
$k_{m}\left(x,x^{\prime}\right)=\delta_{x,x^{\prime}}$ ($m=2,3$). $k_{1}$ is
characteristic (Example 1), $k_{2}$ and $k_{3}$ are universal since the
associated Gram matrix
$\mathbf{G}=[k_{m}(x,x^{\prime})]_{x,x^{\prime}\in\mathscr{X}_{m}}$ is an
identity matrix, which is strictly positive definite ($m=2,3$). However,
$\otimes_{m=1}^{3}{k_{m}}$ is _not_ $\mathcal{I}$-characteristic. See Section
5.7 for details.

###### Remark 7

Note that the l.h.s. in (7) is positive if and only if each term on the r.h.s.
is positive, i.e., if $k=\otimes^{M}_{m=1}k_{m}$ is $\otimes$-characteristic
with $k_{m}$-s being $c_{0}$-kernels on LCP $\mathscr{X}_{m}$-s, then all
$k_{m}$-s are $c_{0}$-universal. A similar result was also proved by Steinwart
and Ziegel (2017, Lemma 3.4). Combining this with Theorem 5 yields that for
tensor product $c_{0}$-kernels, the notions of $\otimes$-characteristic,
characteristic and $c_{0}$-universality are equivalent, which is quite
surprising as for a joint kernel $k$ (that is not of product type), these
notions need not necessarily coincide. In light of this discussion, Figure
2(a) can be simplified to Figure 3.

$\otimes$-char.=characteristic=$c_{0}$-universal$\mathcal{I}$-char.$\otimes_{0}$-char.
Figure 3: Simplification of the $\mathscr{F}$-ispd property of tensor product
kernels; see Remark 7.

##  5 Proofs

In this section, we provide the proofs of our results presented in Section 4.

###  5.1 Proof of Remark 2(iv)

By the second-countability of $\mathscr{X}_{m}$-s,
$\mathscr{B}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)=\otimes_{m=1}^{M}\mathscr{B}(\mathscr{X}_{m})$,
where the r.h.s. is defined as the $\sigma$-field generated by the cylinder
sets $A_{m}\times_{n\neq m}\mathscr{X}_{n}$ where $m\in[M]$ and
$A_{m}\in\mathscr{B}(\mathscr{X}_{m})$. Suppose there exists
$\mathbb{F}\in\left[\otimes_{m=1}^{M}\mathscr{M}_{b}(\mathscr{X}_{m})\right]^{0}\cap\mathcal{I}$
such that $\mathbb{F}\neq 0$. This means there exists
$\mathbb{P}\in\mathscr{M}_{1}^{+}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)$
with $(\mathbb{P}_{m})_{m=1}^{M}$ being the marginals of $\mathbb{P}$ such
that
$\mathbb{F}=\otimes_{m=1}^{M}\mathbb{F}_{m}=\mathbb{P}-\otimes_{m=1}^{M}\mathbb{P}_{m}$.
Since $\mathbb{F}\neq 0$ there exists $A_{m}\times_{n\neq m}\mathscr{X}_{n}$
for some $m\in[M]$ and $A_{m}\in\mathscr{B}(\mathscr{X}_{m})$ such that
$0\neq\mathbb{F}(A_{m}\times_{n\neq
m}\mathscr{X}_{n})=\mathbb{F}_{m}(A_{m})\prod_{n\neq
m}\mathbb{F}_{n}(\mathscr{X}_{n})=\mathbb{P}\left(A_{m}\times_{n\neq
m}\mathscr{X}_{n}\right)-\mathbb{P}_{m}(A_{m})\prod_{n\neq
m}\mathbb{P}_{n}(\mathscr{X}_{n})=\mathbb{P}_{m}(A_{m})-\mathbb{P}_{m}(A_{m})=0$,
leading to a contradiction.

###  5.2 Proof of Example 1

The proof is structured as follows.

  1. 1.

First we show that $k:=k_{1}=k_{2}$ is a kernel and it is characteristic.

  2. 2.

Next it is proved that $k_{1}\otimes k_{2}$ is not $\otimes$-characteristic,
which implies $k_{1}\otimes k_{2}$ is not characteristic.

  3. 3.

Finally, the $\mathcal{I}$-characteristic property of $k_{1}\otimes k_{2}$ is
established.

The individual steps are as follows:
$k$ is a kernel. Assume w.l.o.g. that $x_{1}=\ldots=x_{N}=1$,
$x_{N+1}=\ldots=x_{n}=2$. Then it is easy to verify that the Gram matrix
$\mathbf{G}=[k(x_{i},x_{j})]_{i,j=1}^{n}=\mathbf{a}\mathbf{a}^{\top}$ where
$\mathbf{a}:=\left(\mathbf{1}^{\top}_{N},-\mathbf{1}^{\top}_{n-N}\right)^{\top}$
and $\mathbf{a}^{\top}$ is the transpose of $\mathbf{a}$. Clearly $\mathbf{G}$
is positive semidefinite and so $k$ is a kernel.
$k$ is characteristic. We will show that $k$ satisfies (5). On
$\mathscr{X}=\\{1,2\\}$ a finite signed measure $\mathbb{F}$ takes the form
$\mathbb{F}=a_{1}\delta_{1}+a_{2}\delta_{2}$ for some
$a_{1},a_{2}\in\mathbb{R}$. Thus,

$\mathbb{F}\in\mathscr{M}_{b}(\mathscr{X})\backslash\\{0\\}\Leftrightarrow(a_{1},a_{2})\neq\mathbf{0}\quad\quad\text{and}\quad\quad\mathbb{F}(\mathscr{X})=0\Leftrightarrow
a_{1}+a_{2}=0.$ (9)

Consider

$\displaystyle\int_{\mathscr{X}}\int_{\mathscr{X}}k(x,x^{\prime})\,\mathrm{d}\mathbb{F}(x)\,\mathrm{d}\mathbb{F}(x^{\prime})$
$\displaystyle=a_{1}^{2}k(1,1)+a_{2}^{2}k(2,2)+2a_{1}a_{2}k(1,2)$
$\displaystyle=a_{1}^{2}+a_{2}^{2}-2a_{1}a_{2}=(a_{1}-a_{2})^{2}=4a_{1}^{2}>0,$
(10)

where we used (9) and the facts that $k(1,1)=k(2,2)=1$, $k(1,2)=-1$.
$k_{1}\otimes k_{2}$ is not $\otimes$-characteristic. We construct a witness
$\mathbb{F}=\mathbb{F}_{1}\otimes\mathbb{F}_{2}\in\otimes_{m=1}^{2}\mathscr{M}_{b}(\mathscr{X}_{m})\backslash\\{0\\}$
such that

$\displaystyle\mathbb{F}(\mathscr{X}_{1}\times\mathscr{X}_{2})$
$\displaystyle=\mathbb{F}_{1}(\mathscr{X}_{1})\mathbb{F}_{2}(\mathscr{X}_{2})=0,$
(11)

and

$\displaystyle 0$
$\displaystyle=\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\underbrace{(k_{1}\otimes
k_{2})((i_{1},i_{2}),(i_{1}^{\prime},i_{2}^{\prime}))}_{k_{1}(i_{1},i_{1}^{\prime})k_{2}(i_{2},i_{2}^{\prime})}\,\mathrm{d}\mathbb{F}(i_{1},i_{2})\,\mathrm{d}\mathbb{F}(i_{1}^{\prime},i_{2}^{\prime})$
$\displaystyle=\prod_{m=1}^{2}\int_{\mathscr{X}_{m}}\int_{\mathscr{X}_{m}}k_{m}(i_{m},i_{m}^{\prime})\,\mathrm{d}\mathbb{F}_{m}(i_{m})\,\mathrm{d}\mathbb{F}_{m}(i_{m}^{\prime}).$
(12)

Finite signed measures on $\\{1,2\\}$ take the form
$\mathbb{F}_{1}=\mathbb{F}_{1}(\mathbf{a})=a_{1}\delta_{1}+a_{2}\delta_{2}$,
$\mathbb{F}_{2}=\mathbb{F}_{2}(\mathbf{b})=b_{1}\delta_{1}+b_{2}\delta_{2}$
form, where
$\mathbf{a}=(a_{1},a_{2})\in\mathbb{R}^{2},\mathbf{b}=(b_{1},b_{2})\in\mathbb{R}^{2}$.
With these notations, (11) and (12) can be rewritten as

$\displaystyle 0$ $\displaystyle=(a_{1}+a_{2})(b_{1}+b_{2}),$ $\displaystyle
0$
$\displaystyle=\left[\sum_{i,i^{\prime}=1}^{2}k_{1}(i,i^{\prime})a_{i}a_{i^{\prime}}\right]\left[\sum_{j,j^{\prime}=1}^{2}k_{2}(j,j^{\prime})b_{j}b_{j^{\prime}}\right]=(a_{1}-a_{2})^{2}(b_{1}-b_{2})^{2}.$

Keeping the solutions where neither $\mathbf{a}$ nor $\mathbf{b}$ is the zero
vector, there are 2 (symmetric) possibilities: (i) $a_{1}+a_{2}=0$,
$b_{1}=b_{2}$ and (ii) $a_{1}=a_{2}$, $b_{1}+b_{2}=0$. In other words, for any
$a,b\neq 0$, the possibilities are (i) $\mathbf{a}=(a,-a)$, $\mathbf{b}=(b,b)$
and (ii) $\mathbf{a}=(a,a)$, $\mathbf{b}=(b,-b)$. This establishes the
non-$\left[\otimes_{m=1}^{2}\mathscr{M}_{b}(\mathscr{X}_{m})\right]^{0}$-ispd
property of $k_{1}\otimes k_{2}$.
$k_{1}\otimes k_{2}$ is $\mathcal{I}$-characteristic. Our goal is to show that
$k_{1}\otimes k_{2}$ is $\mathcal{I}$-characteristic, i.e., for any
$\mathbb{P}\in\mathscr{M}_{1}^{+}(\mathscr{X}_{1}\times\mathscr{X}_{2})$,
$\mu_{k_{1}\otimes k_{2}}(\mathbb{F})=0$ implies $\mathbb{F}=0$, where
$\mathbb{F}=\mathbb{P}-\mathbb{P}_{1}\otimes\mathbb{P}_{2}$. We divide the
proof into two parts:

  1. 1.

First we derive the equations of

$\displaystyle\mathbb{F}(\mathscr{X}_{1}\times\mathscr{X}_{2})$
$\displaystyle=0\quad\text{and}\quad\int\int_{(\mathscr{X}_{1}\times\mathscr{X}_{2})^{2}}(k_{1}\otimes
k_{2})\left((i,j),(r,s)\right)\,\mathrm{d}\mathbb{F}(i,j)\,\mathrm{d}\mathbb{F}(r,s)=0$
(13)

for general finite signed measures
$\mathbb{F}=\sum_{i,j=1}^{2}a_{ij}\delta_{(i,j)}$ on
$\mathscr{X}_{1}\times\mathscr{X}_{2}$.

  2. 2.

Then, we apply the $\mathbb{F}=\mathbb{P}-\mathbb{P}_{1}\otimes\mathbb{P}_{2}$
parameterization and solve for $\mathbb{P}$ that satisfies (13) to conclude
that $\mathbb{P}=\mathbb{P}_{1}\otimes\mathbb{P}_{2}$, i.e., $\mathbb{F}=0$.
Note that in the chosen parametrization for $\mathbb{F}$,
$\mathbb{F}(\mathscr{X}_{1}\times\mathscr{X}_{2})=0$ holds automatically.

The details are as follows.
Step 1.

$\displaystyle 0$
$\displaystyle=\mathbb{F}(\mathscr{X}_{1}\times\mathscr{X}_{2})\Leftrightarrow
0=a_{11}+a_{12}+a_{21}+a_{22},$ (14) $\displaystyle 0$
$\displaystyle=\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\underbrace{(k_{1}\otimes
k_{2})\left((i,j),(r,s)\right)}_{k_{1}(i,r)k_{2}(j,s)}\,\mathrm{d}\mathbb{F}(i,j)\,\mathrm{d}\mathbb{F}\left(r,s\right)$
$\displaystyle=\sum_{i,j=1}^{2}\sum_{r,s=1}^{2}k_{1}(i,r)k_{2}(j,s)a_{ij}a_{rs}=\sum_{i,r=1}^{2}k_{1}(i,r)\sum_{j,s=1}^{2}k_{2}(j,s)a_{ij}a_{rs}$
$\displaystyle=k_{1}(1,1)\left[k_{2}(1,1)a_{11}a_{11}+k_{2}(1,2)a_{11}a_{12}+k_{2}(2,1)a_{12}a_{11}+k_{2}(2,2)a_{12}a_{12}\right]$
$\displaystyle\quad+k_{1}(1,2)\left[k_{2}(1,1)a_{11}a_{21}+k_{2}(1,2)a_{11}a_{22}+k_{2}(2,1)a_{12}a_{21}+k_{2}(2,2)a_{12}a_{22}\right]$
$\displaystyle\quad+k_{1}(2,1)\left[k_{2}(1,1)a_{21}a_{11}+k_{2}(1,2)a_{21}a_{12}+k_{2}(2,1)a_{22}a_{11}+k_{2}(2,2)a_{22}a_{12}\right]$
$\displaystyle\quad+k_{1}(2,2)\left[k_{2}(1,1)a_{21}a_{21}+k_{2}(1,2)a_{21}a_{22}+k_{2}(2,1)a_{22}a_{21}+k_{2}(2,2)a_{22}a_{22}\right]$
$\displaystyle=\underbrace{\left(a_{11}^{2}-2a_{11}a_{12}+a_{12}^{2}\right)}_{\left(a_{11}-a_{12}\right)^{2}}+\underbrace{\left(a_{21}^{2}-2a_{21}a_{22}+a_{22}^{2}\right)}_{\left(a_{21}-a_{22}\right)^{2}}-2\underbrace{\left(a_{11}a_{21}-a_{11}a_{22}-a_{12}a_{21}+a_{12}a_{22}\right)}_{(a_{11}-a_{12})(a_{21}-a_{22})}$
$\displaystyle=(a_{11}-a_{12}-a_{21}+a_{22})^{2}.$ (15)

Solving (14) and (15) yields

$\displaystyle a_{11}+a_{22}$ $\displaystyle=0\quad\text{and}\quad
a_{12}+a_{21}=0.$ (16)

Step 2. Any
$\mathbb{P}\in\mathscr{M}_{1}^{+}(\mathscr{X}_{1}\times\mathscr{X}_{2})$ can
be parametrized as

$\displaystyle\mathbb{P}$
$\displaystyle=\sum_{i,j=1}^{2}p_{ij}\delta_{(i,j)},\quad p_{ij}\geq
0,\,\forall\,(i,j)\quad\text{and}\quad\sum_{i,j=1}^{2}p_{ij}=1.$ (17)

Let
$\mathbb{F}=\mathbb{P}-\mathbb{P}_{1}\otimes\mathbb{P}_{2}=\sum_{i,j=1}^{2}a_{ij}\delta_{(i,j)}$;
for illustration see Table 1.

$\mathbb{P}$: $y\backslash x$ | $1$ | $2$ | $\mathbb{P}_{2}$
---|---|---|---
$1$ | $p_{11}$ | $p_{21}$ | $q_{1}=p_{11}+p_{21}$
$2$ | $p_{12}$ | $p_{22}$ | $q_{2}=p_{12}+p_{22}$
$\mathbb{P}_{1}$ | $p_{1}=p_{11}+p_{12}$ | $p_{2}=p_{21}+p_{22}$ |

$\Rightarrow$

$\mathbb{F}:=\mathbb{P}-\mathbb{P}_{1}\otimes\mathbb{P}_{2}$ | $1$ | $2$
---|---|---
$1$ | $a_{11}=p_{11}-(p_{11}+p_{12})(p_{11}+p_{21})$ | $a_{21}=p_{21}-(p_{21}+p_{22})(p_{11}+p_{21})$
$2$ | $a_{12}=p_{12}-(p_{11}+p_{12})(p_{12}+p_{22})$ | $a_{22}=p_{22}-(p_{21}+p_{22})(p_{12}+p_{22})$

Table 1: Joint ($\mathbb{P}$), joint minus product of the marginals
($\mathbb{P}-\mathbb{P}_{1}\otimes\mathbb{P}_{2}$).

It follows from step 1 that $\mathbb{F}$ satisfying (16) is equivalent to
satisfying (13). Therefore, for the choice of
$\mathbb{F}:=\mathbb{P}-\mathbb{P}_{1}\otimes\mathbb{P}_{2}$, we obtain

$\displaystyle
p_{11}-(p_{11}+p_{12})(p_{11}+p_{21})+p_{22}-(p_{21}+p_{22})(p_{12}+p_{22})$
$\displaystyle=0,$ (18) $\displaystyle
p_{12}-(p_{11}+p_{12})(p_{12}+p_{22})+p_{21}-(p_{21}+p_{22})(p_{11}+p_{21})$
$\displaystyle=0,$ (19)

where $(p_{ij})_{i,j\in[2]}$ satisfy (17). Solving (17)–(19), we obtain

$\displaystyle p_{11}$
$\displaystyle=\frac{a[1-(a+b)]}{a+b},\,\,\,p_{12}=\frac{b[1-(a+b)]}{a+b},\,\,\,p_{21}=a\quad\text{and}\quad
p_{22}=b,$

with $0\leq a,b\leq 1$, $a+b\leq 1$ and $(a,b)\neq\mathbf{0}$. The resulting
distribution family with its marginals is summarized in Table 2.

$\mathbb{P}$: $y\backslash x$ | $1$ | $2$ | $\mathbb{P}_{2}$
---|---|---|---
$1$ | $p_{11}=\frac{a[1-(a+b)]}{a+b}$ | $p_{21}=a$ | $q_{1}=\frac{a}{a+b}$
$2$ | $p_{12}=\frac{b[1-(a+b)]}{a+b}$ | $p_{22}=b$ | $q_{2}=\frac{b}{a+b}$
$\mathbb{P}_{1}$ | $p_{1}=1-(a+b)$ | $p_{2}=a+b$ |
Table 2: Family of probability distributions solving (17)–(19).

It can be seen that each member of this family (any $a$, $b$ in the constraint
set) factorizes: $\mathbb{P}=\mathbb{P}_{1}\otimes\mathbb{P}_{2}$. In other
words, $\mathbb{F}=\mathbb{P}-\mathbb{P}_{1}\otimes\mathbb{P}_{2}=0$; hence
$k_{1}\otimes k_{2}$ is $\mathcal{I}$-characteristic.
Remark. We would like to mention that while $k_{1}$ and $k_{2}$ are
characteristic, they are not universal. Since $\mathscr{X}$ is finite, the
usual notion of universality (also called $c$-universality) matches with
$c_{0}$-universality. Therefore, from (10), we have
$\int_{\mathscr{X}}\int_{\mathscr{X}}k(x,x^{\prime})\,\mathrm{d}\mathbb{F}(x)\,\mathrm{d}\mathbb{F}(x)=(a_{1}-a_{2})^{2}$
where $\mathbb{F}=a_{1}\delta_{1}+a_{2}\delta_{2}$ for some
$a_{1},a_{2}\in\mathbb{R}\backslash\\{0\\}$. Clearly, the choice of
$a_{1}=a_{2}$ establishes that there exists
$\mathbb{F}\in\mathscr{M}_{b}(\mathscr{X})\backslash\\{0\\}$ such that
$\int_{\mathscr{X}}\int_{\mathscr{X}}k(x,x^{\prime})\,\mathrm{d}\mathbb{F}(x)\,\mathrm{d}\mathbb{F}(x)=0$.
Hence $k$ is not universal. Note that the constraint in (9), which is needed
to verify the characteristic property of $k$ is not needed to verify its
universality.

###  5.3 Proof of Theorem 3

Define $\mathscr{H}_{m}:=\mathscr{H}_{k_{m}}$.
$(i)$ Suppose $k_{1}$ and $k_{2}$ are characteristic and that for some
$\mathbb{F}=\mathbb{P}-\mathbb{P}_{1}\otimes\mathbb{P}_{2}\in\mathcal{I}$,

$\displaystyle\mathscr{H}_{1}\otimes\mathscr{H}_{2}\ni\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\left(k_{1}\otimes
k_{2}\right)(\cdot,x)\,\mathrm{d}\mathbb{F}(x)=\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}k_{1}(\cdot,x_{1})\otimes
k_{2}(\cdot,x_{2})\,\mathrm{d}\mathbb{F}(x)=0,$ (20)

where $x=(x_{1},x_{2})$. We want to show that $\mathbb{F}=0$. By the second-
countability of $\mathscr{X}_{m}$-s, the product $\sigma$-field, i.e.,
$\otimes_{m=1}^{2}\mathscr{B}(\mathscr{X}_{m})$ generated by the cylinder sets
$B_{1}\times\mathscr{X}_{2}$ and $\mathscr{X}_{1}\times B_{2}$
($B_{m}\in\mathscr{B}(\mathscr{X}_{m}),m=1,2$), coincides with the Borel
$\sigma$-field $\mathscr{B}(\mathscr{X}_{1}\times\mathscr{X}_{2})$ on the
product space (Dudley, 2004, Lemma 4.1.7):

$\displaystyle\otimes_{m=1}^{2}\mathscr{B}\left(\mathscr{X}_{m}\right)=\mathscr{B}\left(\mathscr{X}_{1}\times\mathscr{X}_{2}\right).$

Hence, it is sufficient to prove that $\mathbb{F}\left(B_{1}\times
B_{2}\right)=0$, $\forall\,B_{m}\in\mathscr{B}(\mathscr{X}_{m})$, $m=1,2$. To
this end, it follows from (20) that for all $h_{2}\in\mathscr{H}_{2}$,

$\displaystyle\mathscr{H}_{1}\ni$
$\displaystyle\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}k_{1}(\cdot,x_{1})h_{2}(x_{2})\,\mathrm{d}\mathbb{F}(x)=\int_{\mathscr{X}_{1}}k_{1}(\cdot,x_{1})\,\mathrm{d}\nu(x_{1})=0,$
(21)

where

$\displaystyle\nu(B_{1})$
$\displaystyle:=\nu_{h_{2}}(B_{1})=\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\chi_{B_{1}}(x_{1})h_{2}(x_{2})\,\mathrm{d}\mathbb{F}(x),\quad
B_{1}\in\mathscr{B}(\mathscr{X}_{1}).$

Since $k_{1}$ is characteristic, (21) implies $\nu=0$, provided that
$|\nu|(\mathscr{X}_{1})<\infty$ and $\nu(\mathscr{X}_{1})=0$. These two
requirements hold:

$\displaystyle\nu(\mathscr{X}_{1})$
$\displaystyle=\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}h_{2}(x_{2})\,\mathrm{d}\mathbb{F}(x)=\int_{\mathscr{X}_{2}}h_{2}(x_{2})\,\mathrm{d}[\mathbb{P}_{2}-\mathbb{P}_{2}](x_{2})=0,$
$\displaystyle|\nu|(\mathscr{X}_{1})$
$\displaystyle\leq\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\underbrace{\left|h_{2}(x_{2})\right|}_{\big{|}\left<h_{2},k_{2}(\cdot,x_{2})\right>_{\mathscr{H}_{2}}\big{|}}\mathrm{d}[\mathbb{P}+\mathbb{P}_{1}\otimes\mathbb{P}_{2}](x_{1},x_{2})$
$\displaystyle\leq\left\|h_{2}\right\|_{\mathscr{H}_{2}}\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\sqrt{k_{2}(x_{2},x_{2})}\,\mathrm{d}[\mathbb{P}+\mathbb{P}_{1}\otimes\mathbb{P}_{2}](x_{1},x_{2})$
$\displaystyle\leq
2\left\|h_{2}\right\|_{\mathscr{H}_{2}}\int_{\mathscr{X}_{2}}\sqrt{k_{2}(x_{2},x_{2})}\,\mathrm{d}\mathbb{P}_{2}(x_{2})<\infty,$

where the last inequality follows from the boundedness of $k_{2}$. The
established $\nu=0$ implies that for
$\forall\,B_{1}\in\mathscr{B}(\mathscr{X}_{1})$ and
$\forall\,h_{2}\in\mathscr{H}_{2}$,

$\displaystyle 0$
$\displaystyle=\nu(B_{1})=\left<h_{2},\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\chi_{B_{1}}(x_{1})k_{2}(\cdot,x_{2})\,\mathrm{d}\mathbb{F}(x)\right>_{\mathscr{H}_{2}},$

and hence

$\displaystyle
0=\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\chi_{B_{1}}(x_{1})k_{2}(\cdot,x_{2})\,\mathrm{d}\mathbb{F}(x)=\int_{\mathscr{X}_{2}}k_{2}(\cdot,x_{2})\,\mathrm{d}\theta_{B_{1}}(x_{2}),$
(22)

where

$\displaystyle\theta_{B_{1}}(B_{2})=\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\chi_{B_{1}}(x_{1})\chi_{B_{2}}(x_{2})\,\mathrm{d}\mathbb{F}(x),\quad
B_{2}\in\mathscr{B}(\mathscr{X}_{2}).$

Using the characteristic property of $k_{2}$, it follows from (5.3) that
$\theta_{B_{1}}=0$ for $\forall B_{1}\in\mathscr{B}(\mathscr{X}_{1})$, i.e.,

$\displaystyle 0$ $\displaystyle=\theta_{B_{1}}(B_{2})=\mathbb{F}(B_{1}\times
B_{2}),\quad\forall\,B_{1}\in\mathscr{B}(\mathscr{X}_{1}),\,\forall\,B_{2}\in\mathscr{B}(\mathscr{X}_{2})$

provided that $\theta_{B_{1}}(\mathscr{X}_{2})=0$ and
$|\theta_{B_{1}}|(\mathscr{X}_{2})<\infty$. Indeed, both these conditions
hold:

$\displaystyle\theta_{B_{1}}(\mathscr{X}_{2})$
$\displaystyle=\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\chi_{B_{1}}(x_{1})\,\mathrm{d}\mathbb{F}(x)=\int_{\mathscr{X}_{1}}\chi_{B_{1}}(x_{1})\,\mathrm{d}[\mathbb{P}_{1}-\mathbb{P}_{1}](x_{1})=0,$
$\displaystyle|\theta_{B_{1}}|(\mathscr{X}_{2})$
$\displaystyle\leq\int_{\mathscr{X}_{1}\times\mathscr{X}_{2}}\,\mathrm{d}[\mathbb{P}+\mathbb{P}_{1}\otimes\mathbb{P}_{2}](x)=2.$

$(ii)$ Assume w.l.o.g. that $k_{1}$ is not characteristic. This means there
exists
$\mathbb{P}_{1}\neq\mathbb{P}_{1}^{\prime}\in\mathscr{M}_{1}^{+}(\mathscr{X}_{1})$
such that
$\mu_{k_{1}}(\mathbb{P}_{1})=\mu_{k_{1}}\left(\mathbb{P}_{1}^{\prime}\right)$.
Our goal is to construct an
$\mathbb{F}\in\mathscr{M}_{1}^{+}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)$
such that

$\displaystyle\mu_{\otimes_{m=1}^{M}k_{m}}\left(\mathbb{F}-\otimes_{m=1}^{M}\mathbb{F}_{m}\right)$
$\displaystyle=\int_{\times_{m=1}^{M}}\otimes_{m=1}^{M}k_{m}(\cdot,x_{m})\,\mathrm{d}\big{[}\mathbb{F}-\otimes_{m=1}^{M}\mathbb{F}_{m}\big{]}=0,\text{
but }\mathbb{F}\neq\otimes_{m=1}^{M}\mathbb{F}_{m}.$

Define $\mathbb{I}:=\mathbb{F}-\otimes_{m=1}^{M}\mathbb{F}_{m}\in\mathcal{I}$.
In other words we want to get a witness $\mathbb{I}\in\mathcal{I}$ proving
that $\otimes_{m=1}^{M}k_{m}$ is not $\mathcal{I}$-characteristic. Let us take
$z\neq z^{\prime}\in\mathscr{X}_{2}$, which is possible since
$|\mathscr{X}_{2}|\geq 2$. Let us define $\mathbb{F}$ as666The $\mathbb{F}$
construction specializes to that of Lyons (2013, Proposition 3.15) in the
$M=2$ case; Lyons used it for distance covariances, which is known to be
equivalent to HSIC (Sejdinovic et al., 2013b).

$\displaystyle\mathbb{F}$
$\displaystyle=\frac{\mathbb{P}_{1}\otimes\delta_{z}\otimes(\otimes_{m=3}^{M}\mathbb{Q}_{m})+\mathbb{P}_{1}^{\prime}\otimes\delta_{z^{\prime}}\otimes(\otimes_{m=3}^{M}\mathbb{Q}_{m})}{2}\in\mathscr{M}_{1}^{+}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right).$

It is easy to verify that

$\mathbb{F}_{1}=\frac{\mathbb{P}_{1}+\mathbb{P}_{1}^{\prime}}{2},\,\mathbb{F}_{2}=\frac{\delta_{z}+\delta_{z^{\prime}}}{2}\,\,\text{and}\,\,\mathbb{F}_{m}=\mathbb{Q}_{m}\,\quad(m=3,\ldots,M),$

where $\mathbb{Q}_{3},\ldots,\mathbb{Q}_{M}$ are arbitrary probability
measures on $\mathscr{X}_{3},\ldots,\mathscr{X}_{M}$, respectively. First we
check that $\mathbb{I}\neq 0$. Indeed it is the case since

  * •

$z\neq z^{\prime}$ and $\mathscr{X}_{2}$ is a Hausdorff space, there exists
$B_{2}\in\mathscr{B}(\mathscr{X}_{2})$ such that $z\in B_{2}$,
$z^{\prime}\not\in B_{2}$.

  * •

$\mathbb{P}_{1}\neq\mathbb{P}_{1}^{\prime}$,
$\mathbb{P}_{1}(B_{1})\neq\mathbb{P}_{1}^{\prime}(B_{1})$ for some
$B_{1}\in\mathscr{B}(\mathscr{X}_{1})$.

Let $S=B_{1}\times B_{2}\times\left(\times_{m=3}^{M}\mathscr{X}_{m}\right)$,
and compare its measure under $\mathbb{F}$ and
$\otimes_{m=1}^{M}\mathbb{F}_{m}$:

$\displaystyle\mathbb{F}(S)$
$\displaystyle=\frac{\mathbb{P}_{1}(B_{1})\overbrace{\delta_{z}(B_{2})}^{=1\,(z\in
B_{2})}\prod_{m=3}^{M}\overbrace{\mathbb{Q}_{m}(\mathscr{X}_{m})}^{=1}+\mathbb{P}_{1}^{\prime}(B_{1})\overbrace{\delta_{z^{\prime}}(B_{2})}^{=0\,(z^{\prime}\not\in
B_{2})}\prod_{m=3}^{M}\overbrace{\mathbb{Q}_{m}(\mathscr{X}_{m})}^{=1}}{2}$
$\displaystyle=\frac{\mathbb{P}_{1}(B_{1})}{2},$
$\displaystyle\left(\otimes_{m=1}^{M}\mathbb{F}_{m}\right)(S)$
$\displaystyle=\prod_{m=1}^{M}\mathbb{F}_{m}(B_{m})=\frac{\mathbb{P}_{1}(B_{1})+\mathbb{P}_{1}^{\prime}(B_{1})}{2}\frac{\overbrace{\delta_{z}(B_{2})}^{=1}+\overbrace{\delta_{z^{\prime}}(B_{2})}^{=0}}{2}\prod_{m=3}^{M}\overbrace{\mathbb{Q}_{m}(\mathscr{X}_{m})}^{=1}$
$\displaystyle=\frac{\mathbb{P}_{1}(B_{1})+\mathbb{P}_{1}^{\prime}(B_{1})}{4}\neq\frac{\mathbb{P}_{1}(B_{1})}{2},$

where the last equality holds since
$\mathbb{P}_{1}(B_{1})\neq\mathbb{P}_{1}^{\prime}(B_{1})$. This shows that
$\mathbb{I}=\mathbb{F}-\otimes_{m=1}^{M}\mathbb{F}_{m}\neq 0$ since
$\mathbb{I}(S)\neq 0$.

Next we prove that
$\mu_{\otimes_{m=1}^{M}k_{m}}\left(\mathbb{F}-\otimes_{m=1}^{M}\mathbb{F}_{m}\right)=0$.
Indeed,

$\displaystyle\mu_{\otimes_{m=1}^{M}k_{m}}\left(\mathbb{I}\right)$
$\displaystyle{}={}$
$\displaystyle\int_{\times_{m=1}^{M}\mathscr{X}_{m}}\otimes_{m=1}^{M}k_{m}(\cdot,x_{m})\,\mathrm{d}\left[\mathbb{F}-\otimes_{m=1}^{M}\mathbb{F}_{m}\right](x_{1},\ldots,x_{M})$
$\displaystyle{}={}$
$\displaystyle\int_{\times_{m=1}^{M}\mathscr{X}_{m}}\otimes_{m=1}^{M}k_{m}(\cdot,x_{m})\,\mathrm{d}\left(\left[\frac{\mathbb{P}_{1}\otimes\delta_{z}+\mathbb{P}_{1}^{\prime}\otimes\delta_{z^{\prime}}}{2}-\frac{\mathbb{P}_{1}+\mathbb{P}_{1}^{\prime}}{2}\otimes\frac{\delta_{z}+\delta_{z^{\prime}}}{2}\right]\right.$
$\displaystyle\quad\quad\quad\quad\quad\quad\left.\otimes\left(\otimes_{m=3}^{M}\mathbb{Q}_{m}\right)\right)(x_{1},\ldots,x_{M})$
$\displaystyle{}={}$
$\displaystyle\int_{\times_{m=1}^{M}\mathscr{X}_{m}}\otimes_{m=1}^{M}k_{m}(\cdot,x_{m})\,\mathrm{d}\left(\left[\frac{\mathbb{P}_{1}(x_{1})\otimes\delta_{z}(x_{2})+\mathbb{P}_{1}^{\prime}(x_{1})\otimes\delta_{z^{\prime}}(x_{2})}{2}\right.\right.$
$\displaystyle\quad\quad\quad\quad\left.-\frac{\mathbb{P}_{1}(x_{1})\otimes\delta_{z}(x_{2})+\mathbb{P}_{1}(x_{1})\otimes\delta_{z^{\prime}}(x_{2})}{4}\right.$
$\displaystyle\quad\quad\quad\quad\quad\quad\left.\left.-\frac{\mathbb{P}_{1}^{\prime}(x_{1})\otimes\delta_{z}(x_{2})+\mathbb{P}_{1}^{\prime}(x_{1})\otimes\delta_{z^{\prime}}(x_{2})}{4}\right]\otimes(\otimes_{m=3}^{M}\mathbb{Q}_{m}(x_{m}))\right)$
$\displaystyle{}\stackrel{{\scriptstyle(*)}}{{=}}{}$
$\displaystyle\left[\frac{\mu_{k_{1}}(\mathbb{P}_{1})\otimes
k_{2}(\cdot,z)+\mu_{k_{1}}(\mathbb{P}_{1}^{\prime})\otimes
k_{2}(\cdot,z^{\prime})}{2}\right.$
$\displaystyle\quad\quad-\frac{\mu_{k_{1}}(\mathbb{P}_{1})\otimes
k_{2}(\cdot,z)+\mu_{k_{1}}(\mathbb{P}_{1})\otimes k_{2}(\cdot,z^{\prime})}{4}$
$\displaystyle\quad\quad\quad\quad\left.-\frac{\mu_{k_{1}}(\mathbb{P}_{1}^{\prime})\otimes
k_{2}(\cdot,z)+\mu_{k_{1}}(\mathbb{P}_{1}^{\prime})\otimes
k_{2}(\cdot,z^{\prime})}{4}\right]\otimes\left[\otimes_{m=3}^{M}\mu_{k_{m}}\left(\mathbb{Q}_{m}\right)\right]$
$\displaystyle{}={}$
$\displaystyle\underbrace{0}_{\in\,\mathscr{H}_{k_{1}\otimes
k_{2}}}\otimes\left[\otimes_{m=3}^{M}\mu_{k_{m}}\left(\mathbb{Q}_{m}\right)\right]=0,$

where we used
$\mu_{k_{1}}(\mathbb{P}_{1})=\mu_{k_{1}}\left(\mathbb{P}_{1}^{\prime}\right)$
in $(*)$.

###  5.4 Proof of Example 2

Let $M=3$,
$\times_{m=1}^{M}\mathscr{X}_{m}=\\{(i_{1},i_{2},i_{3}):i_{m}\in\\{1,2\\},\,m\in[3]\\}$,
$k_{m}(x,x^{\prime})=2\delta_{x,x^{\prime}}-1$. Our goal is to show that
$\otimes_{m=1}^{3}k_{m}$ is _not_ $\mathcal{I}$-characteristic. The structure
of the proof is as follows:

  1. 1.

First we describe the equations of the non-characteristic property of
$\otimes_{m=1}^{3}k_{m}$ with a general finite signed measure
$\mathbb{F}=\sum_{i_{1},i_{2},i_{3}=1}^{2}a_{i_{1},i_{2},i_{3}}\delta_{(i_{1},i_{2},i_{3})}$
on $\times_{m=1}^{3}\mathscr{X}_{m}$ where
$a_{i_{1},i_{2},i_{3}}\in\mathbb{R}$ ($\forall\,i_{1},i_{2},i_{3}$).

  2. 2.

Next, we apply the $\mathbb{F}=\mathbb{P}-\otimes_{m=1}^{3}\mathbb{P}_{m}$
parameterization and show that there exists $\mathbb{P}$ that satisfies the
equations of step 1 to conclude that $\otimes_{m=1}^{3}k_{m}$ is not
$\mathcal{I}$-characteristic.

The details are as follows.
Step 1. The equations of non-characteristic property in terms of
$\mathbf{A}=\left[a_{i_{1},i_{2},i_{3}}\right]_{(i_{m})^{3}_{m=1}\in[2]^{3}}\in\mathbb{R}^{2\times
2\times 2}$ are

$\displaystyle\mathbb{F}$
$\displaystyle\in\mathscr{M}_{b}\left(\times_{m=1}^{3}\mathscr{X}_{m}\right)\backslash\\{0\\}\Leftrightarrow\mathbf{A}\neq\mathbf{0},$
$\displaystyle 0$
$\displaystyle=\mathbb{F}(\times_{m=1}^{3}\mathscr{X}_{m})\Leftrightarrow
0=\sum_{i_{1},i_{2},i_{3}=1}^{2}a_{i_{1},i_{2},i_{3}},$ (23) $\displaystyle 0$
$\displaystyle=\int_{\times_{m=1}^{3}\mathscr{X}_{m}}\int_{\times_{m=1}^{3}\mathscr{X}_{m}}\underbrace{(\otimes_{m=1}^{3}k_{m})\left((i_{1},i_{2},i_{3}),(i_{1}^{\prime},i_{2}^{\prime},i_{3}^{\prime})\right)}_{\prod_{m=1}^{3}k_{m}(i_{m},i_{m}^{\prime})}\,\mathrm{d}\mathbb{F}(i_{1},i_{2},i_{3})\,\mathrm{d}\mathbb{F}(i_{1}^{\prime},i_{2}^{\prime},i_{3}^{\prime})$
$\displaystyle=\sum_{i_{1},i_{2},i_{3}=1}^{2}\sum_{i_{1}^{\prime},i_{2}^{\prime},i_{3}^{\prime}=1}^{2}\prod_{m=1}^{3}k_{m}(i_{m},i_{m}^{\prime})a_{i_{1},i_{2},i_{3}}a_{i_{1}^{\prime},i_{2}^{\prime},i_{3}^{\prime}}.$
(24)

Solving (23) and (24) yields

$\displaystyle a_{1,1,1}+a_{1,2,2}+a_{2,1,2}+a_{2,2,1}$
$\displaystyle=0\quad\text{and}\quad
a_{1,1,2}+a_{1,2,1}+a_{2,1,1}+a_{2,2,2}=0.$

Step 2. The equations of non $\mathcal{I}$-characteristic property can be
obtained from step 1 by choosing
$\mathbb{F}=\mathbb{P}-\otimes_{m=1}^{M}\mathbb{P}_{m}$, where

$\mathbb{P}=\sum_{i_{1},i_{2},i_{3}=1}^{2}p_{i_{1},i_{2},i_{3}}\delta_{(i_{1},i_{2},i_{3})}\quad\text{and}\quad\mathbf{P}=\left[p_{i_{1},i_{2},i_{3}}\right]_{(i_{m})^{3}_{m=1}\in[2]^{3}}\in\mathbb{R}^{2\times
2\times 2}.$

In other words, it is sufficient to obtain a $\mathbf{P}$ that solves the
following system of equations for which
$\mathbf{A}=\mathbf{A}(\mathbf{P})\neq\mathbf{0}$:

$\displaystyle\sum_{i_{1},i_{2},i_{3}=1}^{2}p_{i_{1},i_{2},i_{3}}$
$\displaystyle=1,$ (25) $\displaystyle p_{i_{1},i_{2},i_{3}}$
$\displaystyle\geq 0,\,\forall\,(i_{1},i_{2},i_{3})\in[2]^{3},$ (26)
$\displaystyle a_{1,1,1}+a_{1,2,2}+a_{2,1,2}+a_{2,2,1}$ $\displaystyle=0,$
(27) $\displaystyle a_{1,1,2}+a_{1,2,1}+a_{2,1,1}+a_{2,2,2}$
$\displaystyle=0,$ (28)

where

$a_{i_{1},i_{2},i_{3}}=p_{i_{1},i_{2},i_{3}}-p_{1,i_{1}}p_{2,i_{2}}p_{3,i_{3}},$
(29)

and

$p_{1,i_{1}}=\sum_{i_{2},i_{3}=1}^{2}p_{i_{1},i_{2},i_{3}},\quad
p_{2,i_{2}}=\sum_{i_{1},i_{3}=1}^{2}p_{i_{1},i_{2},i_{3}},\quad
p_{3,i_{3}}=\sum_{i_{1},i_{2}=1}^{2}p_{i_{1},i_{2},i_{3}}.$ (30)

One can get an analytical description for the solution of (25)–(30), where the
solution $\mathbf{P}(\mathbf{z})$ is parameterized by
$\mathbf{z}=(z_{0},\ldots,z_{5})\in\mathbb{R}^{6}$. For explicit expressions,
we refer the reader to Appendix A. In the following, we present two examples
of $\mathbf{P}$ that satisfy (25)–(30) such that $\mathbf{A}\neq\mathbf{0}$,
thereby establishing the non $\mathcal{I}$-characteristic property of
$\otimes^{3}_{m=1}k_{m}$.

  1. 1.

$\mathbf{P}$:

$\displaystyle p_{1,1,1}$ $\displaystyle=\frac{1}{5},$ $\displaystyle
p_{1,1,2}$ $\displaystyle=\frac{1}{10},$ $\displaystyle p_{1,2,1}$
$\displaystyle=\frac{1}{10},$ $\displaystyle p_{1,2,2}$
$\displaystyle=\frac{1}{10},$ $\displaystyle p_{2,1,1}$
$\displaystyle=\frac{1}{5},$ $\displaystyle p_{2,1,2}$
$\displaystyle=\frac{1}{10},$ $\displaystyle p_{2,2,1}$
$\displaystyle=\frac{1}{10},$ $\displaystyle p_{2,2,2}$
$\displaystyle=\frac{1}{10},$

and $\mathbf{A}$:

$\displaystyle a_{1,1,1}$ $\displaystyle=\frac{1}{50},$ $\displaystyle
a_{1,1,2}$ $\displaystyle=-\frac{1}{50},$ $\displaystyle a_{1,2,1}$
$\displaystyle=-\frac{1}{50},$ $\displaystyle a_{1,2,2}$
$\displaystyle=\frac{1}{50},$ (31) $\displaystyle a_{2,1,1}$
$\displaystyle=\frac{1}{50},$ $\displaystyle a_{2,1,2}$
$\displaystyle=-\frac{1}{50},$ $\displaystyle a_{2,2,1}$
$\displaystyle=-\frac{1}{50},$ $\displaystyle a_{2,2,2}$
$\displaystyle=\frac{1}{50}.$ (32)

  2. 2.

$\mathbf{P}$:

$\displaystyle p_{1,1,1}$ $\displaystyle=0,$ $\displaystyle p_{1,1,2}$
$\displaystyle=\frac{1}{10},$ $\displaystyle p_{1,2,1}$
$\displaystyle=\frac{1}{10},$ $\displaystyle p_{1,2,2}$
$\displaystyle=\frac{1}{10},$ $\displaystyle p_{2,1,1}$
$\displaystyle=\frac{1}{10},$ $\displaystyle p_{2,1,2}$
$\displaystyle=\frac{1}{10},$ $\displaystyle p_{2,2,1}$
$\displaystyle=\frac{3}{10},$ $\displaystyle p_{2,2,2}$
$\displaystyle=\frac{1}{5},$

and $\mathbf{A}$:

$\displaystyle a_{1,1,1}$ $\displaystyle=-\frac{9}{200},$ $\displaystyle
a_{1,1,2}$ $\displaystyle=\frac{11}{200},$ $\displaystyle a_{1,2,1}$
$\displaystyle=-\frac{1}{200},$ $\displaystyle a_{1,2,2}$
$\displaystyle=-\frac{1}{200},$ $\displaystyle a_{2,1,1}$
$\displaystyle=-\frac{1}{200},$ $\displaystyle a_{2,1,2}$
$\displaystyle=-\frac{1}{200},$ $\displaystyle a_{2,2,1}$
$\displaystyle=\frac{11}{200},$ $\displaystyle a_{2,2,2}$
$\displaystyle=-\frac{9}{200}.$

In fact these examples are obtained with the choices
$\mathbf{z}=(\frac{1}{10},\frac{1}{10},\frac{1}{10},\frac{1}{10},\frac{1}{10},\frac{1}{10})$
and
$\mathbf{z}=(\frac{3}{10},\frac{1}{10},\frac{1}{10},\frac{1}{10},\frac{1}{10},\frac{2}{10})$
respectively. See Appendix A for details.

###  5.5 Proof of Theorem 4

It follows from (8) and Remark 2(iii) that
$(v)\Rightarrow(iii)\Rightarrow(ii)\Leftrightarrow(i)$. It also follows from
(8) and Theorem 3(ii) that $(v)\Rightarrow(iv)\Rightarrow(i)$. We now show
that $(i)\Rightarrow(v)$ which establishes the equivalence of $(i)$–$(v)$.
Suppose $(i)$ holds. Then by Bochner’s theorem (Wendland, 2005, Theorem 6.6),
we have that for all $m\in[M]$,

$\displaystyle
k_{m}(x_{m},y_{m})=\int_{\mathbb{R}^{d_{m}}}e^{-\sqrt{-1}\langle\omega_{m},x_{m}-y_{m}\rangle}\,\mathrm{d}\Lambda_{m}(\omega_{m}),\,\,x_{m},y_{m}\in\mathbb{R}^{d_{m}},$

where $(\Lambda_{m})^{M}_{m=1}$ are finite non-negative Borel measures on
$(\mathbb{R}^{d_{m}})^{M}_{m=1}$ respectively. This implies

$\otimes^{M}_{m=1}k_{m}(x_{m},y_{m})=\otimes^{M}_{m=1}\int_{\mathbb{R}^{d_{m}}}e^{-\sqrt{-1}\langle\omega_{m},x_{m}-y_{m}\rangle}\,\mathrm{d}\Lambda_{m}(\omega_{m})=\int_{\mathbb{R}^{d}}e^{-\sqrt{-1}\langle\omega,x-y\rangle}\,\mathrm{d}\Lambda(\omega),$

where $x=(x_{1},\ldots,x_{M})\in\mathbb{R}^{d}$,
$y=(y_{1},\ldots,y_{M})\in\mathbb{R}^{d}$,
$\omega=(\omega_{1},\ldots,\omega_{M})\in\mathbb{R}^{d}$,
$d=\sum^{M}_{m=1}d_{m}$ and $\Lambda:=\otimes^{M}_{m=1}\Lambda_{m}$.
Sriperumbudur et al. (2010, Theorem 9) showed that $k_{m}$ is characteristic
iff $\text{supp}\left(\Lambda_{m}\right)=\mathbb{R}^{d_{m}}$, where
$\text{supp}(\cdot)$ denotes the support of its argument. Since
$\text{supp}(\Lambda)=\text{supp}\left(\otimes^{M}_{m=1}\Lambda_{m}\right)=\times_{m=1}^{M}\text{supp}\left(\Lambda_{m}\right)=\times_{m=1}^{M}\mathbb{R}^{d_{m}}=\mathbb{R}^{d}$,
it follows that $\otimes^{M}_{m=1}k_{m}$ is characteristic.

###  5.6 Proof of Theorem 5

The $c_{0}$-kernel property of $k_{m}$-s ($m=1,\ldots,M$) implies that of
$\otimes_{m=1}^{M}k_{m}$. Moreover, $\mathscr{X}_{m}$-s are LCP spaces, hence
$\times_{m=1}^{M}\mathscr{X}_{m}$ is also LCP.
($\Leftarrow$) Assume that $\otimes_{m=1}^{M}k_{m}$ is $c_{0}$-universal.
Since
$\otimes_{m=1}^{M}\mathscr{M}_{b}\left(\mathscr{X}_{m}\right)\subseteq\mathscr{M}_{b}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)$,
we have that for all
$\mathbb{F}=\otimes_{m=1}^{M}\mathbb{F}_{m}\in\otimes_{m=1}^{M}\mathscr{M}_{b}(\mathscr{X}_{m})\backslash\\{0\\}$,

$\displaystyle 0$
$\displaystyle<\int_{\times_{m=1}^{M}\mathscr{X}_{m}}\int_{\times_{m=1}^{M}\mathscr{X}_{m}}\underbrace{\left(\otimes_{m=1}^{M}k_{m}\right)(x,x^{\prime})}_{\prod_{m=1}^{M}k_{m}(x_{m},x_{m}^{\prime})}\,\mathrm{d}\mathbb{F}(x)\,\mathrm{d}\mathbb{F}(x^{\prime})$
$\displaystyle=\prod_{m=1}^{M}\int_{\mathscr{X}_{m}\times\mathscr{X}_{m}}k_{m}(x_{m},x_{m}^{\prime})\,\mathrm{d}\mathbb{F}_{m}(x_{m})\,\mathrm{d}\mathbb{F}_{m}\left(x_{m}^{\prime}\right),$

where $x=(x_{1},\ldots,x_{M})$ and
$x^{\prime}=\left(x_{1}^{\prime},\ldots,x_{M}^{\prime}\right)$. The above
inequality implies

$\int_{\mathscr{X}_{m}\times\mathscr{X}_{m}}k_{m}(x_{m},x_{m}^{\prime})\,\mathrm{d}\mathbb{F}_{m}(x_{m})\,\mathrm{d}\mathbb{F}_{m}\left(x_{m}^{\prime}\right)>0,\,\,\forall\,m\in[M].$

Since
$\mathbb{F}\in\otimes^{M}_{m=1}\mathscr{M}_{b}\left(\mathscr{X}_{m}\right)\backslash\\{0\\}$
iff $\mathbb{F}_{m}\in\mathscr{M}_{b}(\mathscr{X}_{m})\backslash\\{0\\}$ for
all $m\in[M]$, the result follows.
($\Rightarrow$) Assume that $k_{m}$-s are $c_{0}$-universal. By the note above
$\otimes_{m=1}^{M}k_{m}$ is $c_{0}$-kernel; its $c_{0}$-universality is
equivalent to the injectivity of $\mu=\mu_{\otimes_{m=1}^{M}k_{m}}$ on
$\mathscr{M}_{b}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)$. In other words,
we want to prove that $\mu(\mathbb{F})=0$ implies $\mathbb{F}=0$, where
$\mathbb{F}\in\mathscr{M}_{b}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)$. We
will use the shorthand $\mathscr{H}_{m}=\mathscr{H}_{k_{m}}$ below.

Suppose there exists
$\mathbb{F}\in\mathscr{M}_{b}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)$
such that

$\displaystyle\mu_{\mathbb{F}}=\int_{\times_{m=1}^{M}\mathscr{X}_{m}}\underbrace{\left(\otimes_{m=1}^{M}k_{m}\right)(\cdot,x)}_{\otimes_{m=1}^{M}k_{m}(\cdot,x_{m})}\,\mathrm{d}\mathbb{F}(x)=0\hskip
2.84544pt(\in\otimes_{m=1}^{M}\mathscr{H}_{m}).$ (33)

Since $\mathscr{X}_{m}$-s are LCP,
$\otimes_{m=1}^{M}\mathscr{B}\left(\mathscr{X}_{m}\right)=\mathscr{B}\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)$
(Steinwart and Christmann, 2008, page 480). Hence, in order to get
$\mathbb{F}=0$ it is sufficient to prove that

$\displaystyle\mathbb{F}\left(\times_{m=1}^{M}B_{m}\right)=0,\quad\forall\,B_{m}\in\mathscr{B}(\mathscr{X}_{m}),m\in[M].$

We will prove by induction that for $m=0,\ldots,M$

$\displaystyle\left(\otimes_{j=m+1}^{M}\mathscr{H}_{j}\ni\right)0$
$\displaystyle=\int_{\times_{j=1}^{M}\mathscr{X}_{j}}\prod_{j=1}^{m}\chi_{B_{j}}(x_{j})\otimes_{j=m+1}^{M}k_{j}(\cdot,x_{j})\,\mathrm{d}\mathbb{F}(x)$
$\displaystyle=:o(B_{1},\ldots,B_{m},k_{m+1},\ldots,k_{M}),\forall\,B_{j}\in\mathscr{B}(\mathscr{X}_{j}),\,j\in[m],$
(34)

which

  * ($*$)

reduces to (33) when $m=0$ by defining
$\prod_{j=1}^{0}\chi_{B_{j}}(x_{j}):=1$;

  * ($\dagger$)

for $m=M$, $\otimes_{m=M+1}^{M}\mathscr{H}_{m}$ is defined to be equal to
$\mathbb{R}$ and $\otimes_{m=M+1}^{M}k_{m}(\cdot,x_{m}):=1$, in which case
$o(B_{1},\ldots,B_{M})=\mathbb{F}\left(\times_{j=1}^{M}B_{j}\right)=0\Rightarrow\mathbb{F}=0$,
the result we want to prove.

From the above, it is clear that (34) holds for $m=0$. Assuming (34) holds for
some $m$, we now prove that it holds for $m+1$. To this end, it follows from
(34) that
$\forall\,h_{m+2}\in\mathscr{H}_{m+2},\ldots,\forall\,h_{M}\in\mathscr{H}_{M}$,

$\displaystyle(\mathscr{H}_{m+1}\ni)\,0$
$\displaystyle=o(B_{1},\ldots,B_{m},k_{m+1},\ldots,k_{M})\left(h_{m+2},\ldots,h_{M}\right)$
$\displaystyle=\left[\int_{\times_{j=1}^{M}\mathscr{X}_{j}}\left(\prod_{j=1}^{m}\chi_{B_{j}}(x_{j})\right)\otimes_{j=m+1}^{M}k_{j}(\cdot,x_{j})\,\mathrm{d}\mathbb{F}(x)\right](h_{m+2},\ldots,h_{M})$
$\displaystyle=\int_{\times_{j=1}^{M}\mathscr{X}_{j}}k_{m+1}(\cdot,x_{m+1})\prod_{j=1}^{m}\chi_{B_{j}}(x_{j})\prod_{j=m+2}^{M}h_{j}(x_{j})\,\mathrm{d}\mathbb{F}(x)$
$\displaystyle=\int_{\mathscr{X}_{m+1}}k_{m+1}(\cdot,x_{m+1})\,\mathrm{d}\nu(x_{m+1}),$

where

$\displaystyle\nu(B)$
$\displaystyle:=\nu_{B_{1},\ldots,B_{m},h_{m+2},\ldots,h_{M}}(B)$
$\displaystyle=\int_{\times_{j=1}^{M}\mathscr{X}_{j}}\left[\prod_{j=1}^{m}\chi_{B_{j}}(x_{j})\right]\chi_{B}(x_{m+1})\left[\prod_{j=m+2}^{M}h_{j}(x_{j})\right]\mathrm{d}\mathbb{F}(x),\,B\in\mathscr{B}(\mathscr{X}_{m+1}).$

By the $c_{0}$-universality of $k_{m+1}$,

$\displaystyle\nu=0\text{ for
}\forall\,h_{m+2}\in\mathscr{H}_{m+2},\ldots,\forall\,h_{M}\in\mathscr{H}_{M}$
(35)

provided that $\nu\in\mathscr{M}_{b}(\mathscr{X}_{m+1})$, in other words if
$|\nu|(\mathscr{X}_{m+1})<\infty$. This condition is met:

$\displaystyle|\nu|(\mathscr{X}_{m+1})$
$\displaystyle\leq\int_{\times_{j=1}^{M}\mathscr{X}_{j}}\prod_{j=m+2}^{M}\underbrace{\left|\left<h_{j},k_{j}(\cdot,x_{j})\right>_{\mathscr{H}_{j}}\right|}_{\leq\left\|h_{j}\right\|_{\mathscr{H}_{j}}\sqrt{k_{j}(x_{j},x_{j})}}\mathrm{d}|\mathbb{F}|(x)$
$\displaystyle\leq|\mathbb{F}|\left(\times_{m=1}^{M}\mathscr{X}_{m}\right)\prod_{j=m+2}^{M}\left\|h_{j}\right\|_{\mathscr{H}_{j}}\sup_{x\in\mathscr{X}_{j},x^{\prime}\in\mathscr{X}_{j}}\sqrt{k_{j}(x,x^{\prime})}<\infty,$

where we used the boundedness of $k_{m}$-s in the last inequality. (35)
implies that for
$\forall\,B_{1}\in\mathscr{B}(\mathscr{X}_{1}),\ldots,\forall\,B_{m+1}\in\mathscr{B}(\mathscr{X}_{m+1})$
and
$\forall\,h_{m+2}\in\mathscr{H}_{m+2},\ldots,\forall\,h_{M}\in\mathscr{H}_{M}$

$\displaystyle 0$
$\displaystyle=\nu(B_{m+1})=\int_{\times_{j=1}^{M}\mathscr{X}_{j}}\left[\prod_{j=1}^{m+1}\chi_{B_{j}}(x_{j})\right]\left[\prod_{j=m+2}^{M}h_{j}(x_{j})\right]\,\mathrm{d}\mathbb{F}(x)$
$\displaystyle=\left<\otimes_{j=m+2}^{M}h_{j},\int_{\times_{j=1}^{M}\mathscr{X}_{j}}\left[\prod_{j=1}^{m+1}\chi_{B_{j}}(x_{j})\right]\otimes_{j=m+2}^{M}k_{j}(\cdot,x_{j})\,\mathrm{d}\mathbb{F}(x)\right>_{\otimes_{j=m+2}^{M}\mathscr{H}_{j}},$

and therefore

$\displaystyle o(B_{1},\ldots,B_{m+1},k_{m+2},\ldots,k_{M})$
$\displaystyle=\int_{\times_{j=1}^{M}\mathscr{X}_{j}}\left[\prod_{j=1}^{m+1}\chi_{B_{j}}(x_{j})\right]\otimes_{j=m+2}^{M}k(\cdot,x_{j})\,\mathrm{d}\mathbb{F}(x)$
$\displaystyle=0\left(\in\otimes_{j=m+2}^{M}\mathscr{H}_{j}\right)$

for
$\forall\,B_{1}\in\mathscr{B}(\mathscr{X}_{1}),\ldots,\forall\,B_{m+1}\in\mathscr{B}(\mathscr{X}_{m+1})$,
i.e., (34) holds for $m+1$. Therefore, by induction, (34) holds for $m=M$ and
the result follows from ($\dagger$). To justify the convention in ($\dagger$),
consider the case of $m=M-1$ in which case (34) can be written as

$\int_{\mathscr{X}_{M}}k_{M}(\cdot,x_{M})\,\mathrm{d}\nu(x_{M})=0,$

where

$\nu(B)=\int_{\times_{j=1}^{M}\mathscr{X}_{j}}\left[\prod_{j=1}^{M-1}\chi_{B_{j}}(x_{j})\right]\chi_{B}(x_{M})\,\mathrm{d}\mathbb{F}(x),\,B\in\mathscr{B}(\mathscr{X}_{M}).$

Then by the $c_{0}$-universal property of $k_{M}$, since

$\displaystyle|\nu|(\mathscr{X}_{M})$
$\displaystyle\leq\int_{\times_{j=1}^{M}\mathscr{X}_{j}}1\,\mathrm{d}|\mathbb{F}|(x)=|\mathbb{F}|\left(\times_{j=1}^{M}\mathscr{X}_{j}\right)<\infty$

we obtain

$\displaystyle\int_{\times_{j=1}^{M}\mathscr{X}_{j}}\prod_{j=1}^{M}\chi_{B_{j}}(x_{j})\,\mathrm{d}\mathbb{F}(x)=\mathbb{F}\left(\times_{j=1}^{M}B_{j}\right)=0,\forall\,B_{1}\in\mathscr{B}(\mathscr{X}_{1}),\ldots,\forall\,B_{M}\in\mathscr{B}(\mathscr{X}_{M}).$

###  5.7 Proof of Example 3

The proof follows by a simple modification of that of Example 2 (Section 5.4).
The equations of a witness
$\mathbf{A}=[a_{i_{1},i_{2},i_{3}}]_{(i_{m})_{m=1}^{3}\in[2]^{3}}$ (and
corresponding
$\mathbf{P}=[p_{i_{1},i_{2},i_{3}}]_{(i_{m})_{m=1}^{3}\in[2]^{3}}$) for the
non-$\mathcal{I}$-characteristic property of $\otimes_{m=1}^{3}k_{m}$ take the
form:

$\displaystyle\mathbf{A}$ $\displaystyle\neq\mathbf{0},$ $\displaystyle 0$
$\displaystyle=\sum_{i_{1},i_{2},i_{3}=1}^{2}a_{i_{1},i_{2},i_{3}},$ (36)
$\displaystyle 0$
$\displaystyle=\sum_{i_{1},i_{2},i_{3}=1}^{2}\sum_{i_{1}^{\prime},i_{2}^{\prime},i_{3}^{\prime}=1}^{2}\prod_{m=1}^{3}k_{m}(i_{m},i_{m}^{\prime})a_{i_{1},i_{2},i_{3}}a_{i_{1}^{\prime},i_{2}^{\prime},i_{3}^{\prime}}$
$\displaystyle=\left(a_{1,1,1}-a_{2,1,1}\right)^{2}+\left(a_{1,1,2}-a_{2,1,2}\right)^{2}+\left(a_{1,2,1}-a_{2,2,1}\right)^{2}+\left(a_{1,2,2}-a_{2,2,2}\right)^{2},$
(37)

where (36) and (37) are equivalent to

$\displaystyle 0$
$\displaystyle=\sum_{i_{1},i_{2},i_{3}=1}^{2}a_{i_{1},i_{2},i_{3}},$
$\displaystyle a_{1,1,1}$ $\displaystyle=a_{2,1,1},$ $\displaystyle a_{1,1,2}$
$\displaystyle=a_{2,1,2},$ $\displaystyle a_{1,2,1}$
$\displaystyle=a_{2,2,1},$ $\displaystyle a_{1,2,2}$
$\displaystyle=a_{2,2,2}.$ (38)

While (38) is more restrictive than (27) and (28) (hence its solution set
might even be empty), one can immediately see that the example of
$\mathbf{A}\neq\mathbf{0}$ given in (31) and (32) fulfills (38) proving the
non-$\mathcal{I}$-characteristic property of $\otimes_{m=1}^{3}k_{m}$.

Acknowledgments

The authors profusely thank Ingo Steinwart for fascinating discussions on
topics related to the paper and for contributing to Remark 7. The authors also
thank the anonymous reviewers for their constructive comments that improved
the manuscript. A part of the work was carried out while BKS was visiting ZSz
at CMAP, École Polytechnique. BKS is supported by NSF-DMS-1713011 and also
thanks CMAP and DSI for their generous support. ZSz is highly grateful for the
Greek hospitality around the Aegean Sea; it greatly contributed to the
development of the induction arguments.

##  A Analytical Solution to (25)–(30) in Example 2

The solution of (25)–(30) takes the form

$\displaystyle p_{1,1,1}$ $\displaystyle=-\frac{\begin{aligned}
&z_{2}+z_{1}+z_{4}+z_{5}-3z_{2}z_{1}-4z_{2}z_{4}-4z_{1}z_{4}-z_{2}z_{3}-2z_{2}z_{0}-2z_{1}z_{3}-3z_{2}z_{5}\\\
&-2z_{4}z_{3}-z_{1}z_{0}-3z_{1}z_{5}-2z_{4}z_{0}-4z_{4}z_{5}-z_{3}z_{0}-z_{3}z_{5}-z_{0}z_{5}+2z_{2}z_{1}^{2}+2z_{2}^{2}z_{1}\\\
&+4z_{2}z_{4}^{2}+2z_{2}^{2}z_{4}+4z_{1}z_{4}^{2}+2z_{1}^{2}z_{4}+2z_{2}^{2}z_{0}+2z_{1}^{2}z_{3}+2z_{2}z_{5}^{2}+2z_{2}^{2}z_{5}+2z_{4}^{2}z_{3}\\\
&+2z_{1}z_{5}^{2}+2z_{1}^{2}z_{5}+2z_{4}^{2}z_{0}+2z_{4}z_{5}^{2}+4z_{4}^{2}z_{5}-z_{2}^{2}-z_{1}^{2}-3z_{4}^{2}+2z_{4}^{3}-z_{5}^{2}\\\
&+6z_{2}z_{1}z_{4}+2z_{2}z_{1}z_{3}+2z_{2}z_{4}z_{3}+2z_{2}z_{1}z_{0}+4z_{2}z_{1}z_{5}+4z_{2}z_{4}z_{0}+4z_{1}z_{4}z_{3}\\\
&+6z_{2}z_{4}z_{5}+2z_{1}z_{4}z_{0}+6z_{1}z_{4}z_{5}+2z_{2}z_{3}z_{0}+2z_{2}z_{3}z_{5}+2z_{1}z_{3}z_{0}+2z_{2}z_{0}z_{5}\\\
&+2z_{1}z_{3}z_{5}+2z_{4}z_{3}z_{0}+2z_{4}z_{3}z_{5}+2z_{1}z_{0}z_{5}+2z_{4}z_{0}z_{5}\end{aligned}}{\begin{aligned}
&2z_{2}z_{1}-z_{1}-2z_{4}-z_{3}-z_{0}-2z_{5}-z_{2}+2z_{2}z_{4}+2z_{1}z_{4}+2z_{2}z_{0}+2z_{1}z_{3}+2z_{2}z_{5}\\\
&+2z_{4}z_{3}+2z_{1}z_{5}+2z_{4}z_{0}+4z_{4}z_{5}+2z_{3}z_{0}+2z_{3}z_{5}+2z_{0}z_{5}+2z_{4}^{2}+2z_{5}^{2}\end{aligned}},$
$\displaystyle p_{1,1,2}$ $\displaystyle=z_{2},$ $\displaystyle p_{1,2,1}$
$\displaystyle=z_{1},$ $\displaystyle p_{1,2,2}$ $\displaystyle=z_{4},$
$\displaystyle p_{2,1,1}$ $\displaystyle=-\frac{\begin{aligned}
&z_{4}+z_{3}+z_{0}+z_{5}-z_{2}z_{1}-z_{2}z_{4}-z_{1}z_{4}-z_{2}z_{3}-2z_{2}z_{0}-2z_{1}z_{3}-2z_{2}z_{5}\\\
&-3z_{4}z_{3}-z_{1}z_{0}-2z_{1}z_{5}-3z_{4}z_{0}-4z_{4}z_{5}-3z_{3}z_{0}-4z_{3}z_{5}-4z_{0}z_{5}+2z_{2}z_{0}^{2}\\\
&+2z_{1}z_{3}^{2}+2z_{2}z_{5}^{2}+2z_{4}z_{3}^{2}+2z_{4}^{2}z_{3}+2z_{1}z_{5}^{2}+2z_{4}z_{0}^{2}+2z_{4}^{2}z_{0}+4z_{4}z_{5}^{2}+2z_{4}^{2}z_{5}\\\
&+2z_{3}z_{0}^{2}+2z_{3}^{2}z_{0}+4z_{3}z_{5}^{2}+2z_{3}^{2}z_{5}+4z_{0}z_{5}^{2}+2z_{0}^{2}z_{5}-z_{4}^{2}-z_{3}^{2}-z_{0}^{2}-3z_{5}^{2}\\\
&+2z_{5}^{3}+2z_{2}z_{1}z_{3}+2z_{2}z_{4}z_{3}+2z_{2}z_{1}z_{0}+2z_{2}z_{1}z_{5}+2z_{2}z_{4}z_{0}+2z_{1}z_{4}z_{3}\\\
&+2z_{2}z_{4}z_{5}+2z_{1}z_{4}z_{0}+2z_{1}z_{4}z_{5}+2z_{2}z_{3}z_{0}+2z_{2}z_{3}z_{5}+2z_{1}z_{3}z_{0}+4z_{2}z_{0}z_{5}\\\
&+4z_{1}z_{3}z_{5}+4z_{4}z_{3}z_{0}+6z_{4}z_{3}z_{5}+2z_{1}z_{0}z_{5}+6z_{4}z_{0}z_{5}+6z_{3}z_{0}z_{5}\\\
\end{aligned}}{\begin{aligned}
&2z_{2}z_{1}-z_{1}-2z_{4}-z_{3}-z_{0}-2z_{5}-z_{2}+2z_{2}z_{4}+2z_{1}z_{4}+2z_{2}z_{0}+2z_{1}z_{3}+2z_{2}z_{5}\\\
&+2z_{4}z_{3}+2z_{1}z_{5}+2z_{4}z_{0}+4z_{4}z_{5}+2z_{3}z_{0}+2z_{3}z_{5}+2z_{0}z_{5}+2z_{4}^{2}+2z_{5}^{2}\\\
\end{aligned}},$ $\displaystyle p_{2,1,2}$ $\displaystyle=z_{3},$
$\displaystyle p_{2,2,1}$ $\displaystyle=z_{0},$ $\displaystyle p_{2,2,2}$
$\displaystyle=z_{5},$

form, where $\mathbf{z}=(z_{0},z_{1},\ldots,z_{5})\in\mathbb{R}^{6}$ satisfies

$\displaystyle 0$
$\displaystyle\leq\left(2z_{0}z_{2}-z_{1}-z_{2}-z_{3}-2z_{4}-2z_{5}-z_{0}+2z_{0}z_{3}+2z_{1}z_{2}+2z_{0}z_{4}+2z_{1}z_{3}+2z_{0}z_{5}\right.$
$\displaystyle\quad\left.+2z_{1}z_{4}+2z_{1}z_{5}+2z_{2}z_{4}+2z_{2}z_{5}+2z_{3}z_{4}+2z_{3}z_{5}+4z_{4}z_{5}+2z_{4}^{2}+2z_{5}^{2}\right)\times$
$\displaystyle\quad\left(z_{0}z_{3}-z_{3}-z_{4}-z_{5}-z_{0}z_{1}-z_{0}-z_{1}z_{2}+z_{0}z_{5}-2z_{1}z_{4}-z_{2}z_{3}-z_{1}z_{5}-2z_{2}z_{4}-z_{2}z_{5}\right.$
$\displaystyle\quad\left.+z_{3}z_{5}+2z_{0}z_{2}^{2}+2z_{1}z_{2}^{2}+2z_{1}^{2}z_{2}+2z_{0}z_{4}^{2}+2z_{1}^{2}z_{3}+4z_{1}z_{4}^{2}+2z_{1}^{2}z_{4}+2z_{1}z_{5}^{2}+4z_{2}z_{4}^{2}\right.$
$\displaystyle\quad\left.+2z_{1}^{2}z_{5}+2z_{2}^{2}z_{4}+2z_{2}z_{5}^{2}+2z_{3}z_{4}^{2}+2z_{2}^{2}z_{5}+2z_{4}z_{5}^{2}+4z_{4}^{2}z_{5}-z_{1}^{2}-z_{2}^{2}-z_{4}^{2}+2z_{4}^{3}+z_{5}^{2}\right.$
$\displaystyle\quad\left.+2z_{0}z_{1}z_{2}+2z_{0}z_{1}z_{3}+2z_{0}z_{1}z_{4}+2z_{0}z_{2}z_{3}+2z_{0}z_{1}z_{5}+4z_{0}z_{2}z_{4}+2z_{1}z_{2}z_{3}+2z_{0}z_{2}z_{5}\right.$
$\displaystyle\quad\left.+2z_{0}z_{3}z_{4}+6z_{1}z_{2}z_{4}+4z_{1}z_{2}z_{5}+4z_{1}z_{3}z_{4}+2z_{0}z_{4}z_{5}+2z_{1}z_{3}z_{5}+2z_{2}z_{3}z_{4}+6z_{1}z_{4}z_{5}\right.$
$\displaystyle\quad\left.+2z_{2}z_{3}z_{5}+6z_{2}z_{4}z_{5}+2z_{3}z_{4}z_{5}\right),$
$\displaystyle 0$
$\displaystyle\leq\left(2z_{0}z_{2}-z_{1}-z_{2}-z_{3}-2z_{4}-2z_{5}-z_{0}+2z_{0}z_{3}+2z_{1}z_{2}+2z_{0}z_{4}+2z_{1}z_{3}+2z_{0}z_{5}\right.$
$\displaystyle\quad\left.+2z_{1}z_{4}+2z_{1}z_{5}+2z_{2}z_{4}+2z_{2}z_{5}+2z_{3}z_{4}+2z_{3}z_{5}+4z_{4}z_{5}+2z_{4}^{2}+2z_{5}^{2}\right)\times$
$\displaystyle\quad\left(z_{1}z_{2}-z_{2}-z_{4}-z_{5}-z_{0}z_{1}-z_{0}z_{3}-z_{1}-z_{0}z_{4}-2z_{0}z_{5}+z_{1}z_{4}-z_{2}z_{3}+z_{2}z_{4}\right.$
$\displaystyle\quad\left.-z_{3}z_{4}-2z_{3}z_{5}+2z_{0}^{2}z_{2}+2z_{0}z_{3}^{2}+2z_{0}^{2}z_{3}+2z_{0}z_{4}^{2}+2z_{1}z_{3}^{2}+2z_{0}^{2}z_{4}+4z_{0}z_{5}^{2}+2z_{0}^{2}z_{5}\right.$
$\displaystyle\quad\left.+2z_{1}z_{5}^{2}+2z_{2}z_{5}^{2}+2z_{3}z_{4}^{2}+2z_{3}^{2}z_{4}+4z_{3}z_{5}^{2}+2z_{3}^{2}z_{5}+4z_{4}z_{5}^{2}+2z_{4}^{2}z_{5}-z_{0}^{2}-z_{3}^{2}+z_{4}^{2}\right.$
$\displaystyle\quad\left.-z_{5}^{2}+2z_{5}^{3}+2z_{0}z_{1}z_{2}+2z_{0}z_{1}z_{3}+2z_{0}z_{1}z_{4}+2z_{0}z_{2}z_{3}+2z_{0}z_{1}z_{5}+2z_{0}z_{2}z_{4}+2z_{1}z_{2}z_{3}\right.$
$\displaystyle\quad\left.+4z_{0}z_{2}z_{5}+4z_{0}z_{3}z_{4}+6z_{0}z_{3}z_{5}+2z_{1}z_{2}z_{5}+2z_{1}z_{3}z_{4}+6z_{0}z_{4}z_{5}+4z_{1}z_{3}z_{5}+2z_{2}z_{3}z_{4}\right.$
$\displaystyle\quad\left.+2z_{1}z_{4}z_{5}+2z_{2}z_{3}z_{5}+2z_{2}z_{4}z_{5}+6z_{3}z_{4}z_{5}\right),$
$\displaystyle
2z_{0}z_{2}+2z_{0}z_{3}+2z_{1}z_{2}+2z_{0}z_{4}+2z_{1}z_{3}+2z_{0}z_{5}+2z_{1}z_{4}+2z_{1}z_{5}+2z_{2}z_{4}+2z_{2}z_{5}$
$\displaystyle\quad+2z_{3}z_{4}+2z_{3}z_{5}+4z_{4}z_{5}+2z_{4}^{2}+2z_{5}^{2}\neq
z_{0}+z_{1}+z_{2}+z_{3}+2z_{4}+2z_{5},$
$\displaystyle\left(2z_{0}z_{2}-z_{1}-z_{2}-z_{3}-2z_{4}-2z_{5}-z_{0}+2z_{0}z_{3}+2z_{1}z_{2}+2z_{0}z_{4}+2z_{1}z_{3}+2z_{0}z_{5}\right.$
$\displaystyle\quad\left.+2z_{1}z_{4}+2z_{1}z_{5}+2z_{2}z_{4}+2z_{2}z_{5}+2z_{3}z_{4}+2z_{3}z_{5}+4z_{4}z_{5}+2z_{4}^{2}+2z_{5}^{2}\right)\times$
$\displaystyle\quad\left(z_{1}+z_{2}+z_{4}+z_{5}-z_{0}z_{1}-2z_{0}z_{2}-z_{0}z_{3}-3z_{1}z_{2}-2z_{0}z_{4}-2z_{1}z_{3}-z_{0}z_{5}-4z_{1}z_{4}\right.$
$\displaystyle\quad\left.-z_{2}z_{3}-3z_{1}z_{5}-4z_{2}z_{4}-3z_{2}z_{5}-2z_{3}z_{4}-z_{3}z_{5}-4z_{4}z_{5}+2z_{0}z_{2}^{2}+2z_{1}z_{2}^{2}+2z_{1}^{2}z_{2}\right.$
$\displaystyle\quad\left.+2z_{0}z_{4}^{2}+2z_{1}^{2}z_{3}+4z_{1}z_{4}^{2}+2z_{1}^{2}z_{4}+2z_{1}z_{5}^{2}+4z_{2}z_{4}^{2}+2z_{1}^{2}z_{5}+2z_{2}^{2}z_{4}+2z_{2}z_{5}^{2}\right.$
$\displaystyle\quad\left.+2z_{3}z_{4}^{2}+2z_{2}^{2}z_{5}+2z_{4}z_{5}^{2}+4z_{4}^{2}z_{5}-z_{1}^{2}-z_{2}^{2}-3z_{4}^{2}+2z_{4}^{3}-z_{5}^{2}+2z_{0}z_{1}z_{2}\right.$
$\displaystyle\quad\left.+2z_{0}z_{1}z_{3}+2z_{0}z_{1}z_{4}+2z_{0}z_{2}z_{3}+2z_{0}z_{1}z_{5}+4z_{0}z_{2}z_{4}+2z_{1}z_{2}z_{3}+2z_{0}z_{2}z_{5}\right.$
$\displaystyle\quad\left.+2z_{0}z_{3}z_{4}+6z_{1}z_{2}z_{4}+4z_{1}z_{2}z_{5}+4z_{1}z_{3}z_{4}+2z_{0}z_{4}z_{5}+2z_{1}z_{3}z_{5}+2z_{2}z_{3}z_{4}+6z_{1}z_{4}z_{5}\right.$
$\displaystyle\quad+2z_{2}z_{3}z_{5}+6z_{2}z_{4}z_{5}+2z_{3}z_{4}z_{5})\leq
0,$
$\displaystyle\left(2z_{0}z_{2}-z_{1}-z_{2}-z_{3}-2z_{4}-2z_{5}-z_{0}+2z_{0}z_{3}+2z_{1}z_{2}+2z_{0}z_{4}+2z_{1}z_{3}+2z_{0}z_{5}\right.$
$\displaystyle\quad\left.+2z_{1}z_{4}+2z_{1}z_{5}+2z_{2}z_{4}+2z_{2}z_{5}+2z_{3}z_{4}+2z_{3}z_{5}+4z_{4}z_{5}+2z_{4}^{2}+2z_{5}^{2}\right)\times$
$\displaystyle\quad\left(z_{0}+z_{3}+z_{4}+z_{5}-z_{0}z_{1}-2z_{0}z_{2}-3z_{0}z_{3}-z_{1}z_{2}-3z_{0}z_{4}-2z_{1}z_{3}-4z_{0}z_{5}\right.$
$\displaystyle\quad\left.-z_{1}z_{4}-z_{2}z_{3}-2z_{1}z_{5}-z_{2}z_{4}-2z_{2}z_{5}-3z_{3}z_{4}-4z_{3}z_{5}-4z_{4}z_{5}+2z_{0}^{2}z_{2}\right.$
$\displaystyle\quad\left.+2z_{0}z_{3}^{2}+2z_{0}^{2}z_{3}+2z_{0}z_{4}^{2}+2z_{1}z_{3}^{2}+2z_{0}^{2}z_{4}+4z_{0}z_{5}^{2}+2z_{0}^{2}z_{5}+2z_{1}z_{5}^{2}+2z_{2}z_{5}^{2}\right.$
$\displaystyle\quad\left.+2z_{3}z_{4}^{2}+2z_{3}^{2}z_{4}+4z_{3}z_{5}^{2}+2z_{3}^{2}z_{5}+4z_{4}z_{5}^{2}+2z_{4}^{2}z_{5}-z_{0}^{2}-z_{3}^{2}-z_{4}^{2}-3z_{5}^{2}+2z_{5}^{3}\right.$
$\displaystyle\quad\left.+2z_{0}z_{1}z_{2}+2z_{0}z_{1}z_{3}+2z_{0}z_{1}z_{4}+2z_{0}z_{2}z_{3}+2z_{0}z_{1}z_{5}+2z_{0}z_{2}z_{4}+2z_{1}z_{2}z_{3}\right.$
$\displaystyle\quad\left.+4z_{0}z_{2}z_{5}+4z_{0}z_{3}z_{4}+6z_{0}z_{3}z_{5}+2z_{1}z_{2}z_{5}+2z_{1}z_{3}z_{4}+6z_{0}z_{4}z_{5}+4z_{1}z_{3}z_{5}\right.$
$\displaystyle\quad\left.+2z_{2}z_{3}z_{4}+2z_{1}z_{4}z_{5}+2z_{2}z_{3}z_{5}+2z_{2}z_{4}z_{5}+6z_{3}z_{4}z_{5}\right)\leq
0,$ and $\displaystyle 0\leq z_{0},z_{1},z_{2},z_{3},z_{4},z_{5}\leq 1.$

The above analytic solution to (25)–(30) is obtained by symbolic math
programming in MATLAB.

## References

  * Aronszajn (1950) N. Aronszajn.  Theory of reproducing kernels.  _Transactions of the American Mathematical Society_ , 68:337–404, 1950.
  * Balasubramanian et al. (2017) K. Balasubramanian, T. Li, and M. Yuan.  On the optimality of kernel-embedding based goodness-of-fit tests.  Technical report, 2017.  (https://arxiv.org/abs/1709.08148).
  * Baringhaus and Franz (2004) L. Baringhaus and C. Franz.  On a new multivariate two-sample test.  _Journal of Multivariate Analysis_ , 88:190–206, 2004.
  * Berlinet and Thomas-Agnan (2004) A. Berlinet and C. Thomas-Agnan.  _Reproducing Kernel Hilbert Spaces in Probability and Statistics_.  Kluwer, 2004.
  * Blanchard et al. (2011) G. Blanchard, G. Lee, and C. Scott.  Generalizing from several related classification tasks to a new unlabeled sample.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 2178–2186, 2011.
  * Blanchard et al. (2017) G. Blanchard, A. A. Deshmukh, U. Dogan, G. Lee, and C. Scott.  Domain generalization by marginal transfer learning.  Technical report, 2017.  (https://arxiv.org/abs/1711.07910).
  * Borgwardt et al. (2006) K. Borgwardt, A. Gretton, M. J. Rasch, H.-P. Kriegel, B. Schölkopf, and A. J. Smola.  Integrating structured biological data by kernel maximum mean discrepancy.  _Bioinformatics_ , 22:e49–e57, 2006.
  * Cardoso (1998) J.-F. Cardoso.  Multidimensional independent component analysis.  In _International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , pages 1941–1944, 1998.
  * Carmeli et al. (2010) C. Carmeli, E. De Vito, A. Toigo, and V. Umanitá.  Vector valued reproducing kernel Hilbert spaces and universality.  _Analysis and Applications_ , 8:19–61, 2010.
  * Chwialkowski et al. (2016) K. Chwialkowski, H. Strathmann, and A. Gretton.  A kernel test of goodness of fit.  In _International Conference on Machine Learning (ICML; PMLR)_ , volume 48, pages 2606–2615, 2016.
  * Dudley (2004) R. M. Dudley.  _Real Analysis and Probability_.  Cambridge University Press, 2004.
  * Fukumizu et al. (2008) K. Fukumizu, A. Gretton, X. Sun, and B. Schölkopf.  Kernel measures of conditional dependence.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 498–496, 2008.
  * Fukumizu et al. (2009) K. Fukumizu, F. Bach, and M. Jordan.  Kernel dimension reduction in regression.  _The Annals of Statistics_ , 37(4):1871–1905, 2009.
  * Fukumizu et al. (2013) K. Fukumizu, L. Song, and A. Gretton.  Kernel Bayes’ rule: Bayesian inference with positive definite kernels.  _Journal of Machine Learning Research_ , 14:3753–3783, 2013\.
  * Gretton (2015) A. Gretton.  A simpler condition for consistency of a kernel independence test.  Technical report, University College London, 2015.  (http://arxiv.org/abs/1501.06103).
  * Gretton et al. (2005a) A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf.  Measuring statistical dependence with Hilbert-Schmidt norms.  In _Algorithmic Learning Theory (ALT)_ , pages 63–78, 2005a.
  * Gretton et al. (2005b) A. Gretton, R. Herbrich, A. Smola, O. Bousquet, and B. Schölkopf.  Kernel methods for measuring independence.  _Journal of Machine Learning Research_ , 6:2075–2129, 2005b.
  * Gretton et al. (2008) A. Gretton, K. Fukumizu, C. H. Teo, L. Song, B. Schölkopf, and A. J. Smola.  A kernel statistical test of independence.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 585–592, 2008.
  * Gretton et al. (2012) A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola.  A kernel two-sample test.  _Journal of Machine Learning Research_ , 13:723–773, 2012\.
  * Harchaoui et al. (2007) Z. Harchaoui, F. Bach, and E. Moulines.  Testing for homogeneity with kernel Fisher discriminant analysis.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 609–616, 2007.
  * Jitkrittum et al. (2017a) W. Jitkrittum, Z. Szabó, and A. Gretton.  An adaptive test of independence with analytic kernel embeddings.  In _International Conference on Machine Learning (ICML; PMLR)_ , volume 70, pages 1742–1751, 2017a.
  * Jitkrittum et al. (2017b) W. Jitkrittum, W. Xu, Z. Szabó, K. Fukumizu, and A. Gretton.  A linear-time kernel goodness-of-fit test.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 261–270, 2017b.
  * Kim et al. (2016) B. Kim, R. Khanna, and O. O. Koyejo.  Examples are not enough, learn to criticize! criticism for interpretability.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 2280–2288, 2016.
  * Klebanov (2005) L. Klebanov.  _N-Distances and Their Applications_.  Charles University, Prague, 2005.
  * Kusano et al. (2016) G. Kusano, K. Fukumizu, and Y. Hiraoka.  Persistence weighted Gaussian kernel for topological data analysis.  In _International Conference on Machine Learning (ICML)_ , pages 2004–2013, 2016.
  * Law et al. (2018) H. C. L. Law, D. J. Sutherland, D. Sejdinovic, and S. Flaxman.  Bayesian approaches to distribution regression.  In _International Conference on Artificial Intelligence and Statistics (AISTATS; PMLR)_ , volume 84, pages 1167–1176, 2018.
  * Liu et al. (2016) Q. Liu, J. Lee, and M. Jordan.  A kernelized Stein discrepancy for goodness-of-fit tests.  In _International Conference on Machine Learning (ICML)_ , pages 276–284, 2016.
  * Lloyd et al. (2014) J. R. Lloyd, D. Duvenaud, R. Grosse, J. B. Tenenbaum, and Z. Ghahramani.  Automatic construction and natural-language description of nonparametric regression models.  In _AAAI Conference on Artificial Intelligence_ , pages 1242–1250, 2014.
  * Lyons (2013) R. Lyons.  Distance covariance in metric spaces.  _The Annals of Probability_ , 41:3284–3305, 2013.
  * Micchelli et al. (2006) C. A. Micchelli, Y. Xu, and H. Zhang.  Universal kernels.  _Journal of Machine Learning Research_ , 7:2651–2667, 2006\.
  * Mooij et al. (2016) J. M. Mooij, J. Peters, D. Janzing, J. Zscheischler, and B. Schölkopf.  Distinguishing cause from effect using observational data: Methods and benchmarks.  _Journal of Machine Learning Research_ , 17:1–102, 2016\.
  * Muandet et al. (2011) K. Muandet, K. Fukumizu, F. Dinuzzo, and B. Schölkopf.  Learning from distributions via support measure machines.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 10–18, 2011.
  * Muandet et al. (2017) K. Muandet, K. Fukumizu, B. Sriperumbudur, and B. Schölkopf.  Kernel mean embedding of distributions: A review and beyond.  _Foundations and Trends® in Machine Learning_ , 10(1-2):1–141, 2017.
  * Park et al. (2016) M. Park, W. Jitkrittum, and D. Sejdinovic.  K2-ABC: Approximate Bayesian computation with kernel embeddings.  In _International Conference on Artificial Intelligence and Statistics (AISTATS; PMLR)_ , volume 51, pages 51:398–407, 2016.
  * Pfister et al. (2017) N. Pfister, P. Bühlmann, B. Schölkopf, and J. Peters.  Kernel-based tests for joint independence.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 80(1):5–31, 2017.
  * Quadrianto et al. (2009) N. Quadrianto, L. Song, and A. Smola.  Kernelized sorting.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 1289–1296, 2009.
  * Schölkopf and Smola (2002) B. Schölkopf and A. J. Smola.  _Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond_.  MIT Press, 2002.
  * Schölkopf et al. (2015) B. Schölkopf, K. Muandet, K. Fukumizu, S. Harmeling, and J. Peters.  Computing functions of random variables via reproducing kernel Hilbert space representations.  _Statistics and Computing_ , 25(4):755–766, 2015\.
  * Sejdinovic et al. (2013a) D. Sejdinovic, A. Gretton, and W. Bergsma.  A kernel test for three-variable interactions.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 1124–1132, 2013a.
  * Sejdinovic et al. (2013b) D. Sejdinovic, B. K. Sriperumbudur, A. Gretton, and K. Fukumizu.  Equivalence of distance-based and RKHS-based statistics in hypothesis testing.  _Annals of Statistics_ , 41:2263–2291, 2013b.
  * Simon-Gabriel and Schölkopf (2016) C.-J. Simon-Gabriel and B. Schölkopf.  Kernel distribution embeddings: Universal kernels, characteristic kernels and kernel metrics on distributions.  Technical report, Max Planck Institute for Intelligent Systems, 2016.  (https://arxiv.org/abs/1604.05251).
  * Smola et al. (2007) A. Smola, A. Gretton, L. Song, and B. Schölkopf.  A Hilbert space embedding for distributions.  In _Algorithmic Learning Theory (ALT)_ , pages 13–31, 2007.
  * Song et al. (2011) L. Song, A. Gretton, D. Bickson, Y. Low, and C. Guestrin.  Kernel belief propagation.  In _International Conference on Artificial Intelligence and Statistics (AISTATS)_ , pages 707–715, 2011.
  * Song et al. (2012) L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt.  Feature selection via dependence maximization.  _Journal of Machine Learning Research_ , 13:1393–1434, 2012\.
  * Sriperumbudur et al. (2010) B. K. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf, and G. R. G. Lanckriet.  Hilbert space embeddings and metrics on probability measures.  _Journal of Machine Learning Research_ , 11:1517–1561, 2010\.
  * Sriperumbudur et al. (2011) B. K. Sriperumbudur, K. Fukumizu, and G. R. G. Lanckriet.  Universality, characteristic kernels and RKHS embedding of measures.  _Journal of Machine Learning Research_ , 12:2389–2410, 2011\.
  * Steinwart (2001) I. Steinwart.  On the influence of the kernel on the consistency of support vector machines.  _Journal of Machine Learning Research_ , 6(3):67–93, 2001.
  * Steinwart and Christmann (2008) I. Steinwart and A. Christmann.  _Support Vector Machines_.  Springer, 2008.
  * Steinwart and Ziegel (2017) I. Steinwart and J. F. Ziegel.  Strictly proper kernel scores and characteristic kernels on compact spaces.  Technical report, Faculty for Mathematics and Physics, University of Stuttgart, 2017.  (https://arxiv.org/abs/1712.05279).
  * Strobl et al. (2017) E. V. Strobl, S. Visweswaran, and K. Zhang.  Approximate kernel-based conditional independence tests for fast non-parametric causal discovery.  Technical report, 2017.  (https://arxiv.org/abs/1702.03877).
  * Szabó et al. (2016) Z. Szabó, B. Sriperumbudur, B. Póczos, and A. Gretton.  Learning theory for distribution regression.  _Journal of Machine Learning Research_ , 17(152):1–40, 2016.
  * Székely and Rizzo (2004) G. J. Székely and M. L. Rizzo.  Testing for equal distributions in high dimension.  _InterStat_ , 5, 2004.
  * Székely and Rizzo (2005) G. J. Székely and M. L. Rizzo.  A new test for multivariate normality.  _Journal of Multivariate Analysis_ , 93:58–80, 2005.
  * Székely and Rizzo (2009) G. J. Székely and M. L. Rizzo.  Brownian distance covariance.  _The Annals of Applied Statistics_ , 3:1236–1265, 2009\.
  * Székely et al. (2007) G. J. Székely, M. L. Rizzo, and N. K. Bakirov.  Measuring and testing dependence by correlation of distances.  _The Annals of Statistics_ , 35:2769–2794, 2007.
  * Wendland (2005) H. Wendland.  _Scattered Data Approximation_.  Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2005.
  * Yamada et al. (2018) M. Yamada, Y. Umezu, K. Fukumizu, and I. Takeuchi.  Post selection inference with kernels.  In _International Conference on Artificial Intelligence and Statistics (AISTATS; PMLR)_ , volume 84, pages 152–160, 2018.
  * Zaheer et al. (2017) M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Póczos, R. R. Salakhutdinov, and A. J. Smola.  Deep sets.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 3394–3404, 2017.
  * Zhang et al. (2013) K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang.  Domain adaptation under target and conditional shift.  _Journal of Machine Learning Research_ , 28(3):819–827, 2013.
  * Zinger et al. (1992) A. A. Zinger, A. V. Kakosyan, and L. B. Klebanov.  A characterization of distributions by mean values of statistics and certain probabilistic metrics.  _Journal of Soviet Mathematics_ , 1992.