$\operatorname{eig}(R_{\lambda_{n}})=\\{0,1,\lambda_{n}\\}$ where
$\lambda_{n}\rightarrow 0$. Therefore, the boundedness depends on the
eigenvalues of the first term. For any $\beta\in\mathcal{M}$ we have

$\displaystyle\beta^{T}\Lambda_{n}\beta$
$\displaystyle=\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}\beta^{T}a(x,z;\theta)a(x,z;\theta)^{T}\beta}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+\beta^{T}R_{\lambda_{n}}\beta$ (126)
$\displaystyle\leq\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}\|a(x,z;\theta)^{T}\beta\|^{2}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+\|\beta\|^{2}$ (127)
$\displaystyle=\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}\|\eta+\langle(k((x,z),\cdot),f\rangle_{\mathcal{F}}-\Psi(x,z;\theta)(h)\|^{2}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+\|\beta\|^{2}$ (128)
$\displaystyle\leq\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}\left(\|\eta\|^{2}+\|\langle(k((x,z),\cdot),f\rangle_{\mathcal{F}}\|^{2}+\|\Psi(x,z;\theta)(h)\|^{2}\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+\|\beta\|^{2}$ (129)
$\displaystyle\leq\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}\left(\|\eta\|^{2}+\|f\|^{2}\|k((x,z),\cdot)\|^{2}+\|h\|^{2}_{\mathcal{H}}\|\Psi(x,z;\theta)\|_{\mathcal{H}^{\ast}}^{2}\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+\|\beta\|^{2}.$ (130)

Now, as $\mathcal{F}$ is an RKHS, the evaluation functional $k((x,z),\cdot)$
can be bounded by a constant $C_{1}$. Moreover, by Assumption d) and f) of
Theorem A.1, we have
$\int\sup_{\theta\in\Theta}\|\Psi(x,z;\theta)\|_{\mathcal{H}^{\ast}}^{2}\mathrm{d}P_{0}<\infty$
and
$\int\sup_{\theta\in\Theta}\|\Psi(x,z;\theta)\|_{\mathcal{H}^{\ast}}^{2}\mathrm{d}Q<\infty$
and thus as $\omega=(1-\alpha)\hat{P}_{n}+\alpha
Q\overset{p}{\rightarrow}(1-\alpha)P_{0}+\alpha Q$ it follows
$\sup_{\theta\in\Theta}\int\|\Psi(x,z;\theta)\|_{\mathcal{H}^{\ast}}^{2}\mathrm{d}\omega\leq\int\sup_{\theta\in\Theta}\|\Psi(x,z;\theta)\|_{\mathcal{H}^{\ast}}^{2}\mathrm{d}\omega<C_{2}$
for some $C_{2}>0$ w.p.a.1. Inserting this back we obtain

$\displaystyle\beta^{T}\Lambda_{\epsilon,\lambda_{n}}\beta$
$\displaystyle\leq\frac{1}{\epsilon}\left(\|\eta\|^{2}+C_{1}\|f\|^{2}+C_{2}\|h\|^{2}_{\mathcal{H}}\right)+\|\beta\|^{2}$
(131) $\displaystyle\leq\left(\frac{C_{3}}{\epsilon}+1\right)\|\beta\|^{2},$
(132)

where $C_{3}=\max(1,C_{1},C_{2})$. It follows w.p.a.1 that the largest
eigenvalue of $\Lambda_{n}$ can be bounded by some constant
$M=\frac{C_{3}}{\epsilon}+1$ for any finite value of $\epsilon>0$. ∎

######  Lemma E.12.

Let the assumptions of Theorem A.1 be satisfied. Additionally let
$\bar{\theta}\in\Theta$, $\bar{\theta}\overset{p}{\rightarrow}\theta_{0}$, and
$\|E_{\hat{P}_{n}}[\Psi(X,Z;\bar{\theta})]\|_{\mathcal{H}^{\ast}}=O_{p}\left(n^{-1/2}\right)$.
Then for $\bar{\beta}=$
$\operatorname*{arg\,max}_{\beta\in\mathcal{M}}\widehat{G}_{\epsilon,\lambda_{n}}(\bar{\theta},\beta)$
we have $\|\bar{\beta}\|_{\mathcal{M}}=O_{p}\left(n^{-1/2}\right)$, and
$\widehat{G}_{\epsilon,\lambda_{n}}(\bar{\theta},\bar{\beta})\leq-\epsilon\varphi^{\ast}(0)+O_{p}\left(n^{-1}\right)$.

###### Proof.

Define $\bar{\Psi}_{i}:=\Psi(x_{i},z_{i};\bar{\theta})$ and
$\bar{\Psi}=\frac{1}{n}\sum_{i=1}^{n}\bar{\Psi}_{i}$. For simplicity of
notation let
$\widehat{G}(\theta,\beta):=\widehat{G}_{\epsilon,\lambda_{n}}(\theta,\beta)$.
The first and second derivative of $\widehat{G}(\bar{\theta},\beta)$ with
respect to $\beta$ are given by

$\displaystyle\frac{\partial\widehat{G}}{\partial\beta}(\bar{\theta},\beta)$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}b_{i}-\int_{\mathcal{X}\times\mathcal{Z}}{a(x,z;\bar{\theta})\varphi_{1}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\bar{\theta})^{T}\beta\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)-R_{\lambda_{n}}\beta$ (133)
$\displaystyle\frac{\partial^{2}\widehat{G}}{(\partial\beta)^{2}}(\bar{\theta},\beta)$
$\displaystyle=-\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}a(x,z;\bar{\theta})a(x,z;\bar{\theta})^{T}\varphi_{2}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\bar{\theta})^{T}\beta\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)-R_{\lambda_{n}}.$ (134)

Consider the optimal dual parameter within the magnitude constrained set
$\mathcal{M}_{n}=\\{\beta\in\mathcal{M}:\|\beta\|_{\mathcal{M}}\leq
n^{-\zeta}\\}$, i.e.,
$\beta_{\zeta}:=\operatorname*{arg\,max}_{\beta\in\mathcal{M}_{n}}\widehat{G}(\bar{\theta},\beta)$
with $\beta_{\zeta}=(\eta_{\zeta},f_{\zeta},h_{\zeta})$. Later on, we will
show that this maximizer can be identified with the maximizer over the
original set $\mathcal{M}$. Using Taylor’s theorem we can expand the empirical
KMM objective about $\beta=0$,

$\displaystyle\widehat{G}(\bar{\theta},\beta_{\zeta})=$
$\displaystyle\widehat{G}(\bar{\theta},0)+\frac{\partial\widehat{G}}{\partial\beta}(\bar{\theta},0)\beta_{\zeta}+\frac{1}{2}\beta_{\zeta}^{T}\frac{\partial^{2}\widehat{G}}{(\partial\beta)^{2}}(\bar{\theta},\dot{\beta})\beta_{\zeta}$
(135) $\displaystyle=$
$\displaystyle-\epsilon\varphi^{\ast}(0)+\int_{\mathcal{X}\times\mathcal{Z}}{\Psi(x,z,\bar{\theta})(h_{\zeta})}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$ (136)
$\displaystyle+\frac{1}{n}\sum_{i=1}^{n}f_{\zeta}(x_{i},z_{i})-\int_{\mathcal{X}\times\mathcal{Z}}{f_{\zeta}(x,z)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$ (137)
$\displaystyle-\frac{1}{2}\beta_{\zeta}^{T}\underbrace{\left(\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}a(x,z;\bar{\theta})a(x,z;\bar{\theta})^{T}\varphi_{2}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\bar{\theta})^{T}\dot{\beta}\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+R_{\lambda_{n}}\right)}_{:=\Lambda_{n}(\dot{\beta},\bar{\theta})}\beta_{\zeta}$
(138)

for some $\dot{\beta}\in\operatorname{conv}(\\{0,\beta_{\zeta}\\})$. Now
adding and subtracting the empirical expectation of the moment functional
$\Psi$ we get

$\displaystyle\widehat{G}(\bar{\theta},\beta_{\zeta})=$
$\displaystyle-\epsilon\varphi^{\ast}(0)-\frac{1}{2}\beta_{\zeta}^{T}\Lambda_{n}(\dot{\beta},\bar{\theta})\beta_{\zeta}+\int_{\mathcal{X}\times\mathcal{Z}}\Psi(x,z;\bar{\theta})(h_{\zeta})\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)$
(139)
$\displaystyle+\int_{\mathcal{X}\times\mathcal{Z}}{\Psi(x,z;\bar{\theta})(h_{\zeta})}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)-\int_{\mathcal{X}\times\mathcal{Z}}\Psi(x,z;\bar{\theta})(h_{\zeta})\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)$
(140)
$\displaystyle+\int_{\mathcal{X}\times\mathcal{Z}}f_{\zeta}(x,z)\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)-\int_{\mathcal{X}\times\mathcal{Z}}{f_{\zeta}(x,z)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$ (141) $\displaystyle\leq$
$\displaystyle-\epsilon\varphi^{\ast}(0)-\frac{1}{2}\beta_{\zeta}^{T}\Lambda_{n}(\dot{\beta},\bar{\theta})\beta_{\zeta}+\|h_{\zeta}\|_{\mathcal{H}}\left\|\int_{\mathcal{X}\times\mathcal{Z}}\Psi(x,z;\bar{\theta})\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)\right\|_{\mathcal{H}^{\ast}}$
(142)
$\displaystyle+\|h_{\zeta}\|_{\mathcal{H}}\left\|\int_{\mathcal{X}\times\mathcal{Z}}\Psi(x,z;\bar{\theta})\left(\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)-\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)\right\|_{\mathcal{H}^{\ast}}$
(143)
$\displaystyle+\|f_{\zeta}\|_{\mathcal{F}}\sup_{f\in\mathcal{F},\|f\|_{\mathcal{F}}=1}\int_{\mathcal{X}\times\mathcal{Z}}f(x,z)\left(\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)-\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)$
(144) $\displaystyle\leq$
$\displaystyle-\epsilon\varphi^{\ast}(0)-\frac{1}{2}\beta_{\zeta}^{T}\Lambda_{n}(\dot{\beta},\bar{\theta})\beta_{\zeta}+\|h_{\zeta}\|_{\mathcal{H}}\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}$
(145)
$\displaystyle+\alpha\|h_{\zeta}\|_{\mathcal{H}}\left(\int_{\mathcal{X}\times\mathcal{Z}}\left\|\Psi(x,z;\bar{\theta})\right\|_{\mathcal{H}^{\ast}}Q(\mathrm{d}x\otimes\mathrm{d}z)+\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}\right)$
(146)
$\displaystyle+\|f_{\zeta}\|_{\mathcal{F}}\sup_{f\in\mathcal{F},\|f\|_{\mathcal{F}}=1}\int_{\mathcal{X}\times\mathcal{Z}}f(x,z)\left(\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)-\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)$
(147) $\displaystyle\leq$
$\displaystyle-\epsilon\varphi^{\ast}(0)-\frac{1}{2}\beta_{\zeta}^{T}\Lambda_{n}(\dot{\beta},\bar{\theta})\beta_{\zeta}+\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}\|h_{\zeta}\|_{\mathcal{H}}$
(148)
$\displaystyle+\alpha\|h_{\zeta}\|_{\mathcal{H}}\left(C_{Q}+\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}\right)+\|f_{\zeta}\|_{\mathcal{F}}\operatorname*{MMD}(\hat{P}_{n},\omega;\mathcal{F})$
(149) $\displaystyle\leq$
$\displaystyle-\epsilon\varphi^{\ast}(0)-\frac{1}{2}\beta_{\zeta}^{T}\Lambda_{n}(\dot{\beta},\bar{\theta})\beta_{\zeta}+\|\beta_{\zeta}\|_{\mathcal{M}}\left(\|\bar{\Psi}\|_{\mathcal{H}}+\alpha(C_{Q}+\|\bar{\Psi}\|_{\mathcal{H}^{\ast}})+\operatorname*{MMD}(\hat{P}_{n},\omega;\mathcal{F})\right)$
(150)

where we repeatedly used the Cauchy-Schwarz inequality and the fact that
$\int_{\mathcal{X}\times\mathcal{Z}}\left\|\Psi(x,z;\bar{\theta})\right\|_{\mathcal{H}^{\ast}}Q(\mathrm{d}x\otimes\mathrm{d}z)<\int_{\mathcal{X}\times\mathcal{Z}}\sup_{\theta\in\Theta}\left\|\Psi(x,z;\theta)\right\|_{\mathcal{H}^{\ast}}Q(\mathrm{d}x\otimes\mathrm{d}z)=:C_{Q}<\infty$
by Assumption f) of Theorem A.1 and Lemma E.7.

Lemma E.11 states that the smallest eigenvalue $C$ of
$\Lambda_{n}(\dot{\beta},\bar{\theta})$ is bounded away from zero w.p.a.1. As
$\beta_{\zeta}$ is a global maximizer of $\widehat{G}(\bar{\theta},\beta)$
over $\mathcal{M}_{n}$ we have that
$\widehat{G}(\bar{\theta},\beta_{\zeta})\geq\widehat{G}(\bar{\theta},\beta)$
for any $\beta\in\mathcal{M}_{n}$ and therefore,

$\displaystyle-\epsilon\varphi^{\ast}(0)$
$\displaystyle=\widehat{G}(\bar{\theta},0)$ (151)
$\displaystyle\leq\widehat{G}(\bar{\theta},\beta_{\zeta})$ (152)
$\displaystyle\leq-\epsilon\varphi^{\ast}(0)-\frac{1}{2}\beta_{\zeta}^{T}\Lambda_{n}(\dot{\beta},\bar{\theta})\beta_{\zeta}+\|\beta_{\zeta}\|_{\mathcal{M}}\left(\|\bar{\Psi}\|_{\mathcal{H}}+\alpha(C+\|\bar{\Psi}\|_{\mathcal{H}^{\ast}})+\operatorname*{MMD}(\hat{P}_{n},\omega;\mathcal{F})\right)$
(153)
$\displaystyle\leq-\epsilon\varphi^{\ast}(0)-C\|\beta_{\zeta}\|_{\mathcal{M}}^{2}+\|\beta_{\zeta}\|_{\mathcal{M}}\left(\|\bar{\Psi}\|_{\mathcal{H}}+\alpha(C+\|\bar{\Psi}\|_{\mathcal{H}^{\ast}})+\operatorname*{MMD}(\hat{P}_{n},\omega;\mathcal{F})\right)$
(154)

Now, adding $-\epsilon\varphi^{\ast}(0)$ on both sides and dividing by
$\|\beta_{\zeta}\|_{\mathcal{M}}$, we have

$\displaystyle
C\|\beta_{\zeta}\|_{\mathcal{M}}\leq\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}+\alpha(C+\|\bar{\Psi}\|_{\mathcal{H}^{\ast}})+\operatorname*{MMD}(\hat{P}_{n},\omega;\mathcal{F}).$
(155)

As $\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$ by assumption and by
Assumption f) $\alpha=O_{p}(n^{-1})$ as well as
$\operatorname*{MMD}(\hat{P}_{n},\omega;\mathcal{F})=O(n^{-1})=o_{p}(n^{-1/2})$
by Lemma E.8, we thus obtain
$\|\beta_{\zeta}\|_{\mathcal{M}}=O_{p}(n^{-1/2})$.

So far, we have restricted the analysis to the maximizer $\beta_{\zeta}$ over
the magnitude constrained set of dual variables $\mathcal{M}_{n}$. In the
following we will show that this maximizer agrees with the maximizer over the
unconstrained (original) set of dual variables $\mathcal{M}$. First, note that
with $\|\beta_{\zeta}\|_{\mathcal{M}}=O_{p}(n^{-1/2})$ and $\zeta<1/2$ we have
that $n^{-\zeta}>n^{-1/2}$, which means that asymptotically $\beta_{\zeta}$ is
contained in the interior of $\mathcal{M}_{n}$, i.e.,
$\beta_{\zeta}\in\operatorname{int}(\mathcal{M}_{n})$. As $\beta_{\zeta}$ is a
maximizer contained in the interior of the domain, it must correspond to a
stationary point of $\widehat{G}$, i.e.,
$\frac{\partial\widehat{G}}{\partial\beta}(\bar{\theta},\beta_{\zeta})=0$.
Clearly $\mathcal{M}_{n}\subset\mathcal{M}$, so the stationary point is
contained also in $\mathcal{M}$. As the empirical objective $\widehat{G}$ is
concave with respect to $\beta$, this means we must have that
$\widehat{G}(\bar{\theta},\beta_{\zeta})=\sup_{\beta\in\mathcal{M}}\widehat{G}(\bar{\theta},\beta)$
and thus $\bar{\beta}=\beta_{\zeta}$, where again
$\bar{\beta}=\operatorname*{arg\,max}_{\beta\in\mathcal{M}}\widehat{G}(\bar{\theta},\beta)$.

As $\bar{\beta}=\beta_{\zeta}$, and
$\|\beta_{\zeta}\|_{\mathcal{M}}=O_{p}(n^{-1/2})$ it directly follows that
$\|\bar{\beta}\|_{\mathcal{M}}=O_{p}(n^{-1/2})$. Finally by assumption we have
$\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$ and thus
$\widehat{G}(\bar{\theta},\bar{\beta})\leq-\epsilon\varphi^{\ast}(0)+\left(\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}+o_{p}(n^{-1/2})\right))\|\bar{\beta}\|_{\mathcal{M}}-C\|\bar{\beta}\|_{\mathcal{M}}^{2}=-\epsilon\varphi^{\ast}(0)+O_{p}(n^{-1})$.
∎

######  Lemma E.13.

Let the assumptions of Theorem A.1 be satisfied and denote the KMM estimator
as
$\hat{\theta}=\operatorname*{arg\,min}_{\theta\in\Theta}\sup_{\beta\in\mathcal{M}}\widehat{G}_{\epsilon,\lambda_{n}}({\theta},\beta)$.
Then
$\|E_{\hat{P}_{n}}[\Psi(X,Z;\hat{\theta})]\|_{\mathcal{H}^{\ast}}=O_{p}\left(n^{-1/2}\right)$.

###### Proof.

Define $\hat{\Psi}_{i}:=\Psi(x_{i},z_{i};\hat{\theta})$ and
$\hat{\Psi}=\frac{1}{n}\sum_{i=1}^{n}\hat{\Psi}_{i}$. For simplicity of
notation let
$\widehat{G}(\theta,\beta):=\widehat{G}_{\epsilon,\lambda_{n}}(\theta,\beta)$.
For any $\eta\in\mathbb{R}$ and $f\in\mathcal{F}$ consider the dual variable
$\bar{\beta}=(\eta,f,\phi(\hat{\Psi}))$ and its normalized version
$\bar{\beta}_{\zeta}=n^{-\zeta}\bar{\beta}/\|\bar{\beta}\|$, where
$\phi(\hat{\Psi})$ denotes the Riesz representer of
$\hat{\Psi}\in\mathcal{H}^{\ast}$ in $\mathcal{H}$ and $0<\zeta<1/2$ as in
Lemma E.9. Taylor expanding the KMM objective about $\beta=0$ again yields

$\displaystyle\widehat{G}(\hat{\theta},\bar{\beta}_{\zeta})=$
$\displaystyle-\epsilon\varphi^{\ast}(0)-\frac{1}{2}\bar{\beta}_{\zeta}^{T}\Lambda_{n}(\dot{\beta},\hat{\theta})\bar{\beta}_{\zeta}+\frac{n^{-\zeta}}{\|\bar{\beta}\|}\int_{\mathcal{X}\times\mathcal{Z}}{\Psi(x,z;\hat{\theta})(\phi(\Psi))}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$ (156)
$\displaystyle+\frac{n^{-\zeta}}{\|\bar{\beta}\|}\int_{\mathcal{X}\times\mathcal{Z}}f(x,z)\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)-\frac{n^{-\zeta}}{\|\bar{\beta}\|}\int_{\mathcal{X}\times\mathcal{Z}}{f(x,z)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$ (157) $\displaystyle=$
$\displaystyle-\epsilon\varphi^{\ast}(0)-\frac{1}{2}\bar{\beta}_{\zeta}^{T}\Lambda_{n}(\dot{\beta},\hat{\theta})\bar{\beta}_{\zeta}+\frac{n^{-\zeta}}{\|\bar{\beta}\|}\int_{\mathcal{X}\times\mathcal{Z}}\Psi(x,z;\hat{\theta})(\phi(\hat{\Psi}))\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)$
(158)
$\displaystyle+\frac{n^{-\zeta}}{\|\bar{\beta}\|}\int_{\mathcal{X}\times\mathcal{Z}}{\Psi(x,z;\hat{\theta})(\phi(\hat{\Psi}))}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)-\frac{n^{-\zeta}}{\|\bar{\beta}\|}\int_{\mathcal{X}\times\mathcal{Z}}\Psi(x,z;\hat{\theta})(\phi(\hat{\Psi}))\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)$
(159)
$\displaystyle+\frac{n^{-\zeta}}{\|\bar{\beta}\|}\int_{\mathcal{X}\times\mathcal{Z}}f(x,z)\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)-\frac{n^{-\zeta}}{\|\bar{\beta}\|}\int_{\mathcal{X}\times\mathcal{Z}}{f(x,z)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$ (160) $\displaystyle\geq$
$\displaystyle-\epsilon\varphi^{\ast}(0)-\frac{1}{2}\bar{\beta}_{\zeta}^{T}\Lambda_{n}(\dot{\beta},\hat{\theta})\bar{\beta}_{\zeta}+\frac{n^{-\zeta}}{\|\bar{\beta}\|}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}^{2}-\frac{n^{-\zeta}}{\|\bar{\beta}\|}\|f\|_{\mathcal{F}}\operatorname*{MMD}(\hat{P}_{n},\omega;\mathcal{F})$
(161) $\displaystyle-\frac{\alpha
n^{-\zeta}}{\|\bar{\beta}\|}\left(\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}\int_{\mathcal{X}\times\mathcal{Z}}\|\Psi(x,z;\hat{\theta})\|_{\mathcal{H}^{\ast}}Q(\mathrm{d}x\otimes\mathrm{d}z)+\|\hat{\Psi}\|^{2}_{\mathcal{H}^{\ast}}\right)$
(162) $\displaystyle\geq$
$\displaystyle-\epsilon\varphi^{\ast}(0)-\frac{1}{2}\bar{\beta}_{\zeta}^{T}\Lambda_{n}(\dot{\beta},\hat{\theta})\bar{\beta}_{\zeta}+C_{\psi}n^{-\zeta}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}$
(163) $\displaystyle-\alpha
n^{-\zeta}C_{\psi}\left(C_{Q}+\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}\right)-n^{-\zeta}C_{f}\operatorname*{MMD}(\hat{P}_{n},\omega;\mathcal{F}),$
(164)

where $C_{\psi},C_{f}\in[0,1]$ as
$\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}/\|\bar{\beta}\|_{\mathcal{M}}\leq 1$ and
$\|f\|_{\mathcal{F}}/\|\bar{\beta}\|_{\mathcal{M}}\leq 1$ by definition of
$\bar{\beta}$ and $\alpha=O_{p}(n^{-1})$ as well as
$\operatorname*{MMD}(\hat{P}_{n},\omega;\mathcal{F})=O_{p}(n^{-1})$ by Lemma
E.8 and Assumption f). Using Lemma E.11 we can bound the largest eigenvalue of
$\Lambda_{n}(\dot{\beta},\hat{\theta})$ by some positive constant $M$ which is
independent of $n$, so we obtain

$\displaystyle\widehat{G}(\hat{\theta},\bar{\beta}_{\zeta})\geq-\epsilon\varphi^{\ast}(0)-Mn^{-2\zeta}+C_{\psi}n^{-\zeta}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}+O_{p}(n^{-1-\zeta}),$
(165)

Now as $(\hat{\theta},\hat{\beta})$ is a saddle point of the _empirical_ KMM
objective, we have
$\widehat{G}(\hat{\theta},\bar{\beta}_{\zeta})\leq\widehat{G}(\hat{\theta},\hat{\beta})\leq\max_{\beta\in\mathcal{M}}\widehat{G}(\theta_{0},\beta)$.
Putting this together with the previous inequality we have

$\displaystyle-\epsilon\varphi^{\ast}(0)+C_{\psi}n^{-\zeta}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}-Mn^{-2\zeta}+O_{p}(n^{-1-\zeta})$
$\displaystyle\leq\widehat{G}(\hat{\theta},\bar{\beta}_{\zeta})$ (166)
$\displaystyle\leq\widehat{G}(\hat{\theta},\hat{\beta})$ (167)
$\displaystyle\leq\max_{\beta\in\mathcal{M}}\widehat{G}(\theta_{0},\beta)$
(168) $\displaystyle\leq-\epsilon\varphi^{\ast}(0)+O_{p}(n^{-1}),$ (169)

where in the last line we used Lemma E.12 with $\bar{\theta}=\theta_{0}$ which
fulfills the corresponding conditions as
$\|E[\Psi(X,Z;\theta_{0})]\|_{\mathcal{H}^{\ast}}=0$ by definition and thus by
the central limit theorem
$\|E_{\hat{P}_{n}}[\Psi(X,Z;\theta_{0})]\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$.
Adding $\epsilon\varphi^{\ast}(0)$ on both sides and solving for
$\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}$ yields

$\displaystyle\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}\leq
O_{p}(n^{-1+\zeta})+O_{p}(n^{-\zeta})=O_{p}(n^{-\zeta}),$ (170)

where the last step follows as $\zeta<1/2$ by definition and thus
$-1+\zeta<-\zeta$. Equation (170) provides an upper bound on the convergence
rate for $\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}$. To further refine this rate
define $\tilde{\beta}:=(0,0,\phi(\hat{\Psi}))$ and for any sequence
$\kappa_{n}\rightarrow 0$ consider $\kappa_{n}\tilde{\beta}$. Then as
$\|\tilde{\beta}\|_{\mathcal{M}}=\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}\leq
O_{p}(n^{-\zeta})$ we immediately have
$\|\kappa_{n}\tilde{\beta}\|_{\mathcal{M}}=o_{p}(n^{-\zeta})$ which implies
$\kappa_{n}\tilde{\beta}\in\mathcal{M}_{n}$ w.p.a.1 and

$\displaystyle-\epsilon\varphi^{\ast}(0)+\kappa_{n}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}^{2}-M\kappa_{n}^{2}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}^{2}+O_{p}(n^{-1-\zeta})$
$\displaystyle\leq\widehat{G}(\hat{\theta},\kappa_{n}\tilde{\beta})$ (171)
$\displaystyle\leq\widehat{G}(\hat{\theta},\hat{\beta})$ (172)
$\displaystyle\leq\max_{\beta\in\mathcal{M}}\widehat{G}(\theta_{0},\beta)$
(173) $\displaystyle\leq-\epsilon\varphi^{\ast}(0)+O_{p}(n^{-1}).$ (174)

This implies
$(1-\kappa_{n}M)\kappa_{n}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}^{2}\leq
O_{p}(n^{-1})$ and as $(1-\kappa_{n}M)$ is bounded away from zero for all
sufficiently large $n$, we get
$\kappa_{n}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}^{2}=O_{p}(n^{-1})$. As this
holds for all $\kappa_{n}\rightarrow 0$, we finally obtain
$\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$. ∎

##### Proof of Theorem A.1

###### Proof.

Define $\hat{\Psi}_{i}=\Psi(x_{i},z_{i};\hat{\theta})$ and
$\hat{\Psi}=\frac{1}{n}\sum_{i=1}^{n}\hat{\Psi}_{i}$. As $\hat{\Psi}$ is the
average of $n$ i.i.d. random variables $\hat{\Psi}_{i}$, by the central limit
theorem and absolute homogeneity of the dual norm, we have
$\|\hat{\Psi}(\theta)-E[\Psi(X,Z;\theta)]\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$
for any $\theta\in\Theta$. From Lemma E.13 we also have
$\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$ and thus using the
triangle inequality we get

$\displaystyle\left\|E[\Psi(X,Z;\hat{\theta})]\right\|_{\mathcal{H}^{\ast}}$
$\displaystyle=\left\|E[\Psi(X,Z;\hat{\theta})]-\hat{\Psi}+\hat{\Psi}\right\|_{\mathcal{H}^{\ast}}$
(175)
$\displaystyle\leq\left\|E[\Psi(X,Z;\hat{\theta})]-\hat{\Psi}\right\|_{\mathcal{H}^{\ast}}+\left\|\hat{\Psi}\right\|_{\mathcal{H}^{\ast}}$
(176) $\displaystyle=O_{p}(n^{-1/2})\overset{p}{\rightarrow}0.$ (177)

As by assumption $\theta=\theta_{0}$ is the unique parameter for which
$\theta\mapsto\|E[\Psi(X,Z;\theta)]\|_{\mathcal{H}^{\ast}}=0$ it follows that
$\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$. To derive a convergence rate
for $\hat{\theta}$ note that by the mean value theorem, there exists
$\bar{\theta}\in\operatorname{conv}(\\{\theta_{0},\hat{\theta}\\})$ such that

$\displaystyle\Psi(X,Z;\hat{\theta})=\Psi(X,Z;\theta_{0})+(\hat{\theta}-\theta_{0})^{T}\nabla_{\theta}\Psi(X,Z;\bar{\theta}).$
(178)

Using this we have

$\displaystyle\|E[\Psi(X,Z;\hat{\theta})]\|^{2}_{\mathcal{H}^{\ast}}$
$\displaystyle=\|\underbrace{E[\Psi(X,Z;\theta_{0})]}_{=0}+(\hat{\theta}-\theta_{0})^{T}E[\nabla_{\theta}\Psi(X,Z;\bar{\theta})]\|^{2}_{\mathcal{H}^{\ast}}$
(179)
$\displaystyle=\left\langle(\hat{\theta}-\theta_{0})^{T}E[\nabla_{\theta}\Psi(X,Z;\bar{\theta})],(\hat{\theta}-\theta_{0})^{T}E[\nabla_{\theta}\Psi(X,Z;\bar{\theta})]\right\rangle_{\mathcal{H}^{\ast}}$
(180) $\displaystyle=(\hat{\theta}-\theta_{0})^{T}\underbrace{\left\langle
E[\nabla_{\theta}\Psi(X,Z;\bar{\theta})],E[\nabla_{\theta}\Psi(X,Z;\bar{\theta})]\right\rangle_{\mathcal{H}^{\ast}}}_{=:\Sigma(\bar{\theta})}(\hat{\theta}-\theta_{0})$
(181)
$\displaystyle\geq\lambda_{\text{min}}\left(\Sigma(\bar{\theta})\right)\|\hat{\theta}-\theta_{0}\|^{2}_{2}$
(182)

Now as $\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$ and
$\bar{\theta}\in\operatorname{conv}(\\{\theta_{0},\hat{\theta}\\})$ we have
$\bar{\theta}\overset{p}{\rightarrow}\theta_{0}$ and thus
$\Sigma(\bar{\theta})\overset{p}{\rightarrow}\Sigma(\theta_{0})=:\Sigma_{0}$
by the continuous mapping theorem. By the non-negativity of the norm
$\Sigma_{0}$ is positive-semi definite and non-singular by Assumption j), thus
the smallest eigenvalue of $\Sigma(\bar{\theta})$,
$\lambda_{\text{min}}(\Sigma(\bar{\theta}))$, is positive and bounded away
from zero w.p.a.1. Finally as $\|E[\Psi(X,Z;\hat{\theta})]\|=O_{p}(n^{-1/2})$
taking the square-root on both sides we have
$\|\hat{\theta}-\theta_{0}\|=O_{p}(n^{-1/2})$. ∎

####  E.4.2 Proof of Theorem A.2

######  Lemma E.14.

For a moment functional $\Psi(X,Z;\theta):\mathcal{H}\rightarrow\mathbb{R}$,
continuously differentiable in $\theta$, define the covariance operator
$\Omega_{0}:=E[\Psi(X,Z;\theta)\otimes\Psi(X,Z;\theta)]$.
Further define the matrix
$\Sigma_{0}:=\langle\nabla_{\theta}E[\Psi(X,Z;\theta_{0})],E[\nabla_{\theta}\Psi(X,Z;\theta_{0})]\rangle_{\mathcal{H}}\in\mathbb{R}^{p\times
p}$, where the inner product is only taken with respect to the $\mathcal{H}$
index. If $\Omega_{0}$ and $\Sigma_{0}$ are non-singular with smallest
eigenvalue bounded away from zero, then the matrix

$\displaystyle\Gamma:=-E[\nabla_{\theta}\Psi(X,Z;\theta_{0})]\Omega_{0}^{-1}E[\nabla_{\theta}\Psi(X,Z;\theta_{0})]$
(183)

is non-singular with smallest eigenvalue bounded away from zero.

###### Proof.

Let $\mathcal{H}=\bigoplus_{i=1}^{m}\mathcal{H}_{i}$ and for $i=1,\ldots,m$
let $\\{h^{i}_{j}\\}_{j=1}^{\infty}$ denote a orthonormal basis of
$\mathcal{H}_{i}$ such that $\langle
h_{i}^{k},h_{j}^{l}\rangle=\delta_{ij}\delta_{kl}$. Then we can write the
identity operator in $\mathcal{H}$ as
$I_{\mathcal{H}}=\sum_{i=1}^{m}\sum_{j=1}^{\infty}h_{j}^{i}\left(h_{j}^{i}\right)^{\ast}$,
where $\left(h_{j}^{i}\right)^{\ast}$ is the Riesz representer of $h_{j}^{i}$
in $\mathcal{H}^{\ast}$. Further, let
$\nabla_{\theta}\Psi_{0}:=E[\nabla_{\theta}\Psi(X,Z;\theta_{0})]$. Then we can
write for any $\theta\in\Theta$,

$\displaystyle-\theta^{T}\Gamma\theta$
$\displaystyle=\theta^{T}\nabla_{\theta}\Psi_{0}\Omega_{0}^{-1}\nabla_{\theta}\Psi_{0}\theta$
(184)
$\displaystyle=\theta^{T}\nabla_{\theta}\Psi_{0}\sum_{i=1}^{m}\sum_{j=1}^{\infty}h_{j}^{i}\left(h_{j}^{i}\right)^{\ast}\Omega_{0}^{-1}\sum_{k=1}^{m}\sum_{l=1}^{\infty}h_{l}^{k}\left(h_{l}^{k}\right)^{\ast}\left(\nabla_{\theta}\Psi_{0}\right)^{T}\theta$
(185)
$\displaystyle=\theta^{T}\nabla_{\theta}\Psi_{0}\left(\sum_{i,k=1}^{m}\sum_{j,l=1}^{\infty}h^{i}_{j}\
\langle h_{j}^{i},\Omega_{0}^{-1}h_{l}^{k}\rangle_{\mathcal{H}}\
h^{k}_{l}\right)\left(\nabla_{\theta}\Psi_{0}\right)^{T}\theta$ (186)
$\displaystyle\geq\lambda_{\textrm{min}}(\Omega_{0}^{-1})\theta^{T}\nabla_{\theta}\Psi_{0}\left(\sum_{i=1}^{m}\sum_{j=1}^{\infty}h^{i}_{j}\left(h_{j}^{i}\right)^{\ast}\right)\left(\nabla_{\theta}\Psi_{0}\right)^{T}\theta$
(187)
$\displaystyle=\lambda_{\textrm{min}}(\Omega_{0}^{-1})\theta^{T}\langle\nabla_{\theta}\Psi_{0},\nabla_{\theta}\Psi_{0}\rangle_{\mathcal{H}}\theta$
(188)
$\displaystyle=\lambda_{\textrm{min}}(\Omega_{0}^{-1})\theta^{T}\Sigma_{0}\theta$
(189)
$\displaystyle\geq\lambda_{\textrm{min}}(\Omega_{0}^{-1})\lambda_{\textrm{min}}(\Sigma_{0})\|\theta\|^{2}>0,$
(190)

where we used that $\Omega_{0}$ is positive semi-definite by construction and
non singular by Assumption e) and thus being the inverse of a strictly
positive definite operator the smallest eigenvalue
$\lambda_{\textrm{min}}(\Omega_{0}^{-1})$ of $\Omega_{0}^{-1}$ is positive and
bounded away from zero. Moreover, $\Sigma_{0}$ is positive semi-definite by
construction and non singular by Assumptions j) and therefore its smallest
eigenvalue $\lambda_{\textrm{min}}(\Sigma_{0})$ positive and bounded away from
zero. From this it immediately follows that $\Gamma$ is strictly negative
definite and thus non-singular. ∎

##### Proof of Theorem A.2

###### Proof.

The KMM estimator $\hat{\theta}$ and the optimal Lagrange parameter
$\hat{\beta}$ are determined via the first order optimality conditions for
$(\theta,\beta)$ which are given by

$\displaystyle
0=\frac{\partial\widehat{G}}{\partial\theta}(\hat{\theta},\hat{\beta})$
$\displaystyle=-\int_{\mathcal{X}\times\mathcal{Z}}{\varphi_{1}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\hat{\theta})^{T}\hat{\beta}\right)\nabla_{\theta}\left(a(x,z;\hat{\theta})^{T}\hat{\beta}\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$ (191) $\displaystyle
0=\frac{\partial\widehat{G}}{\partial\beta}(\hat{\theta},\hat{\beta})$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}b_{i}-\int_{\mathcal{X}\times\mathcal{Z}}{\varphi_{1}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\hat{\theta})^{T}\hat{\beta}\right)a(x,z;\hat{\theta})}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)-R_{\lambda_{n}}\beta$ (192)

Linearizing the first condition (191) about the true parameters
$(\theta_{0},0)$ yields

$\displaystyle 0=$
$\displaystyle\frac{\partial\widehat{G}}{\partial\theta}(\theta_{0},0)+\left(\frac{\partial^{2}\widehat{G}}{\partial\theta\
\partial\theta}(\bar{\theta},\bar{\beta})\right)\left(\hat{\theta}-\theta_{0}\right)+\left(\frac{\partial^{2}\widehat{G}}{\partial\theta\
\partial\beta}(\bar{\theta},\bar{\beta})\right)\hat{\beta}$ (193)
$\displaystyle=$
$\displaystyle-\left(\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}\varphi_{2}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\bar{\theta})^{T}\bar{\beta}\right)\left(\nabla_{\theta}a(x,z;\bar{\theta})^{T}\bar{\beta}\right)\left(\nabla_{\theta}a(x,z;\bar{\theta})^{T}\bar{\beta}\right)^{T}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)\left(\hat{\theta}-\theta_{0}\right)$
(194)
$\displaystyle-\left(\int_{\mathcal{X}\times\mathcal{Z}}{\varphi_{1}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\bar{\theta})^{T}\bar{\beta}\right)\left(\nabla_{\theta}^{2}a(x,z;\bar{\theta})^{T}\bar{\beta}\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)\left(\hat{\theta}-\theta_{0}\right)$
(195)
$\displaystyle-\left(\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}\varphi_{2}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\bar{\theta})^{T}\bar{\beta}\right)\left(\nabla_{\theta}a(x,z;\bar{\theta})^{T}\bar{\beta}\right)a(x,z;\bar{\theta})^{T}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)\hat{\beta}$ (196)
$\displaystyle-\left(\int_{\mathcal{X}\times\mathcal{Z}}{\varphi_{1}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\bar{\theta})^{T}\bar{\beta}\right)\nabla_{\theta}a(x,z;\bar{\theta})^{T}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)\hat{\beta}$ (197)

for some $(\bar{\theta},\bar{\beta})$ on the line between
$(\hat{\theta},\hat{\beta})$ and $(\theta_{0},0)$. Analogously the
linearization of the second condition (192) is given by

$\displaystyle 0=$
$\displaystyle\frac{\partial\widehat{G}}{\partial\beta}(\theta_{0},0)+\left(\frac{\partial^{2}\widehat{G}}{\partial\theta\
\partial\beta}(\dot{\theta},\dot{\beta})\right)\left(\hat{\theta}-\theta_{0}\right)+\left(\frac{\partial^{2}\widehat{G}}{\partial\beta\
\partial\beta}(\dot{\theta},\dot{\beta})\right)\hat{\beta}$ (198)
$\displaystyle=$
$\displaystyle-\frac{1}{n}\sum_{i=1}^{n}b_{i}+\int_{\mathcal{X}\times\mathcal{Z}}{a(x,z;\theta_{0})}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$ (199)
$\displaystyle-\left(\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}\varphi_{2}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\dot{\theta})^{T}\dot{\beta}\right)a(x,z;\dot{\theta})\left(\nabla_{\theta}a(x,z;\dot{\theta})^{T}\dot{\beta}\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)\left(\hat{\theta}-\theta_{0}\right)$
(200)
$\displaystyle-\left(\int_{\mathcal{X}\times\mathcal{Z}}{\varphi_{1}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\dot{\theta})^{T}\dot{\beta}\right)\nabla_{\theta}a(x,z;\dot{\theta})}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)\left(\hat{\theta}-\theta_{0}\right)$
(201)
$\displaystyle-\left(\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}\varphi_{2}^{\ast}\left(\frac{1}{\epsilon}a_{i}(\dot{\theta})^{T}\dot{\beta}\right)a(x,z;\dot{\theta})a(x,z;\dot{\theta})^{T}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+R_{\lambda_{n}}\right)\hat{\beta}$ (202)

Now, as $\bar{\beta},\dot{\beta}$ are on the line between $\hat{\beta}$ and
$0$ and $\hat{\beta}=O_{p}(n^{-1/2})$ by Lemma E.12, we have that all
derivative terms of $\widehat{G}$ involving $\bar{\beta},\dot{\beta}$ linearly
are $O_{p}(n^{-1})$, as each term additionally gets multiplied by
$\hat{\theta}-\theta$ or $\hat{\beta}$ which both are $O_{p}(n^{-1/2})$ in the
respective norms. Further it follows from Lemma E.9 that
$\varphi_{j}(a(X,Z;\theta)^{T}\tilde{\beta})\overset{p}{\rightarrow}1\ \
\omega\text{-a.s.}$ for any $\theta\in\Theta$,
$\tilde{\beta}\in\\{\bar{\beta},\dot{\beta}\\}$ and $j=1,2$. Therefore, the
first linearized first order condition (191) reduces to

$\displaystyle 0=$
$\displaystyle\left(\int_{\mathcal{X}\times\mathcal{Z}}{\nabla_{\theta}a(x,z;\bar{\theta})^{T}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)\hat{\beta}+O_{p}(n^{-1}).$ (203)

For the second condition note that by Lemma E.10

$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}{a(x,z;\theta_{0})}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}a(x_{i},z_{i};\theta_{0})+O_{p}(n^{-1}).$
(204)

Now inserting this into the second linearized first order condition we obtain

$\displaystyle 0=$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\left(b_{i}-a(x_{i},z_{i};\theta_{0})\right)+\int_{\mathcal{X}\times\mathcal{Z}}{\nabla_{\theta}a(x,z;\dot{\theta})^{T}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\left(\hat{\theta}-\theta_{0}\right)$
(205)
$\displaystyle+\underbrace{\left(\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}a(x,z;\dot{\theta})a(x,z;\dot{\theta})^{T}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+R_{\lambda_{n}}\right)}_{=:\Lambda_{n}(\dot{\theta})}\hat{\beta}+O_{p}(n^{-1}),$
(206)

where
$\frac{1}{n}\sum_{i=1}^{n}(b_{i}-a(x_{i},z_{i};\theta_{0}))=\left(0,0,\hat{\Psi}(\theta_{0})\right)^{T}$.
Writing the two linearized first order conditions in matrix-vector form we
obtain

$\displaystyle\begin{pmatrix}0\\\ 0\end{pmatrix}=$
$\displaystyle\begin{pmatrix}0\\\
\frac{1}{n}\sum_{i=1}^{n}\left(b_{i}-a(x_{i},z_{i};\theta_{0})\right)\end{pmatrix}$
(207)
$\displaystyle+\underbrace{\begin{pmatrix}0&\int_{\mathcal{X}\times\mathcal{Z}}{\nabla_{\theta}a(x,z;\bar{\theta})^{T}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\\\
\int_{\mathcal{X}\times\mathcal{Z}}{\nabla_{\theta}a(x,z;\dot{\theta})^{T}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)&\Lambda_{n}(\dot{\theta})\end{pmatrix}}_{:=M_{n}}\begin{pmatrix}\hat{\theta}-\theta_{0}\\\
\hat{\beta}-\beta_{0}\end{pmatrix},$ (208)

where $\beta_{0}=0$. Now $\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$ and
$\dot{\theta}$ and $\bar{\theta}$ are on the line between $\hat{\theta}$ and
$\theta$, and $\omega\overset{p}{\rightarrow}P_{0}$ weakly by definition.
Moreover, $\nabla_{\theta}\Psi(X,Z;\bar{\theta})$ is continuous for any
$\bar{\theta}$ in a neighborhood $\bar{\Theta}$ of $\theta_{0}$ by Assumption
i) and thus by the continuous mapping theorem we have
$\int_{\mathcal{X}\times\mathcal{Z}}{\nabla_{\theta}a(x,z;\bar{\theta})}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\overset{p}{\rightarrow}E[\nabla_{\theta}a(X,Z;\theta_{0})]=:\nabla_{\theta}a_{0}$
and the same holds for the other off-diagonal entry. In addition, the off-
diagonal entries are bounded as Assumption i) with Lemma E.7 implies
$E[\sup_{\theta\in\bar{\Theta}}\|\nabla_{\theta}\Psi(X,Z;\theta)\|]<\infty$.
Finally, by the uniform weak law of large numbers and the continuous mapping
theorem we have
${\Lambda}_{n}(\dot{\theta})\overset{p}{\rightarrow}{\Lambda}(\theta_{0})=E[a(X,Z;\theta_{0})a(X,Z;\theta_{0})^{T}]+R_{\lambda=0}=:\Lambda$.
Let correspondingly $M$ denote the limit operator for $M_{n}$, i.e.,
$M_{n}\overset{p}{\rightarrow}M$. From Lemma E.11 it follows that the smallest
eigenvalue of $\Lambda$ is bounded away from zero and thus it is invertible.
Now suppose the Schur complement of $\Lambda$ in $M$,

$\displaystyle\Gamma:=M/\Lambda=-\left(\nabla_{\theta}a_{0}^{T}\right)\Lambda^{-1}\left(\nabla_{\theta}a_{0}^{T}\right)=-\nabla_{\theta}\Psi_{0}\left(\Lambda^{-1}\right)_{3,3}\nabla_{\theta}\Psi_{0}$
(209)

is invertible, then it follows from standard blockmatrix algebra (see e.g.
Bernstein [2009]), that the inverse of $M$ is given by

$\displaystyle
M^{-1}=\begin{pmatrix}\Gamma^{-1}&\Gamma^{-1}\left(\nabla_{\theta}a_{0}\right)\Lambda^{-1}\\\
\Lambda^{-1}\left(\nabla_{\theta}a_{0}\right)\Gamma^{-1}&\Lambda^{-1}+\Lambda^{-1}\left(\nabla_{\theta}a_{0}\right)\Gamma^{-1}\left(\nabla_{\theta}a_{0}\right)\Lambda^{-1}\end{pmatrix}.$
(210)

Now it remains to find an explicit expression for $(\Lambda^{-1})_{3,3}$ and
to show that $\Gamma$ is indeed invertible. To this aim we write out the outer
product over $\mathcal{M}\times\mathcal{M}$ in $\Lambda$ which yields

$\displaystyle\Lambda=$
$\displaystyle\frac{1}{\epsilon}\begin{pmatrix}1&E[1\otimes
k((X,Z),\cdot)]&-E[1\otimes\Psi(X,Z;\theta_{0})]\\\ E[k((X,Z),\cdot)\otimes
1]&E[k((X,Z),\cdot)\otimes
k((X,Z),\cdot)]+I&-E[k((X,Z),\cdot)\otimes\Psi(X,Z;\theta_{0})]\\\
-E[\Psi(X,Z;\theta_{0})\otimes 1]&-E[\Psi(X,Z;\theta_{0})\otimes
k((X,Z),\cdot)]&E[\Psi(X,Z;\theta_{0})\otimes\Psi(X,Z;\theta_{0})]\end{pmatrix}$
(211) $\displaystyle=$
$\displaystyle\frac{1}{\epsilon}\begin{pmatrix}1&E[1\otimes
k((X,Z),\cdot)]&0\\\ E[k((X,Z),\cdot)\otimes 1]&E[k((X,Z),\cdot)\otimes
k((X,Z),\cdot)]+I&0\\\ 0&0&\Omega_{0}\end{pmatrix}$ (212)

where we used that $\|E[\Psi(X,Z;\theta_{0})]\|_{\mathcal{H}^{\ast}}=0$ by
definition and as $\mathcal{F}$ is an RKHS, the evaluation functional
$k((x,z),\cdot)$ can be bounded by some constant $C$ and thus we have
$\|E[k((X,Z),\cdot)\otimes\Psi(X,Z;\theta_{0})]\|_{\mathcal{F}\times\mathcal{H}^{\ast}}\leq
C\|E[\Psi(X,Z;\theta_{0})]\|_{\mathcal{H}^{\ast}}=0$. Now $\Lambda$ is of
blockdiagonal form and for the upper block $B$ we have

$\displaystyle B=\begin{pmatrix}1&E[1\otimes k((X,Z),\cdot)]\\\
E[k(X,Z),\cdot)\otimes 1]&E[k(X,Z),\cdot)\otimes
k((X,Z),\cdot)]\end{pmatrix}+\begin{pmatrix}1&0\\\ 0&I\end{pmatrix}.$ (213)

The first term is symmetric and thus positive semi-definite and the second
term diagonal with positive entries, thus B is a strictly positive definite
operator and thus invertible. Moreover, by Assumption e) of Theorem A.1
$\Omega_{0}$ is invertible and thus we can conclude

$\displaystyle\Lambda^{-1}=\begin{pmatrix}B^{-1}&0\\\
0&\Omega_{0}^{-1}\end{pmatrix}$ (214)

and $\left(\Lambda^{-1}\right)_{3,3}=\Omega_{0}^{-1}$. Now, from invertibility
of $\Omega_{0}$ we directly obtain that $\Omega_{0}^{-1}$ is non-singular and
thus by Assumption e) and Lemma E.14 we have that $\Gamma$ is non-singular and
invertible which legitimates the inversion of $M$.

With this at hand, we can solve equation (208) for $\hat{\theta}-\theta_{0}$
and obtain

$\displaystyle\sqrt{n}\left(\hat{\theta}-\theta_{0}\right)$
$\displaystyle=\left(\Gamma^{-1}\nabla_{\theta}\Psi_{0}\left(\Lambda^{-1}\right)_{3,3}\right)\sqrt{n}\hat{\Psi}(\theta_{0})$
(215)
$\displaystyle=-\left(\left(\nabla_{\theta}\Psi_{0}\left(\Lambda^{-1}\right)_{3,3}\nabla_{\theta}\Psi_{0}\right)^{-1}\nabla_{\theta}\Psi_{0}\left(\Lambda^{-1}\right)_{3,3}\right)\sqrt{n}\hat{\Psi}(\theta_{0}).$
(216)

By the central limit theorem we have
$\sqrt{n}E_{\hat{P}_{n}}[\Psi(X,Z;\theta_{0})]\sim\mathcal{N}(0,\Omega_{0})$
where as before
$\Omega_{0}=E[\Psi(X,Z;\theta_{0})\otimes\Psi(X,Z;\theta_{0})]$ and thus
inserting into equation (216) we get

$\displaystyle\sqrt{n}\left(\hat{\theta}-\theta_{0}\right)$
$\displaystyle=-\left(\left(\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\left(\nabla_{\theta}\Psi_{0}\right)\right)^{-1}\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\right)\sqrt{n}\hat{\Psi}(\theta_{0})\sim\mathcal{N}(0,\Xi)$
(217)

with

$\displaystyle\Xi$
$\displaystyle=\left(\left(\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\left(\nabla_{\theta}\Psi_{0}\right)\right)^{-1}\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\right)\Omega_{0}\left(\left(\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\left(\nabla_{\theta}\Psi_{0}\right)\right)^{-1}\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\right)^{T}$
(218)
$\displaystyle=\left(\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\left(\nabla_{\theta}\Psi_{0}\right)\right)^{-1}.$
(219)

∎

###  E.5 Asymptotic Properties of KMM for Finite-Dimensional Moment
Restrictions

####  E.5.1 Proof of Theorem B.1 (Consistency for MR)

The consistency for the finite dimensional case follows as a special case of
the consistency result for the functional case (Theorem A.1) by identifying
$\mathcal{H}=\mathbb{R}^{m}$,
$\Psi(x,z;\theta)=\psi(x;\theta)\in\mathbb{R}^{m}$ and $\lambda_{n}=0$. For a
finite dimensional version of Theorem A.1 we need finite dimensional versions
of Lemmas E.9-E.13, which we will state in the following and describe the
differences in the proofs compared to the functional case. Refer to the proof
of Theorem A.1 for details.

######  Lemma E.15.

Let $\mathcal{A}$ denote a $\sigma$-algebra on $\mathcal{X}$ and let
($\mathcal{X},\mathcal{A},\omega)$ be a probability space with measure
$\omega$. For any function
$\psi:\mathcal{X}\times\Theta\rightarrow\mathbb{R}^{m}$ with
$\int_{\mathcal{X}}{\sup_{\theta\in\Theta}\|\psi(x;\theta)\|_{2}^{2}\
}\omega(\mathrm{d}x)<\infty$, it follows that
$\sup_{\theta\in\Theta}\left\|\psi\left(X;\theta\right)\right\|_{2}\leq C\
\omega\text{-a.s.}$ for some constant $C<\infty$.

###### Proof.

The proof follows immediately from the one for Lemma E.7 by exchanging
$\Psi(X,Z;\theta)\rightarrow\psi(X;\theta)$ and
$\mathcal{H}\rightarrow\mathbb{R}^{m}$. ∎

######  Lemma E.16.

Let the assumptions of Theorem B.1 be satisfied, then for any $\zeta$ with
$0<\zeta<1/2$ define the magnitude constrained set of dual variables
$\mathcal{M}_{n}=\\{\beta=(\eta,f,h)\in\mathcal{M}:\|\beta\|_{\mathcal{M}}\leq
n^{-\zeta}\\}$. Then as $n\rightarrow\infty$ with probability approaching 1,

$\displaystyle\sup_{\theta\in\Theta,\
(\eta,f,h)\in\mathcal{M}_{n}}|\psi(X;\theta)^{T}h|=0\ \omega\text{-a.s.},$
(220) $\displaystyle\sup_{\theta\in\Theta,\
\beta\in\mathcal{M}_{n}}|a(X;\theta)^{T}\beta|=0\ \omega\text{-a.s.}.$ (221)

###### Proof.

The proof follows immediately from the one for Lemma E.9 by exchanging
$\Psi(X,Z;\theta)\rightarrow\psi(X;\theta)$ and
$\mathcal{H}\rightarrow\mathbb{R}^{m}$ and using Lemma E.15 instead of Lemma
E.7 to bound
$\sup_{\theta\in\Theta}\left\|\psi\left(X;\theta\right)\right\|_{2}\leq C\
\omega\text{-a.s.}$. ∎

######  Lemma E.17.

Let the assumptions of Theorem B.1 be satisfied and consider
$\bar{\theta}\in\Theta$ such that
$\bar{\theta}\overset{p}{\rightarrow}\theta_{0}$. Further let
$\beta_{\zeta}:=\operatorname*{arg\,max}_{\beta\in\mathcal{M}_{n}}\widehat{G}(\bar{\theta},\beta)$,
where $\mathcal{M}_{n}=\\{\beta\in\mathcal{M}:\|\beta\|_{\mathcal{M}}\leq
n^{-\zeta}\\}$ with $0<\zeta<1/2$. Define the operator
$\Lambda_{n}(\beta,\theta):\mathcal{M}\rightarrow\mathcal{M}$ as

$\displaystyle\Lambda_{n}(\beta,\theta):=\int_{\mathcal{X}}{\frac{1}{\epsilon}a(x;\theta)a(x;\theta)^{T}\varphi^{\ast}_{2}\left(\frac{1}{\epsilon}a(x;\theta)^{T}\beta\right)}\omega(\mathrm{d}x)+R.$
(222)

Then w.p.a.1 for any
$\bar{\beta}\in\operatorname{conv}(\\{0,\beta_{\zeta}\\})$,
$\Lambda_{n}(\bar{\beta},\bar{\theta})$ is strictly positive definite and its
smallest eigenvalue is bounded away from zero. Moreover, for any
$\theta\in\Theta$ the largest eigenvalue of $\Lambda_{n}(\bar{\beta},\theta)$
is bounded from above by a positive constant $M$.

###### Proof.

The proof follows from the one for the functional case Lemma E.11 with the
difference that we directly impose non-singularity of the covariance matrix
$\Omega_{0}=E[\psi(X;\theta_{0})\psi(X;\theta_{0})^{T}]$ by Assumption e) of
Theorem B.1. ∎

######  Lemma E.18.

Let the assumptions of Theorem B.1 be satisfied. Additionally let
$\bar{\theta}\in\Theta$, $\bar{\theta}\overset{p}{\rightarrow}\theta_{0}$, and
$\|E_{\hat{P}_{n}}[\psi(X;\bar{\theta})]\|_{2}=O_{p}\left(n^{-1/2}\right)$.
Then for $\bar{\beta}=$
$\operatorname*{arg\,max}_{\beta\in\mathcal{M}}\widehat{G}_{\epsilon,\lambda_{n}}(\bar{\theta},\beta)$
we have $\|\bar{\beta}\|_{\mathcal{M}}=O_{p}\left(n^{-1/2}\right)$, and
$\widehat{G}_{\epsilon,\lambda_{n}}(\bar{\theta},\bar{\beta})\leq-\epsilon\varphi^{\ast}(0)+O_{p}\left(n^{-1}\right)$.

###### Proof.

The proof follows immediately from the one for the functional case (Lemma
E.12) with the usual substitutions. ∎

######  Lemma E.19.

Let the assumptions of Theorem B.1 be satisfied and denote the KMM estimator
as
$\hat{\theta}=\operatorname*{arg\,min}_{\theta\in\Theta}\sup_{\beta\in\mathcal{M}}\widehat{G}_{\epsilon,\lambda_{n}}({\theta},\beta)$.
Then
$\|E_{\hat{P}_{n}}[\psi(X;\hat{\theta})]\|_{\mathcal{H}^{\ast}}=O_{p}\left(n^{-1/2}\right)$.

###### Proof.

The proof follows immediately from the one for the functional case (Lemma
E.13) with the usual substitutions. ∎

######  Lemma E.20.

Consider a moment function
$\psi:\mathcal{X}\times\Theta\rightarrow\mathbb{R}^{m}$ with
$\nabla_{\theta}\psi(x;\theta)\in\mathbb{R}^{p\times m}$. Then if
$\operatorname{rank}\left(E[\nabla_{\theta}\psi(X;\theta_{0})]\right)=p$ the
matrix
$\Sigma_{0}=E[\nabla_{\theta}\psi(X;\theta_{0})]E[\nabla_{\theta}\psi(X;\theta_{0})]^{T}\in\mathbb{R}^{p\times
p}$ is non-singular with smallest eigenvalue positive and bounded away from
zero.

###### Proof.

As $\operatorname{rank}(E[\nabla_{\theta}\psi(X;\theta_{0})]=p$, its rows are
linearly independent and thus for any $\theta\in\Theta$ with
$0<\|\theta\|<\infty$ there exists $j\in\\{1,\ldots,m\\}$ such that
$\theta^{T}E[\nabla_{\theta}\psi_{j}(X;\theta_{0})]\neq 0$. This means that
$\theta^{T}\Sigma_{0}\theta=\sum_{i=1}^{m}\left(\theta^{T}E[\nabla_{\theta}\psi_{i}(X;\theta_{0})]\right)^{2}\geq\left(\theta^{T}E[\nabla_{\theta}\psi_{j}(X;\theta_{0})]\right)^{2}>0$,
which follows as any term in the sum is non-negative and there exists at least
one term of index $j$ which is positive. Therefore, the smallest eigenvalue of
$\Sigma_{0}$ is positive and bounded away from zero. ∎

##### Proof of Theorem B.1

###### Proof.

Define $\hat{\psi}_{i}=\psi(x_{i};\hat{\theta})$ and
$\hat{\psi}=\frac{1}{n}\sum_{i=1}^{n}\hat{\psi}_{i}$. As $\hat{\psi}$ is the
average of $n$ i.i.d. random variables $\hat{\psi}_{i}$, by the central limit
theorem and absolute homogeneity of the dual norm, we have
$\|\hat{\psi}(\theta)-E[\psi(X;\theta)]\|_{2}=O_{p}(n^{-1/2})$ for any
$\theta\in\Theta$. From Lemma E.19 we also have
$\|\hat{\psi}\|=O_{p}(n^{-1/2})$ and thus using the triangle inequality we get

$\displaystyle\left\|E[\psi(X;\hat{\theta})]\right\|_{2}$
$\displaystyle=\left\|E[\psi(\hat{\theta})]-\hat{\psi}+\hat{\psi}\right\|_{2}$
(223)
$\displaystyle\leq\left\|E[\psi(X;\hat{\theta})]-\hat{\psi}\right\|_{2}+\left\|\hat{\psi}\right\|_{2}$
(224) $\displaystyle=O_{p}(n^{-1/2})\overset{p}{\rightarrow}0.$ (225)

As by assumption $\theta_{0}$ is the unique parameter for which
$E[\psi(X;\theta)]=0$ it follows that
$\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$. To derive a convergence rate
for $\hat{\theta}$ note that by the mean value theorem, there exists
$\bar{\theta}\in\operatorname{conv}(\\{\theta_{0},\hat{\theta}\\})$ such that

$\displaystyle\psi(X;\hat{\theta})=\psi(X;\theta_{0})+(\hat{\theta}-\theta_{0})^{T}\nabla_{\theta}\psi(X;\bar{\theta}),$
(226)

where $\nabla_{\theta}\psi(x;\theta)\in\mathbb{R}^{p\times m}$. Using this we
have

$\displaystyle\|E[\psi(X;\hat{\theta})]\|^{2}_{2}$
$\displaystyle=\|\underbrace{E[\psi(X;\theta_{0})]}_{=0}+(\hat{\theta}-\theta_{0})^{T}E[\nabla_{\theta}\psi(X;\bar{\theta})]\|^{2}_{2}$
(227)
$\displaystyle=\left\langle(\hat{\theta}-\theta_{0})^{T}E[\nabla_{\theta}\psi(X;\bar{\theta})],(\hat{\theta}-\theta_{0})^{T}E[\nabla_{\theta}\psi(X;\bar{\theta})]\right\rangle$
(228)
$\displaystyle=(\hat{\theta}-\theta_{0})^{T}\underbrace{E[\nabla_{\theta}\psi(X;\bar{\theta})]E[\nabla_{\theta}\psi(X;\bar{\theta})]^{T}}_{=:\Sigma(\bar{\theta})}(\hat{\theta}-\theta_{0})$
(229)
$\displaystyle\geq\lambda_{\text{min}}\left(\Sigma(\bar{\theta})\right)\|\hat{\theta}-\theta_{0}\|^{2}_{2}$
(230)

Now as $\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$ and
$\bar{\theta}\in\operatorname{conv}(\\{\theta_{0},\hat{\theta}\\})$ we have
$\bar{\theta}\overset{p}{\rightarrow}\theta_{0}$ and thus
$\Sigma(\bar{\theta})\overset{p}{\rightarrow}\Sigma(\theta_{0})=:\Sigma_{0}$
by the continuous mapping theorem. Further by Assumption i) of Theorem B.1 and
Lemma E.20 it follows that $\Sigma_{0}$ is positive definite and thus as
$\Sigma(\bar{\theta})\overset{p}{\rightarrow}\Sigma_{0}$ it follows that
$\Sigma(\bar{\theta})$ is positive definite with smallest eigenvalue
$\lambda_{\text{min}}\left(\Sigma(\bar{\theta})\right)$ positive and bounded
away from zero w.p.a.1. Finally as
$\|E[\psi(X;\hat{\theta})]\|=O_{p}(n^{-1/2})$ taking the square-root on both
sides we have $\|\hat{\theta}-\theta_{0}\|=O_{p}(n^{-1/2})$. ∎

####  E.5.2 Proof of Theorem B.2 (Asymptotic Normality for MR)

##### Proof of Theorem B.2

###### Proof.

The proof follows directly from the one for functional moment restrictions
Theorem A.2 by identifying $\Psi(x,z;\theta)=\psi(x;\theta)$,
$\mathcal{H}=\mathbb{R}^{m}$, setting $\lambda_{n}=0$ and using Lemma E.20 to
translate the rank condition, Assumption i), into non-singularity of
$\Sigma_{0}=E[\nabla_{\theta}\psi(X;\theta_{0})]E[\nabla_{\theta}\psi(X;\theta_{0})]^{T}\in\mathbb{R}^{p\times
p}$. ∎