plain

# Estimation Beyond Data Reweighting:
Kernel Method of Moments

Heiner Kremer  Max Planck Institute for Intelligent Systems, Tübingen, Germany
Yassine Nemmour  Max Planck Institute for Intelligent Systems, Tübingen,
Germany  Bernhard Schölkopf  Jia-Jie Zhu  Weierstrass Institute for Applied
Analysis and Stochastics, Berlin, Germany

###### Abstract

Moment restrictions and their conditional counterparts emerge in many areas of
machine learning and statistics ranging from causal inference to reinforcement
learning. Estimators for these tasks, generally called _methods of moments_ ,
include the prominent _generalized method of moments_ (GMM) which has recently
gained attention in causal inference. GMM is a special case of the broader
family of _empirical likelihood estimators_ which are based on approximating a
population distribution by means of minimizing a $\varphi$-divergence to an
empirical distribution. However, the use of $\varphi$-divergences effectively
limits the candidate distributions to reweightings of the data samples. We
lift this long-standing limitation and provide a method of moments that goes
beyond data reweighting. This is achieved by defining an empirical likelihood
estimator based on maximum mean discrepancy which we term the _kernel method
of moments_ (KMM). We provide a variant of our estimator for conditional
moment restrictions and show that it is asymptotically first-order optimal for
such problems. Finally, we show that our method achieves competitive
performance on several conditional moment restriction tasks.

##  1 Introduction

Many problems in machine learning, statistics, causal inference and economics
can be formulated as (conditional) moment restrictions (Newey and Powell,
2003; Angrist and Pischke, 2008). Moment restrictions (MR) identify a
parameter of interest by restricting the expectation over a so-called moment
function to a fixed value. From a machine learning perspective, moment
restrictions subsume empirical risk minimization since the corresponding first
order conditions imply that the expectation of the gradient of the loss
function vanishes. A significantly harder problem is posed by conditional
moment restrictions (CMR), which restrict the _conditional_ expectation of the
moment function. In this case estimation effectively requires solving a
continuum of unconditional moment restrictions (Bierens, 1982). A prominent
CMR problem is instrumental variable (IV) regression (Newey and Powell, 2003),
where the expectation of the prediction residual conditioned on the
instruments is required to be zero. The CMR formulation of IV regression is a
powerful way to define estimators that avoid two-step procedures as, e.g., in
the common two-stage least squares method (Angrist and Pischke, 2008). Other
examples of CMR problems include variants of double machine learning
(Chernozhukov et al., 2016, 2017, 2018) and off-policy evaluation in
reinforcement learning (Xu et al., 2021; Chen et al., 2021; Bennett and
Kallus, 2020a; Bennett et al., 2021). Perhaps the most popular approach to
learning with moment restrictions is the generalized method of moments (GMM)
of Hansen (1982), which recently gained popularity in machine learning (Lewis
and Syrgkanis, 2018; Bennett and Kallus, 2020b). GMM belongs to the wider
family of generalized empirical likelihood (GEL) estimators of Owen (1988,
1990); Qin and Lawless (1994); Smith (1997). While moment restrictions are
imposed with respect to a population distribution, in practice one usually
only has access to an empirical sample from this distribution. GEL estimators
are based on simultaneously finding the model parameters and an approximation
of the population distribution by considering distributions with minimal
distance to the empirical distribution for which the moment restrictions can
be fulfilled exactly. The various GEL estimators differ in the choice of
$\varphi$-divergence used to define this distance. In this context, the
continuous updating version of GMM (Hansen et al., 1996) can be interpreted as
a GEL estimator with $\chi^{2}$-divergence. However, the use of
$\varphi$-divergences effectively restricts the set of candidate distributions
to multinomial distributions on the empirical sample, i.e., reweightings of
the data, which can be a crude approximation especially in the low sample
regime. In the present work, we define the first method of moments estimator
that parts with this limitation by defining a GEL framework based on a
fundamentally different notion of distributional distance, namely the maximum
mean discrepancy (MMD). This allows us to consider arbitrary candidate
distributions with support different from the empirical distribution. As in
many cases the population distribution is continuous, this bears the potential
to find better approximations thereof. In principle, our flexible framework
even allows to evolve the class of candidate distributions over the course of
the optimization and thus might benefit from developments in gradient flows
and optimal transport. The practical benefit of our approach is demonstrated
by competitive empirical performance.

##### Our Contributions

  1. 1.

We propose the first method of moments estimator without the limitation to
data reweightings by extending the GEL framework to MMD. We derive the dual
problem of the resulting inner optimization problem which is a semi-infinitely
constrained convex program.

  2. 2.

To overcome computational challenges, we introduce entropy regularization and
show that the dual of the inner problem gives rise to an unconstrained convex
program, turning a semi-infinite formulation into either a soft-constraint or
log-barrier setting.

  3. 3.

We provide the first order asymptotics and demonstrate that our estimator is
asymptotically optimal for CMR estimation in the sense that it achieves the
semi-parametric efficiency bound of Chamberlain (1987).

  4. 4.

We provide details on the practical implementation and empirically demonstrate
state-of-the-art performance of our method on several CMR problems.

  5. 5.

We release an implementation of our method as part of a software package for
(conditional) moment restriction estimation.

The remainder of the paper is structured as follows. Section 2 gives an
overview of method of moments estimation for conditional and unconditional
moment restrictions. Section 3 introduces our estimator, and provides duality
results as well as asymptotic properties and practical considerations. Section
4 provides an empirical evaluation of our estimators on various conditional
moment restriction tasks. Section 5 discusses connections to related methods
and Section 6 concludes.

##  2 Background

##### Method of Moments

Let $X$ be a random variable taking values in
$\mathcal{X}\subseteq\mathbb{R}^{r}$ distributed according to $P_{0}$. In the
following we will denote the expectation with respect to a distribution $P$ by
$E_{P}[\cdot]$ and drop the subscript whenever we refer to the population
distribution $P_{0}$. Moment restrictions identify a parameter of interest
$\theta_{0}\in\Theta\subseteq\mathbb{R}^{p}$ by restricting the expectation of
a so-called moment function
$\psi:\mathcal{X}\times\Theta\rightarrow\mathbb{R}^{m}$, such that

$\displaystyle E[\psi(X;\theta_{0})]=0\in\mathbb{R}^{m}.$

In practice, the true distribution $P_{0}$ is generally unknown and one only
has access to a sample $\\{x_{i}\\}_{i=1}^{n}$ with empirical distribution
$\hat{P}_{n}=\sum_{i=1}^{n}\frac{1}{n}\delta_{x_{i}}$, where $\delta_{x_{i}}$
denotes a Dirac measure centered at $x_{i}$. The corresponding _empirical_
moment restrictions can be defined as

$\displaystyle E_{\hat{P}_{n}}[\psi(X;\theta)]=0,\quad\theta\in\Theta.$

If the number of restrictions $m$ does not exceed the number of parameters
$p$, these can often be solved exactly. For example, suppose we are interested
in estimating the mean $\theta$ of a distribution. Then, solving the empirical
moment restrictions for the moment function $\psi(X;\theta)=X-\theta$ yields
the maximum likelihood estimate $\theta=\frac{1}{n}\sum_{i=1}^{n}x_{i}$.
However, in the so-called _overidentified_ case with $m>p$, the system of
equations is generally over-determined and the empirical moment restrictions
cannot be satisfied exactly. This is the domain of the celebrated generalized
method of moments (GMM) of Hansen (1982). Instead of trying to satisfy the
moment restrictions exactly, GMM relaxes the problem into a minimization of a
quadratic form,

$\displaystyle\theta^{\text{GMM}}\\!=\\!\operatorname*{arg\,min}_{\theta\in\Theta}E_{\hat{P}_{n}}[\psi(X;\theta)]^{T}\\!\left(\widehat{\Omega}(\tilde{\theta})\right)^{-1}\\!E_{\hat{P}_{n}}[\psi(X;\theta)],$

where
$\widehat{\Omega}(\tilde{\theta})=E_{\hat{P}_{n}}[\psi(X;\tilde{\theta})\psi(X;\tilde{\theta})^{T}]\in\mathbb{R}^{m\times
m}$ denotes the empirical covariance matrix evaluated at a first stage
estimate $\tilde{\theta}$.

##### Empirical Likelihood Estimation

GMM is a special case of the wider family of generalized empirical likelihood
estimators (Owen, 1988, 1990; Qin and Lawless, 1994). In an attempt to improve
the finite sample properties of GMM, alternative estimators from this family
have been proposed Smith (1997); Newey and Smith (2004). GEL estimation is
based on the idea that while it might not be possible to satisfy the moment
restrictions with respect to the empirical distribution $\hat{P}_{n}$, the
population distribution $P_{0}$, for which the moment restrictions hold at the
true parameter $\theta_{0}$, will be in a shrinking neighbourhood of
$\hat{P}_{n}$ as the number of samples $n$ grows. Therefore GEL seeks to find
a parameter $\theta$ and a distribution $P$ for which the moment restrictions
hold exactly while staying as close as possible to the empirical distribution.
For a convex function $\varphi:[0,\infty)\rightarrow(-\infty,\infty]$ define
the $\varphi$-divergence between distributions $P$ and $Q$ as
$D_{\varphi}(P||Q)=\int\\!\varphi\\!\left(\frac{\mathrm{d}P}{\mathrm{d}Q}\right)\mathrm{d}Q$,
where $\frac{\mathrm{d}P}{\mathrm{d}Q}$ denotes the Radon-Nikodym derivative
of $P$ with respect to $Q$. Define the _profile divergence_ with respect to a
$\varphi$-divergence as

$\displaystyle R(\theta)=$
$\displaystyle\inf_{P\ll\hat{P}_{n}}D_{\varphi}(P||\hat{P}_{n})\quad\mathrm{s.t.}\quad
E_{P}[\psi(X;\theta)]=0,\quad E_{P}[1]=1,$ (1)

where $P\ll\hat{P}_{n}$ is the set of positive measures $P$ that are
absolutely continuous w.r.t. the empirical distribution $\hat{P}_{n}$. The GEL
estimator then results from minimizing the profile divergence over
$\theta\in\Theta$,

$\displaystyle\theta^{\mathrm{GEL}}=\operatorname*{arg\,min}_{\theta\in\Theta}R(\theta).$

Due to the absolute continuity assumption, the distributions considered by GEL
are reweightings of the empirical data. Being a special case of GEL, GMM
therefore also implicitly corresponds to reweightings of the data as
formalized by the following proposition which follows directly from the
equivalence result of Newey and Smith (2004) (Theorem 2.1).

######  Proposition 2.1.

The first order optimality conditions for the continuous updating GMM
estimator and the GEL estimator with $\chi^{2}$-divergence coincide. As the
optimal distribution of the latter is given by
$P^{\ast}=\sum_{i=1}^{n}p_{i}\delta_{x_{i}}$ for some $p\in\mathbb{R}^{n}$
with $\sum_{i=1}^{n}p_{i}=1$, in consequence, GMM implicitly corresponds to a
reweighting of the data.

##### Conditional Moment Restrictions

In practice, many interesting problems can be formulated as _conditional_
moment restrictions, where the estimating equations are given by a conditional
expectation over the moment function. Let $Z$ be an additional random variable
taking values in $\mathcal{Z}$, then conditional moment restrictions take the
form

$\displaystyle E[\psi(X;\theta_{0})|Z]=0,\ P_{Z}\text{-a.s.},$ (2)

where the restrictions need to hold almost surely (a.s.) with respect to the
marginal distribution $P_{Z}$ of $Z$. As conditional moment restrictions are
difficult to handle in practice, many proposed estimators rely on transforming
them into a corresponding continuum of unconditional restrictions of the form

$\displaystyle E[\psi(X;\theta)^{T}h(Z)]=0\ \ \forall h\in\mathcal{H},$ (3)

where the expectation is taken over the joint distribution of $X$ and $Z$ and
$\mathcal{H}$ is a sufficiently rich function space. Examples of such spaces
are the Hilbert space of square integrable functions or the reproducing kernel
Hilbert space of a universal kernel (Micchelli et al., 2006). Both the GMM and
the GEL framework have been extended to conditional moment restrictions in
multiple ways, building on basis function expansions of $\mathcal{H}$
(Carrasco and Florens, 2000; Tripathi and Kitamura, 2003; Ai and Chen, 2003;
Chaussé, 2012; Carrasco and Kotchoni, 2017) as well as modern machine learning
models (Hartford et al., 2017; Lewis and Syrgkanis, 2018; Bennett et al.,
2019; Kremer et al., 2022).

##### Reproducing Kernel Hilbert Spaces

A reproducing kernel Hilbert space (RKHS) $\mathcal{F}$ is a Hilbert space of
functions $f:\mathcal{X}\rightarrow\mathbb{R}$ in which all point evaluation
functionals are bounded. Let $\langle\cdot,\cdot\rangle_{\mathcal{F}}$ denote
the inner product on $\mathcal{F}$ and define the RKHS norm as the induced
norm $\|f\|_{\mathcal{F}}=\sqrt{\langle f,f\rangle_{\mathcal{F}}}$. With every
RKHS one can associate a unique kernel
$k:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$ with the reproducing
property $\langle k(x,\cdot),f\rangle_{\mathcal{F}}=f(x)$ for any
$f\in\mathcal{F}$ and $x\in\mathcal{X}$. Let $\mathcal{P}$ denote a space of
probability distributions, then we define the kernel mean embedding of
$P\in\mathcal{P}$ as $\mu_{P}=E_{P}[k(X,\cdot)]\in\mathcal{F}$, which has the
property that $\langle\mu_{P},f\rangle_{\mathcal{F}}=E_{P}[f(X)]\ \forall
f\in\mathcal{F}$. This can be used to define a metric on a space of
probability distributions $\mathcal{P}$. For $P,Q\in\mathcal{P}$ the maximum
mean discrepancy (MMD) (Gretton et al., 2012) is defined as
$\operatorname{MMD}(P,Q;\mathcal{F}):=\|\mu_{P}-\mu_{Q}\|_{\mathcal{F}}$.
Refer to, e.g., Schölkopf and Smola (2002); Berlinet and Thomas-Agnan (2011);
Steinwart and Christmann (2008) for comprehensive introductions to kernel
methods for machine learning.

##  3 Kernel Method of Moments

In this section we derive the KMM estimator for unconditional and conditional
moment restrictions and explore its properties. We first derive an exact MMD-
based GEL estimator that leads to a difficult semi-infinitely constrained
optimization problem for the _MMD profile_ $R(\theta)$. We show that an
entropy regularized version of our estimator leads to an unconstrained convex
dual program which can be readily solved with, e.g., first order optimization
methods. We show that our estimator is consistent and optimal for
(conditional) moment restriction problems in the sense that it achieves the
lowest possible asymptotic variance among any class of estimators based solely
on the CMR. Finally we provide details on the computational procedure.

###  3.1 Our Method

Our goal is to define a profile function $R(\theta)$ based on maximum mean
discrepancy instead of $\varphi$-divergences, such that the KMM estimator can
be obtained as
$\hat{\theta}=\operatorname*{arg\,min}_{\theta\in\Theta}R(\theta)$. Let
$\psi:\mathcal{X}\times\Theta\rightarrow\mathbb{R}^{m}$ denote a moment
function and let $\mathcal{P}$ denote the space of positive measures over
$\mathcal{X}$. Further let $\mathcal{F}$ be an RKHS corresponding to a
universal kernel. Then we can define the _MMD-profile_ as

$\displaystyle R(\theta)=$
$\displaystyle\inf_{P\in\mathcal{P}}\frac{1}{2}\operatorname*{MMD}(P,\hat{P}_{n};\mathcal{F})^{2}\quad\mathrm{s.t.}\quad
E_{P}[\psi(X;\theta)]=0,\quad E_{P}[1]=1,$ (4)

where we set $R(\theta)=\infty$ whenever the optimization problem is
infeasible. The restriction to RKHS $\mathcal{F}$ corresponding to universal
kernels ensures that $\operatorname{MMD}(P,Q;\mathcal{F})=0$ if and only if
$P=Q$ and thus ensures uniqueness of the infimum in (4) as
$n\rightarrow\infty$. Using Lagrange duality we can derive the corresponding
dual problem as formalized by the following theorem.

######  Theorem 3.1.

The _MMD profile_ (4) has the strongly dual form

$\displaystyle R(\theta)=$
$\displaystyle\sup_{\begin{subarray}{c}\eta\in\mathbb{R},f\in\mathcal{F},\\\
h\in\mathbb{R}^{m}\end{subarray}}\frac{1}{n}\sum_{i=1}^{n}f(x_{i})+\eta-\frac{1}{2}\|f\|_{\mathcal{F}}^{2}$
(5) $\displaystyle\mathrm{s.t.}\quad f(x)+\eta\leq\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}\quad\forall x\in\mathcal{X}.$

Note that the structure and derivation of (5) resembles recent reformulation
techniques for MMD-based distributionally robust optimization (DRO) by Zhu et
al. (2021) and in fact GEL estimation can be seen as a dual problem to DRO
(Lam, 2019).

While MMD enjoys many favorable properties, the MMD-profile (5), involves a
(semi-)infinite constraint which is difficult to handle in practice,
especially when combined with stochastic-gradient-type algorithms in machine
learning. In the following section, we will show that these limitations can be
lifted by introducing entropy regularization.

###  3.2 Entropy Regularization

Inspired by the interior point method for convex optimization in finite-
dimensions Nesterov and Nemirovskii (1994), we define an entropy-regularized
version of the MMD profile (5). This allows us to translate the semi-infinite
constraint in (5) into an additional term in the objective of an unconstrained
optimization problem. Note that entropy regularization has been used before in
the context of the computation of optimal transport distances (Cuturi, 2013).
Our use here is different as we do not regularize a distance computation but
instead regularize the duality structure to handle the semi-infinite
constraint. To the best of our knowledge entropy regularization has not been
combined with MMD in this context.

For a convex function $\varphi:[0,\infty)\rightarrow(-\infty,\infty]$ define
the relative entropy (or $\varphi$-divergence) between a reference
distribution $\omega$ and a distribution $P$, which admits a density $p$
w.r.t. $\omega$, as

$\displaystyle
D_{\varphi}(P||\omega)=\int_{\mathcal{X}}{\varphi(p(x))}\omega(\mathrm{d}x).$
(6)

Then for any $\epsilon>0$ we define the _entropy-regularized MMD profile_ for
a moment restriction of the form $E[\psi(X;\theta_{0})]=0$ as

$\displaystyle
R^{\varphi}_{\epsilon}(\theta)=\inf_{P\ll\omega}\frac{1}{2}\operatorname*{MMD}(P,\hat{P}_{n};\mathcal{F})^{2}+\epsilon
D_{\varphi}(P||\omega)\quad\mathrm{s.t.}\quad E_{P}[\psi(X;\theta)]=0,\quad
E_{P}[1]=1.$ (7)

In contrast to the classical $\varphi$-divergence-based profile divergence
(1), the regularized MMD profile does not require $P\ll\hat{P}_{n}$. Instead,
absolute continuity is only imposed with respect to an arbitrary (potentially
continuous) reference distribution $\omega$ which can be constructed in a
data-driven way (cf. Section D.3). In practice, by sampling from $\omega$,
this allows us to approximate the population distribution $P_{0}$ in an
arbitrarily fine-grained way instead of using mere reweightings of the
training data as in GEL/GMM, which can be a rough approximation especially in
the low data regime.

Using Lagrangian duality we can derive the dual problem of (7) as formalized
in the following theorem.

######  Theorem 3.2 (KMM Duality).

The entropy-regularized MMD profile (7) has the strongly dual form

$\displaystyle
R^{\varphi}_{\epsilon}(\theta)=\sup_{\begin{subarray}{c}\eta\in\mathbb{R},f\in\mathcal{F},\\\
h\in\mathbb{R}^{m}\end{subarray}}\frac{1}{n}\sum_{i=1}^{n}f(x_{i})+\eta-\frac{1}{2}\|f\|_{\mathcal{F}}^{2}-\epsilon\int_{\mathcal{X}}{\varphi^{\ast}\left(\frac{f(x)+\eta-\langle
h,\psi(x;\theta)\rangle}{\epsilon}\right)}\omega(\mathrm{d}x).$ (8)

where $\varphi^{\ast}(t):=\sup_{s}\langle t,s\rangle-\varphi(s)$ denotes the
convex conjugate of $\varphi$. The optimization problem in (8) is jointly
convex in the dual variables $(\eta,f,h)$.

As opposed to the unregularized version (5) the entropy-regularized MMD
profile (7) is a jointly convex, unconstrained optimization problem over the
dual variables. Finally the KMM estimator can be obtained as the minimizer of
the entropy-regularized MMD profile

$\displaystyle\hat{\theta}=\operatorname*{arg\,min}_{\theta\in\Theta}R^{\varphi}_{\epsilon}(\theta).$

###  3.3 Choices of Entropy Regularizers

Figure 1: Effect of Entropy-Regularization. The red and orange lines
correspond to an exemplary function $\psi(x;\theta)^{T}h$ and its relaxation
$\psi(x;\theta)^{T}h+\epsilon$. The blue line shows the strictly minorizing
RKHS function resulting from enforcing the constraint in (5) exactly. The cyan
and purple lines correspond to the $\varphi$-divergence regularized problem.
The log-divergence works as a barrier-function which allows to violate the
constraint in (5) by up to $\epsilon$. The KL-divergence yields a soft
constraint by penalizing violations exponentially.

Different choices of $\varphi$-divergences in (6) correspond to different
relaxations of the generally intractable semi-infinite constraint in (5).
Choosing the $\varphi$-divergence as the Kullback-Leibler divergence, i.e.,
$\varphi(p)=p\log(p)-p+1$ we obtain

$\displaystyle
R^{\mathrm{KL}}_{\epsilon}(\theta)=\sup_{\begin{subarray}{c}\eta\in\mathbb{R},f\in\mathcal{F},\\\
h\in\mathbb{R}^{m}\end{subarray}}\frac{1}{n}\sum_{i=1}^{n}f(x_{i})+\eta-\frac{1}{2}\|f\|_{\mathcal{F}}^{2}-\epsilon\int_{\mathcal{X}}{\exp\left(\frac{f(x)+\eta-\langle
h,\psi(x;\theta)\rangle}{\epsilon}\right)}\omega(\mathrm{d}x).$

This corresponds to relaxing the constraint in (5) into a soft version, such
that violations can occur but are exponentially penalized. Another
particularly interesting example is obtained by choosing the
$\varphi$-divergence to be the backward KL-divergence or Burg’s entropy
$\varphi(p)=-\log p+p-1$, which leads to a dual regularized MMD profile with a
log-barrier

$\displaystyle
R^{\mathrm{log}}_{\epsilon}(\theta)=\sup_{\begin{subarray}{c}\eta\in\mathbb{R},f\in\mathcal{F},\\\
h\in\mathbb{R}^{m}\end{subarray}}\frac{1}{n}\sum_{i=1}^{n}f(x_{i})+\eta-\frac{1}{2}\|f\|_{\mathcal{F}}^{2}+\epsilon\int_{\mathcal{X}}{\log\left(1-\frac{f(x)+\eta-\langle
h,\psi(x;\theta)\rangle}{\epsilon}\right)}\omega(\mathrm{d}x).$

Numerically, the log-barrier enforces the solution to lie in the interior of
the constraint set, i.e.,

$\displaystyle{f(x_{i})+\eta-\langle h,\psi(x_{i};\theta)\rangle}<{\epsilon}.$

Therefore, this can be seen as an interior-point method for handling the
infinite constraint in (5). Figure 1 provides a visualization of the different
regularization schemes.

###  3.4 KMM for Conditional Moment Restrictions

The KMM estimator can be generalized to conditional moment restrictions via a
functional formulation following the approach of Kremer et al. (2022).

Suppose we have a sufficiently rich Hilbert space $\mathcal{H}$ of functions
$h:\mathcal{Z}\rightarrow\mathbb{R}^{m}$, such that we can write conditional
moment restrictions of the form (2) as a continuum of unconditional
restrictions (3). Let $\mathcal{H}^{\ast}$ denote the dual space of continuous
functionals $\Psi:\mathcal{H}\rightarrow\mathbb{R}$, equipped with the dual
norm $\|\Psi\|_{\mathcal{H}^{\ast}}=\sup_{\|h\|_{\mathcal{H}}=1}\Psi(h)$. Then
for any $(x,z,\theta)\in\mathcal{X}\times\mathcal{Y}\times\Theta$ we define
the _moment functional_ $\Psi(x,z;\theta)\in\mathcal{H}^{\ast}$ such that

$\displaystyle\Psi(x,z;\theta)(h)=\psi(x;\theta)^{T}h(z).$

The continuum of moment restrictions (3) can thus be written as

$\displaystyle E[\Psi(X,Z;\theta)]=0\in\mathcal{H}^{\ast},$ (9)

where $0\in\mathcal{H}^{\ast}$ describes the functional that maps each element
in $\mathcal{H}$ to zero, which is equivalent to requiring
$\|E[\Psi(X,Z;\theta)]\|_{\mathcal{H}^{\ast}}=0$. By the Riesz representation
theorem (Zeidler, 2012), we can identify each element
$\Psi\in\mathcal{H}^{\ast}$ with an element $\phi(\Psi)\in\mathcal{H}$ such
that $\Psi(h)=\langle\phi(\Psi),h\rangle_{\mathcal{H}}$ $\forall
h\in\mathcal{H}$. With a slight abuse of notation, we will use $\phi(\Psi)$
and $\Psi$ interchangeably due to the self-duality of Hilbert spaces (Zeidler,
2012) so that we can denote
$\Psi(h)=\langle\phi(\Psi),h\rangle_{\mathcal{H}}=\langle\Psi,h\rangle_{\mathcal{H}}$.
Generalizing the KMM estimator to conditional moment restrictions then just
becomes a matter of substituting

$\displaystyle h\in\mathbb{R}^{m}$ $\displaystyle\rightarrow h\in\mathcal{H}$
$\displaystyle\langle\psi(x;\theta),h\rangle_{\mathbb{R}^{m}}$
$\displaystyle\rightarrow\langle\Psi(x,z;\theta),h\rangle_{\mathcal{H}}$

and adding a regularization term $-\frac{1}{2}\|h\|_{\mathcal{H}}^{2}$ for the
Lagrange parameter $h\in\mathcal{H}$, which regularizes the first order
conditions for $h$ as argued by Kremer et al. (2022). With this at hand, we
can define the functional version of the entropy-regularized MMD profile for
conditional moment restrictions (refer to Section A.1 for details on the
duality relationship).

######  Definition 3.3 (Functional KMM).

Let $\mathcal{H}$ be a sufficiently rich Hilbert space of functions such that
equivalence between (2) and (3) holds. Then the entropy-regularized MMD
profile for conditional moment restrictions is given as

$\displaystyle R^{\varphi}_{\epsilon,\lambda_{n}}\\!(\theta)=$
$\displaystyle\\!\\!\sup_{\begin{subarray}{c}\eta\in\mathbb{R},f\in\mathcal{F},\\\
h\in\mathcal{H}\end{subarray}}\frac{1}{n}\sum_{i=1}^{n}f(x_{i},z_{i})+\eta-\frac{1}{2}\|f\|_{\mathcal{F}}^{2}-\frac{\lambda_{n}}{2}\|h\|_{\mathcal{H}}^{2}$
(10)
$\displaystyle-\epsilon\int_{\mathcal{X}\times\mathcal{Z}}{\varphi^{\ast}\\!\left(\frac{f(x,z)+\eta-\langle\Psi(x,z;\theta),h\rangle_{\mathcal{H}}}{\epsilon}\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z).$

From the proof of Theorem 3.2 it directly follows that the optimization
problem in (10) is jointly convex in the dual variables. The space
$\mathcal{H}$ can be chosen, e.g., as the RKHS of a universal integrally
strictly positive definite (ISPD) kernel to guarantee the consistency of the
solution for the conditional moment restrictions (Kremer et al., 2022). In
practice, alternative choices, e.g., neural networks which lack these
theoretical guarantees have proven successful and often preferable (Bennett
and Kallus, 2020b; Kremer et al., 2022).

###  3.5 Asymptotic Properties

Consider the regularized KMM estimator with any $\varphi$-divergence such that
$\frac{d}{dt}\varphi^{\ast}(t)|_{t=0}=:\varphi_{1}^{\ast}(0)=1$ and
$\frac{d^{2}}{(dt)^{2}}\varphi^{\ast}(t)|_{t=0}=:\varphi_{2}^{\ast}(0)=1$,
which is fulfilled for, e.g., the forward and backward KL divergence.

For space reasons, here we focus on the estimator for _conditional_ moment
restrictions by combining the theory for the functional KMM estimator (see
Section A.2) with a sufficiently rich space of locally Lipschitz functions
$\mathcal{H}$. The properties of the unconditional/finite-dimensional KMM
estimator are provided in Section B.

######  Theorem 3.4 (Consistency).

Let $\mathcal{H}\subseteq L^{2}(P_{0},\mathcal{Z})$ be a Hilbert space of
locally Lipschitz functions which is sufficiently rich such that (2) is
implied by (3). Further assume that a) $\theta_{0}\in\Theta$ is the unique
solution to $E[\psi(X;\theta)|Z]=0\ P_{Z}\text{-a.s.}$; b)
$\Theta\subset\mathbb{R}^{p}$ is compact; c) $\psi(X;\theta)$ is continuous at
each $\theta\in\Theta$ w.p.1; d)
$E[\sup_{\theta\in\Theta}\|\psi(X;\theta)\|^{2}_{2}|Z]<\infty$ w.p.1; e)
$V_{0}(Z)=E[\psi(X;\theta_{0})\psi(X;\theta_{0})|Z]$ is non-singular w.p.1; f)
$\omega=(1-\alpha)\hat{P}_{n}+\alpha Q$ for $\alpha=O_{p}(n^{-1})$ and any
distribution $Q$ such that
$E_{Q}[\sup_{\theta\in\Theta}\|\psi(X;\theta)\|_{2}^{2}|Z]<\infty$ w.p.1; and
g) $\lambda_{n}=O_{p}(n^{-\xi})$ with $0<\xi<1/2$. Then for the KMM estimator
$\hat{\theta}$ we have $\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$.

If additionally h) $\theta_{0}\in\operatorname{int}(\Theta)$; i)
$\psi(x;\theta)$ is continuously differentiable in a neighborhood
$\bar{\Theta}$ of $\theta_{0}$ and
$E[\sup_{\theta\in\bar{\Theta}}\|\nabla_{\theta}\psi(X;\theta)\|^{2}|Z]<\infty$
w.p.1; as well as j)
$\operatorname{rank}\left(E[\nabla_{\theta}\psi(X;\theta_{0})|Z]\right)=p$
w.p.1, we have $\|\hat{\theta}-\theta_{0}\|=O_{p}(n^{-1/2})$.

######  Remark 3.5.

Assumption f) implies that asymptotically the reference distribution $\omega$
is required to converge weakly to the population distribution $P_{0}$.
However, as $Q$ can be chosen as an arbitrary (continuous) distribution, as
long as $\operatorname{supp}(\hat{P}_{n})\subseteq\operatorname{supp}(Q)$, the
form of $\omega=(1-\alpha)\hat{P}_{n}+\alpha Q$ does not restrict the set of
candidate distributions $P$ in (10) further than to distributions that admit a
density w.r.t. $Q$.

######  Remark 3.6.

A sufficiently rich function space for Theorem 3.4 is for example given by the
RKHS of a universal kernel (see Theorem 3.9 of Kremer et al. (2022)).
Moreover, based on universal approximation theorems (Hornik et al., 1989),
$\mathcal{H}$ can be represented by neural networks of asymptotically growing
width/depth. In this case the local Lipschitz property can be ensured by
restricting the weights to a compact domain (e.g., via weight clipping).

######  Theorem 3.7 (Asymptotic Normality ).

Let Assumptions a)-j) of Theorem 3.4 be satisfied. Then,

$\displaystyle\sqrt{n}(\hat{\theta}-\theta_{0})\overset{d}{\rightarrow}N(0,\Xi_{0})$

where

$\Xi_{0}=E\left[E[\nabla_{\theta}\psi(X;\theta_{0})|Z]\ V_{0}^{-1}(Z)\
E[\nabla_{\theta}\psi(X;\theta_{0})|Z]\right]^{-1}.$

The asymptotic variance in Theorem 3.7 agrees with the semi-parametric
efficiency bound of Chamberlain (1987). This implies that the KMM estimator
achieves the lowest possible asymptotic variance among any estimators based on
the CMR (2) as formalized in the following corollary.

######  Corollary 3.8 (Efficiency).

Let the assumptions of Theorem 3.7 be satisfied. Then the KMM estimator is
efficient with respect to all estimators based on the conditional moment
restrictions $E[\psi(X;\theta_{0})|Z]=0\ P_{Z}\text{-a.s.}$.

This is a particularly strong result, setting our estimator apart from a
number of recently proposed modern mini-max approaches to CMR estimation
(Lewis and Syrgkanis, 2018; Bennett et al., 2019; Zhang et al., 2021) and
which is matched only by the kernel VMM estimator of Bennett and Kallus
(2020b) and the more traditional sieve-based approaches of Ai and Chen (2003)
and Chen and Pouzo (2009). Note that while this shows that our estimator is
asymptotically first-order equivalent to these methods, the finite sample
properties can be vastly different.

###  3.6 Computing KMM Estimators

In the following we restrict the discussion to the conditional version of the
KMM estimator. The version for unconditional MR follows directly by setting
$\lambda_{n}=0$, $\mathcal{H}=\mathbb{R}^{m}$ and
$\Psi(x,z;\theta)=\psi(x;\theta)$.

The functional entropy-regularized MMD profile (10) is a convex optimization
problem over function-valued dual parameters $f\in\mathcal{F}$ and
$h\in\mathcal{H}$ as well as $\eta\in\mathbb{R}$. The dual formulation (10)
allows us to base our method on a dual functional gradient ascent algorithm.
This is in contrast to particle gradient descent methods that address the
primal problem by relying on discretizing measures which is commonly used in
the gradient flow literature. Define the saddle point objective from (10) as

$\displaystyle\widehat{G}_{\epsilon,\lambda_{n}}(\theta,\eta,f,h)=$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}f(x_{i},z_{i})+\eta-\frac{1}{2}\|f\|_{\mathcal{F}}^{2}-\frac{\lambda_{n}}{2}\|h\|_{\mathcal{H}}^{2}$
$\displaystyle-\epsilon\\!\\!\int_{\mathcal{X}\times\mathcal{Z}}{\varphi^{\ast}\left(\frac{f(x,z)+\eta-\langle\Psi(x,z;\theta),h\rangle_{\mathcal{H}}}{\epsilon}\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z).$

Then the MMD-EL estimator is given as the solution to the problem

$\displaystyle\hat{\theta}=\operatorname*{arg\,min}_{\theta\in\Theta}\sup_{\beta\in\mathcal{M}}\widehat{G}_{\epsilon,\lambda_{n}}(\theta,\beta),$
(11)

where we defined
$\beta:=(\eta,f,h)\in\mathcal{M}=\mathbb{R}\times\mathcal{F}\times\mathcal{H}$.

Algorithm 1 Gradient Descent Ascent for KMM

Input: empirical distribution $\hat{P}_{n}$, reference distribution $\omega$,
hyperparameters $\epsilon$, $\lambda$, batchsizes $n_{1}$, $n_{2}$

while not converged do

Sample $\\{(x_{i},z_{i})\\}_{i=1}^{n_{1}}\sim\hat{P}_{n}$,
$\\{(x_{j}^{c},z_{j}^{c})\\}_{j=1}^{n_{2}}\sim\omega$

$G\leftarrow\frac{1}{n_{1}n_{2}}\sum_{i=1}^{n_{1}}\sum_{j=1}^{n_{2}}G_{\epsilon,\lambda}(\theta,\beta,(x_{i},z_{i}),(x^{c}_{j},z_{j}^{c}))$

$\beta\leftarrow\operatorname{AscentStep}(\beta,\nabla_{\beta}G)$

$\theta\leftarrow\operatorname{DescentStep}(\theta,\nabla_{\theta}G)$

end while

Output: Parameter estimate $\theta$

##### Stochastic Approximation

In order to evaluate the KMM objective we use a stochastic approximation of
the integral term in (11). Let the random variables $(X,Z)$ and
$(X^{c},Z^{c})$ be distributed according to $P_{0}$ and $\omega$ respectively
and define the random variable

$\displaystyle
G_{\epsilon,\lambda_{n}}\left(\theta,\beta,(X,Z),(X^{c},Z^{c})\right)\ =$
$\displaystyle\
f(X,Z)+\eta-\frac{1}{2}\|f\|_{\mathcal{F}}^{2}-\frac{\lambda_{n}}{2}\|h\|_{\mathcal{H}}^{2}$
$\displaystyle-\epsilon\varphi^{\ast}\left(\frac{f(X^{c},Z^{c})+\eta-\langle\Psi(X^{c},Z^{c};\theta),h\rangle_{\mathcal{H}}}{\epsilon}\right).$

Then we can express the KMM objective as an expectation with respect to the
empirical and reference distributions $\hat{P}_{n}$ and $\omega$ respectively,

$\displaystyle\widehat{G}_{\epsilon,\lambda_{n}}(\theta,\beta)=E_{\hat{P}_{n}}E_{\omega}[G_{\epsilon,\lambda_{n}}(\theta,\beta,(X,Z),(X^{c},Z^{c}))].$

This has the form required for mini-batch stochastic gradient descent (SGD)
optimization. We solve problem (11) by alternating between functional SGD
steps in the dual variables $\beta$ and SGD steps in $\theta$. Our approach is
detailed in Algorithm 1.

##### Random Feature Approximation

The supremum over $\beta$ in (11) involves the optimization over the function
space $\mathcal{F}$ which is generally intractable. To provide a scalable
estimator that can be optimized with variants of stochastic gradient descent
(SGD), we resort to a random Fourier feature approximation of the RKHS
function $f$ (Rahimi and Recht, 2007). For $\alpha\in\mathbb{R}^{d}$, let
$f(x,z)=\alpha^{T}\phi(x,z)$ define the random feature approximation of $f$
with random Fourier features $\phi(x,z)\in\mathbb{R}^{d}$, where $d$ is the
number of random features. The KMM objective then becomes

$\displaystyle\widehat{G}_{\epsilon,\lambda_{n}}(\theta,\eta,\alpha,h)=$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\alpha^{T}\phi(x_{i},z_{i})+\eta-\frac{1}{2}\|\alpha\|^{2}_{2}-\frac{\lambda_{n}}{2}\|h\|_{\mathcal{H}}^{2}$
$\displaystyle-\epsilon\int_{\mathcal{X}\times\mathcal{Z}}{\varphi^{\ast}\left(\frac{\alpha^{T}\phi(x,z)+\eta-\langle\Psi(x,z;\theta),h\rangle}{\epsilon}\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z).$

Combined with the stochastic approximation and a scalable instrument function
$h$ (e.g., neural network or RKHS function with RF approximation) this allows
our method to scale to large sample sizes.

##  4 Empirical Results

We benchmark our estimator on two different tasks against state-of-the-art
estimators for conditional moment restrictions, including maximum moment
restrictions (MMR) (Zhang et al., 2021), sieve minimum distance (SMD) (Ai and
Chen, 2003), DeepIV (Hartford et al., 2017), DeepGMM (Bennett et al., 2019)
and the neural network version of functional GEL (FGEL) (Kremer et al., 2022).
As an additional baseline we compare to ordinary least squares (OLS) which
ignores the conditioning variable and minimizes
$\frac{1}{n}\sum_{i=1}^{n}\|\psi(x_{i};\theta)\|_{2}^{2}$. For all methods
involving kernels we use a radial basis function (RBF) kernel, whose bandwidth
is set via the median heuristic (Garreau et al., 2018). The hyperparameters of
all methods are set by evaluating the Hilbert-Schmidt independence criterion
(HSIC) between the residues and the conditioning variable
$\operatorname{HSIC}(Y-g(T),Z)$ over a validation set of the size of the
training set. HSIC has been proposed as an objective for CMR problems by Mooij
et al. (2009) and Saengkyongam et al. (2022) and we empirically find it to
yield better estimates than MMR (Muandet et al., 2020; Zhang et al., 2021)
which has been used for as a validation metric in other works (see Section
C.2). For the variational approaches we use a batchsize of $n_{1}=200$.
Additionally, for our KMM esimator we represent the reference distribution
$\omega$ by a kernel density estimator (KDE) trained on the empirical sample
(see Section D.3) from which we sample mini-batches of size $n_{2}=200$. Refer
to Section C for additional details. An implementation of our estimator and
code to reproduce our results is available at
https://github.com/HeinerKremer/conditional-moment-restrictions.

##### Heteroskedastic Instrumental Variable Regression

We adopt the HeteroskedasticIV experiment of Bennett and Kallus (2020b). Let
the data-generating process be given by

$\displaystyle Z\sim\operatorname{Uniform}([-5,5]^{2}),\quad\quad
H,\eta,\varepsilon\sim\mathcal{N}(0,1)$ $\displaystyle
T_{\text{exo}}=Z_{1}+|Z_{2}|,\quad\quad T_{\text{endo}}=5H+0.2\eta$
$\displaystyle T=0.75T_{\text{exo}}+0.25T_{\text{endo}},\quad
S=0.1\log(1+\exp(T_{\text{exo}}))$ $\displaystyle
Y=g(T;\theta_{0})+5H+S\varepsilon,$

where the parameter of interest
$\theta_{0}=[2.0,3.0,-0.5,3.0]\in\mathbb{R}^{4}$ enters the process via the
function

$\displaystyle
g(t;\theta)=\theta_{2}+\theta_{3}(t-\theta_{1})+\frac{\theta_{4}-\theta_{3}}{2}\log(1+e^{2(t-\theta_{1})}).$

This task is particularly challenging as it involves heteroskedastic noise on
the instruments. The true parameter $\theta_{0}$ is identified by imposing the
CMR $E[Y-g(T;\theta)|Z]=0,\ P_{Z}\text{-a.s.}$. Table 1 shows the mean squared
error (MSE) of the predicted function for different methods and sample sizes.
Our method provides a significantly lower MSE for small sample sizes and
approaches the results of DeepGMM and FGEL for larger samples.

Table 1: Instrumental Variable Regression with Heteroskedastic Instrument Noise. Mean of the prediction MSE $E[\|g_{\theta}(T)-g_{0}(T)\|^{2}]$ and its standard error are computed over 20 random runs. | OLS | MMR | DeepIV | DeepGMM | FGEL | KMM
---|---|---|---|---|---|---
n=500 | $1.78\pm 0.21$ | $1.73\pm 0.22$ | $2.57\pm 0.06$ | $1.03\pm 0.17$ | $1.02\pm 0.19$ | $0.40\pm 0.13$
n=1000 | $2.27\pm 0.18$ | $1.97\pm 0.23$ | $2.53\pm 0.08$ | $0.70\pm 0.16$ | $0.45\pm 0.11$ | $0.16\pm 0.04$
n=2000 | $1.79\pm 0.10$ | $2.11\pm 0.20$ | $2.43\pm 0.06$ | $0.15\pm 0.04$ | $0.14\pm 0.03$ | $0.10\pm 0.02$
n=4000 | $1.92\pm 0.06$ | $1.65\pm 0.15$ | $2.41\pm 0.04$ | $0.07\pm 0.02$ | $0.05\pm 0.01$ | $0.07\pm 0.02$
n=10000 | $1.99\pm 0.04$ | N/A | $1.99\pm 0.04$ | $0.02\pm 0.01$ | $0.03\pm 0.01$ | $0.01\pm 0.00$

##### Neural Network Instrumental Variable Regression

To explore the viability of our estimator in the non-uniquely identified
setting, we adopt the non-parametric instrumental variable regression
experiment of Lewis and Syrgkanis (2018) which has also been used by Bennett
et al. (2019), Zhang et al. (2021) and Kremer et al. (2022). Consider a data
generating process given by

$y=g_{0}(t)+e+\delta,$ | $t=z+e+\gamma,$ | $z\sim\operatorname{Uniform}([-3,3]),$
---|---|---
$e\sim N(0,1),$ | $\gamma,\delta\sim N(0,0.1),$ |

where the function $g_{0}$ is chosen from

$\operatorname{sin:}g_{0}(t)=\sin(t)$, |  $\operatorname{abs:}g_{0}(t)=|t|\vspace{.5em}$,
---|---
$\operatorname{linear:}g_{0}(t)=t$, |  $\operatorname{step:}g_{0}(t)=I_{\\{t\geq 0\\}}$.

We try to learn an approximation of $g_{0}$ represented by a shallow neural
network $g_{\theta}(t)$ with 2 layers of $[20,3]$ units and leaky ReLU
activation functions. We identify $g_{\theta}(t)$ by imposing the conditional
moment restrictions $E[Y-g_{\theta}(T)|Z]=0,\ \ P_{Z}\text{-a.s.}$. We use
training and validation sets of size $n=1000$ and evaluate the prediction
error on a test set of size $20000$. Table 2 shows the MSE of the predicted
models trained with different CMR estimation methods. We observe that our
estimator consistently shows competitive performance and slightly outperforms
the baselines on three out of four tasks.

Table 2: Neural Network Instrumental Variable Regression. Mean of the prediction MSE $E[\|g_{\theta}(T)-g_{0}(T)\|^{2}]$ and its standard error are computed over 30 random runs and scaled by a factor of ten for ease of presentation. | OLS | SMD | MMR | DeepIV | DeepGMM | FGEL | KMM
---|---|---|---|---|---|---|---
abs | $3.21\pm 0.14$ | $1.15\pm 0.53$ | $1.41\pm 0.48$ | $2.25\pm 0.68$ | $0.42\pm 0.04$ | $0.37\pm 0.05$ | $0.32\pm 0.06$
step | $3.16\pm 0.05$ | $0.54\pm 0.06$ | $0.58\pm 0.03$ | $0.74\pm 0.04$ | $0.43\pm 0.04$ | $0.40\pm 0.04$ | $0.35\pm 0.02$
sin | $3.33\pm 0.06$ | $1.31\pm 0.08$ | $2.67\pm 0.13$ | $3.75\pm 0.15$ | $0.64\pm 0.05$ | $0.62\pm 0.04$ | $0.88\pm 0.10$
linear | $2.95\pm 0.08$ | $0.47\pm 0.11$ | $0.96\pm 0.20$ | $1.66\pm 0.50$ | $0.49\pm 0.05$ | $0.95\pm 0.27$ | $0.43\pm 0.11$

##  5 Related Work

Conditional moment restrictions have been addressed in multiple ways by
extending the GMM to continua of moment restrictions building on the
equivalence between the conditional (2) and continuum (3) formulations.
Seminal work in this direction has been carried out by (Carrasco and Florens,
2000; Carrasco et al., 2007) and Ai and Chen (2003), which approximate the
continuum of MR by a basis function expansion. Recently, the problem gained
popularity in the machine learning community as many problems in causal
inference can be formulated as CMR, most prominently instrumental variable
regression. These modern approaches represent the continuum of MR via machine
learning models, i.e., RKHS functions or neural networks and solve a mini-max
formulation (Hartford et al., 2017; Lewis and Syrgkanis, 2018; Bennett et al.,
2019; Bennett and Kallus, 2020b; Dikkala et al., 2020). Other GEL methods have
historically played a less prominent role for CMR estimation, most likely due
to their more complex mini-max structure compared to the simple minimization
of traditional GMM-based methods. However, generalizations of GEL to continua
of MR have been developed by Tripathi and Kitamura (2003); Kitamura et al.
(2004); Chaussé (2012); Carrasco and Kotchoni (2017) building on basis
function expansions, which empirically have been competitive with their GMM-
counterparts. Recently the problem has been addressed via modern machine
learning models (Kremer et al., 2022). All the aforementioned methods have in
common that they either explicitly (GEL) or implicitly (GMM) optimize a
$\varphi$-divergence between the candidate distributions and the empirical
distribution and thus only allow for reweightings of the data. To the best of
our knowledge, we provide the first method of moments estimator that lifts
this restriction and allows for arbitrary candidate distributions.

The GEL framework bears a close duality relation to distributionally robust
optimization (DRO) (Lam, 2019). In this context, it has been used to
investigate the statistical properties of DRO (Duchi et al., 2018; Lam, 2019)
and to calibrate the size of the distributional ambiguity set used in the DRO
framework (Lam and Qian, 2017; Lam and Zhou, 2017; Blanchet et al., 2019; He
and Lam, 2021). With the notable exception of Blanchet et al. (2019) these
works build on the standard $\varphi$-divergence-based GEL framework. While
Blanchet et al. (2019) provide a GEL framework based on optimal transport
distances, their goal is to calibrate an ambiguity set for DRO and they do not
provide an estimator for moment restriction problems.

Computationally, an important contribution of this paper is handling the
(semi)-infinite constraint in (5). Classical approaches to handling such
constraints using polynomial sum-of-squares (SOS) Lasserre (2001) do not apply
here since we have a general moment function class outside the polynomials.
Furthermore, both classical and infinite-dimensional SOS techniques Marteau-
Ferey et al. (2020) suffer from scalability issues in high dimensions and
large data sizes. Compared to those, our entropy-regularization approach can
be implemented with general nonlinear problems and stochastic-gradient-type
algorithms.

The objective of our inner optimization is a variational problem in the
measure of the form
$\min_{P}\left\\{\mathrm{F}(P)+\epsilon\mathrm{H}(P,Q)\right\\}$, where $F$ is
some energy functional and $H$ is some metric or divergence measure. This was
notably studied in the seminal work of Jordan et al. (1998) as a time-
discretization scheme of PDEs. In recent literature related to machine
learning, Arbel et al. (2019) studied the variational structure of MMD as
energy in the Wasserstein geometry. Chizat (2022) applied noisy particle
gradient descent to an energy objective similar to ours, i.e.,
$\operatorname*{MMD}+\epsilon D_{\mathrm{KL}}$. Compared with that work, our
goal is to train a model $\theta$ by minimizing this objective over $\theta$.
We also do not rely on gradient descent on the particles obtained from the
discretization of the measure but adopt a dual functional gradient ascent
scheme. Our reformulation technique for the MMD-profile is similar to that of
Zhu et al. (2021), who solved a similar variational problem involving MMD for
DRO. Different from their method, our goal is to provide an estimator for CMR
problems and we introduce entropy regularization as an interior point method
for handling the constraint.

##  6 Conclusion

The emergence of conditional moment restrictions in areas such as causal
inference and robust machine learning has created the need for effective and
robust estimation methods. Existing method of moments estimators (implicitly)
rely on approximating the population distribution by reweighting a discrete
empirical distribution. Our KMM estimator parts with this restrictive
assumption and allows considering arbitrary (continuous) distributions as
candidates for the population distribution. As in many cases the population
distribution is in fact continuous, this has the potential to find more
accurate estimates especially in the low sample regime where reweightings can
provide crude approximations. Our estimator comes with strong theoretical
guarantees showing that it is first order efficient with respect to any
estimator based on CMR and its competitive practical performance is
demonstrated on several CMR tasks.

This paper laid the foundation of the KMM framework, which can inspire future
work in multiple ways. Such work could include the development of more
sophisticated and adaptive reference measures for the regularization scheme,
e.g., by evolving the reference measure over the course of the optimization.
Another important direction would be a statistical learning theory analysis to
provide theoretical properties of our estimator in the non-uniquely identified
case. Other possibilities are extensions of the framework beyond estimation to
construct confidence intervals for the estimates. Lastly, more efficient and
tailored optimization methods can be developed to facilitate the application
at larger scales.

## References

  * Ai and Chen (2003) C. Ai and X. Chen.  Efficient estimation of models with conditional moment restrictions containing unknown functions.  _Econometrica_ , 71(6):1795–1843, 2003.
  * Angrist and Pischke (2008) J. D. Angrist and J.-S. Pischke.  _Mostly harmless econometrics_.  Princeton university press, 2008.
  * Arbel et al. (2019) M. Arbel, A. Korba, A. Salim, and A. Gretton.  Maximum Mean Discrepancy Gradient Flow.  _arXiv:1906.04370 [cs, stat]_ , Dec. 2019.  arXiv: 1906.04370.
  * Bennett and Kallus (2020a) A. Bennett and N. Kallus.  Efficient policy learning from surrogate-loss classification reductions.  In _International Conference on Machine Learning_ , pages 788–798. PMLR, 2020a.
  * Bennett and Kallus (2020b) A. Bennett and N. Kallus.  The variational method of moments, 2020b.
  * Bennett et al. (2019) A. Bennett, N. Kallus, and T. Schnabel.  Deep generalized method of moments for instrumental variable analysis.  _Advances in neural information processing systems_ , 32, 2019.
  * Bennett et al. (2021) A. Bennett, N. Kallus, L. Li, and A. Mousavi.  Off-policy evaluation in infinite-horizon reinforcement learning with latent confounders.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1999–2007. PMLR, 2021.
  * Berlinet and Thomas-Agnan (2011) A. Berlinet and C. Thomas-Agnan.  _Reproducing kernel Hilbert spaces in probability and statistics_.  Springer Science & Business Media, 2011.
  * Bernstein (2009) D. S. Bernstein.  Matrix mathematics.  In _Matrix Mathematics_. Princeton university press, 2009.
  * Bierens (1982) H. J. Bierens.  Consistent model specification tests.  _Journal of Econometrics_ , 20(1):105–134, 1982\.
  * Blanchet et al. (2019) J. Blanchet, Y. Kang, and K. Murthy.  Robust wasserstein profile inference and applications to machine learning.  _Journal of Applied Probability_ , 56(3):830–857, 2019.
  * Carrasco and Florens (2000) M. Carrasco and J.-P. Florens.  Generalization of gmm to a continuum of moment conditions.  _Econometric Theory_ , 16(6):797–834, 2000.  ISSN 02664666, 14694360.
  * Carrasco and Kotchoni (2017) M. Carrasco and R. Kotchoni.  Regularized generalized empirical likelihood estimators.  Technical report, Technical report, 2017.
  * Carrasco et al. (2007) M. Carrasco, M. Chernov, J.-P. Florens, and E. Ghysels.  Efficient estimation of general dynamic models with a continuum of moment conditions.  _Journal of econometrics_ , 140(2):529–573, 2007\.
  * Chamberlain (1987) G. Chamberlain.  Asymptotic efficiency in estimation with conditional moment restrictions.  _Journal of Econometrics_ , 34(3):305–334, 1987\.  ISSN 0304-4076.  doi: https://doi.org/10.1016/0304-4076(87)90015-7.
  * Chaussé (2012) P. Chaussé.  Generalized empirical likelihood for a continuum of moment conditions.  2012\.
  * Chen and Pouzo (2009) X. Chen and D. Pouzo.  Efficient estimation of semiparametric conditional moment models with possibly nonsmooth residuals.  _Journal of Econometrics_ , 152(1):46–60, 2009\.
  * Chen et al. (2021) Y. Chen, L. Xu, C. Gulcehre, T. L. Paine, A. Gretton, N. de Freitas, and A. Doucet.  On instrumental variable regression for deep offline policy evaluation.  _arXiv preprint arXiv:2105.10148_ , 2021.
  * Chernozhukov et al. (2016) V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins.  Double/debiased machine learning for treatment and causal parameters.  _arXiv preprint arXiv:1608.00060_ , 2016.
  * Chernozhukov et al. (2017) V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, and W. Newey.  Double/debiased/neyman machine learning of treatment effects.  _American Economic Review_ , 107(5):261–65, 2017\.
  * Chernozhukov et al. (2018) V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins.  Double/debiased machine learning for treatment and structural parameters, 2018.
  * Chizat (2022) L. Chizat.  Mean-Field Langevin Dynamics: Exponential Convergence and Annealing, Aug. 2022.  arXiv:2202.01009 [math].
  * Cuturi (2013) M. Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  _Advances in neural information processing systems_ , 26, 2013.
  * Daskalakis et al. (2018) C. Daskalakis, A. Ilyas, V. Syrgkanis, and H. Zeng.  Training gans with optimism, 2018.
  * Dikkala et al. (2020) N. Dikkala, G. Lewis, L. Mackey, and V. Syrgkanis.  Minimax estimation of conditional moment models.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 12248–12262. Curran Associates, Inc., 2020.
  * Duchi et al. (2018) J. Duchi, P. Glynn, and H. Namkoong.  Statistics of robust optimization: A generalized empirical likelihood approach, 2018.
  * Garreau et al. (2018) D. Garreau, W. Jitkrittum, and M. Kanagawa.  Large sample analysis of the median heuristic, 2018.
  * Goodfellow et al. (2014) I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gretton et al. (2007) A. Gretton, K. Fukumizu, C. Teo, L. Song, B. Schölkopf, and A. Smola.  A kernel statistical test of independence.  _Advances in neural information processing systems_ , 20, 2007.
  * Gretton et al. (2012) A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola.  A kernel two-sample test.  _The Journal of Machine Learning Research_ , 13(1):723–773, 2012.
  * Hansen (1982) L. P. Hansen.  Large sample properties of generalized method of moments estimators.  _Econometrica_ , 50(4):1029–1054, 1982.  ISSN 00129682, 14680262.
  * Hansen et al. (1996) L. P. Hansen, J. Heaton, and A. Yaron.  Finite-sample properties of some alternative gmm estimators.  _Journal of Business & Economic Statistics_, 14(3):262–280, 1996.  ISSN 07350015.
  * Hartford et al. (2017) J. Hartford, G. Lewis, K. Leyton-Brown, and M. Taddy.  Deep iv: A flexible approach for counterfactual prediction.  In _International Conference on Machine Learning_ , pages 1414–1423. PMLR, 2017.
  * He and Lam (2021) S. He and H. Lam.  Higher-order expansion and bartlett correctability of distributionally robust optimization.  _arXiv preprint arXiv:2108.05908_ , 2021.
  * Ho et al. (2020) J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models, 2020.
  * Hornik et al. (1989) K. Hornik, M. Stinchcombe, and H. White.  Multilayer feedforward networks are universal approximators.  _Neural networks_ , 2(5):359–366, 1989.
  * Jordan et al. (1998) R. Jordan, D. Kinderlehrer, and F. Otto.  The variational formulation of the Fokker–Planck equation.  _SIAM journal on mathematical analysis_ , 29(1):1–17, 1998.  Publisher: SIAM.
  * Kingma and Welling (2022) D. P. Kingma and M. Welling.  Auto-encoding variational bayes, 2022.
  * Kitamura et al. (2004) Y. Kitamura, G. Tripathi, and H. Ahn.  Empirical likelihood-based inference in conditional moment restriction models.  _Econometrica_ , 72(6):1667–1714, 2004.  ISSN 00129682, 14680262.
  * Kremer et al. (2022) H. Kremer, J.-J. Zhu, K. Muandet, and B. Schölkopf.  Functional generalized empirical likelihood estimation for conditional moment restrictions.  In _International Conference on Machine Learning_ , pages 11665–11682. PMLR, 2022.
  * Lam (2019) H. Lam.  Recovering best statistical guarantees via the empirical divergence-based distributionally robust optimization.  _Operations Research_ , 67(4):1090–1105, 2019\.
  * Lam and Qian (2017) H. Lam and H. Qian.  Optimization-based quantification of simulation input uncertainty via empirical likelihood.  _arXiv preprint arXiv:1707.05917_ , 2017.
  * Lam and Zhou (2017) H. Lam and E. Zhou.  The empirical likelihood approach to quantifying uncertainty in sample average approximation.  _Operations Research Letters_ , 45(4):301–307, 2017.
  * Lasserre (2001) J. B. Lasserre.  Global optimization with polynomials and the problem of moments.  _SIAM Journal on optimization_ , 11(3):796–817, 2001.
  * Lewis and Syrgkanis (2018) G. Lewis and V. Syrgkanis.  Adversarial generalized method of moments, 2018.
  * Marteau-Ferey et al. (2020) U. Marteau-Ferey, F. Bach, and A. Rudi.  Non-parametric models for non-negative functions.  _Advances in neural information processing systems_ , 33:12816–12826, 2020.
  * Micchelli et al. (2006) C. Micchelli, Y. Xu, and H. Zhang.  Universal kernels.  _Mathematics_ , 7, 12 2006.
  * Mooij et al. (2009) J. Mooij, D. Janzing, J. Peters, and B. Schölkopf.  Regression by dependence minimization and its application to causal inference.  page 94, 06 2009.  doi: 10.1145/1553374.1553470.
  * Muandet et al. (2020) K. Muandet, W. Jitkrittum, and J. Kübler.  Kernel conditional moment test via maximum moment restriction, 2020.
  * Nesterov and Nemirovskii (1994) Y. Nesterov and A. Nemirovskii.  _Interior-point polynomial algorithms in convex programming_.  SIAM, 1994.
  * Newey and Powell (2003) W. K. Newey and J. L. Powell.  Instrumental variable estimation of nonparametric models.  _Econometrica_ , 71(5):1565–1578, 2003.
  * Newey and Smith (2004) W. K. Newey and R. J. Smith.  Higher order properties of gmm and generalized empirical likelihood estimators.  _Econometrica_ , 72(1):219–255, 2004.  ISSN 00129682, 14680262.
  * Owen (1990) A. Owen.  Empirical likelihood ratio confidence regions.  _The Annals of Statistics_ , 18(1):90–120, 1990\.  ISSN 00905364.
  * Owen (1988) A. B. Owen.  Empirical likelihood ratio confidence intervals for a single functional.  _Biometrika_ , 75(2):237–249, 1988.  ISSN 00063444.
  * Papamakarios et al. (2021) G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  _The Journal of Machine Learning Research_ , 22(1):2617–2680, 2021.
  * Qin and Lawless (1994) J. Qin and J. Lawless.  Empirical likelihood and general estimating equations.  _The Annals of Statistics_ , 22(1):300–325, 1994\.  ISSN 00905364.
  * Rahimi and Recht (2007) A. Rahimi and B. Recht.  Random features for large-scale kernel machines.  In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, _Advances in Neural Information Processing Systems_ , volume 20. Curran Associates, Inc., 2007.
  * Rosenblatt (1956) M. Rosenblatt.  Remarks on some nonparametric estimates of a density function.  _The annals of mathematical statistics_ , pages 832–837, 1956.
  * Saengkyongam et al. (2022) S. Saengkyongam, L. Henckel, N. Pfister, and J. Peters.  Exploiting independent instruments: Identification and distribution generalization, 2022.
  * Schölkopf and Smola (2002) B. Schölkopf and A. J. Smola.  _Learning with kernels: support vector machines, regularization, optimization, and beyond_.  MIT press, 2002.
  * Smith (1997) R. J. Smith.  Alternative semi-parametric likelihood approaches to generalised method of moments estimation.  _The Economic Journal_ , 107(441):503–519, 1997\.
  * Steinwart and Christmann (2008) I. Steinwart and A. Christmann.  _Support vector machines_.  Springer Science & Business Media, 2008.
  * Tripathi and Kitamura (2003) G. Tripathi and Y. Kitamura.  Testing conditional moment restrictions.  _The Annals of Statistics_ , 31(6):2059–2095, 2003.
  * Xu et al. (2021) L. Xu, Y. Chen, S. Srinivasan, N. de Freitas, A. Doucet, and A. Gretton.  Learning deep features in instrumental variable regression.  In _International Conference on Learning Representations_ , 2021.
  * Zeidler (2012) E. Zeidler.  _Applied functional analysis: applications to mathematical physics_ , volume 108.  Springer Science & Business Media, 2012.
  * Zhang et al. (2021) R. Zhang, M. Imaizumi, B. Schölkopf, and K. Muandet.  Maximum moment restriction for instrumental variable regression, 2021\.  arXiv 2010.07684.
  * Zhu et al. (2021) J.-J. Zhu, W. Jitkrittum, M. Diehl, and B. Schölkopf.  Kernel distributionally robust optimization: Generalized duality theorem and stochastic approximation.  In _International Conference on Artificial Intelligence and Statistics_ , pages 280–288. PMLR, 2021.

##  Appendix A The KMM Estimator for Functional Moment Restrictions

###  A.1 Duality

The primal problem of the entropy regularized KMM estimator for conditional
moment restrictions is given by

$\displaystyle
R^{\varphi}_{\epsilon}(\theta)=\inf_{P\in\mathcal{P}}\frac{1}{2}\operatorname*{MMD}(P,\hat{P}_{n};\mathcal{F})^{2}+\epsilon
D_{\varphi}(P||\omega)\quad\mathrm{s.t.}\quad\|E_{P}[\Psi(X,Z;\theta)]\|_{\mathcal{H}^{\ast}}\leq\lambda_{n},\quad
E_{P}[1]=1,$ (12)

where we relaxed the moment restrictions to hold only exactly for
$n\rightarrow\infty$. Note that to be precise, the dual of (12), which can be
obtained following the proof of Theorem 3.2, contains a regularization term
$-\lambda_{n}\|h\|_{\mathcal{H}}$ instead of
$-\frac{1}{2}\|h\|_{\mathcal{H}}^{2}$ as in our Definition 3.3. However, by
Lagrangian duality the regularizer $-\lambda_{n}\|h\|_{\mathcal{H}}$
corresponds to restricting $\mathcal{H}$ to a norm ball of some radius $\rho$,
and equally $-\frac{1}{2}\|h\|_{\mathcal{H}}^{2}$ corresponds to a restriction
to a norm ball of different radius $\rho^{\prime}$. Therefore both
formulations are practically equivalent and we use the squared version for its
greater smoothness and facilitated theoretical analysis. Note that in this
context that a theoretical analysis of the $-\lambda_{n}\|h\|_{\mathcal{H}}$
version would be possible by resorting to the variational formulation of the
norm $\|h\|_{\mathcal{H}}=\sup_{h^{\prime}\in\mathcal{H},\|h^{\prime}\|\leq
1}\langle h^{\prime},h\rangle_{\mathcal{H}}$. For an appropriate choice of
reference distribution $\omega$ a solution to the KMM problem (12) at the true
parameter $\theta_{0}\in\Theta$ always exists as the true distribution $P_{0}$
is contained in an MMD ball around the empirical distribution with probability
$1$. This is in stark contrast to the $\varphi$-divergence based FGEL
estimator of Kremer et al. [2022], as a $\varphi$-divergence ball around the
empirical distribution $\hat{P}_{n}$ generally contains the corresponding
continuous true distribution with probability $0$, as the $\varphi$ divergence
between a discrete distribution $\hat{P}_{n}$ and continuous distribution
$P_{0}$ diverges. However, at different parameters $\theta\in\Theta$ existence
of a distribution $P\in\mathcal{P}$ for which the functional moment
restrictions hold exactly cannot be guaranteed which implies
$R(\theta)=\infty$ and thus gradient-based optimization over $\theta\in\Theta$
can become difficult. Therefore the role of the relaxation parameter
$\lambda_{n}$ here is to smooth the MMD profile such that $R(\theta)<\infty$
in a neighbourhood of the true parameter to facilitate gradient-based
optimization over $\theta$. Note that even for fixed values of $\lambda_{n}$,
i.e., $\lambda_{n}=O_{p}(1)$, as $n\rightarrow\infty$ the objective has its
global minimum of $0$ at $P=P_{0}$ as
$\hat{P}_{n}\overset{p}{\rightarrow}P_{0}$ and
$\omega\overset{p}{\rightarrow}P_{0}$ weakly and thus we will retrieve the
true solution $\theta_{0}$. Therefore, compared to Kremer et al. [2022] where
the relaxation scheme is a fundamental necessity to restore strong duality,
here the regularization parameter can be seen merely as a computational tool.

###  A.2 Asymptotic Properties

For the KMM estimator for functional moment restrictions (FMR) of the form (9)
based on the functional MMD profile (10) we have the following properties.

######  Theorem A.1 (Consistency for FMR).

Assume that a) $\theta_{0}\in\Theta$ is the unique solution to
$E[\Psi(X,Z;\theta)]=0\in\mathcal{H}^{\ast}$; b) $\Theta\subset\mathbb{R}^{p}$
is compact; c) $\Psi(X,Z;\theta)$ is continuous in $\theta$ at any
$\theta\in\Theta$ with probability one; d)
$E[\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}^{2}]<\infty$;
e) $\Omega_{0}=E[\Psi(X,Z;\theta_{0})\otimes\Psi(X,Z;\theta_{0})]$ is non-
singular; f) $\omega=(1-\alpha)\hat{P}_{n}+\alpha Q$ for
$\alpha=O_{p}(n^{-1})$ and any distribution $Q$ such that
$E_{Q}[\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}^{2}]<\infty$;
and g) $\lambda_{n}=O_{p}(n^{-\xi})$ with $0<\xi<1/2$. Let $\hat{\theta}$
denote the functional KMM estimator for $\theta_{0}$, then
$\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$.

If additionally h) $\theta_{0}\in\operatorname{int}(\Theta)$; i)
$\Psi(x,z;\theta)$ is continuously differentiable in a neighborhood
$\bar{\Theta}$ of $\theta_{0}$ and
$E[\sup_{\theta\in\bar{\Theta}}\|\nabla_{\theta}\Psi(X,Z;\theta)\|^{2}_{\mathcal{H}^{\ast}}]<\infty$;
as well as j) $\Sigma_{0}=\left\langle
E[\nabla_{\theta}\Psi(X,Z;{\theta_{0}})],E[\nabla_{\theta}\Psi(X,Z;{\theta_{0}})]\right\rangle_{\mathcal{H}^{\ast}}\in\mathbb{R}^{p\times
p}$ is non-singular, we have $\|\hat{\theta}-\theta_{0}\|=O_{p}(n^{-1/2})$.

######  Theorem A.2 (Asymptotic Normality for FMR).

Let Assumptions a)-j) of Theorem A.1 be satisfied. Then,

$\displaystyle\sqrt{n}(\hat{\theta}-\theta_{0})\overset{d}{\rightarrow}N(0,\Xi_{0})$

where
$\Xi_{0}=\left(E[\nabla_{\theta}\Psi(X,Z;\theta_{0})]\Omega_{0}^{-1}E[\nabla_{\theta}\Psi(X,Z;\theta_{0})]\right)^{-1}$.

##  Appendix B Asymptotic Properties of the Finite-Dimensional KMM Estimator

For the KMM estimator for finite dimensional moment restrictions based on (8)
we have the following results.

######  Theorem B.1 (Consistency for MR).

Assume that a) $\theta_{0}\in\Theta$ is the unique solution to
$E[\psi(X;\theta)]=0\in\mathbb{R}^{m}$; b) $\Theta\subset\mathbb{R}^{p}$ is
compact; c) $\psi(X;\theta)$ is continuous at each $\theta\in\Theta$ with
probability one; d)
$E[\sup_{\theta\in\Theta}\|\psi(X;\theta)\|_{2}^{2}]<\infty$; e) The
covariance matrix
$\Omega_{0}:=E\left[\psi(X,\theta_{0})\psi(X;\theta_{0})^{T}\right]$ is non-
singular; and f) $\omega=(1-\alpha)\hat{P}_{n}+\alpha Q$ for
$\alpha=O_{p}(n^{-1})$ and any distribution $Q$ such that
$E_{Q}[\sup_{\theta\in\Theta}\|\psi(X;\theta)\|_{2}^{2}]<\infty$. Let
$\hat{\theta}$ denote the KMM estimator for $\theta_{0}$, then
$\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$.

If additionally g) $\theta_{0}\in\operatorname{int}(\Theta)$; h)
$\psi(x;\theta)$ is continuously differentiable in a neighborhood
$\bar{\Theta}$ of $\theta_{0}$ and
$E[\sup_{\theta\in\bar{\Theta}}\|\nabla_{\theta}\psi(X;\theta)\|^{2}|Z]<\infty$
w.p.1 as well as i)
$\operatorname{rank}\left(E[\nabla_{\theta}\psi(X;\theta_{0})]\right)=p$, we
have $\|\hat{\theta}-\theta_{0}\|=O_{p}(n^{-1/2})$.

######  Theorem B.2 (Asymptotic Normality for MR).

Let Assumptions a)-i) of Theorem B.1 be satisfied. Then,

$\displaystyle\sqrt{n}(\hat{\theta}-\theta_{0})\overset{d}{\rightarrow}N(0,\Xi_{0})$

where $\Xi_{0}=\left(E[\nabla_{\theta}\psi(X;\theta_{0})]\ \Omega_{0}^{-1}\
E[\nabla_{\theta}\psi(X;\theta_{0})]\right)^{-1}$.

######  Remark B.3.

The asymptotic variance $\Xi_{0}$ of the KMM estimator agrees with the one of
the optimally weighted GMM estimator [Hansen, 1982], thus for finite
dimensional moment restrictions KMM and OW-GMM are asymptotically first-order
equivalent.

##  Appendix C Additional Experimental Details

###  C.1 Hyperparameter choices

For the KMM estimator and the baselines we set the hyperparameters within the
values described below using the setting with the minimal value for
$\operatorname{HSIC}(\psi(X;\theta),Z)$ evaluated on a validation set of the
same size as the training set. As for large samples the $\operatorname{HSIC}$
computation becomes increasingly expensive we partition the validation data
into batches of size $n_{b}=2000$ and average $\operatorname{HSIC}$ over the
batches.

For the variational methods we use an optimistic Adam [Daskalakis et al.,
2018] implementation with a mini-batch size of $n=200$ and a learning rate of
$\tau_{\theta}=5\cdot 10^{-4}$ for optimization over $\theta$ and
$\tau_{h}=2.5\cdot 10^{-3}$ for optimization over $h$ and $\beta=(\eta,f,h)$
respectively. The regularization parameter for the instrument function
$h\in\mathcal{H}$ is picked from the grid $\lambda\in[0,10^{-4},10^{-2},1]$.

Specific to FGEL we treat the divergence as a hyperparameter which we pick
from $\varphi\in[\operatorname{KL},\operatorname{log},\chi^{2}]$.

Specific to KMM we use $n_{\textrm{RF}}=2000$ random Fourier features and for
every batch of size $n_{\textrm{batch}}=200$ sampled from $\hat{P}_{n}$ we
attach $n_{\text{reference}}=200$ samples from a reference distribution $Q$
which we represent by a kernel density estimator with Gaussian kernel and
bandwidth of $\sigma=0.1$ trained on $\hat{P}_{n}$. We observed that the
results are largely insensitive to the choice of bandwidth parameter $\sigma$.
The entropy regularizer is chosen as the Kullback-Leibler divergence as in the
first part of Section 3.3. In agreement with the observations of Kremer et al.
[2022] we noticed experimentally that the choice of $\varphi$-divergence has
only a minor effect on the obtained estimator.

###  C.2 Choice of Validation Metric and Failure of MMR

Figure 2: Effects of Validation Metrics for Early Stopping. Visualization of
different validation losses for $10$ training samples and different
estimators. Goal of the estimation is to minimize the error with respect to
the true function $g_{0}$ shown on the right which is unknown in practice. We
observe that among the considered validation metrics, HSIC is the only one
that approximately follows the behavior of the error with respect to the true
function and thus allows for effective early stopping. The author’s
implementations of DeepGMM and FGEL use MMR as validation loss. Switching to
HSIC allowed us to improve the performance of these baselines by a factor of
2-10.

The computation of modern CMR estimators including DeepGMM [Bennett et al.,
2019], Functional GEL [Kremer et al., 2022] and our KMM estimator generally
requires solving a mini-max or saddle point problem where the minimization is
with respect to the model parameters and the maximization with respect to the
instrument function $h$ (and the RKHS function $f$ in the case of KMM). For
such problems it is not obvious how to monitor the success of the training
procedure as for conditional moment restriction problems it is not clear which
validation objective is supposed to be optimized, which makes tuning of
hyperparameters and early stopping cumbersome and ambiguous. This is in
contrast to standard supervised learning via empirical risk minimization where
training and target objectives are usually aligned and thus one can simply
evaluate the loss function over a suitable validation set. There exist
different approaches to quantify how well the restrictions
$E[\psi(X;\theta)|Z]=0\ P_{Z}\text{-a.s.}$ are satisfied. The authors of
Bennett et al. [2019] and Kremer et al. [2022] used the maximum moment
restriction objective [Zhang et al., 2021]
$\operatorname{MMR}(\theta)=E_{\hat{P}_{n}}[\psi(X;\theta)K(Z,Z^{\prime})\psi(X^{\prime};\theta)]$
which results from the variational formulation
$\operatorname{MMR}(\theta)=\sup_{h\in\mathcal{H}}\left(E_{\hat{P}_{n}}[\psi(X;\theta)^{T}h(Z)]\right)^{2}$
with $\mathcal{H}$ corresponding to a unit ball of a reproducing kernel
Hilbert space. While Zhang et al. [2021] show that this leads to a consistent
estimator for $\theta_{0}$ when optimized over $\theta\in\Theta$ and thus
quantifies the satisfaction of the CMR in a meaningful way, it is a priori not
clear if it provides a suitable validation metric in finite samples.

As an alternative Saengkyongam et al. [2022] proposed to measure the
satisfaction of $E[\psi(X;\theta)|Z]=0\ P_{Z}\text{-a.s.}$ by quantifying the
independence of the random variables $\psi(X;\theta)$ and $Z$ via the Hilbert-
Schmidt independence criterion (HSIC) [Gretton et al., 2007].

We tested these two validation metrics for hyperparameter optimization and
early stopping and observed that using HSIC instead of MMR as validation
metric leads to improvements of the predictive MSE of the variational
estimators (DeepGMM, FGEL, KMM) by a factor of $2$-$10$, when keeping all
other settings (i.e. hyperparameter grids) fixed.

We exemplarily visualize the effect of different validation metrics for early
stopping for the heteroskedastic IV experiment in Figure 2. We train all
estimators for $10$ different random samples and use HSIC with a loose
stopping criterion as validation metric in order to train beyond the optimal
validation loss for visualization. The left column shows the prediction MSE of
the learned function. While we aim to optimize this quantity, in practice we
do not have access to it as the true function $g_{0}$ is unknown. The
remaining columns show the different validation metrics over the course of the
optimization. We observe that using the simplistic unconditional moment
violation generally leads to overfitting as the estimator would be trained
beyond the minimum of the true objective of interest. Interestingly, in most
cases the MMR objective does not decrease over the course of the optimization
procedure and thus any early stopping strategy might stop the training at
random. Of the three metrics, HSIC is the only one that approximately mimics
the behavior of the true objective of interest and thus allows for an
effective early stopping strategy to prevent overfitting and unnecessary long
training.

##  Appendix D Entropy Regularization

###  D.1 Effect of the Regularization Parameter

As discussed in the Section 3.2, for the case of the backward KL divergence or
Burg entropy, our entropy regularization can be interpreted as a barrier
function in an interior-point method, see [Nesterov and Nemirovskii, 1994].
For decreasing values of $\epsilon$, the _entropy-regularized MMD profile_
approaches the unregularized _MMD-profile_. To validate this empirically we
carry out the maximization over the dual parameters $(\eta,f)$ in equation (8)
while keeping $h$ and $\theta$ fixed, which preserves the convex structure of
the problem. In Figure 3(a) we observe that for smaller $\epsilon$ we get
closer and closer to the original _MMD-profile_ , which we obtain from
equation (5) by using a sample approximation of the semi-infinite constraint
in a convex solver.

###  D.2 Annealing of Entropy Regularization

Instead of keeping the regularization parameter $\epsilon$ fixed during the
optimization procedure as in Figure 3(a), we study an annealing schedule in
which it is gradually decreased, similar to actual interior-point methods.
Chizat [2022] also studied effects of annealing in a setting where they use
particle-gradient descent. While their work also builds on an energy
functional consisting of a combination of MMD and KL-divergence, the
dissipation is done in the Wasserstein geometry. In comparison, we do not
carry out the optimization by moving in the Wasserstein space, but instead in
the dual RKHS. To visualize the effect of annealing, we keep $h$ and $\theta$
fixed and only maximize with respect to the remaining dual variables
$(\eta,f)$, while gradually decreasing $\epsilon$ with the number of
iterations. We empirically observe that the annealing procedure eventually
leads to a solution that satisfies the (semi-)infinite constraint (11). This
is visualized in Figure 3(b) where the shaded black curves, corresponding to
$f(x)+\eta$ at different iterations, are slowly pushed below the red curve in
the course of the optimization.

(a) Regularization Parameters (b) Annealing to Approximate the (Semi-)Infinite
Constraint

Figure 3: Effect of Entropy Regularization. Figure a) shows the effect of
entropy regularization for fixed parameters $\epsilon$. The gray lines
correspond to logarithmically decreasing values of $\epsilon$ between $1000$
and $0.01$. Figure b) shows the annealing procedure for entropy
regularization, where the shaded curves show the intermediate progress of the
optimization.

###  D.3 Choice of Reference Measure

The KMM estimator based on (8) and (10) respectively requires a choice of
distribution $Q$ that enters the reference distribution
$\omega=(1-\alpha)\hat{P}_{n}+\alpha Q$ to define the entropy regularizer. As
the candidate distributions are required to admit a density with respect to
$\omega$, the choice of $Q$ directly determines the class of distributions
considered in the minimization over $P$. Optimally $Q$ should be chosen as
close as possible to the population distribution $P_{0}$. As generally $P_{0}$
is unknown, in the following we discuss several (data-driven) choices for $Q$.

##### Lebesgue Measure

Choosing $Q$ as the Lebesgue measure or the uniform distribution over
$\mathcal{X}\times\mathcal{Z}$ respectively, allows for considering arbitrary
distributions on $\mathcal{X}\times\mathcal{Z}$. At the same time, this choice
corresponds to an uninformative prior which discards the information contained
in the sample and does not converge to the population distribution as
$n\rightarrow\infty$. Empirically the Lebesgue measure did not show
competitive performance.

##### Empirical Distribution

The empirical distribution converges to the population distribution as
$n\rightarrow\infty$, and therefore it provides a viable candidate as
reference distribution. However, as the empirical distribution is a discrete
distribution supported on the samples, the considered candidate distributions
are again reweightings of the data and thus some of the competitive advantage
of KMM over other method of moments estimators is lost. Note however, that MMD
provides different gradient information compared to $\varphi$-divergences as
used in GEL/GMM and therefore the obtained estimator will still be different.

##### Kernel Density Estimation

In order to combine the strength of considering continuous candidate
distributions in (8) with the information contained in the empirical
distribution, one can represent the reference distribution $Q$ by a kernel
density estimator (KDE) [Rosenblatt, 1956] trained on the empirical sample.
This allows us to sample from a continuous distribution in Algorithm 1 and
thus taking into account candidate distributions with support different from
the empirical distribution, while still converging to the population
distribution as $n\rightarrow\infty$. Representing $Q$ by a KDE proved to be
the most effective choice in practice.

##### Modern Machine Learning Models

As a straight-forward extension of the KDE approach, one can represent $Q$ by
any density estimator from which one can sample and can thus leverage the
potential of modern machine learning approaches like generative adversarial
networks [Goodfellow et al., 2014], variational auto-encoders [Kingma and
Welling, 2022], normalizing flows [Papamakarios et al., 2021] or diffusion
models [Ho et al., 2020]. This seems particularly promising for complex high-
dimensional data, where KDE estimators become increasingly inaccurate. Note
however, that while better density estimators most likely improve the finite
sample performance of our estimator, the role of $Q$ is to define the class of
(continuous) candidate distributions via its support. As long as $P_{0}\ll Q$,
we can find a $P$ arbitrarily close to $P_{0}$ and better choices of $Q$
(closer to $P_{0}$) mostly only facilitate finding these.

##### Time Evolution of $Q$ via Primal-Dual Schemes

Instead of using a fixed choice of reference distribution $Q$, one could
choose the reference distribution adaptively over the course of the
optimization via primal-dual schemes. To this aim, take $P^{0}=\hat{P}_{n}$
and consider for timesteps $k=1,\ldots,T$ problem (7) as

$\displaystyle R^{\varphi}_{\epsilon}(\theta)=$
$\displaystyle\inf_{P\in\mathcal{P}}\frac{1}{2}\operatorname*{MMD}(P,\hat{P}_{n})^{2}+\epsilon
D(P||P_{k})\quad\mathrm{s.t.}\quad E_{P}[1]=1,\quad E_{P}[\psi(X;\theta)]=0.$
(13)

Carrying out the optimization over $\mu\in\mathcal{F}$ and defining
$\beta=(\eta,f,h)\in\mathcal{M}$, the Lagrangian of the MMD profile (7) can be
cast in the (semi-dual) form,

$\displaystyle
L(P,\beta,\theta)=\frac{1}{n}\sum_{i=1}^{n}f(x_{i})+\eta-\frac{1}{2}\|f\|_{\mathcal{H}}^{2}+\int\left(-f(x)-\eta+\langle
h,\psi(x;\theta)\rangle\right)dP(x)+\epsilon D_{\varphi}(P||P_{k})$ (14)

Now, instead of the dual approach used in our dual MMD profile, one could
consider a primal dual update where one alternates between updates of the
primal measure $P$ via a proximal term using a Bregman divergence $D$

$\displaystyle
P_{k+1}\in\operatorname*{arg\,min}_{P}\int\left(-f(x)-\eta+\langle
h,\psi(x;\theta)\rangle\right)dP(x)+\epsilon D(P\|P_{k}).$ (15)

and subsequently update the dual variables $\beta$, where we solve

$\displaystyle\beta_{k+1}\in\operatorname*{arg\,max}_{\beta\in\mathcal{M}}\frac{1}{n}\sum_{i=1}^{n}f(x_{i})+\eta-\frac{1}{2}\|f\|_{\mathcal{H}}^{2}+\int\left(-f(x)-\eta+\langle
h,\psi(x;\theta)\rangle\right)dP_{k+1}(x)+\frac{1}{2t}\|\beta-\beta_{k}\|^{2}_{\mathcal{M}}.$
(16)

The update rule (16) is a standard convex optimization problem. We leave a
specific implementation of this approach to future work.

##  Appendix E Proofs

###  E.1 Definitions and Preliminaries

To simplify notation and analysis we first provide a compact formulation of
the functional KMM objective (10) given by

$\displaystyle\widehat{G}_{\epsilon,\lambda_{n}}(\theta,\eta,f,h)=\frac{1}{n}\sum_{i=1}^{n}f(x_{i},z_{i})+\eta-\frac{1}{2}\|f\|_{\mathcal{F}}^{2}-\epsilon\int_{\mathcal{X}\times\mathcal{Z}}{\\!\\!\\!\\!\\!\\!\\!\varphi^{\ast}\left(\frac{f(x,z)+\eta-\langle\Psi(x,z;\theta),h\rangle_{\mathcal{H}}}{\epsilon}\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)-\frac{\lambda_{n}}{2}\|h\|_{\mathcal{H}}^{2}.$

As $\mathcal{F}$ is an RKHS of functions
$\mathcal{X}\times\mathcal{Z}\rightarrow\mathbb{R}$, the evaluation functional
in $\mathcal{F}$ is given by
$k\left((x,z),\cdot\right):\mathcal{F}\rightarrow\mathbb{R}$, such that
$\langle k((x,z),\cdot),f\rangle_{\mathcal{F}}=f(x,z)\ \forall
f\in\mathcal{F}$ and $(x,z)\in\mathcal{X}\times\mathcal{Z}$. Let
$\mathcal{M}:=\mathbb{R}\times\mathcal{F}\times\mathcal{H}$. For
$\beta=(\eta,f,h)\in\mathcal{M}$ define a norm on $\mathcal{M}$ as
$\|\beta\|_{\mathcal{M}}=\sqrt{|\eta|^{2}+\|f\|^{2}_{\mathcal{F}}+\|h\|^{2}_{\mathcal{H}}}$.
Define for $i=1,\ldots,n$,

$\displaystyle b_{i}=\begin{pmatrix}1\\\ k((x_{i},z_{i}),\cdot)\\\
0\end{pmatrix}\in\mathcal{M},\quad a(x,z;\theta)=\begin{pmatrix}1\\\
k((x,z),\cdot)\\\ -\Psi(x,z;\theta)\end{pmatrix}\in\mathcal{M},\quad
R_{\lambda}=\begin{pmatrix}0&&\\\ &I&\\\ &&\lambda
I\end{pmatrix}\in\mathcal{M\times M}.$ (17)

where we used that we can identify $\mathcal{M}^{\ast}$ with $\mathcal{M}$ by
the self-duality property of Hilbert spaces. Then the functional KMM objective
(10) can be written in the compact form

$\displaystyle
G_{\epsilon,\lambda_{n}}(\theta,\beta)=\frac{1}{n}\sum_{i=1}^{n}b_{i}^{T}\beta-\epsilon\int_{\mathcal{X}\times\mathcal{Z}}{\varphi^{\ast}\left(\frac{1}{\epsilon}a(x,z;\theta)^{T}\beta\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)-\frac{1}{2}\beta^{T}R_{\lambda_{n}}\beta.$
(18)

Analogously for the objective of the finite dimensional KMM estimator (8) we
have $\mathcal{H}=\mathbb{R}^{m}$ and
$\mathcal{M}=\mathbb{R}\times\mathcal{F}\times\mathbb{R}^{m}$ and further
define for $i=1,\ldots,n$,

$\displaystyle b_{i}=\begin{pmatrix}1\\\ k(x_{i},\cdot)\\\
0\end{pmatrix}\in\mathcal{M},\quad a(x;\theta)=\begin{pmatrix}1\\\
k(x,\cdot)\\\ -\psi(x;\theta)\end{pmatrix}\in\mathcal{M},\quad
R=\begin{pmatrix}0&&\\\ &I&\\\ &&0\end{pmatrix}\in\mathcal{M\times M}.$ (19)

Then the unconditional KMM objective (8) can be written in the compact form

$\displaystyle\widehat{G}_{\epsilon}(\theta,\beta)=\frac{1}{n}\sum_{i=1}^{n}b_{i}^{T}\beta-\epsilon\int_{\mathcal{X}}{\varphi^{\ast}\left(\frac{1}{\epsilon}a(x;\theta)^{T}\beta\right)}\omega(\mathrm{d}x)-\frac{1}{2}\beta^{T}R\beta.$
(20)

In the proofs we will consider derivatives of the KMM objective with respect
to the dual parameters $\beta\in\mathcal{M}$, the second and the third
component of which live in function spaces $\mathcal{F}$ and $\mathcal{H}$
respectively. We define the corresponding functional derivative as follows.

######  Definition E.1 (Functional Derivative).

Let $\mathcal{H}$ be a vector space of functions. For a functional
$G:\mathcal{H}\rightarrow\mathbb{R}$ and a pair of functions
$h_{0},h_{1}\in\mathcal{H}$, we define the derivative operator
$\frac{\partial}{\partial h}G(h_{0})$ at $h_{0}$ via $\frac{\partial}{\partial
h}G(h_{0})(h_{1})=\left.\frac{d}{dt}G(h_{0}+th_{1})\right|_{t=0}$. Likewise,
we define the $k$-th functional derivative $\frac{\partial^{k}}{(\partial
h)^{k}}G(h_{0})$ at $h_{0}$ via

$\displaystyle\frac{\partial^{k}}{(\partial
h)^{k}}G(h_{0})\left(h_{1},\ldots,h_{k}\right)=\left.\frac{\partial^{k}}{\partial
t_{1}\ldots\partial
t_{k}}G\left(h_{0}+t_{1}h_{1}+\ldots+t_{k}h_{k}\right)\right|_{t_{1}=\cdots=t_{k}=0}.$
(21)

Moreover, we write $\frac{\partial^{k}}{(\partial h)^{k}}G(h_{0})=0$ as a
shorthand for $\frac{\partial^{k}}{(\partial
h)^{k}}G(h_{0})\left(h_{1},\ldots,h_{k}\right)=0$ for all
$h_{1},\ldots,h_{k}\in\mathcal{H}$. Similarly, when considering a vector-
valued function of a vector-valued parameter,
$G:\Theta\subseteq\mathbb{R}^{p}\rightarrow\mathbb{R}^{m}$, we denote the
$k$-th standard directional derivative at $\theta_{0}\in\Theta$ as
$\frac{\partial^{k}}{(\partial\theta)^{k}}G(\theta_{0})\in\mathbb{R}^{p\times
m}$ and in the case $k=1$ we write the Jacobian as
$\frac{\partial}{(\partial\theta)}G(\theta_{0})=\left(\nabla_{\theta}G\right)(\theta_{0})=:\nabla_{\theta}G(\theta_{0})$.

Additionally we will make use of the functional version of Taylor’s theorem
with Lagrange remainder, which we state here for completeness.

######  Proposition E.2 (Taylor’s Theorem).

Let $G:\mathcal{H}\rightarrow\mathbb{R}$, where $\mathcal{H}$ is a vector
space of functions. For any $h,h^{\prime}\in\mathcal{H}$, if $t\mapsto
G\left(th+(1-t)h^{\prime}\right)$ is $(k+1)$-times differentiable over an open
interval containing $[0,1]$, then there exists
$\bar{h}\in\operatorname{conv}\left(\left\\{h,h^{\prime}\right\\}\right)$ such
that

$\displaystyle G\left(h^{\prime}\right)=$ $\displaystyle\
G(h)+\sum_{i=1}^{k}\frac{1}{i!}\frac{\partial^{i}}{(\partial
h)^{i}}G(h)[\underbrace{h^{\prime}-h,\ldots,h^{\prime}-h}_{i\text{ times }}]$
(22) $\displaystyle+\frac{1}{(k+1)!}\frac{\partial^{k+1}}{(\partial
h)^{k+1}}G(\bar{h})[\underbrace{h^{\prime}-h,\ldots,h^{\prime}-h}_{k+1\text{
times }}].$ (23)

###  E.2 Duality Results

####  E.2.1 Proof of Theorem 3.1

###### Proof.

Let
$\mu_{\hat{P}_{n}}=E_{\hat{P}_{n}}[k(X,\cdot)]=\frac{1}{n}\sum_{i=1}^{n}k(x_{i},\cdot)$
denote the kernel mean embedding of the empirical distribution. Instead of
working with the measure $P$ directly, we introduce an auxiliary variable
$\mu\in\mathcal{F}$, which serves as the kernel mean embedding of $P$, so we
can write the MMD profile as

$\displaystyle
R(\theta)=\inf_{P\in\mathcal{P},\mu\in\mathcal{H}}\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}\quad\mathrm{s.t.}\quad\int
k(x,\cdot)dP(x)=\mu,\quad\int dP(x)=1,\quad\int\psi(x;\theta)dP(x)=0.$ (24)

Introducing Lagrange parameters $\eta\in\mathbb{R}$, $f\in\mathcal{F}$ and
$h\in\mathbb{R}^{m}$ we can define the Lagrangian as

$\displaystyle L(P,\mu,\eta,f,h)$
$\displaystyle=\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{H}}^{2}+\langle
f,\mu-\int k(x,\cdot)dP(x)\rangle_{\mathcal{F}}+\eta\left(1-\int
dP(x)\right)+\langle h,\int\psi(x;\theta)dP(x)\rangle_{\mathbb{R}^{m}}$ (25)
$\displaystyle=\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\int\left(-f(x)-\eta+\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}\right)dP(x)+\langle
f,\mu\rangle_{\mathcal{F}}+\eta.$ (26)

Now as we minimize the Lagrangian with respect to all positive measures $P$,
this only yields a finite expression as long as $-f(x)-f_{0}+\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}\geq 0$ $\forall x\in\mathcal{X}$.
This directly translates into a semi-infinite constraint and the problem
becomes

$\displaystyle\sup_{f_{0},f,h}\inf_{\mu\in\mathcal{F}}\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\langle
f,\mu\rangle_{\mathcal{F}}+f_{0}$ (27) $\displaystyle\mathrm{s.t.}\quad\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}\geq f(x)+f_{0}\quad\forall
x\in\mathcal{X}.$ (28)

Now, the first order optimality conditions for $\mu$ yield (see subsection
Optimality Condition in $\mu$ in the following for details)

$\displaystyle\mu=\mu_{\hat{P}_{n}}-f,$ (29)

and reinserting yields the final dual problem

$\displaystyle\sup_{f_{0},f,h}\frac{1}{n}\sum_{i=1}^{n}f(x_{i})-\frac{1}{2}\|f\|_{\mathcal{H}}^{2}+f_{0}$
(30) $\displaystyle\mathrm{s.t.}\quad\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}\geq f(x)+f_{0}\quad\forall
x\in\mathcal{X}.$ (31)

Strong duality holds trivially as the primal problem only contains equality
constraints. ∎

####  E.2.2 Proof of Theorem 3.2

###### Proof.

Let again
$\mu_{\hat{P}_{n}}=E_{\hat{P}_{n}}[k(X,\cdot)]=\frac{1}{n}\sum_{i=1}^{n}k(x_{i},\cdot)$
denote the KME of the empirical distribution. Following the proof of Theorem
3.1 we can write the entropy regularized MMD profile as

$\displaystyle
R_{\epsilon}(\theta)=\inf_{P\ll\omega,\mu\in\mathcal{H}}\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\epsilon
D_{\varphi}(P||\omega)\quad\mathrm{s.t.}\quad\int
k(x,\cdot)dP(x)=\mu,\quad\int dP(x)=1,\quad\int\psi(x;\theta)dP(x)=0.$

Let $p(x)$ denote the density of $P$ with respect to the reference measure
$\omega$. Introducing dual variables $\eta\in\mathbb{R}$, $f\in\mathcal{F}$
and $h\in\mathbb{R}^{m}$, the Lagrangian of the problem can be obtained as

$\displaystyle L(p,\mu,f,h)=$
$\displaystyle\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\epsilon\int_{\mathcal{X}}{\varphi(p(x))}\omega(\mathrm{d}x)+\langle
f,\mu-\int_{\mathcal{X}}{k(x,\cdot)p(x)}\omega(\mathrm{d}x)\rangle_{\mathcal{F}}$
(32)
$\displaystyle+\eta\left(1-\int_{\mathcal{X}}{p(x)}\omega(\mathrm{d}x)\right)+\langle
h,\int_{\mathcal{X}}{\psi(x;\theta)p(x)}\omega(\mathrm{d}x)\rangle_{\mathbb{R}^{m}}.$
(33)

Now, collecting terms containing $p$ we get

$\displaystyle L(p,\mu,f,h)=$
$\displaystyle\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\langle
f,\mu\rangle_{\mathcal{F}}+\eta-\epsilon\int_{\mathcal{X}}{\left(\frac{f(x)+\eta-\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}}{\epsilon}p(x)-\varphi(p(x))\right)}\omega(\mathrm{d}x).$
(34)

The dual formulation follows from minimizing the Lagrangian with respect to
the primal variables $\mu\in\mathcal{F}$ and $p\in\Pi(\omega)$, where
$\Pi(\omega)$ denotes the set of all densities with respect to $\omega$.
Taking the infimum of $L$ with respect to $p$ we obtain

$\displaystyle\inf_{p\in\Pi(\omega)}\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\langle
f,\mu\rangle_{\mathcal{F}}+\eta-\epsilon\int_{\mathcal{X}}{\left(\frac{f(x)+\eta-\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}}{\epsilon}p(x)-\varphi(p(x))\right)}\omega(\mathrm{d}x)$
(35) $\displaystyle=$
$\displaystyle\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\langle
f,\mu\rangle_{\mathcal{F}}+\eta-\epsilon\sup_{p\in\Pi(\omega)}\int_{\mathcal{X}}{\left(\frac{f(x)+\eta-\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}}{\epsilon}p(x)-\varphi(p(x))\right)}\omega(\mathrm{d}x)$
(36) $\displaystyle=$
$\displaystyle\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\langle
f,\mu\rangle_{\mathcal{F}}+\eta-\epsilon\int_{\mathcal{X}}{\sup_{t\in\mathbb{R}_{+}}\left(\frac{f(x)+\eta-\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}}{\epsilon}t-\varphi(t)\right)}\omega(\mathrm{d}x)$
(37) $\displaystyle=$
$\displaystyle\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\langle
f,\mu\rangle_{\mathcal{F}}+\eta-\epsilon\int_{\mathcal{X}}{\varphi^{\ast}\left(\frac{f(x)+\eta-\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}}{\epsilon}\right)}\omega(\mathrm{d}x),$
(38)

where we used the definition of the Fenchel conjugate function
$\varphi^{\ast}(q)=\sup_{p}\langle q,p\rangle-\varphi(p)$. In the third line
we used that as $p:\mathcal{X}\rightarrow\mathbb{R}_{+}$ is an arbitrary
function, we can swap the supremum outside the integral for a pointwise
supremum over $t:=p(x)$ for each $x\in\mathcal{X}$ in the integral. Now, the
first order optimality conditions for the function $\mu$ yield (refer to the
following subsection for details)

$\displaystyle\mu=\mu_{\hat{P}_{n}}-f.$ (39)

Inserting this back into the Lagrangian we get

$\displaystyle L(f,\eta,h)$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}f(x_{i})+\eta-\frac{1}{2}\|f\|_{\mathcal{H}}^{2}-\epsilon\int_{\mathcal{X}}{\varphi^{\ast}\left(\frac{f(x)+\eta-\langle
h,\psi(x;\theta)\rangle_{\mathbb{R}^{m}}}{\epsilon}\right)}\omega(\mathrm{d}x),$
(40)

from which the dual program follows. Strong duality follows trivially as the
primal problem only contains equality constraints. In order to show convexity,
consider the compact notation (20),

$\displaystyle
G_{\epsilon}(\theta,\beta)=\frac{1}{n}\sum_{i=1}^{n}b_{i}^{T}\beta-\epsilon\int_{\mathcal{X}}{\varphi^{\ast}\left(\frac{1}{\epsilon}a(x)^{T}\beta\right)}\omega(\mathrm{d}x)-\frac{1}{2}\beta^{T}R\beta.$
(41)

The first term is linear in $\beta$ and thus trivially concave. The second
term is concave as by definition the Fenchel conjugate of any function is
convex (and thus its negative concave) and the composition of a concave
function with a linear function yields a concave function. Finally the third
term is a negative semi-definite quadratic form for any $\lambda_{n}\geq 0$
and thus concave. As an unconstrained maximization over a jointly concave
objective the optimization over the dual parameters is a convex program. ∎

##### Optimality condition in $\mu$

It is easily verified that the functional

$\displaystyle\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\langle
f,\mu\rangle_{\mathcal{F}}$ (42)

is (strongly) convex in $\mu$. In fact, its minimizer can be seen by a
straightforward manipulation of the terms

$\displaystyle\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\langle
f,\mu\rangle_{\mathcal{F}}=$
$\displaystyle\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}+\langle
f,\mu-\mu_{\hat{P}_{n}}\rangle_{\mathcal{F}}+\frac{1}{2}\|f\|_{\mathcal{F}}^{2}+\langle
f,\mu_{\hat{P}_{n}}\rangle_{\mathcal{F}}-\frac{1}{2}\|f\|_{\mathcal{F}}^{2}$
(43) $\displaystyle=$
$\displaystyle\frac{1}{2}\|\mu-\mu_{\hat{P}_{n}}+f\|_{\mathcal{F}}^{2}+\langle
f,\mu_{\hat{P}_{n}}\rangle_{\mathcal{F}}-\frac{1}{2}\|f\|_{\mathcal{F}}^{2}$
(44) $\displaystyle\geq$ $\displaystyle\langle
f,\mu_{\hat{P}_{n}}\rangle_{\mathcal{F}}-\frac{1}{2}\|f\|_{\mathcal{F}}^{2},$
(45)

where the optimum is attained at $\mu=\mu_{\hat{P}_{n}}-f$.

Alternatively, we can also characterize the optimality condition via the
differentiability structure. Since $\mathcal{F}$ is a normed space, we use
$\nabla G(\mu)$ to denote the Fréchet derivative of a functional $G$. Suppose
$\mu$ is a minimizer of the problem

$\displaystyle\min_{\mu^{\prime}}\ \left\\{G(\mu^{\prime}):=\langle
f,\mu^{\prime}\rangle_{\mathcal{F}}+\frac{1}{2}\|\mu^{\prime}-\mu_{\hat{P}_{n}}\|_{\mathcal{F}}^{2}\right\\}.$
(46)

Then $\nabla G(\mu)=0$ and a straightforward calculation yields
$\mu=\mu_{\hat{P}_{n}}-f$.

###  E.3 Asymptotic Properties of KMM for Conditional Moment Restrictions

The asymptotic properties of the KMM estimator for conditional moment
restrictions follow from phrasing the conditional moment restrictions (2) as
functional moment restrictions of the form (3) over a sufficiently rich
Hilbert space of functions. In the following we show that the assumptions of
Theorem 3.4 suffice to fulfill the assumptions of the theorems for the
functional KMM estimator (Theorems A.1 and A.2) from which the results follow.
The proofs for the functional case are deferred to Section E.4.

####  E.3.1 Proof of Theorem 3.4 (Consistency for CMR)

######  Lemma E.3.

For a moment function $\psi(x;\theta)$ taking values in $\mathbb{R}^{m}$
define the _conditional covariance matrix_
$V_{0}(Z)=E[\psi(X;\theta_{0})\psi(X;\theta_{0})^{T}|Z]$ as a function of the
conditioning random variable $Z$ taking values in $\mathcal{Z}$. Let
$\mathcal{H}$ be a Hilbert space of square integrable functions equipped with
the norm
$h\mapsto\|h\|_{L^{2}(\mathcal{H},P_{0})}=\left(\int_{\mathcal{Z}}\|h(z)\|_{2}^{2}\
P_{0}(\mathrm{d}z)\right)^{1/2}$. Define the moment functional
$\Psi(x,z;\theta):\mathcal{H}\rightarrow\mathbb{R}$ such that
$\Psi(x,z;\theta)(h)=\psi(x;\theta)^{T}h(z)$ for any
$(x,z)\in\mathcal{X}\times\mathcal{Z}$, $\theta\in\Theta$ and
$h\in\mathcal{H}$. Then the covariance operator
$\Omega_{0}:\mathcal{H}\rightarrow\mathcal{H}$ defined as

$\displaystyle\Omega_{0}=E[\Psi(X,Z;\theta_{0})\otimes\Psi(X,Z;\theta_{0})]$
(47)

is non-singular if $V_{0}(Z)$ is non-singular with probability 1.

###### Proof.

Note that $\Omega_{0}$ is non-singular if
$\|\Omega_{0}h\|_{L^{2}(\mathcal{H},P_{0})}>0$ for any $h\in\mathcal{H}$ with
$\|h\|_{L^{2}(\mathcal{H},P_{0})}>0$, or equivalently if $\langle
h,\Omega_{0}h\rangle\neq 0$. Consider any $h\in\mathcal{H}$ with
$\|h\|_{L^{2}(\mathcal{H},P_{0})}>0$, then by the law of iterated expectation
we have

$\displaystyle\langle h,\Omega_{0}h\rangle_{\mathcal{H}}$
$\displaystyle=E\left[\langle\Psi(X,Z;\theta_{0})(h),\Psi(X,Z;\theta_{0})(h)\rangle\right]$
(48)
$\displaystyle=E\left[\left(\psi(X;\theta_{0})^{T}h(Z)\right)^{T}\left(\psi(X;\theta_{0})^{T}h(Z)\right)\right]$
(49)
$\displaystyle=E\left[h(Z)^{T}E[\psi(X;\theta_{0})\psi(X;\theta_{0})^{T}|Z]h(Z)\right]$
(50) $\displaystyle=\int h(z)^{T}V_{0}(z)h(z)\mathrm{d}P_{0}(z)$ (51)

Now, $V_{0}(Z)$ is a positive-semi definite matrix by construction and non-
singular $P_{0}\text{-a.s.}$ by assumption and thus its smallest eigenvalue
$C$ is bounded away from zero. Therefore we have

$\displaystyle\langle h,\Omega_{0}h\rangle_{\mathcal{H}}\geq
C\int\|h(z)\|_{2}^{2}\mathrm{d}P_{0}(z)=\|h\|_{L^{2}(\mathcal{H},P_{0})}^{2}>0$
(52)

and thus $\Omega_{0}$ is non-singular with smallest eigenvalue bounded away
from zero. ∎

######  Lemma E.4.

Let the assumptions of Theorem 3.4 be satisfied and for a moment function
$\psi$ define the moment functional
$\Psi(x,z;\theta):\mathcal{H}\rightarrow\mathbb{R}$ as
$\Psi(x,z;\theta)(h)=\psi(x;\theta)^{T}h(z)$ $\forall h\in\mathcal{H}$. Then
the matrix $\Sigma_{0}=\left\langle
E[\nabla_{\theta}\Psi(X,Z;\theta_{0})],E[\nabla_{\theta}\Psi(X,Z;\theta_{0})]\right\rangle_{\mathcal{H}}\in\mathbb{R}^{p\times
p}$ is strictly positive definite and non-singular with smallest eigenvalue
bounded away from zero.

###### Proof.

Let the Hilbert space $\mathcal{H}=\bigoplus_{i=1}^{m}\mathcal{H}_{i}$ be
equipped with the inner product $\langle
h,h^{\prime}\rangle_{L^{2}(\mathcal{H},P_{0})}=\sum_{i=1}^{m}\int_{\mathcal{Z}}h_{i}(z)^{T}h_{i}^{\prime}(z)P_{0}(\mathrm{d}z)$
which induces the weighted $L^{2}$ norm
$\|\cdot\|_{L^{2}(\mathcal{H},P_{0})}:h\rightarrow\left(\int_{\mathcal{Z}}\|h(z)\|^{2}_{\mathbb{R}^{m}}P_{0}(\mathrm{d}z)\right)^{1/2}$.
Let $h_{j}^{i}\in\mathcal{H}_{i}$ for $j=1,\ldots,\infty$ and let
$\\{h_{j}^{i}\\}_{j=1}^{\infty}$ denote an orthonormal basis of
$\mathcal{H}_{i}$ such that $\langle
h_{i}^{k},h_{j}^{l}\rangle=\delta_{ij}\delta_{kl}$. Consider any
$\theta\in\Theta$ with $0<\|\theta\|<\infty$, then

$\displaystyle\theta^{T}\Sigma_{0}\theta=$ $\displaystyle\left\langle
E[\theta^{T}\nabla_{\theta}\Psi(X,Z;\theta_{0})],E[\theta^{T}\nabla_{\theta}\Psi(X,Z;\theta_{0})]\right\rangle_{\mathcal{H}}$
(53) $\displaystyle=$ $\displaystyle\left\langle
E[\theta^{T}\nabla_{\theta}\Psi(X,Z;\theta_{0})],\left(\sum_{i=1}^{m}\sum_{j=1}^{\infty}h_{j}^{i}\left(h_{j}^{i}\right)^{T}\right)E[\theta^{T}\nabla_{\theta}\Psi(X,Z;\theta_{0})]\right\rangle_{\mathcal{H}}$
(54) $\displaystyle=$
$\displaystyle\sum_{i=1}^{m}\sum_{j=1}^{\infty}E[\theta^{T}\nabla_{\theta}\psi_{i}(X;\theta_{0})h^{i}_{j}(Z)]^{2}$
(55) $\displaystyle=$
$\displaystyle\sum_{i=1}^{m}\sum_{j=1}^{\infty}\left(\int_{\mathcal{Z}}\theta^{T}D^{i}_{0}(z)h^{i}_{j}(z)P_{0}(\mathrm{d}z)\right)^{2}$
(56)

where
$D_{0}^{i}(z)=E[\nabla_{\theta}\psi_{i}(X;\theta_{0})|Z=z]\in\mathbb{R}^{p}$
denotes the columns of
$D_{0}(z)=E[\nabla_{\theta}\psi(X;\theta_{0})|Z=z]\in\mathbb{R}^{p\times m}$.
Now as $\operatorname{rank}(D_{0}(Z))=p$ w.p.1 by Assumption j), for any
$\theta\in\Theta$ with $0<\|\theta\|<\infty$ there exists
$i\in\\{1,\ldots,m\\}$ such that $\theta^{T}D_{0}^{i}(Z)\neq 0$ w.p.1.
Moreover, Assumption i) we have
$E[\|\nabla_{\theta}\psi(X;\theta_{0})\|^{2}|Z]<\infty$ w.p.1 which implies
$\int_{\mathcal{Z}}\|D_{0}^{i}(z)\|^{2}P_{0}(z)<\infty$ and thus $D_{0}^{i}\in
L^{2}(P_{0},\mathcal{H})$ and by linearity $\theta^{T}D_{0}^{i}\in
L^{2}(P_{0},\mathcal{H})$. Therefore, we can expand $\theta^{T}D_{0}^{i}$ in
terms of basis functions $\\{h_{k}^{i}\\}_{k=1}^{\infty}$ of
$L^{2}(P_{0},\mathcal{H}_{i})$ as
$\theta^{T}D_{0}^{i}(z)=\sum_{k=1}^{\infty}\alpha_{k}h_{k}^{i}(z)$ with
$\alpha_{k}\in\mathbb{R}$. Inserting this back into (56) we get

$\displaystyle\theta^{T}\Sigma_{0}\theta=$
$\displaystyle\sum_{i=1}^{m}\int_{\mathbb{R}}\left(\int_{\mathcal{X}}\theta^{T}D^{i}_{0}(z)h^{i}_{\tau}(z)P_{0}(\mathrm{d}z)\right)^{2}$
(57) $\displaystyle>$
$\displaystyle\sum_{k=1}^{\infty}\left(\int_{\mathcal{X}}\theta^{T}D^{i}_{0}(z)h^{i}_{j}(z)P_{0}(\mathrm{d}z)\right)^{2}$
(58) $\displaystyle=$
$\displaystyle\sum_{j=1}^{\infty}\left(\int_{\mathcal{X}}\sum_{k=1}^{\infty}\alpha_{k}h^{i}_{k}(z)h^{i}_{j}(z)P_{0}(\mathrm{d}z)\right)^{2}$
(59) $\displaystyle=$
$\displaystyle\sum_{j=1}^{\infty}\left(\sum_{k=1}^{\infty}\alpha_{k}\langle
h_{k}^{i},h^{i}_{j}\rangle_{L^{2}(P_{0},\mathcal{H})}\right)^{2}$ (60)
$\displaystyle=$
$\displaystyle\sum_{j=1}^{\infty}\left(\sum_{k=1}^{\infty}\alpha_{k}\delta_{kj}\right)^{2}$
(61) $\displaystyle=$ $\displaystyle\sum_{j=1}^{\infty}\alpha_{j}^{2}>0$ (62)

where the last inequality follows as $\theta^{T}D_{0}^{i}(Z)\neq 0$ w.p.1 so
with probability 1 there exists at least one $j\in\\{1,\ldots,\infty\\}$ with
$\alpha_{j}\neq 0$. From this it follows that $\Sigma_{0}$ is non-singular
with probability 1. ∎

##### Proof of Theorem 3.4

###### Proof.

By definition the function space $\mathcal{H}$ is expressive enough such that
we can express the conditional moment restriction $E[\psi(X;\theta)|Z]=0\
P_{Z}\text{-a.s.}$ in functional form as

$\displaystyle E[\Psi(X,Z;\theta)]=0\in\mathcal{H}^{\ast}.$ (63)

It remains to be shown that the assumptions imposed on $\psi$ are sufficient
for $\Psi$ to fulfill the conditions of Theorem A.1. Assumptions a) and b)
directly translate to the corresponding assumptions in Theorem A.1. Assumption
c) of Theorem 3.4 follows directly from Assumption c) as
$\Psi(X,Z;\theta)(h)=\psi(X;\theta)^{T}h(Z)$ is continuous in $\theta$ for any
$\theta\in\Theta$ if $\psi(X;\theta)$ is continuous in $\theta$ for any
$\theta\in\Theta$. As this holds for any $h\in\mathcal{H}$ continuity of
$\Psi(X,Z;\theta)$ in $\theta$ follows. Assumption d) for Theorem A.1 follows
as

$\displaystyle
E[\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}^{2}]$ (64)
$\displaystyle=E[\sup_{\theta\in\Theta}\sup_{h\in\mathcal{H},\|h\|\leq
1}\|\psi(X;\theta)^{T}h(Z)\|^{2}$ (65) $\displaystyle\leq
E[E[\sup_{\theta\in\Theta}\|\psi(X;\theta)\|_{2}^{2}\
|Z]\sup_{h\in\mathcal{H},\|h\|\leq 1}\|h(Z)\|_{2}^{2}]$ (66)
$\displaystyle\leq C\int_{\mathcal{Z}}\sup_{h\in\mathcal{H},\|h\|\leq
1}\|h(z)\|_{2}^{2}\ P_{0}(\mathrm{d}z)$ (67)

where we used that $E[\sup_{\theta\in\Theta}\|\psi(X;\theta)\|_{2}^{2}|Z]\leq
C$ with probability $1$ by Assumption d). Now for any function
$h\in\mathcal{H}$ with $\|h\|_{L^{2}(P_{0},\mathcal{H})}\leq 1$ we must have
that $h(Z)<\infty$ w.p.1 and thus by the local Lipschitz property it follows
$h(z)\leq M<\infty$ for any $z\in\operatorname{supp}(P_{0})$. As this holds
for any $h\in\mathcal{H}$, in particular it also holds for the supremum over
$\mathcal{H}$ and thus $\sup_{h\in\mathcal{H},\|h\|\leq 1}\|h(z)\|_{2}^{2}\leq
M$ $\forall z\in\mathcal{Z}$. Therefore, we obtain

$\displaystyle
E[\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}^{2}]\leq
CM\int_{\mathcal{Z}}P_{0}(\mathrm{d}z)=CM<\infty.$ (68)

Assumption e) of Theorem A.1 follows from Assumption e) and Lemma E.3.
Assumption f) is identical to the corresponding Assumption f) in Theorem A.1
using the same argument as for Assumption d) for the integrability condition.
Assumption g) of Theorem A.1 is identical to Assumption g). Therefore
Assumptions a)-g) of Theorem A.1 are fulfilled and it follows that
$\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$.

Now further, Assumption h) of Theorem A.1 is identical with Assumption h).
Assumption i) of Theorem A.1 follow from Assumption i) by the same argument
presented earlier for Assumption c) and d) of Theorem A.1. Finally Assumption
j) of Theorem A.1 follows from Assumption j) by Lemma E.4. Therefore,
Assumptions h)-j) of Theorem A.1 are fulfilled and we have
$\|\hat{\theta}-\theta_{0}\|=O_{p}(n^{-1/2})$. ∎

####  E.3.2 Proof of Theorem 3.7 (Asymptotic Normality for CMR)

The asymptotic normality of the KMM estimator for conditional moment
restrictions follows directly from the result for functional moment
restrictions Theorem A.2 using that by Theorem 3.4 the assumptions of Theorem
3.4 are sufficient to satisfy the assumptions of Theorem A.1. What remains to
be shown is that we can translate the asymptotic covariance of the KMM
estimator for functional moment restrictions into an expression containing the
conditional quantities. To this aim, first, we show that we can express the
asymptotic covariance of the KMM estimator for FMR in a variational form
following Lemma 15 of Bennett and Kallus [2020b].

######  Lemma E.5.

Let the assumptions of Lemma E.6 be fulfilled. Then we have

$\displaystyle
E[\nabla_{\theta}\Psi(X,Z;\theta_{0})\Omega_{0}^{-1}\nabla_{\theta}\Psi(X,Z;\theta_{0})]=\sup_{h\in\mathcal{H}}E[\nabla_{\theta}\psi(X;\theta_{0})^{T}h(Z)]-\frac{1}{4}E\left[\left(\psi(X;\theta_{0})^{T}h(Z)\right)^{2}\right]$
(69)

###### Proof.

By Lemma 14 of Bennett and Kallus [2020b] we have for any Hilbert space
$\mathcal{H}$, and element $h\in\mathcal{H}$, that

$\displaystyle\|h\|_{\mathcal{H}}^{2}=\sup_{h^{\prime}\in\mathcal{H}}\langle
h,h^{\prime}\rangle-\frac{1}{4}\|h^{\prime}\|^{2}.$ (70)

Moreover, as the dual space of $\mathcal{H}$, $\mathcal{H}^{\ast}$ is a
Hilbert space itself, we can write for any
$(x,z)\in\mathcal{X}\times\mathcal{Z}$,

$\displaystyle\nabla_{\theta}\Psi(x,z;\theta_{0})\Omega_{0}^{-1}\nabla_{\theta}\Psi(x,z;\theta_{0})=$
$\displaystyle\|\Omega_{0}^{-1/2}\nabla_{\theta}\Psi(x,z;\theta)\|^{2}_{\mathcal{H}^{\ast}}$
(71) $\displaystyle=$
$\displaystyle\sup_{h^{\prime}\in\mathcal{H}^{\ast}}\langle\Omega_{0}^{-1/2}\nabla_{\theta}\Psi(x,z;\theta),h^{\prime}\rangle_{\mathcal{H}^{\ast}}-\frac{1}{4}\|h^{\prime}\|^{2}_{\mathcal{H}^{\ast}}$
(72) $\displaystyle=$
$\displaystyle\sup_{h^{\prime}\in\mathcal{H}^{\ast}}\langle\nabla_{\theta}\Psi(x,z;\theta_{0}),\Omega_{0}^{-1/2}h^{\prime}\rangle_{\mathcal{H}^{\ast}}-\frac{1}{4}\|h^{\prime}\|^{2}_{\mathcal{H}^{\ast}}$
(73) $\displaystyle=$
$\displaystyle\sup_{h^{\prime}\in\operatorname{Range}(\Omega^{-1/2})}\langle\nabla_{\theta}\Psi(x,z;\theta_{0}),h^{\prime}\rangle-\frac{1}{4}\langle\Omega^{1/2}h^{\prime},\Omega^{1/2}h^{\prime}\rangle_{\mathcal{H}^{\ast}}$
(74) $\displaystyle=$
$\displaystyle\sup_{h^{\prime}\in\mathcal{H}^{\ast}}\langle\nabla_{\theta}\Psi(x,z;\theta_{0}),h^{\prime}\rangle-\frac{1}{4}\langle
h^{\prime},\Omega h^{\prime}\rangle_{\mathcal{H}^{\ast}}$ (75)
$\displaystyle=$
$\displaystyle\sup_{h\in\mathcal{H}}\nabla_{\theta}\psi(x;\theta_{0})^{T}h(z)-\frac{1}{4}h(z)^{T}\psi(x;\theta_{0})\psi(x;\theta_{0})^{T}h(z)$
(76)

where we used that
$\operatorname{Range}(\Omega_{0}^{-1/2})=\mathcal{H}^{\ast}$. This follows as
$\Omega_{0}$ is defined on all of $\mathcal{H}$ and invertible which
immediately implies $\Omega_{0}^{1/2}$ is defined on all of $\mathcal{H}$ and
invertible. This means that $\Omega_{0}^{1/2}$ is injective and thus
$\operatorname{Range}(\Omega_{0}^{-1/2})=\mathcal{H}^{\ast}$. The result
follows by taking the expectation over $(x,z)$ on both sides. ∎

With the variational formulation at hand we can translate the expression of
the covariance of the KMM estimator for FMR into an expression for CMR. The
following result is a special case of Lemma 25 of Bennett and Kallus [2020b].

######  Lemma E.6.

For the moment functional $\Psi(x,z;\theta):\mathcal{H}\rightarrow\mathbb{R}$
defined for any $h\in\mathcal{H}$ via
$\Psi(x,z;\theta)(h)=\psi(x;\theta)^{T}h(z)$ define the covariance operator
$\Omega_{0}=E[\Psi(X,Z;\theta_{0})\otimes\Psi(X,Z;\theta_{0})]$ and further
define the conditional covariance matrix
$V_{0}(z)=E[\psi(X;\theta_{0})\psi(X;\theta_{0})^{T}|Z=z]$ as a function
$\mathcal{Z}\rightarrow\mathbb{R}^{m\times m}$ and assume $V_{0}(Z)$ to be
non-singular with probability $1$. Then we have

$\displaystyle
E[\nabla_{\theta}\Psi(X,Z;\theta_{0})\Omega_{0}^{-1}\Psi(X,Z;\theta_{0})]=E\left[E[\nabla_{\theta}\psi(X;\theta_{0})|Z]\
V_{0}^{-1}(Z)\ E[\nabla_{\theta}\psi(X;\theta_{0})|Z]\right]$ (77)

###### Proof.

Using Lemma E.5 we can write

$\displaystyle
E[\nabla_{\theta}\Psi(X,Z;\theta_{0})\Omega_{0}^{-1}\Psi(X,Z;\theta_{0})]=\sup_{h\in\mathcal{H}}E[\nabla_{\theta}\psi(X;\theta_{0})^{T}h(Z)]-\frac{1}{4}E\left[\left(\psi(X;\theta_{0})^{T}h(Z)\right)^{2}\right]=:L(h)$
(78)

The functional derivative of $L$ at $h^{\ast}\in\mathcal{H}$ in direction
$\epsilon\in\mathcal{H}$ is given by

$\displaystyle\left(\frac{\partial}{\partial h}L(h^{\ast})\right)(\epsilon)$
$\displaystyle=E[\nabla_{\theta}\psi(X;\theta_{0})^{T}\epsilon(Z)]-\frac{1}{2}E[\epsilon(Z)^{T}\psi(X;\theta_{0})\psi(X;\theta_{0})^{T}h^{\ast}(Z)]$
(79)
$\displaystyle=E\left[\epsilon(Z)^{T}\left(\nabla_{\theta}\psi(X;\theta_{0})-\frac{1}{2}\psi(X;\theta_{0})\psi(X;\theta_{0})^{T}h^{\ast}(Z)\right)\right]$
(80)
$\displaystyle=E\left[\epsilon(Z)^{T}\left(E[\nabla_{\theta}\psi(X;\theta_{0})|Z]-\frac{1}{2}V_{0}(Z)h^{\ast}(Z)\right)\right]$
(81)

Now, by assumption $V_{0}(Z)$ is non-singular and thus invertible, moreover
$L$ is a concave functional in $h$ and thus the global maximizer is given for
any $z\in\mathcal{Z}$ by

$\displaystyle
h^{\ast}(z)=2V_{0}(z)^{-1}E[\nabla_{\theta}\psi(X;\theta_{0})|Z=z].$ (82)

Inserting back into equation (78) and denoting
$D_{0}(z):=E[\nabla_{\theta}\psi(X;\theta_{0})|Z=z]$ we have

$\displaystyle
E[\nabla_{\theta}\Psi(X,Z;\theta_{0})\Omega_{0}^{-1}\Psi(X,Z;\theta_{0})]$
$\displaystyle=2E[D_{0}(Z)^{T}V_{0}(Z)^{-1}D_{0}(Z)]-E[D_{0}(Z)^{T}V_{0}(Z)^{-1}V_{0}(Z)V_{0}(Z)^{-1}D_{0}(Z)]$
(83) $\displaystyle=E[D_{0}(Z)^{T}V_{0}(Z)^{-1}D_{0}(Z)].$ (84)

∎

##### Proof of Theorem 3.7

###### Proof.

The conditions of Theorem A.2 are fulfilled by the conditions of Theorem 3.7
by the proof of Theorem 3.4. We can translate the expression for the
asymptotic variance in terms of the moment functional into the conditional
counterpart by applying Lemma E.6 whose conditions are fulfilled by Assumption
e) of Theorem 3.4. ∎

####  E.3.3 Proof of Corollary 3.8 (Efficiency for CMR)

###### Proof.

This is a direct implication of Theorem 3.7 as the asymptotic variance of the
KMM estimator achieves the semi-parametric efficiency bound of Chamberlain
[1987]. ∎

###  E.4 Asymptotic Properties of KMM for Functional Moment Restrictions

The consistency proofs roughly follow the general idea laid out in the seminal
paper by Newey and Smith [2004] with the adaption to functional moment
restrictions by Kremer et al. [2022]. The proof for the finite dimensional
case is mostly a special case of the proof of the functional version.
Therefore, we provide a detailed proof for the arguably more interesting
functional case and a short version for the finite dimensional case,
emphasizing the differences to the former.

####  E.4.1 Proof of Theorem A.1

######  Lemma E.7.

Let $\mathcal{A}$ denote a $\sigma$-algebra on $\mathcal{X}\times\mathcal{Z}$
and let ($\mathcal{X}\times\mathcal{Z},\mathcal{A},\omega)$ be a probability
space with measure $\omega$. For any functional
$\Psi:(\mathcal{X}\times\mathcal{Z})\times\Theta\times\mathcal{H}\rightarrow\mathbb{R}$
with
$\int_{\mathcal{X}\times\mathcal{Z}}{\sup_{\theta\in\Theta}\|\Psi(x,z;\theta)\|_{\mathcal{H}^{\ast}}^{2}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)<\infty$, it follows that
$\sup_{\theta\in\Theta}\left\|\Psi\left(X,Z;\theta\right)\right\|_{\mathcal{H}^{\ast}}\leq
C\ \omega\text{-a.s.}$ for some constant $C<\infty$.

###### Proof.

The proof is trivially implied by the definition of the almost surely
property. If the event
$\mathcal{E}=\left\\{(x,z)\in\mathcal{X}\times\mathcal{Z}\ :\
\sup_{\theta\in\Theta}\left\|\Psi\left(x,z;\theta\right)\right\|_{\mathcal{H}^{\ast}}=\infty\right\\}$
has non-zero measure, i.e., $\omega\left[\mathcal{E}\right]\neq 0$, then
$\int_{\mathcal{X}\times\mathcal{Z}}{\sup_{\theta\in\Theta}\|\Psi(x,z;\theta)\|_{\mathcal{H}^{\ast}}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)=\infty$ and thus
$\int_{\mathcal{X}\times\mathcal{Z}}{\sup_{\theta\in\Theta}\|\Psi(x,z;\theta)\|_{\mathcal{H}^{\ast}}^{2}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)=\infty$. Therefore we must have
$\omega[\mathcal{E}]=0$ and there exists some constant $C$ such that
$\sup_{\theta\in\Theta}\left\|\Psi\left(X,Z;\theta\right)\right\|_{\mathcal{H}^{\ast}}\leq
C\ \omega\text{-a.s.}$. ∎

######  Lemma E.8.

For two distributions $Q_{1}$ and $Q_{2}$ on $\mathcal{X}\times\mathcal{Y}$
define the mixing distribution $\omega=(1-\alpha)Q_{1}+\alpha Q_{2}$, with
$\alpha=O_{p}(n^{-\zeta})$ and $\zeta>0$. Then
$\operatorname*{MMD}(Q_{1},\omega;\mathcal{F})=O_{p}(n^{-\zeta})$ for any RKHS
$\mathcal{F}$ of functions
$\mathcal{X}\times\mathcal{Z}\rightarrow\mathbb{R}$. In particular it follows
for any distribution $Q$ and $\omega=(1-\alpha)\hat{P}_{n}+\alpha Q$ that
$\operatorname*{MMD}(\hat{P}_{n},\omega;\mathcal{F})=O_{p}(n^{-\zeta})$.

###### Proof.

The proof follows directly by using the definition of MMD,

$\displaystyle\operatorname*{MMD}(Q_{1},\omega;\mathcal{F})=$
$\displaystyle\sup_{f\in\mathcal{F},\|f\|_{\mathcal{F}}=1}\left(\int_{\mathcal{X}\times\mathcal{Z}}f(x,z)Q_{1}(\mathrm{d}x\otimes\mathrm{d}z)-\int_{\mathcal{X}\times\mathcal{Z}}{f(x,z)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)\right)$ (85) $\displaystyle=$
$\displaystyle\alpha\sup_{f\in\mathcal{F},\|f\|_{\mathcal{F}}=1}\left(\int_{\mathcal{X}\times\mathcal{Z}}f(x,z)Q_{1}(\mathrm{d}x\otimes\mathrm{d}z)-\int_{\mathcal{X}\times\mathcal{Z}}f(x,z)Q_{2}(\mathrm{d}x\otimes\mathrm{d}z)\right)$
(86) $\displaystyle=$
$\displaystyle\alpha\operatorname*{MMD}(Q_{1},Q_{2};\mathcal{F})$ (87)
$\displaystyle=$ $\displaystyle\alpha C$ (88) $\displaystyle=$ $\displaystyle
O_{p}(n^{-\zeta}),$ (89)

where we used that $\operatorname{MMD}(Q_{1},Q_{2};\mathcal{F})$ can be
bounded by some positive constant $C$ for any $Q_{1}$, $Q_{2}$ and
$\mathcal{F}$ which directly follows from the fact that by definition of an
RKHS the evaluation functional in $\mathcal{F}$ is bounded and $Q_{1}$,
$Q_{2}$ are finite measures normalized to $1$. The second statement is a
direct application of the former. ∎

######  Lemma E.9.

Let the assumptions of Theorem A.1 be satisfied, then for any $\zeta$ with
$0<\zeta<1/2$ define the magnitude constrained set of dual variables
$\mathcal{M}_{n}=\\{\beta=(\eta,f,h)\in\mathcal{M}:\|\beta\|_{\mathcal{M}}\leq
n^{-\zeta}\\}$. Then as $n\rightarrow\infty$ with probability approaching 1,

$\displaystyle\sup_{\theta\in\Theta,\
(\eta,f,h)\in\mathcal{M}_{n}}|\Psi(X,Z;\theta)(h)|=0\ \omega\text{-a.s.},$
(90) $\displaystyle\sup_{\theta\in\Theta,\
\beta\in\mathcal{M}_{n}}|a(X,Z;\theta)^{T}\beta|=0\ \omega\text{-a.s.}.$ (91)

###### Proof.

Using the Cauchy-Schwarz inequality and Assumption d) with Lemma E.7,

$\displaystyle\sup_{\theta\in\Theta,\beta\in\mathcal{M}_{n}}|\Psi(X,Z;\theta)(h)|$
(92) $\displaystyle\leq$
$\displaystyle\sup_{\theta\in\Theta,\beta\in\mathcal{M}_{n}}\left(\|h\|_{\mathcal{H}}\cdot\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}\right)$
(93) $\displaystyle\leq$
$\displaystyle\sup_{\theta\in\Theta,\beta\in\mathcal{M}_{n}}\left(\|\beta\|_{\mathcal{M}}\cdot\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}\right)$
(94) $\displaystyle\leq$ $\displaystyle
n^{-\zeta}\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}.$
(95)

Now, by Assumptions d) and f) of Theorem A.1 and Lemma E.7 we have that
$\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}<C\
P_{0}\text{-a.s.}$ and
$\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}<C\
Q\text{-a.s.}$ respectively and as $\omega\rightarrow P_{0}$ weakly we have
w.p.a.1 that
$\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}<C\ \
\omega\text{-a.s.}$ and thus w.p.a.1
$\sup_{\theta\in\Theta,\beta\in\mathcal{M}_{n}}|\Psi(X,Z;\theta)(h)|=0\
\omega\text{-a.s.}$.

For the second part note that if $\|\beta\|_{\mathcal{M}}\leq n^{-\zeta}$, we
must have that $|\eta|,\|f\|_{\mathcal{F}},\|h\|_{\mathcal{H}}\leq
n^{-\zeta}$. Then we have

$\displaystyle\sup_{\theta\in\Theta,\
\beta\in\mathcal{M}_{n}}|a(X,Z;\theta)^{T}\beta|$ (96) $\displaystyle=$
$\displaystyle\sup_{\theta\in\Theta,\
(\eta,f,h)\in\mathcal{M}_{n}}|\eta+\langle
k((X,Z),\cdot),f\rangle_{\mathcal{F}}+\Psi(X,Z;\theta)(h)|$ (97)
$\displaystyle\leq$
$\displaystyle\sup_{(\eta,f,h)\in\mathcal{M}_{n}}|\eta|+\sup_{(\eta,f,h)\in\mathcal{M}_{n}}|\langle
k((X,Z),\cdot),f\rangle_{\mathcal{F}}|+\sup_{\theta\in\Theta,\beta\in\mathcal{M}_{n}}|\Psi(X,Z;\theta)(h)|$
(98) $\displaystyle\leq$ $\displaystyle
n^{-\zeta}+\sup_{(\eta,f,h)\in\mathcal{M}_{n}}\|f\|_{\mathcal{F}}\|k((X,Z),\cdot)\|_{\mathcal{F}}+\sup_{\theta\in\Theta,\beta\in\mathcal{M}_{n}}|\Psi(X,Z;\theta)(h)|$
(99) $\displaystyle\leq$ $\displaystyle
n^{-\zeta}+Cn^{-\zeta}+\sup_{\theta\in\Theta,\beta\in\mathcal{M}_{n}}|\Psi(X,Z;\theta)(h)|$
(100) $\displaystyle\leq$
$\displaystyle\left(n^{-\zeta}+Cn^{-\zeta}+Cn^{-\zeta}\right)\
\omega\text{-a.s.}$ (101) $\displaystyle\leq$ $\displaystyle O(n^{-\zeta})\
\omega\text{-a.s.},$ (102)

where for the second term in the fourth line we applied the Cauchy-Schwarz
inequality and used the fact that in an RKHS the evaluation function is
bounded by some constant $C>0$. ∎

######  Lemma E.10.

Under the assumptions of Theorem A.1 we have for any $\theta\in\Theta$,

$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}{a(x,z;\theta)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)=\frac{1}{n}\sum_{i=1}^{n}a(x_{i},z_{i};\theta)+O_{p}(n^{-1})$
(103)

and for any $\beta\in\mathcal{M}$ with $\|\beta\|_{\mathcal{M}}<\infty$,

$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}{a(x,z;\theta)a(x,z;\theta)^{T}\beta}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)=\frac{1}{n}\sum_{i=1}^{n}a(x_{i},z_{i};\theta)a(x_{i},z_{i};\theta)^{T}\beta+O_{p}(n^{-1}).$
(104)

###### Proof.

For the first statement note that

$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}{a(x,z;\theta)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$
$\displaystyle=\int_{\mathcal{X}\times\mathcal{Z}}{a(x,z;\theta)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+\int
a(x,z;\theta)\mathrm{d}\hat{P}_{n}-\int a(x,z;\theta)\mathrm{d}\hat{P}_{n}$
(105) $\displaystyle=\frac{1}{n}\sum_{i=1}^{n}a(x_{i},z_{i};\theta)+\alpha\int
a(x,z;\theta)(\mathrm{d}Q-\mathrm{d}\hat{P}_{n})$ (106)

Now
$a(x,z;\theta_{0})=\left(1,k\left((x,z),\cdot\right),\Psi(x,z;\theta)\right)^{T}$
is trivially integrable in the first component and second component with
respect to any probability distribution as the evaluation functional
$k((x,z),\cdot)$ in $\mathcal{F}$ is bounded by definition of an RKHS. For the
third component integrability with respect to $Q$ follows by Assumption f) of
Theorem A.1. Moreover, by Assumption d) of Theorem A.1 and Lemma E.7 we have
$\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|\leq C$ w.p.1 with respect to
$P_{0}$. And thus as $\hat{P}_{n}\overset{p}{\rightarrow}P_{0}$ weakly, we
have
$\int\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|\hat{P}_{n}(\mathrm{d}x\otimes\mathrm{d}z)<\infty$
w.p.a.1. In conclusion we have $\int
a(x,z;\theta)(\mathrm{d}Q-\mathrm{d}\hat{P}_{n})<\infty$ w.p.a.1 and as
$\alpha=O_{p}(n^{-1})$ we finally get

$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}{a(x,z;\theta)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)=\frac{1}{n}\sum_{i=1}^{n}a(x_{i},z_{i};\theta)+O_{p}(n^{-1}).$
(107)

For the second statement consider any $\beta\in\mathcal{M}$ with
$\|\beta\|_{\mathcal{M}}<\infty$,

$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}{a(x,z;\theta)a(x,z;\theta)^{T}\beta}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)=$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}a(x_{i},z_{i};\theta)a(x_{i},z_{i};\theta)^{T}\beta$
(108)
$\displaystyle+\alpha\int_{\mathcal{X}\times\mathcal{Z}}a(x,z;\theta)a(x,z;\theta)^{T}\beta\left(\mathrm{d}Q-\mathrm{d}\hat{P}_{n}\right).$
(109)

Now for the second term we have

$\displaystyle\|\int_{\mathcal{X}\times\mathcal{Z}}a(x,z;\theta)a(x,z;\theta)^{T}\beta\left(\mathrm{d}Q-\mathrm{d}\hat{P}_{n}\right)\|_{\mathcal{M}^{\ast}}^{2}$
(110) $\displaystyle\leq$
$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}\|a(x,z;\theta)a(x,z;\theta)^{T}\beta\|^{2}\left(\mathrm{d}Q+\mathrm{d}\hat{P}_{n}\right)$
(111) $\displaystyle=$
$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}|a(x,z;\theta)^{T}\beta|^{2}\|a(x,z;\theta)\|^{2}\left(\mathrm{d}Q+\mathrm{d}\hat{P}_{n}\right)$
(112) $\displaystyle\leq$
$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}|a(x,z;\theta)^{T}\beta|^{2}\left(\mathrm{d}Q+\mathrm{d}\hat{P}_{n}\right)\int_{\mathcal{X}\times\mathcal{Z}}\|a(x,z;\theta)\|^{2}\left(\mathrm{d}Q+\mathrm{d}\hat{P}_{n}\right)$
(113) $\displaystyle\leq$
$\displaystyle\|\beta\|_{\mathcal{M}}^{2}\left(\int_{\mathcal{X}\times\mathcal{Z}}\|a(x,z;\theta)\|^{2}\left(\mathrm{d}Q+\mathrm{d}\hat{P}_{n}\right)\right)^{2}$
(114) $\displaystyle\leq$
$\displaystyle\|\beta\|_{\mathcal{M}}^{2}\left(\int_{\mathcal{X}\times\mathcal{Z}}1+\|k((x,z),\cdot)\|_{\mathcal{F}}^{2}+\|\Psi(x,z;\theta)\|_{\mathcal{H}^{\ast}}^{2}\left(\mathrm{d}Q+\mathrm{d}\hat{P}_{n}\right)\right)^{2}.$
(115)

The first term is trivially bounded, the second bounded as $\mathcal{F}$ is an
RKHS and thus its evaluation functional $k((x,z),\cdot)$ is bounded. The third
term is bounded as
$E_{Q}[\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}^{2}<\infty$
by Assumption f) of Theorem A.1 and
$E_{\hat{P}_{n}}[\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}^{2}<\infty$
w.p.a.1 as
$E[\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}^{2}<\infty$
by Assumption d) of Theorem A.1 and $\hat{P}_{n}\rightarrow P_{0}$ weakly. In
conclusion the norm of the integral in equation (109) is bounded by some
constant $C$ and as $\alpha=O_{p}(n^{-1})$ the statement follows. ∎

######  Lemma E.11.

Let the assumptions of Theorem A.1 be satisfied and consider
$\bar{\theta}\in\Theta$ such that
$\bar{\theta}\overset{p}{\rightarrow}\theta_{0}$. Further let
$\beta_{\zeta}:=\operatorname*{arg\,max}_{\beta\in\mathcal{M}_{n}}\widehat{G}(\bar{\theta},\beta)$,
where $\mathcal{M}_{n}=\\{\beta\in\mathcal{M}:\|\beta\|_{\mathcal{M}}\leq
n^{-\zeta}\\}$ with $0<\zeta<1/2$. Define the operator
$\Lambda_{n}(\beta,\theta):\mathcal{M}\rightarrow\mathcal{M}$ as

$\displaystyle\Lambda_{n}(\beta,\theta):=\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}a(x,z;\theta)a(x,z;\theta)^{T}\varphi^{\ast}_{2}\left(\frac{1}{\epsilon}a(x,z;\theta)^{T}\beta\right)}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+R_{\lambda_{n}}.$ (116)

Then w.p.a.1 for any
$\bar{\beta}\in\operatorname{conv}(\\{0,\beta_{\zeta}\\})$,
$\Lambda_{n}(\bar{\beta},\bar{\theta})$ is strictly positive definite and its
smallest eigenvalue is bounded away from zero. Moreover, for any
$\theta\in\Theta$ the largest eigenvalue of $\Lambda_{n}(\bar{\beta},\theta)$
is bounded from above by a positive constant $M$.

###### Proof.

As $\bar{\beta}\in\operatorname{conv}(\\{0,\beta_{\zeta}\\})$ we have
$\bar{\beta}\in\mathcal{M}_{n}$, and hence Lemma E.9 implies that w.p.a.1
$\sup_{\theta\in\Theta}|a(x,z;\theta)^{T}\bar{\beta}|=0\ \omega\text{-a.s.}$,
which implies for every fixed value of $\epsilon>0$,
$\varphi_{2}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\theta)^{T}\bar{\beta}\right)\overset{p}{\rightarrow}\varphi_{2}(0)=1\
\omega\text{-a.s.}$ by the continuous mapping theorem. This means that for
every value of $(x,z)$ that provides a non-vanishing contribution to the
integral, we have
$\varphi_{2}^{\ast}\left(\frac{1}{\epsilon}a(x,z;\theta)^{T}\bar{\beta}\right)\overset{p}{\rightarrow}1$
and so as $n\rightarrow\infty$ the first term is equivalent to
$\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}a(x,z;\theta)a(x,z;\theta)^{T}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$ which clearly is a positive semi-
definite operator. In the following we will show that its smallest eigenvalue
is bounded away from zero w.p.a.1. First note that for any vector
$\beta=(\eta,f,h)\in\mathcal{M}$ with $f\neq 0$ we have

$\displaystyle\beta^{T}\Lambda_{n}\beta$
$\displaystyle=\int_{\mathcal{X}\times\mathcal{Z}}{\underbrace{(a(x,z;\theta)^{T}\beta)^{2}}_{\geq
0}}\ \omega(\mathrm{d}x\otimes\mathrm{d}z)+\|f\|^{2}+\lambda_{n}\|h\|^{2}$
(117) $\displaystyle\geq\|f\|^{2}+\lambda_{n}\|h\|^{2}$ (118)
$\displaystyle>0,$ (119)

and thus such vector cannot correspond to an eigenvalue of $0$. Therefore
consider any vector $\beta=(\eta,0,h)\in\mathcal{M}$, then if such vector
corresponds to an eigenvalue of zero we must have

$\displaystyle\begin{pmatrix}0\\\ 0\\\ 0\end{pmatrix}=$
$\displaystyle\Lambda_{n}(\bar{\beta},\bar{\theta})\begin{pmatrix}\eta\\\ 0\\\
h\end{pmatrix}$ (120) $\displaystyle=$
$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}a(x,z;\bar{\theta})a(x,z;\bar{\theta})^{T}\begin{pmatrix}\eta\\\
0\\\ h\end{pmatrix}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)+\begin{pmatrix}0&0&0\\\ 0&I&0\\\
0&0&\lambda_{n}I\end{pmatrix}\begin{pmatrix}\eta\\\ 0\\\ h\end{pmatrix}$ (121)
$\displaystyle=$
$\displaystyle\int_{\mathcal{X}\times\mathcal{Z}}{\frac{1}{\epsilon}\begin{pmatrix}\eta-\Psi(x,z;\bar{\theta})(h)\\\
k((x,z),\cdot)\eta-k((x,z),\cdot)\Psi(x,z;\bar{\theta})(h)\\\
-\eta\Psi(x,z;\bar{\theta})+\Psi(x,z;\bar{\theta})\Psi(x,z;\bar{\theta})(h)+\lambda_{n}h\end{pmatrix}}\
\omega(\mathrm{d}x\otimes\mathrm{d}z)$ (122) $\displaystyle=$
$\displaystyle\frac{1}{\epsilon}\frac{1}{n}\sum_{i=1}^{n}\begin{pmatrix}\eta-\Psi(x_{i},z_{i};\bar{\theta})(h)\\\
k((x_{i},z_{i}),\cdot)\eta-k((x_{i},z_{i}),\cdot)\Psi(x_{i},z_{i};\bar{\theta})(h)\\\
-\eta\Psi(x_{i},z_{i};\bar{\theta})+\Psi(x_{i},z_{i};\bar{\theta})\Psi(x_{i},z_{i};\bar{\theta})(h)+\lambda_{n}h\end{pmatrix}+O_{p}(n^{-1}),$
(123)

where we used Lemma E.10 to express the integral term in terms of the
empirical average. Now the first row gives
$\eta=E_{\hat{P}_{n}}[\Psi(X,Z;\bar{\theta})(h)]+O_{p}(n^{-1})$ which inserted
in the last row gives

$\displaystyle
0=\left(\underbrace{E_{\hat{P}_{n}}[\Psi(x_{i},z_{i};\bar{\theta})\otimes\Psi(x_{i},z_{i};\bar{\theta})]-E_{\hat{P}_{n}}[\Psi(x_{i},z_{i};\bar{\theta})]\otimes
E_{\hat{P}_{n}}[\Psi(x_{i},z_{i};\bar{\theta})]}_{=:\hat{\Omega}(\bar{\theta})}+\lambda_{n}\right)h+O_{p}(n^{-1}).$
(124)

As $n\rightarrow\infty$ we have $\bar{\theta}\rightarrow\theta_{0}$ and by the
uniform weak law of large numbers and the continuous mapping theorem (as
$\Psi$ is continuous in $\theta$ by Assumption c) of Theorem A.1)
$\hat{\Omega}(\bar{\theta})\rightarrow
E[\Psi(X,Z;\theta_{0})\otimes\Psi(X,Z;\theta_{0})]-E[\Psi(X,Z;\theta_{0})]\otimes
E[\Psi(X,Z;\theta_{0})]=E[\Psi(X,Z;\theta_{0})\otimes\Psi(X,Z;\theta_{0})]=\Omega_{0}$.
Thus as $n\rightarrow\infty$ we have

$\displaystyle 0=\left(\Omega_{0}+\lambda_{n}\right)h+O_{p}(n^{-1})$ (125)

From Assumption e) of Theorem A.1 it follows that $\Omega_{0}$ is non-singular
and thus $0$ is not in its spectrum. Moreover, by Assumption g) of Theorem A.1
$\lambda_{n}=O_{p}(n^{-\xi})$ with $0<\xi<1/2$, so the RHS is $\neq 0$ w.p.a.1
and the eigenvalue equations can only be fulfilled with $h=0$ which implies
$\eta=0$ and thus $\beta=0$. Therefore it follows that the smallest eigenvalue
of $\Lambda_{n}(\bar{\beta},\bar{\theta})$ is bounded away from zero w.p.a.1.

In order to bound the largest eigenvalue of $\Lambda_{n}(\bar{\beta},\theta)$
for any $\theta\in\Theta$ recall that for the second term we have