#  Statistical inference for generative adversarial networks††thanks: The
author thanks the Academy of Finland, Foundation for the Advancement of
Finnish Securities Markets, and OP Group Research Foundation for financial
support. Contact address: Mika Meitz, Department of Economics, University of
Helsinki, P. O. Box 17, FI–00014 University of Helsinki, Finland; e-mail:
<EMAIL_ADDRESS>
Mika Meitz
University of Helsinki

(April 2021)

###### Abstract

This paper studies generative adversarial networks (GANs) from a statistical
perspective. A GAN is a popular machine learning method in which the
parameters of two neural networks, a generator and a discriminator, are
estimated to solve a particular minimax problem. This minimax problem
typically has a multitude of solutions and the focus of this paper are the
statistical properties of these solutions. We address two key issues for the
generator and discriminator network parameters, consistent estimation and
confidence sets. We first show that the set of solutions to the sample GAN
problem is a (Hausdorff) consistent estimator of the set of solutions to the
corresponding population GAN problem. We then devise a computationally
intensive procedure to form confidence sets and show that these sets contain
the population GAN solutions with the desired coverage probability. The
assumptions employed in our results are weak and hold in many practical GAN
applications. To the best of our knowledge, this paper provides the first
results on statistical inference for GANs in the empirically relevant case of
multiple solutions.

MSC2020 subject classifications: Primary 62F12; secondary 68T07.

Keywords: Generative adversarial network, GAN, machine learning, minimax
problem, multiple solutions, statistical inference, consistent estimation,
confidence set.

##  1 Introduction

A generative adversarial network (GAN) is a machine learning method introduced
in Goodfellow et al. (2014). The basic purpose of a GAN is to learn how to
generate synthetic data based on a training set of real-world examples. While
a traditional use of GANs has been to generate authentic-looking photographs
based on real example images (for a recent example, see Karras et al., 2020),
they are now in use across various scientific disciplines. To give brief
examples, variants of GANs have been used in biology to generate synthetic
protein-coding DNA sequences (Gupta and Zou, 2019), in physics to simulate
subatomic particle collisions at the Large Hadron Collider (Paganini et al.,
2018), in astronomy to de-noise images of galaxies (Schawinski et al., 2017),
and in medicine to improve near-infrared fluorescence imaging (Ma et al.,
2021). GANs have also been used in popular culture and arts to re-create video
games (Kim et al., 2020), to make computer-generated art (Miller, 2019), and
to compose music (Briot et al., 2020). To put the recent popularity of GANs
into perspective, the original Goodfellow et al. (2014) article received over
10.000 Google Scholar citations in the year 2020 alone. For recent surveys of
GANs and for further references, see Creswell et al. (2018), Pan et al.
(2019), or Goodfellow et al. (2020).

A GAN is typically comprised of two neural networks called a generator and a
discriminator. (For background material on neural networks, see the books of
Bishop (2006) or Goodfellow et al. (2016).) The generator network produces
synthetic data whose distribution aims to mimic that of real data, while the
discriminator network evaluates whether the data produced by the generator is
real or fake. Suppose the observed vector $x$ mathematically represents an
observed image, DNA sequence, or some other object of interest. This $x$ is
viewed as a realization from an underlying random vector $X$ whose
distribution remains unknown to us. Obtaining a large number of realizations
from $X$ may be difficult or costly, and the researcher desires to produce
synthetic replicas in an easy manner. These replicas are the output of a
generating mechanism $G(z,\gamma)$ that in practice is a complicated neural
network. This generator takes as an input noise variables $z$ that are drawn
from some underlying random vector $Z$ and depends on parameters $\gamma$ that
are tuned so that the output $G(z,\gamma)$ would be a close replica of $x$. To
assess the quality of the synthetic data produced, a discriminating mechanism
$D(\cdot,\delta)$ indicates how likely it is an input, whether original data
$x$ or a replica $G(z,\gamma)$, is real data from the underlying distribution
$X$. In practice, the discriminator $D(\cdot,\delta)$ is again a complicated
neural network with parameters $\delta$ to be estimated. The ‘adversarial
principle’ of generative adversarial networks works as follows. Given an
original $x$ and any synthetic $G(z,\gamma)$, the discriminator aims to give a
high rating $D(x,\delta)$ to the real $x$ and a low rating
$D(G(z,\gamma),\delta)$ to the artificial $G(z,\gamma)$ by choosing $\delta$
to maximize the objective function

$D(x,\delta)\times(\thinspace 1-D(G(z,\gamma),\delta)\thinspace)$

for any given fixed $\gamma$ at a time. In contrast, the generator aims to
make $G(z,\gamma)$ as hard to distinguish from $x$ as possible by choosing the
$\gamma$ to minimize the discriminator’s maximized objective function.

To formalize the above discussion, the GAN problem can be expressed as the
minimax problem

(1)
$\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}f(\gamma,\delta)\text{\
\ with\ \
}f(\gamma,\delta)=\mathbb{E}[\ln(D(X,\delta))]+\mathbb{E}[\ln(1-D(G(Z,\gamma),\delta))],$

where the infimum and supremum are taken over sets $\Gamma$ and $\Delta$
denoting the ranges of permissible values for $\gamma$ and $\delta$,
respectively, and $\mathbb{E}$ denotes expectation with respect to the joint
distribution of $X$ and $Z$. The formalities will be discussed in detail in
Section 2. For now it suffices to remark that $X$ and $Z$ are typically of
rather large dimension, that the neural networks $G(\cdot,\cdot)$ and
$D(\cdot,\cdot)$ are in practice quite complicated and parameterized via very
high-dimensional $\gamma$ and $\delta$, and that in GAN applications $f$ is as
a rule non-convex and non-concave (in contrast to the traditional convex-
concave minimax setting in which $f(\cdot,\delta)$ is convex for all fixed
$\delta\in\Delta$ and $f(\gamma,\cdot)$ is concave for all fixed
$\gamma\in\Gamma$). Given solutions, say $\gamma_{0}$ and $\delta_{0}$, to
problem (1), especially $\gamma_{0}$ is of interest as $G(Z,\gamma_{0})$ gives
the researcher a mechanism to produce the desired synthetic data. This
description of GANs corresponds to the original formulation in Goodfellow et
al. (2014). The original GAN and its numerous extensions and variants have in
recent years attracted remarkable interest in applications in which having
access to large quantities of synthetic data is beneficial. Examples of such
applications were mentioned above, and the surveys listed above contain
further references as well as details of the many extensions of GANs.

In practical GAN applications, one would use a large training sample of
observations to estimate the GAN. Let the $n$ observations correspond to
independent and identically distributed (IID) random vectors
$X_{1},\ldots,X_{n}$, all distributed as the variable $X$, and suppose the $n$
IID random vectors $Z_{1},\ldots,Z_{n}$ have the same distribution as the
noise variable $Z$. Then the objective is to solve the sample minimax problem
(cf. Goodfellow et al., 2014; Biau et al., 2020)

(2)
$\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}\hat{f}_{n}(\gamma,\delta)\text{\
\ with\ \
}\hat{f}_{n}(\gamma,\delta)=\frac{1}{n}\sum_{i=1}^{n}\ln(D(X_{i},\delta))+\frac{1}{n}\sum_{i=1}^{n}\ln(1-D(G(Z_{i},\gamma),\delta)).$

The primary objective in GAN applications in the machine learning literature
is to find solutions to this sample problem, or to ‘train’ the GAN. In
statistical terminology, this corresponds to estimating the GAN parameters
$\gamma$ and $\delta$. (Of course, choosing the parametric forms of the neural
networks $G(\cdot,\cdot)$ and $D(\cdot,\cdot)$, or specifying the network
architecture, is done before this.) Devising algorithms that solve the sample
minimax problem (2) is very challenging and a large body of machine learning
literature on GANs focuses on this. Discussions of convergence and stability
of various GAN training algorithms can be found for instance in Arjovsky and
Bottou (2017), Nagarajan and Kolter (2017), and Mescheder et al. (2018); more
recent contributions include Diakonikolas et al. (2021), Fiez and Ratliff
(2021), and Mangoubi and Vishnoi (2021). Another theoretical aspect of GANs
studied in the machine learning literature is how well the distribution of
$G(Z,\gamma)$ can approximate the target distribution of $X$ (e.g., Arora et
al., 2017; Liu et al., 2017).

This paper studies GANs from a statistical perspective. We do not address
algorithmic issues and simply assume a method is available for solving the
sample GAN problem (2). The focus of this paper are the statistical properties
of these solutions, say $\hat{\gamma}_{n}$ and $\hat{\delta}_{n}$, as
estimators of the solutions $\gamma_{0}$ and $\delta_{0}$ to the population
problem (1). Such questions of statistical inference are quite orthogonal to
much of the machine learning literature on GANs as in most GAN applications
the ability to produce synthetic data is the ultimate goal. Our study can be
seen as complementary to these existing works and aims to contribute to a
better understanding of the statistical properties of GANs.

The study of statistical inference in GANs was initiated in the recent
important paper of Biau et al. (2020). These authors were the first to
consider the consistency and asymptotic normality of the sample GAN solutions
$\hat{\gamma}_{n}$ and $\hat{\delta}_{n}$ as estimators of the population GAN
solutions $\gamma_{0}$ and $\delta_{0}$. A key assumption these authors make
is that the population GAN problem (1) has a single, unique solution
$(\gamma_{0},\delta_{0})$. However, as Biau et al. (2020, pp. 1560–1561)
acknowledge, this assumption is unrealistic in practical GAN applications. On
the contrary, as the objective function $f(\gamma,\delta)$ in (1) is
parameterized using complicated neural networks $G$ and $D$, it is essentially
guaranteed to have an extremely large number of solutions (for related
discussion, see Goodfellow et al., 2016, Sec 8.2). One reason for this
prevalence of multiple solutions is the inherent non-identifiability of
heavily parameterized neural networks.

In this paper, we consider statistical inference for GANs in the empirically
relevant case of multiple solutions. We focus on two key issues, consistent
estimation and confidence sets, and the assumptions we make are weak and hold
in many real GAN applications. To briefly describe our results, let
$\Theta_{0}$ and $\hat{\Theta}_{n}$ denote the sets of solutions to the
population minimax problem (1) and the sample GAN problem (2), respectively.
We first consider an appropriately defined notion of consistency for the set-
valued estimator $\hat{\Theta}_{n}$, namely Hausdorff consistency (precise
definition will be given in Section 3). Without any restrictions on the
(potentially infinite) number of solutions, we show that $\hat{\Theta}_{n}$ is
a Hausdorff consistent estimator of $\Theta_{0}$. We then consider confidence
sets, random sets that contain $\Theta_{0}$ with a prespecified coverage
probability. In the traditional point-identified setting in which $\Theta_{0}$
is a singleton, confidence sets would often be formed based on the asymptotic
distribution of an estimator of the parameters of interest. In the present
set-identified case, we follow an alternative approach. We devise a
computationally intensive resampling procedure based on appropriate lower
contour sets of a particular criterion function to form the confidence sets,
and show that these sets contain the population GAN solutions with (at least)
the desired coverage probability. The details will be given in Section 4.

The theoretical developments in this paper build on two strands of literature.
Our results for consistent estimation and for constructing confidence sets are
based on recent developments in estimation and inference for general set-
identified parameters in the partial identification literature in
econometrics; our approach relies in particular on the pioneering work of
Chernozhukov et al. (2007) and Romano and Shaikh (2010) (for further related
references, see the recent survey of this literature in Molinari, 2020). The
research in the partial identification literature does not, however, consider
minimax problems such as (1) or GANs. General minimax problems (but no GANs)
have previously been considered in the stochastic programming literature in
operations research. Our results are particularly closely related to those in
Shapiro (2008) and Shapiro et al. (2009, Sec 5.1.4) who consider consistent
estimation (but no confidence sets) in minimax problems, as well as to the
foundational results for minimization problems in Shapiro (1991). Further
comparison to all these previous works will be given later in the paper.

The present paper contributes to the literature by providing the first results
for statistical inference in GANs in the practically relevant case of multiple
solutions. Our results may also be of interest in general minimax problems,
not just in GANs. As a technical device in our proofs, we establish the
Hadamard directional differentiability of certain mappings related to GANs and
minimax, and these results may be of independent interest (see Appendix B).

The rest of the paper is organized as follows. The next section sets the stage
by considering the GAN problem more formally. Consistent estimation of
$\Theta_{0}$ is the topic of Section 3, while Section 4 discusses confidence
sets for $\Theta_{0}$. Section 5 concludes. All technical derivations and
proofs are relegated to Appendices A–C.

##  2 The GAN problem

We now consider the GAN problem outlined in the Introduction more formally. In
what follows, all the random quantities are defined on some appropriate
underlying probability space but typically there is no need to emphasize this.
The available data corresponds to a random vector $X$ taking values in some
Euclidean subset $\mathcal{X}\subseteq\mathbb{R}^{d_{X}}$ and whose
distribution remains unknown. The noise variables $Z$ with values in
$\mathcal{Z}\subseteq\mathbb{R}^{d_{Z}}$ come from a distribution chosen by
the researcher, often the multivariate uniform or Gaussian distribution. The
generator function $G(\cdot,\cdot)$ is a neural network that transforms the
noise variables $Z$ into synthetic data $G(Z,\gamma)$. The discriminator
function $D(\cdot,\cdot)$ is another neural network indicating how likely it
is that a real observation $x$ or a replica $G(z,\gamma)$ comes from the
distribution of $X$. The following assumption summarizes these concepts from a
technical point of view.

######  Assumption GAN.

  * (a)

Suppose $X_{1},\ldots,X_{n}$ and $Z_{1},\ldots,Z_{n}$ are independent and
identically distributed random vectors with the same distributions as $X$ and
$Z$ and taking values in $\mathcal{X}\subseteq\mathbb{R}^{d_{X}}$ and
$\mathcal{Z}\subseteq\mathbb{R}^{d_{Z}}$, respectively.

  * (b)

The set $\Theta=\Gamma\times\Delta\subseteq\mathbb{R}^{d_{\gamma}+d_{\delta}}$
is compact and non-empty.

  * (c)

The generator function $G:\mathcal{Z}\times\Gamma\to\mathcal{X}$, the
discriminator function $D:\mathcal{X}\times\Delta\to(0,1)$, and the function
$F:\mathcal{X}\times\mathcal{Z}\times\Gamma\times\Delta\to\mathbb{R}$ defined
by

$F(x,z,\gamma,\delta)=\ln(D(x,\delta))+\ln(1-D(G(z,\gamma),\delta))$

are such that $F(X,Z,\gamma,\delta)$ is measurable for all
$(\gamma,\delta)\in\Theta$ and continuous on $\Theta$ with probability one.

This assumption contains some minimal requirements for the original GAN
framework of Goodfellow et al. (2014). The IID assumption in part (a) is
standard in the GAN setting, and the same holds for the compactness assumption
for the permissible parameter space in part (b). Part (c) is a minimal
continuity assumption. Similar assumptions have been used for instance by Biau
et al. (2020). Note also that the GAN formulation allows for misspecification:
It is not assumed that $G(Z,\gamma)$ would for some $\gamma$ have the same
distribution as $X$ does.

Descriptions of neural network architectures that can be used to specify the
generator $G$ and the discriminator $D$ can be found in the book of Goodfellow
et al. (2016) and in the many references therein. Often the generator and
discriminator networks would satisfy additional differentiability and moment
condition assumptions. Although such extra assumptions are not necessarily
required for the results that follow, such a ‘smooth GAN’ setting serves as a
convenient example that will be used to illustrate our results. In the
following assumption and in what follows, we use either notation $\theta$ or
notation $(\gamma,\delta)$ for the elements of $\Theta=\Gamma\times\Delta$;
for instance, $F(x,z,\theta)$ and $F(x,z,\gamma,\delta)$ are used
interchangeably. Also, $\left|\cdot\right|$ denotes the Euclidean norm.

######  Assumption Smooth GAN.

Assumption GAN holds, the function $F(x,z,\theta)$ is continuously
differentiable on $\Theta$ for all $(x,z)\in\mathcal{X}\times\mathcal{Z}$ with
a square integrable derivative$(\mathbb{E}[\sup_{\theta\in\Theta}|\partial
F(X,Z,\theta)/\partial\theta|^{2}]<\infty)$, and
$\inf_{\theta\in\Theta}\operatorname{Var}[F(X,Z,\theta)]>0$.

These conditions require the GAN problem to be somewhat more well-behaved.
Differentiability is a common requirement for training methods employed in GAN
applications (such as variants of the gradient descent-ascent algorithm),
while the mild moment conditions imposed facilitate checking later high-level
assumptions and rule out degenerate cases.

Using the notation in Assumption GAN, we can now restate the GAN minimax
problem (1) from the Introduction as

(3)
$\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}f(\gamma,\delta)\qquad\text{with}\qquad
f(\gamma,\delta)=\mathbb{E}[F(X,Z,\gamma,\delta)].$

In the traditional convex-concave setting ($f(\cdot,\delta)$ convex for all
fixed $\delta\in\Delta$ and $f(\gamma,\cdot)$ concave for all fixed
$\gamma\in\Gamma$), the classical von Neumann minimax theorem implies that
$\inf_{\gamma\in\Gamma}\sup_{\delta\in\Delta}f(\gamma,\delta)=\sup_{\delta\in\Delta}\inf_{\gamma\in\Gamma}f(\gamma,\delta)$
under mild conditions. In contrast to this, Jin et al. (2020) among others
have emphasized that in GAN applications $f$ is as a rule non-convex and non-
concave and the order in which minimization and maximization are performed
matters. Another point to note is that in GAN applications the object of
interest is not the optimal value of problem (3), that is,

(4)
$V_{0}=\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}f(\gamma,\delta),$

but rather the optimal solutions, the parameter values
$(\gamma_{0},\delta_{0})$ that solve problem (3). Of main interest are the
$\gamma$-parameters appering in the generator network as these facilitate
producing synthetic data according to $G(Z,\gamma_{0})$; for completeness we
consider also the $\delta$-parameters in the discriminator network.

Let $\Theta_{0}\subseteq\Theta$ denote the set of optimal solutions to (3). To
describe $\Theta_{0}$, we introduce the max-function

(5) $\varphi(\gamma)=\sup_{\delta\in\Delta}f(\gamma,\delta).$

A point $(\gamma_{0},\delta_{0})\in\Theta$ solves the GAN problem (3) when it
is a solution both to the inner maximization problem (hence satisfying
$f(\gamma_{0},\delta_{0})=\sup_{\delta\in\Delta}f(\gamma_{0},\delta)=\varphi(\gamma_{0})$)
and to the outer minimization problem (hence satisfying
$\varphi(\gamma_{0})=\inf_{\gamma\in\Gamma}\varphi(\gamma)=V_{0}$). Therefore
the set of solutions $\Theta_{0}$ can be expressed as

(6)
$\Theta_{0}=\\{(\gamma_{0},\delta_{0})\in\Theta:f(\gamma_{0},\delta_{0})=\sup_{\delta\in\Delta}f(\gamma_{0},\delta)=\varphi(\gamma_{0})\text{
and }\varphi(\gamma_{0})=\inf_{\gamma\in\Gamma}\varphi(\gamma)=V_{0}\\}.$

Equivalently, $(\gamma_{0},\delta_{0})\in\Theta_{0}$ if and only if
$\max\\{\varphi(\gamma_{0})-f(\gamma_{0},\delta_{0}),\varphi(\gamma_{0})-V_{0}\\}=0$.
This motivates us to define the (population) criterion function

(7)
$Q(\theta)=Q(\gamma,\delta)=\max\\{\varphi(\gamma)-f(\gamma,\delta),\varphi(\gamma)-V_{0}\\}.$

The function $Q(\theta)$ is nonnegative for all $\theta$ and
$\theta_{0}=(\gamma_{0},\delta_{0})\in\Theta_{0}$ if and only if
$Q(\theta_{0})=0$. Therefore the set of solutions (6) can alternatively and
concisely be characterized as

(8) $\Theta_{0}=\\{\theta_{0}\in\Theta:Q(\theta_{0})=0\\}.$

Now consider the corresponding sample GAN problem (2), which can be written as

(9)
$\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}\hat{f}_{n}(\gamma,\delta)\qquad\text{with}\qquad\hat{f}_{n}(\gamma,\delta)=\frac{1}{n}\sum_{i=1}^{n}F(X_{i},Z_{i},\gamma,\delta)$

and define the sample analogues of the quantities in (4)–(7) as

(10) $\displaystyle\hat{V}_{n}$
$\displaystyle=\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}\hat{f}_{n}(\gamma,\delta),$
(11) $\displaystyle\hat{\varphi}_{n}(\gamma)$
$\displaystyle=\sup_{\delta\in\Delta}\hat{f}_{n}(\gamma,\delta),$ (12)
$\displaystyle\hat{\Theta}_{n}$
$\displaystyle=\bigl{\\{}(\hat{\gamma}_{n},\hat{\delta}_{n})\in\Theta:\;\hat{f}_{n}(\hat{\gamma}_{n},\hat{\delta}_{n})=\sup_{\delta\in\Delta}\hat{f}_{n}(\hat{\gamma}_{n},\delta)=\hat{\varphi}_{n}(\hat{\gamma}_{n})\text{
and }$ $\displaystyle\hskip
90.0pt\hat{\varphi}_{n}(\hat{\gamma}_{n})=\inf_{\gamma\in\Gamma}\hat{\varphi}_{n}(\gamma)=\hat{V}_{n}\bigr{\\}},$
(13) $\displaystyle\hat{Q}_{n}(\theta)$
$\displaystyle=\hat{Q}_{n}(\gamma,\delta)=\max\\{\hat{\varphi}_{n}(\gamma)-\hat{f}_{n}(\gamma,\delta),\hat{\varphi}_{n}(\gamma)-\hat{V}_{n}\\}.$

These quantities have interpretations similar to their population
counterparts: $\hat{V}_{n}$ is the optimal value of the sample GAN problem
(9), $\hat{\varphi}_{n}(\gamma)$ is a sample max-function, $\hat{\Theta}_{n}$
denotes the set of (exact) solutions to the sample GAN problem (9), and
$\hat{Q}_{n}(\theta)$ is a (nonnegative) sample criterion function that allows
us to express the set $\hat{\Theta}_{n}$ as

(14)
$\hat{\Theta}_{n}=\\{\hat{\theta}_{n}\in\Theta:\hat{Q}_{n}(\hat{\theta}_{n})=0\\}.$

Finding a solution to the sample GAN problem (9), let alone the entire set of
optimal solutions $\hat{\Theta}_{n}$, is a challenging task. From a practical
machine learning perspective, this ‘training of the GAN’ is of principle
interest and extensive research efforts have been made to devise algorithms to
do this (see the references listed in the Introduction). In this paper, we do
not consider these algorithms but rather just assume that the sample GAN
problem is solved using some available method. These algorithms typically
search for approximate rather than exact solutions: for some small non-
negative constant $\tau$, approximate solutions to the sample GAN problem (9)
are points $(\hat{\gamma}_{n},\hat{\delta}_{n})\in\Gamma\times\Delta$
satisfying

$\hat{f}_{n}(\hat{\gamma}_{n},\hat{\delta}_{n})\geq\sup_{\delta\in\Delta}\hat{f}_{n}(\hat{\gamma}_{n},\delta)-\tau\quad\text{and}\quad\hat{\varphi}_{n}(\hat{\gamma}_{n})\leq\inf_{\gamma\in\Gamma}\hat{\varphi}_{n}(\gamma)+\tau.$

Such points $(\hat{\gamma}_{n},\hat{\delta}_{n})$ approximately solve both the
inner maximization problem and the outer minimization problem, with $\tau$
determining the slackness allowed in these maximization and minimization
problems. Somewhat more generally, let $\tau_{n}$ be a sequence of non-
negative random variables such that $\tau_{n}\overset{p}{\to}0$ (where
$\overset{p}{\to}$ denotes convergence in probability). Define in analogy with
the above

(15) $\displaystyle\hat{\Theta}_{n}(\tau_{n})$
$\displaystyle=\bigl{\\{}(\hat{\gamma}_{n},\hat{\delta}_{n})\in\Theta:\;\hat{f}_{n}(\hat{\gamma}_{n},\hat{\delta}_{n})\geq\sup_{\delta\in\Delta}\hat{f}_{n}(\hat{\gamma}_{n},\delta)-\tau_{n}\text{
and }$ $\displaystyle\hskip
90.0pt\hat{\varphi}_{n}(\hat{\gamma}_{n})\leq\inf_{\gamma\in\Gamma}\hat{\varphi}_{n}(\gamma)+\tau_{n}\bigr{\\}}$

as the set of approximate solutions to the sample GAN problem (9). Noting that
the two inequalities in (15) can be expressed as
$\hat{\varphi}_{n}(\hat{\gamma}_{n})-\hat{f}_{n}(\hat{\gamma}_{n},\hat{\delta}_{n})\leq\tau_{n}$
and $\hat{\varphi}_{n}(\hat{\gamma}_{n})-\hat{V}_{n}\leq\tau_{n}$,
respectively, and recalling the definition of the sample criterion function
$\hat{Q}_{n}(\theta)$ in (13) allows us to characterize the set
$\hat{\Theta}_{n}(\tau_{n})$ as

(16)
$\hat{\Theta}_{n}(\tau_{n})=\\{\hat{\theta}_{n}\in\Theta:\hat{Q}_{n}(\hat{\theta}_{n})\leq\tau_{n}\\}.$

Setting $\tau_{n}=0$ one obtains as a special case the set of exact solutions,
$\hat{\Theta}_{n}=\hat{\Theta}_{n}(0)$. We next consider the properties of
$\hat{\Theta}_{n}(\tau_{n})$ as an estimator of $\Theta_{0}$.

##  3 Consistent estimation

In the GAN setting both $\hat{\Theta}_{n}(\tau_{n})$ and $\Theta_{0}$ are
typically set-valued and not singletons and an appropriate notion of distance
between sets is required. One commonly used generalization of the Euclidean
distance $|\cdot|$ is the Hausdorff distance (see, e.g., Rockafellar and Wets,
2009, Sec 4.C, or Molchanov, 2017, Appendix D). For any two non-empty bounded
subsets $A$ and $B$ of some Euclidean space, the Hausdorff distance between
$A$ and $B$ is defined as

$d_{H}(A,B)=\max\bigl{\\{}\sup_{a\in A}d(a,B),\sup_{b\in B}d(b,A)\bigr{\\}},$

where $d(a,B)=\inf_{b\in B}|a-b|$ is the shortest distance from the point $a$
to the set $B$. That is, the Hausdorff distance is the greatest distance from
an arbitrary point in one of the sets to the closest neighboring point in the
other set. The Hausdorff distance $d_{H}$ is a metric for the family of non-
empty compact sets and for such sets $d_{H}(A,B)=0$ if and only if $A=B$.

The consistency result we aim to prove is Hausdorff consistency in the sense
that$d_{H}(\hat{\Theta}_{n}(\tau_{n}),\Theta_{0})\overset{p}{\to}0$.
Establishing this requires us to show that both of the ‘one-sided Hausdorff
consistency’ conditions

(17)
$\text{(a)}\;\sup_{\theta\in\hat{\Theta}_{n}(\tau_{n})}d(\theta,\Theta_{0})\overset{p}{\to}0\qquad\text{and}\qquad\text{(b)}\;\sup_{\theta\in\Theta_{0}}d(\theta,\hat{\Theta}_{n}(\tau_{n}))\overset{p}{\to}0$

hold. Heuristically, the former condition in (17) guarantees that
$\hat{\Theta}_{n}(\tau_{n})$ is not too large compared to $\Theta_{0}$,
whereas the latter condition ensures $\hat{\Theta}_{n}(\tau_{n})$ is large
enough to cover all of $\Theta_{0}$. Establishing (17a) follows the pattern of
a standard consistency proof and relies on a suitable uniform law of large
numbers combined with an appropriate set-identification condition for
$\Theta_{0}$. Proving (17b) relies on also knowing the rate of this uniform
convergence. The following assumption formally states the required high-level
conditions; here $O_{p}(1)$ stands for a sequence of random variables that is
bounded in probability.

######  Assumption 1.

  * (a)

$\sup_{\theta\in\Theta}|\hat{f}_{n}(\theta)-f(\theta)|\overset{p}{\to}0$ with
the function $f(\theta)$ continuous in $\theta$.

  * (b)

Part (a) holds with
$\sup_{\theta\in\Theta}n^{1/2}|\hat{f}_{n}(\theta)-f(\theta)|=O_{p}(1)$.

The high-level conditions in this assumption can be verified using various
sets of sufficient conditions. For instance, Assumption 1(a) holds when
Assumption GAN is combined with the mild moment condition
$\mathbb{E}[\sup_{\theta\in\Theta}|F(X,Z,\theta)|]<\infty$, and 1(b) holds
under Assumption Smooth GAN (justifications for these statemens are given in
Appendix C). Assumption 1 thus holds in most practical GAN applications.

Our consistency results also require certain conditions for the slackness
sequence $\tau_{n}$.

######  Assumption 2.

  * (a)

$\tau_{n}$ is a sequence of non-negative random variables such that
$\tau_{n}\overset{p}{\to}0$.

  * (b)

$\tau_{n}$ is a sequence of positive random variables such that
$\tau_{n}\overset{p}{\to}0$ and $n^{-1/2}/\tau_{n}\overset{p}{\to}0$.

Part (a) of Assumption 2 allows for the possibility that $\tau_{n}$ is
identically zero while this is ruled out in part (b). In (b) it is
additionally assumed that the convergence of $\tau_{n}$ to zero is slower than
that of $n^{-1/2}$ in the sense that $n^{-1/2}/\tau_{n}\overset{p}{\to}0$; for
instance, $\tau_{n}=n^{-0.49}$ is a possibility. Of course, (b) implies (a).

We can now state our consistency theorem for the solutions of the GAN problem
(the proof is given in Appendix A).

######  Theorem 1.

Suppose Assumption GAN holds.

  * (a)

Assumptions 1(a) and 2(a) imply that (17a) holds (and also that
$\hat{V}_{n}\overset{p}{\to}V_{0}$).

  * (b)

Assumptions 1(b) and 2(b) imply that (17b) holds so that
$d_{H}(\hat{\Theta}_{n}(\tau_{n}),\Theta_{0})\overset{p}{\to}0$.

Part (a) of Theorem 1 establishes the former one-sided Hausdorff consistency
condition in (17); for completeness, consistency of the optimal value
$\hat{V}_{n}$ is also given. Under the stronger conditions in part (b), the
latter condition in (17) is also obtained, thus establishing the desired
Hausdorff consistency result.

Theorem 1 is based on the Hausdorff consistency results of Chernozhukov et al.
(2007, Thm 3.1) in general set-identified situations. Shapiro et al. (2009,
Thm 5.9) give (an almost sure version) of part (a) of Theorem 1 in a general
minimax setting. In the GAN setting, a consistency result in the case of a
unique solution is given in Biau et al. (2020, Thm 4.2). (Note that when
$\Theta_{0}$ is a singleton-set consisting of one point $\theta_{0}$, Theorem
1(a) shows that any element of $\hat{\Theta}_{n}(\tau_{n})$ is consistent for
$\theta_{0}$.) We are not aware of consistency results in the GAN setting in
the case of multiple solutions and the result of Theorem 1 is reassuring in
that estimation in the GAN setting will be consistent regardless of the
(potentially infinite) number of solutions. The one-sided consistency result
of Theorem 1(a) covers the case of exact solutions ($\tau_{n}=0$), whereas the
two-sided case in part (b) requires us to (somewhat arbitrarily) choose a
strictly positive slackness sequence $\tau_{n}$. Such a choice is not needed
in the procedure for forming confidence sets for $\Theta_{0}$ that we consider
next.

##  4 Confidence sets

A confidence set for $\Theta_{0}$ is a random set that covers the entire
$\Theta_{0}$ with a prespecified probability. (As in GANs the individual
elements $\theta_{0}$ of $\Theta_{0}$ do not have any special interpretation
attached to them, a confidence set covering all of $\Theta_{0}$ and not just
some particular element of it is appropriate.) Let $1-\alpha$ denote the
desired coverage probability (such as $95\%$) where $\alpha\in(0,1)$. We aim
to construct confidence sets $\hat{\operatorname{CS}}_{n,1-\alpha}$ that
contain $\Theta_{0}$ with at least probability $1-\alpha$,

(18)
$\liminf_{n\to\infty}\mathbb{P}[\Theta_{0}\subseteq\hat{\operatorname{CS}}_{n,1-\alpha}]\geq
1-\alpha,$

that is, sets that are conservatively asymptotically consistent at level
$1-\alpha$. (Here $\mathbb{P}$ refers to the probability measure of $(X,Z)$
which we consider fixed.)

In the traditional point-identified setting in which $\Theta_{0}$ consists of
a single point $\theta_{0}$, a conventional route to forming confidence sets
is to consider a Taylor approximation of a certain function around
$\theta_{0}$; this leads to the (often Gaussian) distribution of an
appropriately rescaled estimator from which confidence sets for $\theta_{0}$
are then obtained in a straightforward manner. Biau et al. (2020) consider
this point-identified case and, focusing on the generator network
$\gamma$-parameters, in their Theorem 4.3 state that
$n^{1/2}(\hat{\gamma}_{n}-\gamma_{0})$ converges in distribution to a Gaussian
random variable. When $\Theta_{0}$ is not a singleton set alternative
approaches are called for. Given the results of Biau et al. (2020), we focus
only on the case of multiple (more than one) solutions. This is the typical
case in GAN applications. Our approach follows the recent developments in the
partial identification literature in econometrics; we rely in particular on
the results in Chernozhukov et al. (2007) and Romano and Shaikh (2010) (for
further related references, see the survey of this literature in Molinari,
2020).

The confidence sets we consider are based on appropriate lower contour sets of
the rescaled criterion function $n^{1/2}\hat{Q}_{n}(\theta)$. To motivate the
subsequent technical developments, we begin with some informal remarks. First,
if the $1-\alpha$ quantile $c_{n,1-\alpha}$ of the distribution of
$\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)$ was known, one could
form the (infeasible) confidence set
$\operatorname{CS}_{n,1-\alpha}=\\{\theta\in\Theta:n^{1/2}\hat{Q}_{n}(\theta)\leq
c_{n,1-\alpha}\\}$ that satisfies
$\mathbb{P}[\Theta_{0}\subseteq\operatorname{CS}_{n,1-\alpha}]\geq 1-\alpha$
(because $\Theta_{0}\subseteq\operatorname{CS}_{n,1-\alpha}$ if and only if
$\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)\leq c_{n,1-\alpha}$).
Second, consider the statistic $\sup_{\theta\in S}n^{1/2}\hat{Q}_{n}(\theta)$
where $S$ is some nonempty subset of $\Theta$; under appropriate conditions,
it can be shown that as $n\to\infty$, $\sup_{\theta\in
S}n^{1/2}\hat{Q}_{n}(\theta)$ remains stochastically bounded for
$S\subseteq\Theta_{0}$ and diverges to infinity for $S\nsubseteq\Theta_{0}$.
Now, to form our confidence sets we approximate the $1-\alpha$ quantile of
$\sup_{\theta\in S}n^{1/2}\hat{Q}_{n}(\theta)$ for various sets $S$ using a
suitable resampling method, and then locate a confidence set for $\Theta_{0}$
based on the different behavior of $\sup_{\theta\in
S}n^{1/2}\hat{Q}_{n}(\theta)$ in the two cases $S\subseteq\Theta_{0}$ and
$S\nsubseteq\Theta_{0}$. As our situation involves non-standard features,
resampling based on standard bootstrap is not appropriate and instead we use a
procedure based on subsampling (see Politis et al., 1999).

To formalize this discussion, we next introduce some notation. For any
nonempty subset $S$ of $\Theta$, let $L_{n}(S,x)$ denote the cumulative
distribution function (cdf) of the statistic $\sup_{\theta\in
S}n^{1/2}\hat{Q}_{n}(\theta)$ and $c_{n,1-\alpha}(S)$ the corresponding
(smallest) $1-\alpha$ quantile, that is,

$\displaystyle L_{n}(S,x)$ $\displaystyle=\mathbb{P}[\sup_{\theta\in
S}n^{1/2}\hat{Q}_{n}(\theta)\leq x]\quad\text{and}$ $\displaystyle
c_{n,1-\alpha}(S)$ $\displaystyle=\inf\\{x\in\mathbb{R}:L_{n}(S,x)\geq
1-\alpha\\}.$

Let $b$ denote the subsample size that is assumed to satisfy the usual
requirements $b\to\infty$ and $b/n\to 0$ as $n\to\infty$; for notational
simplicity, the dependence of $b$ on $n$ is not made explicit. The number of
different subsamples is denoted by $N_{n}$ ($=n!/(b!(n-b)!)$) and the
subsample statistics $\hat{Q}_{n,b,i}(\theta)$ ($i=1,\ldots,N_{n}$) are
defined exactly as $\hat{Q}_{n}(\theta)$ in (13) but based on the $i$th
subsample of size $b$ rather than the full sample. The subsampling
counterparts of $L_{n}(S,x)$ and $c_{n,1-\alpha}(S)$ are $\hat{L}_{n,b}(S,x)$,
the empirical distribution function of the subsample statistics
$\\{\sup_{\theta\in S}b^{1/2}\hat{Q}_{n,b,i}(\theta):i=1,\ldots,N_{n}\\}$, and
$\hat{c}_{n,b,1-\alpha}(S)$, the corresponding $1-\alpha$ sample quantile,
defined as

$\displaystyle\hat{L}_{n,b}(S,x)$
$\displaystyle=\frac{1}{N_{n}}\sum_{i=1}^{N_{n}}1\Bigl{(}\sup_{\theta\in
S}b^{1/2}\hat{Q}_{n,b,i}(\theta)\leq x\Bigr{)}\quad\text{and}$
$\displaystyle\hat{c}_{n,b,1-\alpha}(S)$
$\displaystyle=\inf\\{x\in\mathbb{R}:\hat{L}_{n,b}(S,x)\geq 1-\alpha\\}.$

We can now state the iterative procedure that we use to construct the desired
confidence sets. This procedure is akin to step-down procedures used in
multiple hypothesis testing problems and follows Lehmann and Romano (2005, Sec
9.1) and Romano and Shaikh (2010).

######  Procedure 1.

Set $S_{1}=\Theta$ and for $j=1,2,\ldots$ do the following: If

$\sup_{\theta\in
S_{j}}n^{1/2}\hat{Q}_{n}(\theta)\leq\hat{c}_{n,b,1-\alpha}(S_{j}),$

then set $\hat{\operatorname{CS}}_{n,1-\alpha}=S_{j}$ and stop; otherwise, set
$S_{j+1}=\\{\theta\in\Theta:n^{1/2}\hat{Q}_{n}(\theta)\leq\hat{c}_{n,b,1-\alpha}(S_{j})\\}$
and continue iteration.

This procedure starts from the full parameter space $S_{1}=\Theta$ and
iteratively discards $\theta$’s from the $S_{j}$-sets until a suitable
confidence set is formed. Overall, the procedure is certainly computer-
intensive yet feasible (to reduce computational costs, one could use just a
subset of the $N_{n}$ subsamples without affecting the validity of our
results; cf. Politis et al., 1999, Cor 2.4.1). Theorem 2 below shows that,
under appropriate assumptions, the confidence sets
$\hat{\operatorname{CS}}_{n,1-\alpha}$ formed using Procedure 1 are valid
confidence sets in the sense that (18) holds. The key requirement for this
subsampling-based procedure to work is the following high-level assumption.

######  Assumption 3.

The statistic $\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)$ converges
in distribution to some limiting random variable $L$.

The convergence requirement in Assumption 3 is a standard high-level condition
needed for the validity of subsampling procedures (see Politis et al., 1999,
Sec 2.2). Ensuring that Assumption 3 holds in the GAN setting turns out to be
particularly challenging and we resort to empirical process theory for this.
(A thorough account of the needed empirical process theory can be found in the
monograph of van der Vaart and Wellner (1996).) With the set $\Theta$ compact
as before, let $l^{\infty}(\Theta)$ denote the space of all bounded real-
valued functions on $\Theta$ equipped with the supremum norm and $C(\Theta)$
stand for the subspace of those functions that are continuous. Random
functions such as $\hat{f}_{n}(\cdot)$ are viewed as maps from appropriate
underlying probability spaces to $l^{\infty}(\Theta)$, and $\rightsquigarrow$
denotes weak convergence as defined in van der Vaart and Wellner (1996, Sec
1.3).

A convenient and broadly applicable high-level assumption that we make is that
a suitable functional central limit theorem (Donsker property) holds.

######  Assumption 4.

Suppose $n^{1/2}(\hat{f}_{n}-f)\rightsquigarrow\mathbb{G}$ in
$l^{\infty}(\Theta)$ with $f\in C(\Theta)$ and $\mathbb{G}$ a tight mean-zero
Gaussian process taking values in $C(\Theta)$ with probability one and such
that $\inf_{\theta\in\Theta_{0}}\mathbb{E}[(\mathbb{G}(\theta))^{2}]>0$.

Assumption 4 is very general and rather technical. For the present purposes it
suffices to note that Assumption Smooth GAN implies the validity of Assumption
4 (justification of this is given in Appendix C). Alternatively, Assumption 4
also holds under much weaker conditions and can be verified using a variety of
different methods in empirical process theory (for details, see van der Vaart
and Wellner, 1996).

Showing that Assumption 3 follows from Assumption 4 (and other additional
conditions) requires rather long and technical details. In order to not
distract from the main issue, we outline the key points of the argument here
and relegate the technical details to Appendix B. First, it is shown that for
a suitably defined map $\phi:l^{\infty}(\Theta)\to\mathbb{R}$, the statistic
$\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)$ in Assumption 3 can be
expressed as
$\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)=n^{1/2}(\phi(\hat{f}_{n})-\phi(f))$.
Second, the map $\phi$ is shown to be (directionally) differentiable in an
appropriate sense. Third, the previous facts enable us to apply a particular
version of the functional delta method to deduce that
$n^{1/2}(\phi(\hat{f}_{n})-\phi(f))\rightsquigarrow\phi_{f}^{\prime}(\mathbb{G})$
where $\phi_{f}^{\prime}$ denotes a certain (directional) derivative of the
mapping $\phi$ at $f$ (and $\mathbb{G}$ is the Gaussian process in Assumption
4). Thus Assumption 3 follows. To prove the differentiability result
mentioned, additional assumptions are needed and we focus on the following
leading case.

######  Assumption 5.

The set
$\Gamma_{0}=\\{\gamma_{0}\in\Gamma:(\gamma_{0},\delta_{0})\in\Theta_{0}\text{
for some }\delta_{0}\in\Delta\\}$ is finite.

Assumption 5 requires that the outer minimization problem in the population
minimax problem (3) is solved at a finite number of $\gamma$-values only (the
inner maximization problem is allowed to have infinitely many solutions). This
assumption is quite reasonable in practical GAN applications and is needed to
prove the mentioned differentiability result.

All the details of the preceding discussion are available in Appendix B, where
we prove the following lemma.

######  Lemma 1.

Suppose Assumptions GAN, 4, and 5 hold. Then Assumption 3 holds with the
limiting distribution of
$\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)$ given in expression
(25) in Appendix B.

Lemma 1 offers one convenient way to verify that Assumption 3 holds in the GAN
setting. Even with the finiteness requirement of Assumption 5, the technical
details in Appendix B are rather long and the limiting distribution in (25)
quite complicated. Assumption 3 could potentially be verified using weaker
conditions but we do not pursue this.

After these preparations, we are now ready to state our main result regarding
the confidence sets formed by Procedure 1. Let $L(\cdot)$ denote the cdf of
the limiting random variable $L$ in Assumption 3,
$c_{1-\alpha}=\inf\\{x\in\mathbb{R}:L(x)\geq 1-\alpha\\}$ the corresponding
$1-\alpha$ quantile, and $\lim_{c\mathrel{\nonscript\mkern-1.2mu\mkern
1.2mu{\scriptscriptstyle\uparrow}}c_{1-\alpha}}L(c)$ the limit from the left
of $L(\cdot)$ at $c_{1-\alpha}$. The following theorem is proved in Appendix
B.

######  Theorem 2.

Suppose Assumptions GAN and 3 hold, the population GAN problem (3) has
multiple solutions, the confidence sets $\hat{\operatorname{CS}}_{n,1-\alpha}$
($n=1,2,\ldots$) are formed using Procedure 1, and the subsampling size $b$ is
such that $b\to\infty$ and $b/n\to 0$ as $n\to\infty$. Then the confidence
sets $\hat{\operatorname{CS}}_{n,1-\alpha}$ satisfy

$\liminf_{n\to\infty}\mathbb{P}[\Theta_{0}\subseteq\hat{\operatorname{CS}}_{n,1-\alpha}]\geq\lim_{c\mathrel{\nonscript\mkern-1.2mu\mkern
1.2mu{\scriptscriptstyle\uparrow}}c_{1-\alpha}}L(c).$

Theorem 2 shows that the confidence sets
$\hat{\operatorname{CS}}_{n,1-\alpha}$ have an asymptotic coverage probability
of at least $1-\alpha$ when the cdf $L(\cdot)$ is continuous at $c_{1-\alpha}$
(as then $\lim_{c\mathrel{\nonscript\mkern-1.2mu\mkern
1.2mu{\scriptscriptstyle\uparrow}}c_{1-\alpha}}L(c)=1-\alpha$; if $L(\cdot)$
has a jump at $c_{1-\alpha}$, then
$\lim_{c\mathrel{\nonscript\mkern-1.2mu\mkern
1.2mu{\scriptscriptstyle\uparrow}}c_{1-\alpha}}L(c)<1-\alpha\leq
L(c_{1-\alpha})$). In general the limiting distribution $L$ is rather
complicated and verifying the continuity of $L(\cdot)$ requires some more
concrete assumptions. In Appendix B we show that this continuity holds for
instance when $\Theta_{0}$ is a finite set of the form
$\Theta_{0}=\\{(\gamma_{01},\delta_{01}),\ldots,(\gamma_{0K},\delta_{0K})\\}$
for some finite $K>1$ and with distinct $\gamma$’s and $\delta$’s (i.e.,
$\gamma_{0i}\neq\gamma_{0j}$ and $\delta_{0i}\neq\delta_{0j}$ for all $i\neq
j$). This case with multiple but finite number of distinct solutions covers
many practical GAN applications.

In previous work, confidence regions for general set-identified parameters
were considered by Chernozhukov et al. (2007) and Romano and Shaikh (2010).
Our Theorem 2 is based on their results, especially on Theorem 2.2 of Romano
and Shaikh (2010). Shapiro (2008, Thm 3.1) and Shapiro et al. (2009, Thm 5.10)
study the asymptotic distribution of the optimal value in convex-concave
minimax problems but do not consider inference for the solutions of minimax
problems. Biau et al. (2020, Thm 4.3) consider the GAN problem assuming a
single unique solution exists and give a result on the asymptotic normality of
the generator network parameters; however, they remark (pp. 1560–1561) that
the assumed uniqueness is “hardly satisfied in the high-dimensional context of
(deep) neural networks” and call for generalizations. Theorem 2 builds on
these previous results and provides the first inference procedure for GAN
parameters in the presence of multiple solutions. Our construction of the
confidence sets is specifically tailored for the case of multiple solutions;
in the case of a single solution, confidence sets can be formed based on the
results of Biau et al. (2020) (for details, see Appendix B).

Regarding the potential practical use of these confidence sets in GAN
applications, recall from the Introduction that when
$(\gamma_{0},\delta_{0})\in\Theta_{0}$, the generator $G(Z,\gamma_{0})$ gives
the researcher a mechanism to produce synthetic data. In recent work, Karras
et al. (2020, Sec 3.2, Fig 4) have considered the effect of small variations
in the input noise on the synthetic images produced. Similarly, one could
consider how the produced images (or, more generally, the distribution of
$G(Z,\gamma_{0})$) change when the generator parameters vary within a certain
confidence set. Exploring the effect of such variations on the synthetic data
produced in other GAN applications (such as DNA sequences, art, or music)
would also be interesting. We leave these issues for future research.

##  5 Conclusion

In this paper, we have considered statistical inference for GANs in the
empirically relevant case of multiple solutions. We first considered the
consistency of (approximate) solutions to the sample GAN problem, and showed
that the set of these solutions is a Hausdorff consistent estimator of the
corresponding set of solutions to the population GAN problem, $\Theta_{0}$. We
then presented a subsampling-based iterative procedure for forming confidence
sets for $\Theta_{0}$, and showed that these confidence sets are
conservatively asymptotically consistent. The consistency result was shown to
hold without any restrictions on the number of solutions to the GAN problem,
whereas our results for confidence sets were given for the common case of
multiple but finite number of solutions. These results extend on the results
of Biau et al. (2020) who considered the case of a single unique solution.

Our study aims for a better understanding of the statistical properties of
GANs and can be seen as complementary to the existing vast literatures on
computational GAN training algorithms and practical GAN applications. In this
paper we have considered only the original Goodfellow et al. (2014)
formulation of the GAN problem. Extensions to the numerous other existing GAN
variants, such as the popular Wasserstein GAN of Arjovsky et al. (2017), would
be of interest and would bring the statistical theory closer to the machine
learning practice. Related minimax problems in non-GAN settings in machine
learning and elsewhere could also be considered. Finally, the focus in this
paper has been purely theoretical and exploring the use of our results in
practical GAN applications would be interesting.

## Appendix A: Proof of Theorem 1

###### Proof of Theorem 1.

For completeness, first note that measurability issues and non-emptyness of
$\Theta_{0}$ and $\hat{\Theta}_{n}$ (with probability one) are discussed in
Shapiro et al. (2009, pp. 170–171). For ease of reference, also note that for
any two bounded real-valued functions $u$ and $v$ defined on a domain $D$ (a
subset of some Euclidean space),

(19) $|\sup u(x)-\sup v(x)|\leq\sup|u(x)-v(x)|,\;\;\>|\inf u(x)-\inf
v(x)|\leq\sup|u(x)-v(x)|,$

where all the supremums and infimums are taken over $x\in D$.

(a) To establish the consistency of the optimal value $\hat{V}_{n}$, it
suffices to note that

(20)
$|\hat{V}_{n}-V_{0}|=|\inf_{\gamma\in\Gamma}\hat{\varphi}_{n}(\gamma)-\inf_{\gamma\in\Gamma}\varphi(\gamma)|\leq\sup_{\gamma\in\Gamma}|\hat{\varphi}_{n}(\gamma)-\varphi(\gamma)|\leq\sup_{\theta\in\Theta}|\hat{f}_{n}(\theta)-f(\theta)|\overset{p}{\to}0,$

where we have used the definitions of $\hat{V}_{n}$, $V_{0}$,
$\hat{\varphi}_{n}(\gamma)$, and $\varphi(\gamma)$, the two inequalities in
(19), and the uniform convergence in Assumption 1(a).

We next show that

(21)
$\sup_{\theta\in\Theta}|\hat{Q}_{n}(\theta)-Q(\theta)|\overset{p}{\to}0\quad\text{with
the function $Q(\theta)$ continuous in $\theta$.}$

As
$\hat{Q}_{n}(\theta)-Q(\theta)=\hat{\varphi}_{n}(\gamma)-\varphi(\gamma)-(\min\\{\hat{f}_{n}(\gamma,\delta),\hat{V}_{n}\\}-\min\\{f(\gamma,\delta),V_{0}\\})$,
the triangle inequality, elementary properties of min and max, and the
inequalities in (20) imply that

(22) $\displaystyle|\hat{Q}_{n}(\theta)-Q(\theta)|$
$\displaystyle\leq|\hat{\varphi}_{n}(\gamma)-\varphi(\gamma)|+\max\\{|\hat{f}_{n}(\gamma,\delta)-f(\gamma,\delta)|,|\hat{V}_{n}-V_{0}|\\}$
$\displaystyle\leq 2\sup_{\theta\in\Theta}|\hat{f}_{n}(\theta)-f(\theta)|$

for all $\theta\in\Theta$. Thus by Assumption 1(a),
$\sup_{\theta\in\Theta}|\hat{Q}_{n}(\theta)-Q(\theta)|\overset{p}{\to}0$. As
for the continuity in (21), by Assumptions GAN and 1(a) the function
$f(\theta)$ is continuous on the compact set $\Theta$, and thus by Berge’s
maximum theorem the function
$\varphi(\gamma)=\sup_{\delta\in\Delta}f(\gamma,\delta)$ is continuous on
$\Gamma$. Consequently, $Q(\theta)$ is continuous. Also note that the
continuity of $Q$ and the definition of $\Theta_{0}$ imply that for all
$\epsilon>0$ there exists an $\eta(\epsilon)>0$ such that

(23)
$\inf_{\theta\in\Theta\setminus\Theta_{0}^{\epsilon}}Q(\theta)\geq\eta(\epsilon),$

where $\Theta_{0}^{\epsilon}$ denotes the $\epsilon$-expansion of the set
$\Theta_{0}$ in $\Theta$ defined as
$\Theta_{0}^{\epsilon}:=\\{\theta\in\Theta:d(\theta,\Theta_{0})\leq\epsilon\\}$
and $\Theta\setminus\Theta_{0}^{\epsilon}$ is the complement of
$\Theta_{0}^{\epsilon}$ in $\Theta$.

Establishing
$\sup_{\theta\in\hat{\Theta}_{n}(\tau_{n})}d(\theta,\Theta_{0})\overset{p}{\to}0$
now follows the pattern of a standard consistency proof and relies on the
uniform convergence result (21) and the set-identification condition for
$\Theta_{0}$ in (23). To this end, let small $\epsilon_{p},\epsilon_{d}>0$ be
arbitrary, choose an $\eta=\eta(\epsilon_{d})$ as in (23) so that
$\inf_{\theta\in\Theta\setminus\Theta_{0}^{\epsilon_{d}}}Q(\theta)\geq\eta$
holds, and by (21) and Assumption 2(a) choose an $n_{\epsilon_{p}}$ such that
for all $n\geq n_{\epsilon_{p}}$ both
$\sup_{\theta\in\Theta}|\hat{Q}_{n}(\theta)-Q(\theta)|\leq\eta/4$ and
$\tau_{n}\leq\eta/4$ hold with probability larger than $1-\epsilon_{p}$. We
now have

$\displaystyle\sup_{\theta\in\hat{\Theta}_{n}(\tau_{n})}Q(\theta)$
$\displaystyle\leq\sup_{\theta\in\hat{\Theta}_{n}(\tau_{n})}|\hat{Q}_{n}(\theta)-Q(\theta)|+\sup_{\theta\in\hat{\Theta}_{n}(\tau_{n})}\hat{Q}_{n}(\theta)$
$\displaystyle\leq\sup_{\theta\in\Theta}|\hat{Q}_{n}(\theta)-Q(\theta)|+\tau_{n}\leq\eta/2<\inf_{\theta\in\Theta\setminus\Theta_{0}^{\epsilon_{d}}}Q(\theta),$

and therefore also
$\hat{\Theta}_{n}(\tau_{n})\subseteq\Theta_{0}^{\epsilon_{d}}$ and
$\sup_{\theta\in\hat{\Theta}_{n}(\tau_{n})}d(\theta,\Theta_{0})\leq\epsilon_{d}$,
for all $n\geq n_{\epsilon_{p}}$ with probability larger than
$1-\epsilon_{p}$. Thus
$\sup_{\theta\in\hat{\Theta}_{n}(\tau_{n})}d(\theta,\Theta_{0})\overset{p}{\to}0$.

(b) Assumption 1(b) and (22) imply that
$\sup_{\theta\in\Theta}n^{1/2}|\hat{Q}_{n}(\theta)-Q(\theta)|=O_{p}(1)$. Note
that

$\sup_{\theta\in\Theta_{0}}\hat{Q}_{n}(\theta)\leq\sup_{\theta\in\Theta_{0}}|\hat{Q}_{n}(\theta)-Q(\theta)|+\sup_{\theta\in\Theta_{0}}Q(\theta),$

where
$\sup_{\theta\in\Theta_{0}}|\hat{Q}_{n}(\theta)-Q(\theta)|\leq\sup_{\theta\in\Theta}|\hat{Q}_{n}(\theta)-Q(\theta)|$
as $\Theta_{0}\subseteq\Theta$ and $\sup_{\theta\in\Theta_{0}}Q(\theta)=0$ by
the definition of $\Theta_{0}$. By Assumption 2(b) we have
$n^{-1/2}/\tau_{n}\overset{p}{\to}0$ and thus for any $\epsilon_{p}>0$ we can
find an $n_{\epsilon_{p}}$ such that for all $n\geq n_{\epsilon_{p}}$

$\sup_{\theta\in\Theta_{0}}\hat{Q}_{n}(\theta)\leq
O_{p}(n^{-1/2})=O_{p}(1)(n^{-1/2}/\tau_{n})\tau_{n}\leq\tau_{n}$

with probability larger than $1-\epsilon_{p}$. By the definition of
$\hat{\Theta}_{n}(\tau_{n})$ we now have
$\Theta_{0}\subseteq\hat{\Theta}_{n}(\tau_{n})$ and thus
$\sup_{\theta\in\Theta_{0}}d(\theta,\hat{\Theta}_{n}(\tau_{n}))=0$ (for all
$n\geq n_{\epsilon_{p}}$ with probability larger than $1-\epsilon_{p}$). This
shows that
$\sup_{\theta\in\Theta_{0}}d(\theta,\hat{\Theta}_{n}(\tau_{n}))\overset{p}{\to}0$.
Combining this with the result in (a), we have established that
$d_{H}(\hat{\Theta}_{n}(\tau_{n}),\Theta_{0})\overset{p}{\to}0$. ∎

## Appendix B: Proofs of Lemma 1 and Theorem 2

We begin with some preparatory discussion. As was outlined in Section 4, the
statistic$\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)$ can be
expressed as $n^{1/2}(\phi(\hat{f}_{n})-\phi(f))$ for a suitably defined map
$\phi:l^{\infty}(\Theta)\to\mathbb{R}$ (to be given below in the proof of
Lemma 1), this map will be shown to be differentiable in an appropriate sense,
and these facts enable us to apply a suitable functional delta method to
obtain the result of Lemma 1. To derive the necessary results, it is
convenient to define the relevant differentiability concepts for maps between
abstract Banach spaces; the specific spaces used below include
$l^{\infty}(\Theta)$, $C(\Theta)$, and the Euclidean space $\mathbb{R}$.

Let $X$ and $Y$ be real Banach spaces with norms $\left\|\cdot\right\|_{X}$
and $\left\|\cdot\right\|_{Y}$, respectively. Let the domain $X_{D}\subseteq
X$ be some subset of $X$, and consider an arbitrary map $\phi:X_{D}\to Y$. Let
$X_{0}\subseteq X$ be another subset of $X$. The map $\phi$ is said to be
Hadamard directionally differentiable at $x\in X_{D}$ tangentially to $X_{0}$
with a derivative $\phi_{x}^{\prime}:X_{0}\to Y$ if

(24)
$\lim_{n\to\infty}\Bigl{\|}\frac{\phi(x+t_{n}h_{n})-\phi(x)}{t_{n}}-\phi_{x}^{\prime}(h)\Bigr{\|}_{Y}=0$

for all sequences $\\{h_{n}\\}\subset X$ and
$\\{t_{n}\\}\subset\mathbb{R}_{+}$ such that $t_{n}\downarrow 0$ and $h_{n}\to
h\in X_{0}$ as $n\to\infty$ and $x+t_{n}h_{n}\in X_{D}$ for all $n$. Two
related differentiability concepts are defined as follows. If the convergence
(24) is required to hold only for all fixed $h_{n}\equiv h\in X_{0}$, the map
$\phi$ is said to be Gateaux directionally differentiable (at $x\in X_{D}$
tangentially to $X_{0}$). Alternatively, if in the above definition the
requirement “$\\{t_{n}\\}\subset\mathbb{R}_{+}$ such that $t_{n}\downarrow 0$”
is replaced with “$\\{t_{n}\\}\subset\mathbb{R}$ such that $t_{n}\to 0$”, the
map is said to be (fully) Hadamard differentiable (at $x\in X_{D}$
tangentially to $X_{0}$). In statistical literature, these differentiability
concepts are discussed and developed in Reeds (1976), Gill (1989), and Shapiro
(1990; 1991), among others.

Our main interest in these differentiability concepts comes from the fact that
the functional delta method may become applicable. For (fully) Hadamard
differentiable maps this is well known; see, e.g., van der Vaart and Wellner
(1996, Thm 3.9.4). For Hadamard directionally differentiable maps the
functional delta method is given in Shapiro (1991, Thm 2.1); recently
attention to this result has been drawn for instance by Fang and Santos (2019,
Thm 2.1) and Cárcamo et al. (2020, Propn 2.1). For convenience, we reproduce
this result as the following lemma.

######  Lemma 2.

Suppose the map $\phi:X_{D}\to Y$ is Hadamard directionally differentiable at
$x\in X_{D}$ tangentially to $X_{0}$ with a derivative
$\phi_{x}^{\prime}:X_{0}\to Y$. Let $x_{1},x_{2},\ldots$ be $X_{D}$-valued
random elements such that $r_{n}(x_{n}-x)\rightsquigarrow x_{0}$ in $X$ for
some random element $x_{0}$ taking its values in $X_{0}$ with probability one
and for some constants $r_{n}\to\infty$. Then
$r_{n}(\phi(x_{n})-\phi(x))\rightsquigarrow\phi_{x}^{\prime}(x_{0})$ in $Y$.

Gateaux directional differentiability is in general too weak of a
differentiability concept to ensure the validity of the functional delta
method. However, when the map $\phi$ is also locally Lipschitz at $x\in X_{D}$
(in the sense that there exists a $C>0$ such that for all
$x^{\prime},x^{\prime\prime}\in X_{D}$ in some neighborhood of $x$,
$\lVert\phi(x^{\prime})-\phi(x^{\prime\prime})\rVert_{Y}\leq C\lVert
x^{\prime}-x^{\prime\prime}\rVert_{X}$), Gateaux and Hadamard directional
differentiability become equivalent (Shapiro, 1990, Propn 3.5). This is
convenient and holds for the maps considered in the next lemma. In this lemma,
we let $\Gamma$ and $\Delta$ denote any compact and non-empty Euclidean sets
(which need not equal the sets $\Gamma$ and $\Delta$ elsewhere in the paper;
the same notation is used for convenience). Furthermore, for any function
$x\in C(\Delta)$ we denote
$\Delta_{0}(x)=\\{\delta_{0}\in\Delta:x(\delta_{0})=\sup_{\delta\in\Delta}x(\delta)\\}$,
and for any function $x\in C(\Gamma\times\Delta)$ we let
$\bar{x}(\gamma)=\sup_{\delta\in\Delta}x(\gamma,\delta)$ denote the max-
function and define the sets

$\displaystyle\Theta_{0}(x)$
$\displaystyle=\\{(\gamma_{0},\delta_{0})\in\Gamma\times\Delta:x(\gamma_{0},\delta_{0})=\sup_{\delta\in\Delta}x(\gamma_{0},\delta)=\bar{x}(\gamma_{0})\text{
and }\bar{x}(\gamma_{0})=\inf_{\gamma\in\Gamma}\bar{x}(\gamma)\\},$
$\displaystyle\Gamma_{0}(x)$
$\displaystyle=\\{\gamma_{0}\in\Gamma:(\gamma_{0},\delta_{0})\in\Theta_{0}(x)\text{
for some }\delta_{0}\in\Delta\\},\vphantom{\sup_{\delta\in\Delta}}$
$\displaystyle\Delta_{0}(\gamma,x)$
$\displaystyle=\\{\delta_{0}\in\Delta:x(\gamma,\delta_{0})=\sup_{\delta\in\Delta}x(\gamma,\delta)\\}\quad(\gamma\in\Gamma).$

For any functions $x,y\in l^{\infty}(\Gamma\times\Delta)$, we also allow
ourselves to write

$\underset{(\gamma,\delta)\in\Theta_{0}(x)}{{\textstyle\inf_{\gamma}}\thinspace{\textstyle\sup_{\delta}}}\thinspace
y(\gamma,\delta)\quad\text{instead
of}\quad\adjustlimits{\inf}_{\gamma\in\Gamma_{0}(x)}{\sup}_{\delta\in\Delta_{0}(\gamma,x)}\thinspace
y(\gamma,\delta).$

Finally, for notational ease the three different maps in parts (a)–(c) of the
following lemma are all denoted by $\phi$ as confusion should not arise.

######  Lemma 3.

  * (a)

The map $\phi:l^{\infty}(\Delta)\to\mathbb{R}$ given by
$\phi(x)=\sup_{\delta\in\Delta}x(\delta)$ for $x\in l^{\infty}(\Delta)$ is
Hadamard directionally differentiable at any $x\in C(\Delta)$ tangentially to
$C(\Delta)$ with the derivative
$\phi^{\prime}_{x}(h)=\sup_{\delta\in\Delta_{0}(x)}h(\delta)$.

  * (b)

The map $\phi:l^{\infty}(\Gamma\times\Delta)\to\mathbb{R}$ given by
$\phi(x)=\inf_{\gamma\in\Gamma}\sup_{\delta\in\Delta}x(\gamma,\delta)$ for
$x\in l^{\infty}(\Gamma\times\Delta)$ is Hadamard directionally differentiable
at any $x\in C(\Gamma\times\Delta)$ tangentially to $C(\Gamma\times\Delta)$
with the derivative

$\phi^{\prime}_{x}(h)=\underset{(\gamma,\delta)\in\Theta_{0}(x)}{{\textstyle\inf_{\gamma}}\thinspace{\textstyle\sup_{\delta}}}\thinspace
h(\gamma,\delta).$

  * (c)

Consider a fixed function $x_{0}\in C(\Gamma\times\Delta)$. If the set
$\Gamma_{0}(x_{0})$ is finite, then the map
$\phi:l^{\infty}(\Gamma\times\Delta)\to\mathbb{R}$ given by

$\phi(x)=\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\Bigl{\\{}\sup_{\delta\in\Delta}x(\gamma_{0},\delta)-\min\bigl{\\{}x(\gamma_{0},\delta_{0}),\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x(\gamma,\delta)\bigr{\\}}\Bigr{\\}}$

for $x\in l^{\infty}(\Gamma\times\Delta)$ is Hadamard directionally
differentiable at $x_{0}$ tangentially to $C(\Gamma\times\Delta)$ with the
derivative

$\phi_{x_{0}}^{\prime}(h)=\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\Bigl{\\{}\sup_{\delta\in\Delta_{0}(\gamma_{0},x_{0})}h(\gamma_{0},\delta)-\min\bigl{\\{}h(\gamma_{0},\delta_{0}),\underset{(\gamma,\delta)\in\Theta_{0}(x_{0})}{{\textstyle\inf_{\gamma}}\thinspace{\textstyle\sup_{\delta}}}\thinspace
h(\gamma,\delta)\bigr{\\}}\Bigr{\\}}.$

The proof of Lemma 3 is given at the end of this appendix. The result in part
(a) is well-known and versions of it can be found in Shapiro (1991, Thm 3.1),
Fang and Santos (2019, Lemma S.4.9), and Cárcamo et al. (2020, Corollary 2.3),
among others. Result (b) in a convex-concave special case ($x$ and $h$ convex
in $\gamma$ and concave in $\delta$) has been obtained by Shapiro (2008, Propn
2.1). Our main interest is in part (c), the proof of which relies on parts (a)
and (b). The finiteness of $\Gamma_{0}(x_{0})$ is assumed in part (c) because
the mapping taking functions $x$ in $l^{\infty}(\Gamma\times\Delta)$ to
functions $\sup_{\delta\in\Delta}x(\gamma,\delta)$, $\gamma\in\Gamma$, in
$l^{\infty}(\Gamma)$ is in general not Hadamard directionally differentiable
without some additional assumptions.

With the general Lemmas 2 and 3 available, we now return to the GAN problem
and first give the proof of Lemma 1.

###### Proof of Lemma 1.

With the set $\Theta_{0}$ as in Section 2, consider the map
$\phi:l^{\infty}(\Gamma\times\Delta)\to\mathbb{R}$ defined, for any function
$x\in l^{\infty}(\Gamma\times\Delta)$, by

$\phi(x)=\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}}\Bigl{\\{}\sup_{\delta\in\Delta}x(\gamma_{0},\delta)-\min\bigl{\\{}x(\gamma_{0},\delta_{0}),\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x(\gamma,\delta)\bigr{\\}}\Bigr{\\}}.$

Noting that the function $\hat{Q}_{n}(\theta)$ in (13), here for clarity
evaluated at any $\theta_{0}\in\Theta$, can be written as

$\hat{Q}_{n}(\theta_{0})=\sup_{\delta\in\Delta}\hat{f}_{n}(\gamma_{0},\delta)-\min\bigl{\\{}\hat{f}_{n}(\gamma_{0},\delta_{0}),\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}\hat{f}_{n}(\gamma,\delta)\bigr{\\}},$

it is immediate that
$\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)=n^{1/2}\phi(\hat{f}_{n})$.
Moreover, by the definition of
$\Theta_{0}$,$\sup_{\delta\in\Delta}f(\gamma_{0},\delta)=f(\gamma_{0},\delta_{0})=\inf_{\gamma\in\Gamma}\sup_{\delta\in\Delta}f(\gamma,\delta)$
for all $(\gamma_{0},\delta_{0})\in\Theta_{0}$, so that $\phi(f)=0$ and
$\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)=n^{1/2}(\phi(\hat{f}_{n})-\phi(f))$.
Applying Lemma 3(c) with the function $x_{0}=f\in C(\Gamma\times\Delta)$, the
sets $\Theta_{0}(f)$ and $\Gamma_{0}(f)$ in Lemma 3(c) reduce to the sets
$\Theta_{0}$ and $\Gamma_{0}$ in (6) and in Assumption 5. On the other hand,
by Assumption 4,
$n^{1/2}(\hat{f}_{n}(\theta)-f(\theta))\rightsquigarrow\mathbb{G}(\theta)$ in
$l^{\infty}(\Theta)$ with $\mathbb{G}$ taking values in $C(\Theta)$ with
probability one. Combining these facts and using the functional delta method
of Lemma 2 now yields the desired result
$\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)=n^{1/2}(\phi(\hat{f}_{n})-\phi(f))\rightsquigarrow\phi_{f}^{\prime}(\mathbb{G})$
where

(25)
$\phi_{f}^{\prime}(\mathbb{G})=\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}}\Bigl{\\{}\sup_{\delta\in\Delta_{0}(\gamma_{0},f)}\mathbb{G}(\gamma_{0},\delta)-\min\bigl{\\{}\mathbb{G}(\gamma_{0},\delta_{0}),\underset{(\gamma,\delta)\in\Theta_{0}}{{\textstyle\inf_{\gamma}}\thinspace{\textstyle\sup_{\delta}}}\thinspace\mathbb{G}(\gamma,\delta)\bigr{\\}}\Bigr{\\}}.$

∎

Expression (25) gives the limiting distribution $L$ in Assumption 3. In
general this distribution is rather complicated. To illustrate, consider the
case with multiple but finite number of distinct solutions mentioned after
Theorem 2
($\Theta_{0}=\\{(\gamma_{01},\delta_{01}),\ldots,(\gamma_{0K},\delta_{0K})\\}$
with $K>1$ and $\gamma_{0i}\neq\gamma_{0j}$, $\delta_{0i}\neq\delta_{0j}$ for
all $i\neq j$). It may be helpful to note that (25) can equivalently be
expressed as

$\displaystyle\adjustlimits{\sup}_{\gamma_{0}\in\Gamma_{0}}{\sup}_{\delta_{0}\in\Delta_{0}(\gamma_{0},f)}\Bigl{\\{}\max\bigl{\\{}$
$\displaystyle\sup_{\delta\in\Delta_{0}(\gamma_{0},f)}\mathbb{G}(\gamma_{0},\delta)-\mathbb{G}(\gamma_{0},\delta_{0}),$
$\displaystyle\qquad\sup_{\delta\in\Delta_{0}(\gamma_{0},f)}\mathbb{G}(\gamma_{0},\delta)-\adjustlimits{\inf}_{\gamma\in\Gamma_{0}}{\sup}_{\delta\in\Delta_{0}(\gamma,f)}\thinspace\mathbb{G}(\gamma,\delta)\bigr{\\}}\Bigr{\\}}$

and in the said special case this expression reduces to

(26)
$\max_{k=1,\ldots,K}\mathbb{G}(\gamma_{0k},\delta_{0k})-\min_{k=1,\ldots,K}\mathbb{G}(\gamma_{0k},\delta_{0k})$

which also satisfies the continuity requirement of Theorem 2 due to Davydov et
al. (1998, Thm 11.1) (as the mapping from
$(\mathbb{G}(\gamma_{01},\delta_{01}),\ldots,\mathbb{G}(\gamma_{0K},\delta_{0K}))$
to (26) is convex and
$\inf_{\theta\in\Theta_{0}}\mathbb{E}[(\mathbb{G}(\theta))^{2}]\linebreak>0$
by Assumption 4).

When the set $\Theta_{0}$ is a singleton, the derivative map
$\phi_{f}^{\prime}$ in (25) is identically zero and the corresponding limiting
distribution $L$ is degenerate. To obtain an informative limiting distribution
in this case, our approach would need to be modified. One possibility is to
apply a higher-order functional delta method based on second-order directional
derivatives similarly as in the second-order analysis for minimization
problems in Shapiro et al. (2009, Sec 5.1.3). However, we do not pursue this
further as in the case of a single solution confidence sets can be directly
formed based on the asymptotic normality result of Biau et al. (2020, Thm
4.3).

We are now ready to prove Theorem 2.

###### Proof of Theorem 2.

Suppose the confidence sets $\hat{\operatorname{CS}}_{n,1-\alpha}$
($n=1,2,\ldots$) are formed using Procedure 1 so that
$\hat{\operatorname{CS}}_{n,1-\alpha}=S_{\hat{j}_{n}}$ for some indices
$\hat{j}_{n}$. First consider the probabilities
$\mathbb{P}[\Theta_{0}\nsubseteq\hat{\operatorname{CS}}_{n,1-\alpha}]$. In the
event $\Theta_{0}\nsubseteq\hat{\operatorname{CS}}_{n,1-\alpha}$, necessarily
$\hat{j}_{n}\geq 2$ (as $\Theta_{0}\subseteq\Theta=S_{1}$) and, by the
construction of $\hat{\operatorname{CS}}_{n,1-\alpha}$ in Procedure 1, there
exists an index $\hat{i}_{n}\in\\{2,\ldots,\hat{j}_{n}\\}$ such that
$\Theta_{0}\subseteq{S\/}_{\hat{i}_{n}-1}$ but
$\Theta_{0}\nsubseteq{S\/}_{\hat{i}_{n}}$. For such an index $\hat{i}_{n}$,
there exists a $\theta^{\bullet}$ such that $\theta^{\bullet}\in\Theta_{0}$
but $\theta^{\bullet}\notin{S\/}_{\hat{i}_{n}}$ and then

$\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)\geq
n^{1/2}\hat{Q}_{n}(\theta^{\bullet})>\hat{c}_{n,b,1-\alpha}({S\/}_{\hat{i}_{n}-1})\geq\hat{c}_{n,b,1-\alpha}(\Theta_{0}),$

where the first inequality holds as $\theta^{\bullet}\in\Theta_{0}$, the
second inequality is due to the definition of ${S\/}_{\hat{i}_{n}}$, and the
third inequality follows from the fact that
$\Theta_{0}\subseteq{S\/}_{\hat{i}_{n}-1}$ (this last fact implies that
$\hat{L}_{n,b}(\Theta_{0},x)\geq\hat{L}_{n,b}({S\/}_{\hat{i}_{n}-1},x)$ for
all $x$, and thus the third inequality in the previous display follows).
Therefore it holds that

(27)
$\mathbb{P}[\Theta_{0}\subseteq\hat{\operatorname{CS}}_{n,1-\alpha}]=1-\mathbb{P}[\Theta_{0}\nsubseteq\hat{\operatorname{CS}}_{n,1-\alpha}]\geq\mathbb{P}[\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)\leq\hat{c}_{n,b,1-\alpha}(\Theta_{0})].$

Now consider the sample quantiles $\hat{c}_{n,b,1-\alpha}(\Theta_{0})$
appearing in (27) and the associated empirical distribution function
$\hat{L}_{n,b}(\Theta_{0},x)$. By Assumption 3 and the conditions imposed on
the subsampling size $b$ in Theorem 2, the requirements of Theorem 2.2.1(i) of
Politis et al. (1999) are satisfied so that
$\hat{L}_{n,b}(\Theta_{0},x)\overset{p}{\to}L(x)$ for any $x$ that is a
continuity point of the cdf $L(\cdot)$. In particular, for any $\epsilon>0$
for which $c_{1-\alpha}-\epsilon$ is such a continuity point,
$\hat{L}_{n,b}(\Theta_{0},c_{1-\alpha}-\epsilon)\overset{p}{\to}L(c_{1-\alpha}-\epsilon)<1-\alpha$
where the inequality holds by the definition of $c_{1-\alpha}$. Together with
the definition of $\hat{c}_{n,b,1-\alpha}(\Theta_{0})$, this implies that
$\lim_{n\to\infty}\mathbb{P}[c_{1-\alpha}-\epsilon\leq\hat{c}_{n,b,1-\alpha}(\Theta_{0})]=1$.
Elementary probability rules now imply that

$\displaystyle\mathbb{P}[\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)\leq\hat{c}_{n,b,1-\alpha}(\Theta_{0})]$
$\displaystyle\quad\geq\mathbb{P}[\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)\leq\hat{c}_{n,b,1-\alpha}(\Theta_{0})\text{
and }c_{1-\alpha}-\epsilon\leq\hat{c}_{n,b,1-\alpha}(\Theta_{0})]$
$\displaystyle\quad\geq\mathbb{P}[\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)\leq
c_{1-\alpha}-\epsilon\text{ and
}c_{1-\alpha}-\epsilon\leq\hat{c}_{n,b,1-\alpha}(\Theta_{0})]$
$\displaystyle\quad=\mathbb{P}[\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)\leq
c_{1-\alpha}-\epsilon]+o(1),$

where the $o(1)$ term converges to zero as $n\to\infty$ due to
$\lim_{n\to\infty}\mathbb{P}[c_{1-\alpha}-\epsilon\leq\hat{c}_{n,b,1-\alpha}(\Theta_{0})]=1$.
Therefore

(28)
$\liminf_{n\to\infty}\mathbb{P}[\sup_{\theta\in\Theta_{0}}n^{1/2}\hat{Q}_{n}(\theta)\leq\hat{c}_{n,b,1-\alpha}(\Theta_{0})]\geq
L(c_{1-\alpha}-\epsilon).$

Using (27) and (28) and choosing a sequence $\epsilon_{k}>0$, $k=1,2,\ldots$,
such that $c_{1-\alpha}-\epsilon_{k}$ is a continuity point of $L(\cdot)$ for
each $k=1,2,\ldots$ and such that $\epsilon_{k}\to 0$ as $k\to\infty$ now
yields the result of Theorem 2. ∎

We close this appendix with the proof of Lemma 3.

###### Proof of Lemma 3.

We omit the proof of (the well-known) part (a). To prove part (b), first note
that for any $x^{\prime},x^{\prime\prime}\in l^{\infty}(\Gamma\times\Delta)$
we have (making use of the two inequalities in (19)) that

(29)
$|\phi(x^{\prime})-\phi(x^{\prime\prime})|\leq\sup_{\gamma\in\Gamma}|\sup_{\delta\in\Delta}x^{\prime}(\gamma,\delta)-\sup_{\delta\in\Delta}x^{\prime\prime}(\gamma,\delta)|\leq\sup_{(\gamma,\delta)\in\Gamma\times\Delta}|x^{\prime}(\gamma,\delta)-x^{\prime\prime}(\gamma,\delta)|$

so that the map $\phi$ is Lipschitz and it therefore suffices to consider
Gateaux directional differentiability (recall the remarks following Lemma 2).
To this end, pick any $x,h\in C(\Gamma\times\Delta)$ and a sequence
$\\{t_{n}\\}\subset\mathbb{R}_{+}$ such that $t_{n}\downarrow 0$ as
$n\to\infty$. Denote $x_{(n)}=x+t_{n}h$. For any $\gamma$ and each $n$, set
$\Delta_{0}(\gamma,x)=\\{\delta_{0}\in\Delta:x(\gamma,\delta_{0})=\sup_{\delta\in\Delta}x(\gamma,\delta)\\}$
and
$\Delta_{0}(\gamma,x_{(n)})=\\{\delta_{0}\in\Delta:x_{(n)}(\gamma,\delta_{0})=\sup_{\delta\in\Delta}x_{(n)}(\gamma,\delta)\\}$;
by continuity and compactness, these sets are non-empty. In what follows, we
consider the difference quotient

$D_{n}=t_{n}^{-1}\Bigl{[}\thinspace\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x_{(n)}(\gamma,\delta)-\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x(\gamma,\delta)\thinspace\Bigr{]}$

and prove that

(30)
$\limsup_{n\to\infty}D_{n}\leq\underset{(\gamma,\delta)\in\Theta_{0}(x)}{{\textstyle\inf_{\gamma}}\thinspace{\textstyle\sup_{\delta}}}\thinspace
h(\gamma,\delta)\leq\liminf_{n\to\infty}D_{n},$

implying the desired result.

To study the limes superior of $D_{n}$, first choose an arbitrary
$\gamma^{*}\in\Gamma$ such that $(\gamma^{*},\delta)\in\Theta_{0}(x)$ for some
$\delta\in\Delta$. Then, for each $n$, choose a
$\delta_{n}\in\Delta_{0}(\gamma^{*},x_{(n)})$ (we suppress the dependence of
$\delta_{n}$ on $\gamma^{*}$). By the definitions of the infimum and
$\Delta_{0}(\gamma^{*},x_{(n)})$,

$\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x_{(n)}(\gamma,\delta)\leq\sup_{\delta\in\Delta}x_{(n)}(\gamma^{*},\delta)=x_{(n)}(\gamma^{*},\delta_{n})$

and by the definitions of $\gamma^{*}$, $\Theta_{0}(x)$, and the supremum,

$\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x(\gamma,\delta)=\sup_{\delta\in\Delta}x(\gamma^{*},\delta)\geq
x(\gamma^{*},\delta_{n}).$

Therefore

$\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x_{(n)}(\gamma,\delta)-\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x(\gamma,\delta)\leq
x_{(n)}(\gamma^{*},\delta_{n})-x(\gamma^{*},\delta_{n})=t_{n}h(\gamma^{*},\delta_{n})$

so that $D_{n}\leq h(\gamma^{*},\delta_{n})$. Now let $\delta_{n_{k}}$be an
arbitrary convergent subsequence of $\delta_{n}$ (compactness of $\Delta$
ensures the existence of such subsequences) and suppose
$\delta_{n_{k}}\to\delta^{*}$ with $\delta^{*}\in\Delta$ as $k\to\infty$. As
(for all $k$ and all $\delta\in\Delta$)

$x(\gamma^{*},\delta_{n_{k}})+t_{n_{k}}h(\gamma^{*},\delta_{n_{k}})=x_{(n_{k})}(\gamma^{*},\delta_{n_{k}})\geq
x_{(n_{k})}(\gamma^{*},\delta)=x(\gamma^{*},\delta)+t_{n_{k}}h(\delta),$

it also holds (for all $k$ and all $\delta\in\Delta$) that

$x(\gamma^{*},\delta^{*})\geq
x(\gamma^{*},\delta)+t_{n_{k}}h(\gamma^{*},\delta)-t_{n_{k}}h(\gamma^{*},\delta_{n_{k}})+x(\gamma^{*},\delta^{*})-x(\gamma^{*},\delta_{n_{k}}).$

Letting $k$ tend to infinity, continuity of $x$ implies that
$x(\gamma^{*},\delta^{*})\geq x(\gamma^{*},\delta)$ for all $\delta\in\Delta$.
Therefore, by the definition of $\gamma^{*}$,

$x(\gamma^{*},\delta^{*})=\sup_{\delta\in\Delta}x(\gamma^{*},\delta)=\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x(\gamma,\delta)$

so that $\delta^{*}$ is such that $\delta^{*}\in\Delta_{0}(\gamma^{*},x)$ and
$(\gamma^{*},\delta^{*})\in\Theta_{0}(x)$. Continuity of $h$ now implies that
$h(\gamma^{*},\delta_{n_{k}})\to
h(\gamma^{*},\delta^{*})\leq\sup_{\delta\in\Delta_{0}(\gamma^{*},x)}h(\gamma^{*},\delta)$
as $k\to\infty$. As the convergent subsequence $\delta_{n_{k}}$was arbitrary,
$\limsup_{n\to\infty}h(\gamma^{*},\delta_{n})\leq\sup_{\delta\in\Delta_{0}(\gamma^{*},x)}h(\gamma^{*},\delta)$.
Therefore also
$\limsup_{n\to\infty}D_{n}\linebreak\leq\sup_{\delta\in\Delta_{0}(\gamma^{*},x)}h(\gamma^{*},\delta)$.
As $\gamma^{*}\in\Gamma$ was arbitrary except for satisfying
$(\gamma^{*},\delta)\in\Theta_{0}(x)$ for some $\delta\in\Delta$,

$\limsup_{n\to\infty}D_{n}\leq\underset{(\gamma,\delta)\in\Theta_{0}(x)}{{\textstyle\inf_{\gamma}}\thinspace{\textstyle\sup_{\delta}}}\thinspace
h(\gamma,\delta).$

For the limes inferior of $D_{n}$, it suffices to note that for any
$\gamma\in\Gamma$ and for all $n$

$\sup_{\delta\in\Delta}x_{(n)}(\gamma,\delta)\geq\sup_{\delta\in\Delta_{0}(\gamma,x)}x_{(n)}(\gamma,\delta)=\sup_{\delta\in\Delta}x(\gamma,\delta)+t_{n}\sup_{\delta\in\Delta_{0}(\gamma,x)}h(\gamma,\delta)$

so that (for any $\gamma\in\Gamma$ and for all $n$)

$\displaystyle
t_{n}^{-1}[\sup_{\delta\in\Delta}x_{(n)}(\gamma,\delta)-\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x(\gamma,\delta)]$
$\displaystyle\qquad\geq
t_{n}^{-1}[\sup_{\delta\in\Delta}x(\gamma,\delta)-\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x(\gamma,\delta)]+\\!\sup_{\delta\in\Delta_{0}(\gamma,x)}\\!h(\gamma,\delta)$
$\displaystyle\qquad\geq
t_{n}^{-1}[\sup_{\delta\in\Delta}x(\gamma,\delta)-\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x(\gamma,\delta)]$
$\displaystyle\qquad\qquad\qquad+\sup_{\delta\in\Delta_{0}(\gamma,x)}h(\gamma,\delta)-\underset{(\gamma,\delta)\in\Theta_{0}(x)}{{\textstyle\inf_{\gamma}}\thinspace{\textstyle\sup_{\delta}}}\thinspace
h(\gamma,\delta)+\underset{(\gamma,\delta)\in\Theta_{0}(x)}{{\textstyle\inf_{\gamma}}\thinspace{\textstyle\sup_{\delta}}}\thinspace
h(\gamma,\delta)$
$\displaystyle\qquad\geq\underset{(\gamma,\delta)\in\Theta_{0}(x)}{{\textstyle\inf_{\gamma}}\thinspace{\textstyle\sup_{\delta}}}\thinspace
h(\gamma,\delta).$

Taking an infimum over $\gamma\in\Gamma$ and then the limes inferior as
$n\to\infty$ establishes the latter inequality in (30), hence completing the
proof of part (b).

To prove part (c), we first use the former inequality in (19) and the triangle
inequality to note that for any $x^{\prime},x^{\prime\prime}\in
l^{\infty}(\Gamma\times\Delta)$

$\displaystyle|\phi(x^{\prime})-\phi(x^{\prime\prime})|$
$\displaystyle=\;\Bigr{|}\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\Bigl{\\{}\sup_{\delta\in\Delta}x^{\prime}(\gamma_{0},\delta)-\min\bigl{\\{}x^{\prime}(\gamma_{0},\delta_{0}),\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x^{\prime}(\gamma,\delta)\bigr{\\}}\Bigr{\\}}$
$\displaystyle\quad-\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\Bigl{\\{}\sup_{\delta\in\Delta}x^{\prime\prime}(\gamma_{0},\delta)-\min\bigl{\\{}x^{\prime\prime}(\gamma_{0},\delta_{0}),\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x^{\prime\prime}(\gamma,\delta)\bigr{\\}}\Bigr{\\}}\Bigl{|}$
$\displaystyle\leq\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\Bigr{|}\sup_{\delta\in\Delta}x^{\prime}(\gamma_{0},\delta)-\sup_{\delta\in\Delta}x^{\prime\prime}(\gamma_{0},\delta)\Bigl{|}$
$\displaystyle\quad+\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\Bigr{|}\min\bigl{\\{}x^{\prime}(\gamma_{0},\delta_{0}),\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x^{\prime}(\gamma,\delta)\bigr{\\}}-\min\bigl{\\{}x^{\prime\prime}(\gamma_{0},\delta_{0}),\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x^{\prime\prime}(\gamma,\delta)\bigr{\\}}\Bigl{|}.$

Using the former inequality in (19) again it is easy to see that the first
term on the majorant side of the previous display is dominated by
$\sup_{(\gamma,\delta)\in\Gamma\times\Delta}|x^{\prime}(\gamma,\delta)-x^{\prime\prime}(\gamma,\delta)|$.
Also the second term is dominated by the same quantity, as can be seen by
using the elementary inequality
$|\min\\{a,b\\}-\min\\{c,d\\}|\leq\max\\{|a-c|,|b-d|\\}$ and the inequalities
in (29). Thus the map $\phi$ in part (c) is Lipschitz and it suffices to
consider Gateaux directional differentiability.

Pick any $h\in C(\Gamma\times\Delta)$, a sequence
$\\{t_{n}\\}\subset\mathbb{R}_{+}$ such that $t_{n}\downarrow 0$ as
$n\to\infty$, and denote $x_{(n)}=x_{0}+t_{n}h$. Note that by the definition
of $\Theta_{0}(x_{0})$,

$\sup_{\delta\in\Delta}x_{0}(\gamma_{0},\delta)=x_{0}(\gamma_{0},\delta_{0})=\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x_{0}(\gamma,\delta)\quad\text{for
all }(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})$

so that $\phi(x_{0})=0$. These equalities also imply that
$\phi(x_{0}+t_{n}h)=\phi(x_{(n)})$ can be written as

$\displaystyle\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\Bigl{\\{}\sup_{\delta\in\Delta}x_{(n)}(\gamma_{0},\delta)-\sup_{\delta\in\Delta}x_{0}(\gamma_{0},\delta)$
$\displaystyle\qquad\qquad\qquad\;\;-\min\bigl{\\{}x_{(n)}(\gamma_{0},\delta_{0})-x_{0}(\gamma_{0},\delta_{0}),\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x_{(n)}(\gamma,\delta)-\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x_{0}(\gamma,\delta)\bigr{\\}}\Bigr{\\}}$

and therefore the difference quotient
$t_{n}^{-1}\\{\phi(x_{0}+t_{n}h)-\phi(x_{0})\\}=t_{n}^{-1}\\{\phi(x_{(n)})-\phi(x_{0})\\}$
can be written as

$\displaystyle\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\Bigl{\\{}t_{n}^{-1}\bigl{\\{}\sup_{\delta\in\Delta}x_{(n)}(\gamma_{0},\delta)-\sup_{\delta\in\Delta}x_{0}(\gamma_{0},\delta)\bigr{\\}}$
$\displaystyle\qquad\qquad\qquad-\min\bigl{\\{}h(\gamma_{0},\delta_{0}),t_{n}^{-1}\bigl{\\{}\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x_{(n)}(\gamma,\delta)-\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x_{0}(\gamma,\delta)\bigr{\\}}\bigr{\\}}\Bigr{\\}}.$

Introducing the shorthand notation

$\displaystyle A_{n}(\gamma)$
$\displaystyle=t_{n}^{-1}\bigl{\\{}\sup_{\delta\in\Delta}x_{(n)}(\gamma,\delta)-\sup_{\delta\in\Delta}x_{0}(\gamma,\delta)\bigr{\\}},$
$\displaystyle B_{n}$
$\displaystyle=t_{n}^{-1}\bigl{\\{}\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x_{(n)}(\gamma,\delta)-\adjustlimits{\inf}_{\gamma\in\Gamma}{\sup}_{\delta\in\Delta}x_{0}(\gamma,\delta)\bigr{\\}},$
$\displaystyle C_{n}(\gamma)$
$\displaystyle=\sup_{\delta\in\Delta_{0}(\gamma,x_{0})}h(\gamma,\delta),\qquad\quad
D_{n}=\underset{(\gamma,\delta)\in\Theta_{0}(x_{0})}{{\textstyle\inf_{\gamma}}\thinspace{\textstyle\sup_{\delta}}}\thinspace
h(\gamma,\delta),$

and using the inequality in (19) and the triangle inequality allow us to write

$\displaystyle\bigl{|}t_{n}^{-1}\bigl{\\{}\phi(x_{(n)})-\phi(x_{0})\bigr{\\}}-\phi_{x_{0}}^{\prime}(h)\bigr{|}$
$\displaystyle\qquad=\bigl{|}\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\bigl{\\{}A_{n}(\gamma_{0})-\min\\{h(\gamma_{0},\delta_{0}),B_{n}\\}\bigr{\\}}$
$\displaystyle\qquad\qquad\qquad-\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\bigl{\\{}C_{n}(\gamma_{0})-\min\\{h(\gamma_{0},\delta_{0}),D_{n}\\}\bigr{\\}}\bigr{|}$
$\displaystyle\qquad\leq\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\bigl{|}A_{n}(\gamma_{0})-C_{n}(\gamma_{0})\bigr{|}$
$\displaystyle\qquad\qquad\qquad+\sup_{(\gamma_{0},\delta_{0})\in\Theta_{0}(x_{0})}\bigl{|}\min\\{h(\gamma_{0},\delta_{0}),B_{n}\\}-\min\\{h(\gamma_{0},\delta_{0}),D_{n}\\}\bigr{|}.$

By the definition of the set $\Gamma_{0}(x_{0})$, the former term on the
majorant side of the previous display can be written as
$\sup_{\gamma_{0}\in\Gamma_{0}(x_{0})}|A_{n}(\gamma_{0})-C_{n}(\gamma_{0})|$.
For each fixed $\gamma_{0}\in\Gamma_{0}(x_{0})$ at a time,
$|A_{n}(\gamma_{0})-C_{n}(\gamma_{0})|\to 0$ as $n\to\infty$ by the
definitions of $A_{n}(\gamma)$ and $C_{n}(\gamma)$, part (a) of this lemma,
and the definition of Hadamard directional differentiability. By assumption,
the set $\Gamma_{0}(x_{0})$ is finite, implying that
$\sup_{\gamma_{0}\in\Gamma_{0}(x_{0})}|A_{n}(\gamma_{0})-C_{n}(\gamma_{0})|\to
0$ as $n\to\infty$ (note that this result does not generally hold if
$\Gamma_{0}(x_{0})$ is infinite). On the other hand, the elementary inequality
$|\min\\{a,b\\}-\min\\{c,d\\}|\leq\max\\{|a-c|,|b-d|\\}$ can be used to show
that the latter term on the majorant side of the previous display is dominated
by $|B_{n}-D_{n}|$. By the definitions of $B_{n}$ and $D_{n}$, part (b) of
this lemma, and the definition of Hadamard directional differentiability,
$|B_{n}-D_{n}|\to 0$ as $n\to\infty$. Overall, we have shown that
$|t_{n}^{-1}\\{\phi(x_{(n)})-\phi(x_{0})\\}-\phi_{x_{0}}^{\prime}(h)|\to 0$ as
$n\to\infty$, which completes the proof of part (c). ∎

## Appendix C: Sufficient conditions for Assumptions 1 and 4

Assumption GAN and condition
$\mathbb{E}[\sup_{\theta\in\Theta}|F(X,Z,\theta)|]<\infty$ imply Assumption
1(a) by Hoffmann-Jørgensen (1994, Sec 9.17) (cf. also the more transparent
Example 19.8 and Theorem 19.4 of van der Vaart (1998) where slightly stronger
assumptions are made).

When Assumption Smooth GAN holds, the mean value theorem implies that the
Lipschitz condition
$|F(x,z,\theta_{1})-F(x,z,\theta_{2})|\leq\sup_{\theta\in\Theta}|\partial
F(x,z,\theta)/\partial\theta|\,|\theta_{1}-\theta_{2}|$ holds for all
$\theta_{1},\theta_{2}\in\Theta$ where
$\mathbb{E}[\sup_{\theta\in\Theta}|\partial
F(X,Z,\theta)/\partial\theta|^{2}]<\infty$ by assumption. Example 19.7 and
Theorem 19.5 of van der Vaart (1998) now imply the weak convergence in
Assumption 4; moreover, it holds that
$\mathbb{E}[\sup_{\theta\in\Theta}(F(X,Z,\theta))^{2}]<\infty$ (van der Vaart
and Wellner, 1996, p. 129). Condition
$\inf_{\theta\in\Theta_{0}}\mathbb{E}[(\mathbb{G}(\theta))^{2}]>0$ follows
from the requirement
$\inf_{\theta\in\Theta}\operatorname{Var}[F(X,Z,\theta)]>0$ in Assumption
Smooth GAN. Therefore Assumptions 1(b) and 4 hold.

## References

  * Arjovsky and Bottou (2017) Arjovsky, M., Bottou, L., 2017. Towards principled methods for training generative adversarial networks. In: International Conference on Learning Representations.
  * Arjovsky et al. (2017) Arjovsky, M., Chintala, S., Bottou, L., 2017. Wasserstein generative adversarial networks. In: International Conference on Machine Learning. Vol. 70. pp. 214–223.
  * Arora et al. (2017) Arora, S., Ge, R., Liang, Y., Ma, T., Zhang, Y., 2017. Generalization and equilibrium in generative adversarial nets (GANs). In: International Conference on Machine Learning. Vol. 70. pp. 224–232.
  * Biau et al. (2020) Biau, G., Cadre, B., Sangnier, M., Tanielian, U., 2020. Some theoretical properties of GANs. Annals of Statistics 48, 1539–1566.
  * Bishop (2006) Bishop, C. M., 2006. Pattern Recognition and Machine Learning. Springer, New York.
  * Briot et al. (2020) Briot, J.-P., Hadjeres, G., Pachet, F., 2020. Deep Learning Techniques for Music Generation. Springer, Cham.
  * Cárcamo et al. (2020) Cárcamo, J., Cuevas, A., Rodríguez, L.-A., 2020. Directional differentiability for supremum-type functionals: statistical applications. Bernoulli 26, 2143–2175.
  * Chernozhukov et al. (2007) Chernozhukov, V., Hong, H., Tamer, E., 2007. Estimation and confidence regions for parameter sets in econometric models. Econometrica 75, 1243–1284.
  * Creswell et al. (2018) Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., Bharath, A. A., 2018. Generative adversarial networks: an overview. IEEE Signal Processing Magazine 35, 53–65.
  * Davydov et al. (1998) Davydov, Yu. A., Lifshits, M. A., Smorodina, N. V., 1998. Local Properties of Distributions of Stochastic Functionals. American Mathematical Society, Providence.
  * Diakonikolas et al. (2021) Diakonikolas, J., Daskalakis, C., Jordan, M. I., 2021. Efficient methods for structured nonconvex-nonconcave min-max optimization. In: Artificial Intelligence and Statistics. Vol. 130. pp. 2746–2754.
  * Fang and Santos (2019) Fang, Z., Santos, A., 2019. Inference on directionally differentiable functions. Review of Economic Studies 86, 377–412.
  * Fiez and Ratliff (2021) Fiez, T., Ratliff, L. J., 2021. Local convergence analysis of gradient descent ascent with finite timescale separation. In: International Conference on Learning Representations.
  * Gill (1989) Gill, R. D., 1989. Non-and semi-parametric maximum likelihood estimators and the von Mises method (part 1). Scandinavian Journal of Statistics 16, 97–128.
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., Courville, A., 2016. Deep Learning. MIT Press, Cambridge.
  * Goodfellow et al. (2014) Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., Bengio, Y., 2014. Generative adversarial nets. In: Advances in Neural Information Processing Systems. Vol. 27. pp. 2672–2680.
  * Goodfellow et al. (2020) Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., Bengio, Y., 2020. Generative adversarial networks. Communications of the ACM 63, 139–144.
  * Gupta and Zou (2019) Gupta, A., Zou, J., 2019. Feedback GAN for DNA optimizes protein functions. Nature Machine Intelligence 1, 105–111.
  * Hoffmann-Jørgensen (1994) Hoffmann-Jørgensen, J., 1994. Probability with a View Towards Statistics. Vol. II. Chapman & Hall, New York.
  * Jin et al. (2020) Jin, C., Netrapalli, P., Jordan, M., 2020. What is local optimality in nonconvex-nonconcave minimax optimization? In: International Conference on Machine Learning. Vol. 119. pp. 4880–4889.
  * Karras et al. (2020) Karras, T., Laine, S., Aila, T., 2020. A style-based generator architecture for generative adversarial networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. Early Access, https://doi.org/10.1109/TPAMI.2020.2970919.
  * Kim et al. (2020) Kim, S. W., Zhou, Y., Philion, J., Torralba, A., Fidler, S., 2020. Learning to simulate dynamic environments with GameGAN. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1228–1237.
  * Lehmann and Romano (2005) Lehmann, E. L., Romano, J. P., 2005. Testing Statistical Hypotheses, 3rd Edition. Springer, New York.
  * Liu et al. (2017) Liu, S., Bousquet, O., Chaudhuri, K., 2017. Approximation and convergence properties of generative adversarial learning. In: Advances in Neural Information Processing Systems. Vol. 30. pp. 5545–5553.
  * Ma et al. (2021) Ma, Z., Wang, F., Wang, W., Zhong, Y., Dai, H., 2021. Deep learning for in vivo near-infrared imaging. Proceedings of the National Academy of Sciences 118, e2021446118.
  * Mangoubi and Vishnoi (2021) Mangoubi, O., Vishnoi, N. K., 2021. Greedy adversarial equilibrium: an efficient alternative to nonconvex-nonconcave min-max optimization. In: ACM Symposium on Theory of Computing. To appear.
  * Mescheder et al. (2018) Mescheder, L., Geiger, A., Nowozin, S., 2018. Which training methods for GANs do actually converge? In: International Conference on Machine Learning. Vol. 80. pp. 3481–3490.
  * Miller (2019) Miller, A. I., 2019. The Artist in the Machine. MIT Press, Cambridge.
  * Molchanov (2017) Molchanov, I., 2017. Theory of Random Sets, 2nd Edition. Springer, London.
  * Molinari (2020) Molinari, F., 2020. Microeconometrics with partial identification. In: Handbook of Econometrics. Vol. 7A. Elsevier, pp. 355–486.
  * Nagarajan and Kolter (2017) Nagarajan, V., Kolter, J. Z., 2017. Gradient descent GAN optimization is locally stable. In: Advances in Neural Information Processing Systems. Vol. 30. pp. 5585–5595.
  * Paganini et al. (2018) Paganini, M., de Oliveira, L., Nachman, B., 2018. Accelerating science with generative adversarial networks: an application to 3D particle showers in multilayer calorimeters. Physical Review Letters 120, 042003.
  * Pan et al. (2019) Pan, Z., Yu, W., Yi, X., Khan, A., Yuan, F., Zheng, Y., 2019. Recent progress on generative adversarial networks (GANs): a survey. IEEE Access 7, 36322–36333.
  * Politis et al. (1999) Politis, D. N., Romano, J. P., Wolf, M., 1999. Subsampling. Springer, New York.
  * Reeds (1976) Reeds, J., 1976. On the definition of von Mises functionals. Ph.D. thesis, Harvard University.
  * Rockafellar and Wets (2009) Rockafellar, R. T., Wets, R. J.-B., 2009. Variational Analysis. Springer, Berlin.
  * Romano and Shaikh (2010) Romano, J. P., Shaikh, A. M., 2010. Inference for the identified set in partially identified econometric models. Econometrica 78, 169–211.
  * Schawinski et al. (2017) Schawinski, K., Zhang, C., Zhang, H., Fowler, L., Santhanam, G. K., 2017. Generative adversarial networks recover features in astrophysical images of galaxies beyond the deconvolution limit. Monthly Notices of the Royal Astronomical Society 467, L110–L114.
  * Shapiro (1990) Shapiro, A., 1990. On concepts of directional differentiability. Journal of Optimization Theory and Applications 66, 477–487.
  * Shapiro (1991) Shapiro, A., 1991. Asymptotic analysis of stochastic programs. Annals of Operations Research 30, 169–186.
  * Shapiro (2008) Shapiro, A., 2008. Asymptotics of minimax stochastic programs. Statistics & Probability Letters 78, 150–157.
  * Shapiro et al. (2009) Shapiro, A., Dentcheva, D., Ruszczyński, A., 2009. Lectures on Stochastic Programming. SIAM, Philadelphia.
  * van der Vaart (1998) van der Vaart, A. W., 1998. Asymptotic Statistics. Cambridge University Press, Cambridge.
  * van der Vaart and Wellner (1996) van der Vaart, A. W., Wellner, J. A., 1996. Weak Convergence and Empirical Processes. Springer, New York.