amounts to taking $\zeta$ to be the category with the largest probability,
i.e., a rich-get-richer rule. Then the predictive IM has contour

$\pi_{y}(z)=\sup_{\theta}\mathsf{P}_{Y,Z|\theta}\\{R(Y,Z)\leq R(y,z)\\},\quad
z\in\\{1,2,\ldots,K\\}.$

For illustration, consider the application in Goodman, (1965) and Denœux,
(2006) with $n=220$ psychiatric patients and $K=4$ categories corresponding to
four diagnoses: neurotic, depressed, schizophrenic, or having a personality
disorder. The observed counts are $y=(91,49,37,43)$, so there’s a clear
tendency towards the first category, corresponding to neurosis. Figure 9 shows
a plot of the IM’s predictive plausibility contour. As expected, the
plausibility contour mode is the first category, but the other three
categories have non-negligible plausibility too. In fact, all reasonable
$100(1-\alpha)$% prediction sets for $Z$ contain all four categories. The same
conclusion is drawn from other approaches, e.g., Denœux, (2006) develops a
belief function fo prediction and, for small $\alpha$, the smallest set that
his method assigns at least $1-\alpha$ belief to is all four categories.

Figure 9: Plot of the predictive IM plausibility contour, $\pi_{y}(z)$, for
the real-data illustration in Example 10 involving $K=4$ categories. Circles
correspond to the maximum likelihood estimator of
$\theta=(\theta_{1},\ldots,\theta_{4})$.

######  Example 11 (Gamma).

Consider a gamma model like in Example 6, where $Y=(Y_{1},\ldots,Y_{n})$
denotes an iid sample of size $n$ from a gamma distribution with an unknown
$\Theta=(\Theta_{1},\Theta_{2})$ consisting of the model’s shape and scale
parameter, respectively. The focus here, however, is on prediction of a future
observable, say, $Z$. One case, of course, is where $Z=Y_{n+1}$ is a
subsequent observation from the same gamma model, but that’s not the only
possibility. Another practically relevant case, not uncommon in reliability
applications (e.g., Hamada et al., 2004; Wang et al., 2012), is that where
$Z=\max\\{Y_{n+1},\ldots,Y_{n+k}\\}$ is the maximum of $k$-many future
observations, with $k$ a given non-negative integer. Note that $Z$ is
independent of $Y$. For this example here, I’ll focus on “Option 1” where the
IM plausibility contour $\pi_{y}(\theta)$ as presented in Example 10 of Part
II is combined with information about the $\theta$\- and $k$-dependent
distribution of $Z$ through the relationship (12), with the Fisher-based
function ${\cal K}$ recommended above. This is applicable since $Y$ and $Z$
are independent and I’m assuming, as before, that the prior information about
$\Theta$ is vacuous.

It’s easy to check that the probability density function for $Z$, given
$\Theta=\theta$ and a fixed $k$, is given by

$p_{\theta}^{(k)}(z)=k\,p_{\theta}(z)\,\\{1-P_{\theta}(z)\\}^{k-1},\quad z>0,$

where $p_{\theta}$ and $P_{\theta}$ are the density and distribution functions
of ${\sf Gamma}(\theta_{1},\theta_{2})$, with
$\theta=(\theta_{1},\theta_{2})$. Then the possibilistic representation of the
distribution of $Z$ is

$f_{\theta}(z)=\mathsf{P}_{Z|\theta}\\{p_{\theta}^{(k)}(Z)\leq
p_{\theta}^{(k)}(z)\\},\quad z>0,$

which, for given $\theta$ and $k$, can easily be evaluated based on Monte
Carlo. Plugging this and the joint contour $\pi_{y}(\theta)$ as presented in
Example 10 of Part II into the formula (12) gives a (strongly valid)
predictive IM with contour $\pi_{y}(z)$ for $Z$, given $Y=y$, with $k=3$. For
the same data as in Example 6, Figure 10 shows both the joint contour and this
predictive plausibility contour. The sample mean is about 113 and the sample
standard deviation is about 36, so one can’t rule out the possibility that $Z$
is considerably larger than the maximum in the sample, which is 165. So the
fact that the IM’s predictive contour has a long tail and the corresponding
90% prediction interval stretches has upper bound near 244 is not unexpected.

(a) Joint contour for $\Theta=(\text{shape, scale})$

(b) Predictive contour for $Z$

Figure 10: Panel (a) shows the joint IM contour for $\Theta$ based on the same
rat survival time data analyzed in Example 6. Panel (b) shows the IM’s
predictive contour for $Z$, the maximum of $k=3$ many future gamma
observations.

##  5 Non-parametric possibilistic IMs

###  5.1 Setup

To set the notation and terminology, suppose that the distribution
$\mathsf{P}_{Y|\theta}$ of the data $Y$ is indexed by $\theta\in\mathbb{T}$,
where $\theta$ is an infinite-dimensional index such as the density. The point
is that I’m just using $\theta$ to label the distribution
$\mathsf{P}_{Y|\theta}$, so it’s not a parametric model in the usual sense. To
make this point clear, I’ll directly write $\theta$ for the density/mass
function of $\mathsf{P}_{Y|\theta}$ in some of the expressions below. The non-
parametric problem assumes that the distribution of $Y$ is unknown, which
means there’s an uncertain variable $\Theta\in\mathbb{T}$ about which
inference is to be drawn. Just like in the previous sections of this paper,
it’ll often be the case that only some (finite-dimensional) feature $\Phi$ of
$\Theta$ is of interest; that is, $\Phi=f(\Theta)$ for some functional
$f:\mathbb{T}\to\mathbb{F}$.

In addition, there might also be (partial) prior information about $\Theta$ or
directly about $\Phi$ to be included in the model formulation. If $\Theta$
denotes the density function, then prior information might come in the form of
lending more credibility to densities that are smoother. An advantage,
however, of the partial prior IM framework is that one need not say anything
about the density $\Theta$ directly, available prior information about the
relevant feature $\Phi$ can be incorporated directly. Indeed, all one needs is
a contour $q_{\Phi}(\phi)$ and they’re ready to construct a valid, partial
prior-dependent IM for $\Phi$.

###  5.2 Valid IM construction

None of the theory presented in the previous sections of the paper required
that the unknowns were finite-dimensional, so there’s nothing new here. For
the non-parametric case, if partial prior information for $\Theta$ is
available and encoded in the contour $q_{\Theta}$, then I’ll first construct
the plausibility order

$R_{q}(y,\theta)=\frac{\theta(y)\,q_{\Theta}(\theta)}{\sup_{\vartheta\in\mathbb{T}}\\{\vartheta(y)\,q_{\Theta}(\vartheta)\\}},\quad(y,\theta)\in\mathbb{Y}\times\mathbb{T},$

and then define the IM for $\Theta$—or, equivalently, for the distribution of
$Y$ itself—as

$\pi_{y}(\theta)=\overline{\mathsf{P}}_{Y,\Theta}\\{R_{q}(Y,\Theta)\leq
R_{q}(y,\theta)\\},\quad\theta\in\mathbb{T},$

where $\overline{\mathsf{P}}_{Y,\Theta}$ is the upper joint distribution of
$(Y,\Theta)$ based on the non-parametric model and the partial prior
information. Obviously, since $\Theta$ is such a complex object and the
partial prior is completely general, there’s no opportunity for dimension
reduction and efficiency gain—we simply get what we get. But strong validity
of this IM for $\Theta$ holds by the general theory in Part II. In case prior
information about $\Theta$ is vacuous, then there’s an opportunity to reduce
dimension as before,

$\pi_{y}(\theta)=\mathsf{P}_{Y|\theta}\\{R(Y,\theta)\leq
R(y,\theta)\\},\quad\theta\in\mathbb{T},$ (13)

where, now, $R(y,\theta)=\theta(y)/\sup_{\vartheta\in\mathbb{T}}\vartheta(y)$
is the non-parametric relative likelihood. A complete prior can be handled in
this non-parametric case too, as described in Part II, i.e., rather than
fixing the value of $\theta$ as in the above display, I fix the value of $y$.
The point is that, in the vacuous- or complete-prior cases, there’s some
opportunity for dimension reduction: the vacuous prior collapses the Choquet
integral in the $\Theta$ dimension whereas the complete prior does so in the
$Y$ dimension.

Next, suppose the goal is inference on a finite- and probably relatively low-
dimensional feature $\Phi=f(\Theta)$, e.g., a moment or quantile. That the
quantity of interest is low-dimensional creates an opportunity for efficiency
gain compared to starting with the IM for $\Theta$ and marginalizing to $\Phi$
from there via the extension principle. Following the general framework
presented in Section 3, the construction of an IM for $\Phi$, based on partial
prior information encoded in the contour $q_{\Phi}$, starts with a likelihood-
driven plausibility ordering

$R_{q}(y,\phi)=\frac{\\{\sup_{\theta:f(\theta)=\phi}\theta(y)\\}\,q_{\Phi}(\phi)}{\sup_{\theta}\\{\theta(y)\,q_{\Phi}(f(\theta))\\}},\quad(y,\phi)\in\mathbb{Y}\times\mathbb{F}.$

In what follows, it’s important to remember that $\theta(y)$ represents
$\mathsf{P}_{Y|\theta}(\\{y\\})$, the distribution determined by $\theta$
evaluated at the data $y$. So the optimization over $\theta$ in the above
display is closely related to the so-called empirical likelihood framework
described by, e.g., Owen, (1988, 1990, 2001), Qin and Lawless, (1994), Tang
and Leng, (2010), and others. Then the IM construction proceeds exactly as
before, producing a contour

$\pi_{y}(\phi)=\overline{\mathsf{P}}_{Y,\Theta}\\{R_{q}(Y,f(\Theta))\leq
R_{q}(y,\phi)\\},\quad\phi\in\mathbb{F}.$

Strong validity holds by the general theory, so the critical question is how
to carry out the necessary computations. For this, let me consider the special
case of vacuous prior information, so that the plausibility ordering takes the
form

$R(y,\phi)=\frac{\sup_{\theta:f(\theta)=\phi}\theta(y)}{\sup_{\theta}\theta(y)},\quad(y,\phi)\in\mathbb{Y}\times\mathbb{F}.$
(14)

This is exactly the empirical likelihood ratio statistic discussed and
analyzed extensively in Owen, (2001). In that case, the IM contour function is

$\pi_{y}(\phi)=\sup_{\theta:f(\theta)=\phi}\mathsf{P}_{Y|\theta}\\{R(Y,\phi)\leq
R(y,\phi)\\},\quad\phi\in\mathbb{F},$ (15)

which is simply the p-value function for the empirical likelihood ratio test
of $H_{0}:\Phi=\phi$ based on the statistic (14). Under regularity conditions
(e.g., Owen, 2001, Theorem 3.6), when $Y=(Y_{1},\ldots,Y_{n})$ is an iid
sample and $n\to\infty$, a version of Wilks’s theorem applies to $-2$ times
the log empirical likelihood ratio, that is, it has a limiting chi-square
distribution that’s independent of $\theta$. Therefore, the strongly valid,
vacuous-prior IM for $\Phi$ can be well approximated by a chi-square tail
probability fairly generally. Other kinds of approximations are possible as
well; see Section 5.3. My point is that the very same general framework
developed above and illustrated in simple, low-dimensional parametric
inference problems can be readily applied to non-parametric problems. The
catch, however, is that computation is obviously a much more serious challenge
in these high-complexity settings. I imagine that there will be cases in which
it’s much more efficient—computationally and/or statistically—to construct an
approximate pivot by some other means than via the relative empirical
likelihood. The downside to a non-likelihood-based construction is that it’s
no longer clear how partial prior information can be incorporated in a
principled way. More on this in Section 6.

I should also mention briefly about how the ideas developed above would apply
to the case of prediction without parametric model assumptions, as I hinted at
in Section 4. Suppose, for simplicity, that $Y=(Y_{1},\ldots,Y_{n})$ and
$Z=Y_{n+1}$ consist of iid samples from a distribution
$\mathsf{P}_{Y|\Theta}$, where $\Theta$ is uncertain, and the goal is
prediction of $Z$. Following the prediction strategy above, and assuming prior
information about $\Theta$ is vacuous, I’d take the plausibility ordering to
be

$R(y,z)=\frac{\sup_{\theta\in\mathbb{T}}\\{\theta(y)\,\theta(z)\\}}{\sup_{x\in\mathbb{Y},\theta\in\mathbb{T}}\\{\theta(y)\,\theta(x)\\}},\quad(y,z)\in\mathbb{Y}^{n}\times\mathbb{Y}.$

From here, it’s straightforward to write down an expression for the predictive
IM for $Z$, given $Y=y$, that’s free of parametric model assumptions. The
problem, however, is that it’s not clear how the predictive IM can be
computed. If it turned out that $R(Y,Z)$ were a pivot, with a known
distribution independent of $\Theta$, then the predictive IM computations
would be immediate. Since it’s not clear if/when this pivotal structure holds,
one might consider an alternative strategy that sacrifices some of the
efficiency-related benefits of working exclusively with likelihoods for the
benefit of a pivotal structure that aids in computation; see Section 6 for
more discussion on this.

###  5.3 Approximations

In the previous subsection I already suggested the option to approximate the
IM’s plausibility contour for $\Phi$, in the vacuous prior case, using the
Wilks-like limiting distribution of the empirical likelihood ratio statistic.
One can also make certain adjustments (one-to-one transformations) to the
plausibility ordering, e.g., Bartlett correction (e.g., Owen, 2001, Sec. 3.3)
that would improve the accuracy of of this limiting approximation. But
asymptotic approximations aren’t the only approximation games in town and here
I want to mention two such alternatives. This discussion will be relatively
high-level, focusing on the main ideas rather than details. Also, for ease of
connecting this new framework to the existing literature, I’ll focus here
exclusively on the vacuous-prior case; extending these ideas to handle the
case with partial prior information is a huge open question that will have to
be resolved elsewhere.

The first of these alternatives is a relatively obvious one: just use the
bootstrap (e.g., Efron, 1979; Efron and Tibshirani, 1993; Owen, 1988). There
are, however, some subtleties in getting this to work properly, as I explain
next. To start, for simplicity, let $Y=(Y_{1},\ldots,Y_{n})$ consist of $n$
many iid observations from $\mathsf{P}_{Y|\Psi}$ depending on the unknown
(infinite-dimensional) $\Theta$. Suppose the goal is inference on
$\Phi=f(\Theta)$ and let $\hat{\phi}_{Y}$ denote the maximum likelihood
estimator of $\Phi$, i.e.,

$\hat{\phi}_{Y}=f(\hat{\theta}_{Y}),$

where $\hat{\theta}_{Y}$ is the (non-parametric) maximum likelihood estimator
of $\Theta$. For a bootstrap sample size $B$, let $y^{b}$ denote a random
sample of size $n$, with replacement, drawn from the observed data values
$y=(y_{1},\ldots,y_{n})$. Then a bootstrap approximation of the IM contour
$\pi_{y}(\phi)$ for $\Phi$, given $Y=y$, in (15) is

$\pi_{y}(\phi)\approx\pi_{y}^{\text{boot}}(\phi):=\frac{1}{B}\sum_{b=1}^{B}1\\{R(y^{b},\hat{\phi}_{y})\leq
R(y,\phi)\\},\quad\phi\in\mathbb{F},$ (16)

where $R$ is as given in (14). Two remarks deserve to be made here.

  * •

First, note that $\hat{\phi}_{y}$ remains fixed as $b=1,\ldots,B$. The reason
is that the $y^{b}$’s represent samples from the “population” $y$ and the
“true $\Phi$” corresponding to the “population $y$” is $\hat{\phi}_{y}$. As
the bootstrap story goes, if the $y$ sample is sufficiently informative, say,
as $n\to\infty$, then the “population $y$” approximates
$\mathsf{P}_{Y|\Theta}$ and, therefore, the $R(y^{b},\hat{\phi}_{y})$’s are
approximately representative samples of $R(Y,f(\Theta))$, so the approximation
in (16) should be relatively accurate.

  * •

Second, note that there’s no “$\sup_{\theta:f(\theta)=\phi}$” in (16) like
there is in (15). The reason is that, technically, strong validity only
requires that $\pi_{Y}(\Phi)$ be stochastically bounded at the “true $\Phi$”
or, in this case, at $f(\Theta)$. This control at the “true value” can’t be
achieved exactly with our less-than-fully-informative finite samples, so the
supremum is a conservative adjustment to make up for this shortcoming. In the
Utopian bootstrap world, where $n\to\infty$ and samples are fully informative,
the true values are recovered and there’s no need for a supremum. It’s no
different than the situation described above where the relative profile
empirical likelihood is an asymptotic pivot and, therefore, the supremum over
$\theta$ such that $f(\theta)=\phi$ drops out. The point is that there’s no
finite-sample strong validity guarantees for the bootstrap approximation in
(16), only asymptotically approximation strong validity as $n\to\infty$; see,
e.g., Cella and Martin, 2022a (, Theorem 1).

Another interesting but very different kind of approximation is that based on
the so-called universal inference framework developed in Wasserman et al.,
(2020). The developments here are based on (profile) relative likelihood
functions, but the previous make use of a split (profile) relative likelihood
function, which I explain below. Suppose, for simplicity, that $Y$ consists of
$n$ many iid observations like above. Then split this collection into two
chunks, denoted by $Y^{(1)}$ and $Y^{(2)}$, where, for concreteness, $Y^{(1)}$
corresponds to the first $\lceil n/2\rceil$ many observations and $Y^{(2)}$
the rest. Let $\hat{\theta}_{Y^{(2)}}$ denote the maximum likelihood estimator
of $\Theta$ based on the second chunk of data, i.e.,

$\hat{\theta}_{Y^{(2)}}=\arg\max_{\theta\in\mathbb{T}}\theta(Y^{(2)}).$

Now, for the quantity of interest $\Phi=f(\Theta)$, define the split-dependent
plausibility ordering given by

$R_{\text{split}}(y,\phi)=\frac{\sup_{\theta:f(\theta)=\phi}\theta(y^{(1)})}{\hat{\theta}_{y^{(2)}}(y^{(1)})},\quad(y,\phi)\in\mathbb{Y}\times\mathbb{F}.$

This is just a ratio of $y^{(1)}$-data likelihoods, the numerator profiled
over those $\psi$ satisfying $f(\theta)=\phi$ and the denominator evaluated at
$\hat{\theta}_{y^{(2)}}$. If both chunks of data are “similarly informative,”
then one would expect $\hat{\theta}_{y^{(1)}}\approx\hat{\theta}_{y^{(2)}}$,
in which case $R_{\text{split}}$ is just the relative profile likelihood based
on $y^{(1)}$. But note that $R_{\text{split}}(y,\phi)$ is not bounded above by
1; when this not-bounded-by-1 feature might be an issue, see below, it’s easy
enough to just truncate it at 1. Applying the general IM construction above to
this split-based plausibility order gives

$\pi_{y}(\phi)=\sup_{\theta:f(\theta)=\phi}\mathsf{P}_{Y|\theta}\\{R_{\text{split}}(Y,\phi)\leq
R_{\text{split}}(y,\phi)\\}.$

This is no more straightforward to compute than the original, no-split
contour, but it’s easy to approximate. Indeed, by Markov’s inequality,

$\displaystyle\pi_{y}(\phi)$
$\displaystyle=\sup_{\theta:f(\theta)=\phi}\mathsf{P}_{Y|\theta}\\{R_{\text{split}}(Y,\phi)\leq
R_{\text{split}}(y,\phi)\\}$
$\displaystyle=\sup_{\theta:f(\theta)=\phi}\mathsf{P}_{Y|\theta}\\{R_{\text{split}}(Y,\phi)^{-1}\geq
R_{\text{split}}(y,\phi)^{-1}\\}$ $\displaystyle\leq
1\wedge\Bigl{[}R_{\text{split}}(y,\phi)\,\sup_{\theta:f(\theta)=\phi}\mathsf{E}_{Y|\theta}\\{R_{\text{split}}(Y,\phi)^{-1}\\}\Bigr{]}$
$\displaystyle\leq 1\wedge R_{\text{split}}(y,\phi),$

where I’ve made use of the key result in Equation (6) of Wasserman et al.,
(2020), a simple consequence of the law of iterated expectation, which states
that the $\mathsf{P}_{Y|\theta}$-expected value of
$R_{\text{split}}(Y,\phi)^{-1}$ is no more than 1, uniformly in $\theta$ with
$f(\theta)=\phi$. Then the approximation I propose is to take

$\pi_{y}^{\text{split}}(\phi)=1\wedge R_{\text{split}}(y,\phi),$

which is relatively easy to compute—it only requires evaluating the split
relative profile likelihood function, no probability calculations necessary.
Moreover, it follows from Theorem 3 in Wasserman et al., (2020) that the
contour in the above display defines a strongly valid IM for $\Phi$, given
$Y=y$. This might appear too good to be true, but there’s a price for the
apparent simplicity: the data-splitting strategy generally results in a loss
of efficiency. This could be an acceptable trade-off in complex non-parametric
problems where there might not be any other options for constructing a valid
IM.

###  5.4 Examples

######  Example 12 (Multinomial).

Reconsider the multinomial model from Example 10, with
$\Theta=(\Theta_{1},\ldots,\Theta_{K})$ the $K$-dimensional vector in the
probability simplex $\mathbb{T}$, so that $\Theta_{k}$ denotes the probability
of class $k$, for $k=1,\ldots,K$. Since every discrete distribution on
$\\{1,\ldots,K\\}$ can be described by such a $\Theta$ vector, I refer to this
as the “discrete non-parametric” model. It’s for this reason that the
multinomial model, while relatively simple, is of fundamental importance. Many
of the more general non-parametric developments, such as Bayesian non-
parametrics via the Dirichlet process (e.g., Ferguson, 1973), can be seen as
extensions of the multinomial model.

More specifically, let $(Y\mid\Theta=\theta)\sim{\sf Mult}_{K}(n,\theta)$, so
that $Y=(Y_{1},\ldots,Y_{K})$ where $Y_{k}$ is the sample frequency count for
category $k$, with $k=1,\ldots,K$. As before, this determines a likelihood
function which, using the notation of this section, is given by
$\theta(y)\propto\prod_{k=1}^{K}\theta_{k}^{y_{k}}$, for
$\theta\in\mathbb{T}$. Assuming vacuous prior information for $\Theta$, just
for simplicity, the plausibility ordering is determined by the relative
likelihood alone,

$R(y,\theta)=\prod_{k=1}^{K}\Bigl{(}\frac{n\theta_{k}}{y_{k}}\Bigr{)}^{y_{k}},$

where I’ve plugged in the likelihood function maximizer, which is available in
closed form. From here it’s straightforward to evaluate the contour function
of the (strongly valid) IM for $\Theta$, via Monte Carlo, using the formula
(13).

This is the same multinomial problem considered in a recent discussion paper
(Jacob et al., 2021a ; Jacob et al., 2021b ) published in the Journal of the
American Statistical Association. They’re approaching the problem from the
perspective of Dempster–Shafer inference, so a comparison with the proposed
solution here makes sense. Jacob et al. compared their solution to another
with an IM-like flavor (Lawrence et al., 2009). Jacob, et al. consider two
real-data examples, both involving scientifically relevant questions
concerning the multinomial parameter $\Theta$, and I’ll reanalyze one of them
here.

In the $K=4$ case, Rao, (1973, p. 368) considered a so-called linkage model
wherein the parameter $\Theta=(\Theta_{1},\ldots,\Theta_{4})$ has the
following low-dimensional structure:

$\vartheta(\omega)=\Bigl{(}\frac{1}{2}+\frac{\omega}{4},\frac{1-\omega}{4},\frac{1-\omega}{4},\frac{\omega}{4}\Bigr{)},\quad\omega\in(0,1).$

In Rao’s example, the categories represent four different phenotypes in
animals, so the above model represents a simple(r) relationship between the
proportions of these phenotypes in the animal population in question. The data
analyzed in Jacob et al., 2021a  and in Lawrence et al., (2009) has observed
cell counts $y=(25,3,4,7)$. My focus here is on the question of whether the
above linkage model is plausible given the observed data. So I have in mind
the following assertion/hypothesis about the uncertain $\Theta$:

$A=\\{\text{$\Theta=\vartheta(\omega)$ for some $\omega\in(0,1)$}\\},$

and the goal is to calculate $\overline{\Pi}_{y}(A)$, the IM’s upper
probability assigned to the above assertion; if this quantity is small, then I
might be willing to reject the claim that the low-dimensional structure
imposed by the linkage model is present in this data example. By the IM’s
consonance structure, this amounts to solving a simple, one-dimensional
optimization problem

$\overline{\Pi}_{y}(A)=\sup_{\omega\in(0,1)}\pi_{y}\\{\vartheta(\omega)\\}.$

Figure 11 shows a plot444Note that this is different from marginalization via
the extension principle. In marginalization, every value of the full parameter
determines a value of the reduced parameter. Here, however, not all
$\Theta\in\mathbb{T}$ correspond to a $\vartheta(\omega)$ for some $\omega$.
This explains why the curve in Figure 11 doesn’t reach the value 1 as we’d
expect in the case of marginalization, e.g., Figure 2. of
$\omega\mapsto\pi_{y}\\{\vartheta(\omega)\\}$ and, clearly, the supremum is
attained at $\omega\approx 0.61$ and the plausibility there is $\approx 0.97$.
Here, obviously the plausibility is large, so the data shows effectively no
signs of disagreement with Rao’s linkage model that assumes $\Theta$ is of the
form $\vartheta(\omega)$ for some $\omega\in(0,1)$. This conclusion isn’t
surprising, given that Rao—who’s a pretty smart guy—already took the linkage
model as given for his analysis of these data. From here, one can take the
reduced model and carry out the IM construction to make inference on the
uncertain linkage parameter $\Omega\in(0,1)$ directly.

Figure 11: Plot of the function $\omega\mapsto\pi_{y}\\{\vartheta(\omega)\\}$
in the multinomial/genetic linkage model in Example 12. The maximum value is
$\overline{\Pi}_{y}(A)$, the plausibility that the unknown $\Theta$ satisfies
the linkage model constraint indexed by $\omega$.

######  Example 13 (Non-parametric quantile).

Suppose the goal is inference on, say, the $r^{\text{th}}$ quantile
$\Phi=\Phi_{r}$ of a completely unknown distribution, $\mathsf{P}_{Y|\Theta}$,
i.e., where $\Phi$ is such that

$\mathsf{P}_{Y|\Theta}(Y\leq\Phi)=r,\quad r\in(0,1).$

Let $Y=(Y_{1},\ldots,Y_{n})$ be an iid sample of size $n$ from this unknown
distribution. Following the general framework as described above, I want
$R(y,\phi)$ to be the empirical likelihood ratio statistic in (14) which, in
this case (e.g., Wasserman, 1990, Theorem 5), is

$R(y,\phi)=\Bigl{\\{}\frac{r}{u(y,\phi)}\Bigl{\\}}^{u(y,\phi)}\Bigl{\\{}\frac{1-r}{n-u(y,\phi)}\Bigr{\\}}^{n-u(y,\phi)},$

where

$u(y,\phi)=\begin{cases}|\\{i:y_{i}\leq\phi\\}|&\text{if
$\phi<\hat{\phi}_{y}$}\\\ nr&\text{if $\phi=\hat{\phi}_{y}$}\\\
|\\{i:y_{i}<\phi\\}|&\text{if $\phi>\hat{\phi}_{y}$},\end{cases}$

and $\hat{\phi}_{y}$ the $r^{\text{th}}$ quantile of the sample $y$. This is
assuming, for computational simplicity, that the prior information about
$\Phi$ is vacuous. In this case, it’s straightforward to get a bootstrap
approximation, $\pi_{y}^{\text{boot}}(\phi)$, of the marginal IM’s
plausibility contour for $\Phi$ as in (16). I simulated $n=25$ observations
from a ${\sf Gamma}(3,1)$ distribution and that bootstrap-based approximate
plausibility contour is plotted in Figure 12(a). The stair-step pattern is a
result of $R$ only depending on certain sample counts rather than the
numerical values. The true quantile in this case is $\approx 3.6$, which is
right near the peak of the plausibility contour, as desired.

Recall that this is only an approximate IM, so there’s no guarantee that
strong validity holds exactly. It’s not difficult, however, to get empirical
confirmation that validity does hold, at least approximately. Figure 12(b)
shows a plot of the distribution function

$\alpha\mapsto\mathsf{P}_{Y|\theta}\\{\pi_{Y}^{\text{boot}}(f(\theta))\leq\alpha\\},\quad\alpha\in[0,1],$
(17)

and the fact that this curve falls below the diagonal line is an indication
that strong validity holds. This is only for one distribution, namely, ${\sf
Gamma}(3,1)$ but the other experiments I conducted lead to the same
conclusion, namely, that strong validity holds for the bootstrap-based
approximation, even for relatively small $n$.

(a) Plausibility contour

(b) Distribution function (17)

Figure 12: Plots pertaining to Example 13. Panel (a) is the bootstrap
approximation (16) of the marginal IM’s plausibility contour for $\Phi$ based
on a sample of size $n=25$ from a ${\sf Gamma}(3,1)$ distribution; vertical
line is the true quantile, $\approx 3.6$. Panel (b) shows a Monte Carlo
approximation of the distribution function (17), for the same gamma model, and
that the curve falls below the diagonal line confirms strong validity.

######  Example 14 (Non-parametric mean).

Arguably the most fundamental problems in statistics is inference on the mean
of a population based on random sampling. Here I let $\Phi$ denote that
unknown mean but I assume nothing more about the underlying distribution,
$\Theta$, other than that its tails are such that it admits a finite mean. For
this case, assuming vacuous prior information about $\Phi$, just for
simplicity, I can follow the IM construction in (15), with $R(y,\phi)$ being
the empirical likelihood ratio statistic for the mean, which is fleshed out in
detail in, e.g., Owen, (2001, Ch. 2.9). For the computation of $R(y,\phi)$, I
used the function el.test in the R package emplik (Zhou, 2022). Using the
bootstrap approximation suggested above, I found the plausibility contour
$\pi_{y}^{\text{boot}}(\phi)$ for based on a real data set consisting of
$n=29$ observations on the density of the Earth relative to water taken by
Cavendish back in 1798 (Stigler, 1977, Table 8). The peak of the contour is,
of course, at the sample mean $\bar{y}=5.48$, and the circle marks the
plausibility contour $\pi_{y}^{\text{boot}}(\phi^{\star})$ at the “true value”
of $\Phi$, which is $\phi^{\star}=5.517$. The horizontal line at $\alpha=0.05$
determines the upper-$\alpha$ level set, so it’s clear that the true value is
contained in the $100(1-\alpha)$% plausibility region based on the analysis
here.

Figure 13: Plot of the IM plausibility contour for the mean $\Phi$, based on
the bootstrap approximation (16), in the nonparametric case considered in
Example 14. The data used here in this illustration are those measurements of
the density of the Earth relative to water, taken by Cavendish back in 1798
(Stigler, 1977, Table 8). The circle marks the plausibility evaluated at the
“true value” of $\Phi$, which is 5.517.

##  6 Possibilistic IMs without a likelihood

###  6.1 Setup

In all of the cases discussed previously in the paper and almost all of the
previous literature on IMs, the focus was on cases involving parametric models
that connect the observable data to the unknown quantities of interest, e.g.,
model parameters and/or future observations. But there is a wide class of
classical and modern problems that don’t fit this mold. Perhaps the simplest
of these problems is inference on an unknown quantile. More specifically,
suppose that $Y=(Y_{1},\ldots,Y_{n})$ is an iid sample from a distribution
$\mathsf{P}$ supported on $\mathbb{R}$ and, for a given $q\in(0,1)$, the
quantity of interest is the $q$-quantile of $\mathsf{P}$, defined by the
equation $\mathsf{P}(Y_{1}\leq\phi)=q$. It’s straightforward to estimate
$\theta$, via the corresponding sample quantile, and, if desired,
(approximate) confidence intervals are available. But what about
“probabilistic inference” in the sense that I’m concerned with here? Do I
first have to infer the whole (infinite-dimensional) $\mathsf{P}$ and then
marginalize to the scalar $\phi$? Wasserman, (2008) describes this gap
poignantly:

> The idea that statistical problems do not have to be solved as one coherent
> whole is anathema to Bayesians but is liberating for frequentists. To
> estimate a quantile, an honest Bayesian needs to put a prior on the space of
> all distributions and then find the marginal posterior. The frequentist need
> not care about the rest of the distribution and can focus on much simpler
> tasks.

My claim is that it’s the frequentists’ implicit imprecision that’s
liberating—they leave unspecified (via vacuous models) those things that
aren’t relevant to their analysis. My modest goal here is to suggest a
framework that would allow probabilistic inference without the anathema,
without the Bayesians’ requirement to have a (precise) model for everything.
The full force of this will be developed in a subsequent, follow-up paper.

Issues similar to those in the quantile problem arise is more modern problems.
Machine learning applications often start with a loss function
$(y,\phi)\mapsto\ell_{\phi}(y)$ mapping data and decision rule pairs
$(y,\phi)\in\mathbb{Y}\times\mathbb{F}$ to a loss incurred by applying rule
$\phi$ when data is $y$. Then the data analyst’s task boils down to estimation
of and inference on the rule that minimizes the risk, or expected loss, i.e.,
$\Phi=\arg\min_{\phi\in\mathbb{F}}\mathsf{P}\ell_{\phi}$. Alternatively, it
might be that $\Phi$ is defined as the solution to a so-called estimating
equation (e.g., Godambe, 1960; Huber, 1981), i.e., $\mathsf{P}z_{\Phi}=0$ for
a given (possibly vector-valued) function $(y,\phi)\mapsto z_{\phi}(y)$. In
any case, since a parametric statistical model isn’t required to define the
inferential target, the risk minimizer, estimating equation solution, etc.,
Manski’s law (Manski, 2003, p. 1) dictates that the data analyst make as few
model assumptions as possible. Lots of problems, including the quantile
example above, fit in this inference-on-risk-minimizers setting, so it’s
important to address this gap between frequentist, Bayesian, and other
probabilistic inference frameworks. Our work on generalized posterior
distributions (e.g., Syring and Martin, 2017, 2019, 2023; Martin and Syring,
2022; Wu and Martin, 2022, and the references therein) shows how to construct
“posterior distributions” without a likelihood function, thus providing a
generalization of Bayesian inference that’s particularly suited for cases
where the quantity of interest is a risk minimizer.

The key point at the heart of Wasserman’s remark and of what I’m suggesting
here is that, while it’s possible to connect the observable data $Y$ to
quantity of interest $\Phi$ by thinking in terms of a non-parametric model
(e.g., with an empirical likelihood as described in Section 5), this might not
be the most statistically, computationally, or conceptually efficient
solution. As an alternative, one might consider defining a plausibility order
for $\Phi$ in terms of a generic mapping $(y,\phi)\mapsto\rho(y,\phi)$ that
makes no reference to a (empirical, marginal, or profile) likelihood. Of
course, this lacks the principles developed in Part II for the case when a
model/likelihood is available, but efficient marginal inference in non-
parametric problems will likely require bending the rules a bit.

Since it’s currently not clear how one can incorporate partial prior
information about $\Phi$ into these no-likelihood applications—that’s an
important open problem—I’ll assume in what follows that the prior information
is vacuous. In that case, all we have to use is the mapping $\rho$, but the
principles detailed in Part II and applied above offer some guidance. If
$\Phi=f(\mathsf{P})$ is a relevant feature, some functional applied to the
uncertain distribution $\mathsf{P}$ for data $Y$, then I suggest constructing
a (marginal) IM for $\Phi$ with contour

$\pi_{y}(\phi)=\sup_{\mathsf{P}:f(\mathsf{P})=\phi}\mathsf{P}\\{\rho(Y,\phi)\leq\rho(y,\phi)\\},\quad\phi\in\mathbb{F}.$
(18)

All the desirable properties of the IMs constructed in this manner carry over
to this likelihood-free case. In particular, the plausibility regions
$\\{\phi:\pi_{y}(\phi)>\alpha\\}$ are exact $100(1-\alpha)$% confidence
regions. If one could choose $\rho$ such that $\rho(Y,\phi)$ is a pivot under
$\mathsf{P}$ with $f(\mathsf{P})=\phi$, then this would be easy to implement
via Monte Carlo. While this can be done in certain applications (Cella, 2023),
there are currently no broadly general strategies available for constructing
pivots. Below I’ll highlight, in two practically relevant scenarios, where
alternative strategies can be applied to make the above computation
manageable, and with little or no sacrifice in validity. These surveys are
meant to just to give an idea of what’s possible, further investigations are
needed.

###  6.2 Inference on risk minimizers

As described above, for a given loss function $(y,\phi)\mapsto\ell_{\phi}(y)$,
suppose that $\Phi$ is defined as the minimizer of the corresponding risk
(expected loss) function, i.e.,

$\Phi=\arg\min_{\phi\in\mathbb{F}}r(\phi),\quad\text{where}\quad
r(\phi)=\mathsf{P}\ell_{\phi}.$

This is an unknown/uncertain quantity because $\mathsf{P}$ itself is
unknown/uncertain. Since the goal is direct inference on $\Phi$, I don’t want
to introduce an indirectly-relevant “model parameter” $\Theta$ so that I can
form a likelihood as in the previous sections. Fortunately, the definition of
$\Phi$ as a risk minimizer is enough structure to suggest a plausibility
ordering $\rho$ and a corresponding marginal IM for $\Phi$.

Let $Y^{n}=(Y_{1},\ldots,Y_{n})$ consist of an iid sample from $\mathsf{P}$;
note that these could be independent–dependent variable pairs with joint
distribution $\mathsf{P}$, but I’ll not make this explicit in the notation.
For the observed $y^{n}$, the corresponding empirical risk is

$r_{y^{n}}(\phi)=\frac{1}{n}\sum_{i=1}^{n}\ell_{\phi}(y_{i}),$

and a natural estimate of $\Phi$ is obtained by minimizing the empirical risk:

$\hat{\phi}_{y^{n}}=\arg\min_{\phi}r_{y^{n}}(\phi).$

Analogous to the relative likelihood plausibility ordering, I propose the
following:

$\rho(y^{n},\phi)=\exp[-\\{r_{y^{n}}(\phi)-r_{y^{n}}(\hat{\phi}_{y^{n}})\\}]\in[0,1].$

(The exponential form isn’t necessary, that’s just to make it resemble the
relative likelihood.) From here, one can define a marginal IM for $\Phi$ with
contour as in (18). This is exactly the IM solution presented in Cella and
Martin, 2022a , and they proposed a bootstrap approximation analogous to that
in Section 5.3 above to carry out the necessary computations. With a bootstrap
approximation, there’s virtually no hope of having an exact validity result,
but they proved an asymptotic validity theorem and demonstrated the IM’s
strong finite-sample performance in simulations.

###  6.3 Prediction

Let’s revisit the prediction problem discussed in a few places above. Assume
that $Y^{n}=(Y_{1},\ldots,Y_{n})$ consists of iid observations from a common
distribution $\mathsf{P}$, and that the goal is to predict the next
observation $Y_{n+1}$. In fact, I can be even more general and assume that the
$Y$-process is exchangeable and that $\mathsf{P}$ is the full joint
distribution for the process. This is an extreme case of marginal inference,
where the entirety of the highly-complex $\mathsf{P}$ is a nuisance parameter
to be eliminated. It may not be realistic/attractive to introduce a density
function and a corresponding likelihood as suggested in Section 5.2 above, so
here I’ll avoid the use of likelihood. A popular prediction method in the
literature these days is conformal prediction (e.g., Vovk et al., 2005; Shafer
and Vovk, 2008). A close connection between conformal prediction and IMs has
already been demonstrated in Cella and Martin, 2022c ; Cella and Martin, 2022b
, and what I present below offers some new perspectives.

Let $\rho:\mathbb{Y}^{n}\times\mathbb{Y}$ be a mapping with two inputs: one is
a data set and the other is a candidate value for the next observation.
Without loss of generality, I’ll assume that $\rho(y^{n},y_{n+1})$ is a
measure of “conformity” of the candidate value $y_{n+1}$ with the data set
$y^{n}$; that is, larger values correspond to $y_{n+1}$ that’s consistent with
the values in $y^{n}$. For example, if $\hat{y}_{y^{n}}$ is a point prediction
of the next observation, then the plausibility order $\rho$ can be defined as

$\rho(y^{n},y_{n+1})=-d(\hat{y}_{y^{n}},y_{n+1}),$

where $d\geq 0$ is any suitable measure of distance between two points in
$\mathbb{Y}$. The only other constraint is that $\rho$ be symmetric in its
first argument, i.e., the data set $y^{n}$ can be shuffled arbitrarily without
affecting the value of $\rho$. Having specified this plausibility order, the
predictive IM contour for $Y_{n+1}$ is

$\pi_{y^{n}}(y_{n+1})=\sup_{\mathsf{P}}\mathsf{P}\\{\rho(Y^{n},Y_{n+1})\leq\rho(y^{n},y_{n+1})\\},\quad
y_{n+1}\in\mathbb{Y},$

where the supremum is over all exchangeable joint distributions for the full
$Y$-process. Note that the probability calculation above is with respect to
the joint distribution of $(Y^{n},Y_{n+1})$ under $\mathsf{P}$. Even though
this looks a little different than the setup above, all the same strong
(prediction) validity properties hold, in particular,

$\sup_{\mathsf{P}}\mathsf{P}\\{\pi_{Y^{n}}(Y_{n+1})\leq\alpha\\}\leq\alpha,\quad\text{all
$\alpha\in[0,1]$, all $n$}.$ (19)

The problem, of course, is that the supremum makes evaluation of the IM
contour unattainable. A key point, however, is that the supremum also makes
the resulting IM unnecessarily conservative. By applying the Principle of
Minimal Complexity from Part II, it’s possible to reduce the dimension of the
aforementioned Choquet integral, which makes the computation simpler and the
IM more efficient. As explained in Part II and applied in a few places above,
the implementation of the Principle often boils down to conditioning on things
that can be meaningfully conditioned on. In this case, the structure of the
problem makes it possible to condition on the set of values
$\\{y_{1},\ldots,y_{n},y_{n+1}\\}$ while leaving their arrangement
unspecified. This set is, of course, a minimal sufficient statistic, so
conditioning on this feature will eliminate the dependence on the unknown
$\mathsf{P}$, so the supremum drops out completely. That is, the new
predictive IM contour—based on conditioning and the aforementioned
Principle—is given by

$\displaystyle\pi_{y^{n}}(y_{n+1})$
$\displaystyle=\sup_{\mathsf{P}}\mathsf{P}\bigl{[}\rho(Y^{n},Y_{n+1})\leq\rho(y^{n},y_{n+1})\mid\\{y_{1},\ldots,y_{n},y_{n+1}\\}\bigr{]}$
$\displaystyle=\frac{1}{(n+1)!}\sum_{\sigma}1\\{\rho(y^{\sigma(1:n)},y_{\sigma(n+1)})\leq\rho(y^{n},y_{n+1})\\},$

where the sum is over all $(n+1)!$ many permutations, $\sigma$, of the
integers $1,\ldots,n,n+1$. Finally, since $\rho$ is symmetric in its first
argument, the right-hand side above can be further simplfied:

$\pi_{y^{n}}(y_{n+1})=\frac{1}{n+1}\sum_{i=1}^{n+1}1\\{\rho(y_{-i}^{n+1},y_{i})\leq\rho(y^{n},y_{n+1})\\},\quad
y_{n+1}\in\mathbb{Y}.$

The reader will surely recognize the right-hand side above as the so-called
“transducer” or “p-value” output produced by the inductive conformal
prediction algorithm. In particular, a result establishing what is equivalent
to the prediction validity property in (19) can be found in Corollary 2.9 of
Vovk et al., (2005). The derivation above, which makes use of conditioning and
sufficiency of the empirical distribution more closely resembles that in
Faulkenberry, (1973) and, more recently, Hoff, (2023). That one can arrive at
the very powerful conformal prediction methodology through a (generalized—in
the sense of allowing generic orderings $\rho$) IM-driven line of reasoning is
quite remarkable.

##  7 Conclusion

This paper, Part III of the series, considered the problem of efficient
marginal inference on an interest parameter through a suitable elimination of
the underlying nuisance parameters. Depending on the problem at hand, this can
be relatively straightforward or quite difficult (at least computationally).
For inference on parameters in a posited statistical model, if there’s an
“ideal factorization,” then valid and efficient marginal inference is almost
immediate, through a general relative profile likelihood-based formulation.
Outside the “ideal” class of problems, the same proposal still works and is
shown to very strong solutions in some challenging applications, namely, the
gamma mean and Behrens–Fisher problem. In fact, based on the results presented
in Example 5 above, my conjecture is that the proposed IM solution is the best
available among in the sense of being exactly valid and also empirically
efficient. There are certain cases where the profile relative likelihood
strategy is inefficient, in particular, when there’s a large number of
nuisance parameters; but this risk can be anticipated, and other
marginalization strategies can be applied, as I showed in Examples 7–8.

Prediction problems can be viewed as extreme cases of marginal inference,
where all of the model parameters are nuisance and to be eliminated. Here the
same relative profile likelihood-based construction is possible, leading to
what I called a predictive IM that is provably valid and, among other things,
can be used to construct prediction regions for features of future
observables. For instance, in Example 11, I showed how to construct an valid
predictive IM for the maximum of the next $k$ realizations in a sequence of
gamma observables. This same problem has been investigated in, e.g., Hamada et
al., (2004), Wang et al., (2012), and Martin and Lingham, (2016), but none of
these proposals come equipped with exact prediction coverage guarantees.

The first part of this paper, and most of the previous literature on IMs,
focused on the case of a finite-dimensional parametric model for the
observable data. Section 5 lays the groundwork for a new, (empirical)
likelihood-based approach for marginal inference on certain features of the
non-parametric model, indexed by an infinite-dimensional unknown. In this
case, as expected, computation is a more serious challenge, and there I put
forward some first thoughts on efficient approximations via, say, bootstrap. A
few examples of this were presented, in particular, non-parametric inference
on a mean and on a quantile. There are other “non-parametric” problems in
which it may be preferable to proceed without thinking in terms of an
infinite-dimensional unknown, e.g., in machine learning problems where the
quantity of interest is defined as a risk minimizer. Section 6 briefly
describes how an approach similar to what was developed in the first part of
the paper can be applied even in this seemingly-very-different context. In
fact, for prediction, I showed how the powerful and now widely-used conformal
prediction algorithm can be derived from (a slightly broader perspective on)
this general IM framework.

I’ll conclude this discussion with a brief mention of some directions for
future investigation. These and/or other things will be addressed in
subsequent parts of this series.

  * •

An important problem that’s closely related to the elimination of nuisance
parameters is model assessment and, in turn, the task of model selection. The
point is that, if the model and, as usual, the model parameters are both
unknown, then there’s really an uncertain pair $(\Gamma,\Theta_{\Gamma})$,
where $\Gamma\in\mathbb{G}$ is the uncertain model index and
$\Theta_{\Gamma}\in\mathbb{T}_{\Gamma}$ is the uncertain, model-specific
parameter. When it comes to model assessment, the entirety of
$\Theta_{\Gamma}$ is a nuisance parameter and the goal is marginal inference
on $\Gamma$. From this perspective, it’s only natural to consider the same
relative profile likelihood-based IM construction presented here. The result
would be a strongly valid IM on the model space $\mathbb{G}$, offering
provably reliable possibilistic uncertainty quantification about the model,
something no other frameworks are able to offer. This can also accommodate
partial prior information, e.g., to encourage
simplicity/sparsity/parsimony/etc.

  * •

In the context considered in Section 6, when there’s no likelihood directly in
consideration, it’s no longer clear how to incorporate partial prior
information. For sure, it’s not so simple as normalizing the likelihood times
prior to get a relative likelihood function. One of the challenges is in
defining the upper joint distribution “$\overline{\mathsf{P}}_{Y,\Phi}$” that
would be used to carry out the Choquet integration. While there might still be
a work-around, I can also see that this difficulty is to be expected and
perhaps insurmountable: the Choquet integral requires specification of an
imprecise probability, which in turn requires a probabilistic link between
data and parameters, hence a sort of model, likelihood, etc.

  * •

Finally, computation of the marginal IM clearly is feasible in the examples
presented here in this paper. There’s also lots of other similar examples
where the same (naive) Monte Carlo-driven strategies can be put to work. For
problems that involve a lot of nuisance parameters, however, this might be
quite expensive. One option would be to give up some information/efficiency
about the interest parameter in exchange for computational benefits, e.g., to
work with a marginal- instead of profile-based relative likelihood in a “not-
so-ideal factorization” case. Another option is to develop some new and less-
naive strategies for Monte Carlo-based optimization using, say, stochastic
gradient decent. Some initial work on this was presented in Syring and Martin,
(2021), but I think more can be done.

## Acknowledgments

Thanks to Leonardo Cella for helpful discussions and comments on an earlier
draft. This work is partially supported by the U.S. National Science
Foundation, SES–2051225.

## References

  * Anscombe, (1964) Anscombe, F. J. (1964).  Normal likelihood functions.  Ann. Inst. Statist. Math., 16:1–19.
  * Aspin, (1948) Aspin, A. A. (1948).  An examination and further development of a formula arising in the problem of comparing two mean values.  Biometrika, 35:88–96.
  * Balch et al., (2019) Balch, M. S., Martin, R., and Ferson, S. (2019).  Satellite conjunction analysis and the false confidence theorem.  Proc. Royal Soc. A, 475(2227):2018.0565.
  * Barndorff-Nielsen, (1973) Barndorff-Nielsen, O. (1973).  Exponential Families and Conditioning.  PhD thesis, University of Copenhagen, Department of Mathematics.
  * Basu, (1977) Basu, D. (1977).  On the elimination of nuisance parameters.  J. Amer. Statist. Assoc., 72(358):355–366.
  * Basu, (1978) Basu, D. (1978).  On partial sufficiency: a review.  J. Statist. Plann. Inference, 2(1):1–13.
  * Berger and Bernardo, (1992) Berger, J. O. and Bernardo, J. M. (1992).  On the development of reference priors.  In Bayesian Statistics, 4 (Peñíscola, 1991), pages 35–60. Oxford Univ. Press, New York.
  * Berger et al., (2009) Berger, J. O., Bernardo, J. M., and Sun, D. (2009).  The formal definition of reference priors.  Ann. Statist., 37(2):905–938.
  * Berger et al., (1999) Berger, J. O., Liseo, B., and Wolpert, R. L. (1999).  Integrated likelihood methods for eliminating nuisance parameters.  Statist. Sci., 14(1):1–28.
  * Bhaumik et al., (2009) Bhaumik, D. K., Kapur, K., and Gibbons, R. D. (2009).  Testing parameters of a gamma distribution for small samples.  Technometrics, 51(3):326–334.
  * Boos and Stefanski, (2013) Boos, D. D. and Stefanski, L. A. (2013).  Essential Statistical Inference.  Springer Texts in Statistics. Springer, New York.
  * Cella, (2023) Cella, L. (2023).  Finite sample valid probabilistic inference on quantile regression.  In Miranda, E., Montes, I., Quaeghebeur, E., and Vantaggi, B., editors, Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, volume 215 of Proceedings of Machine Learning Research, pages 109–118. PMLR.
  * (13) Cella, L. and Martin, R. (2022a).  Direct and approximately valid probabilistic inference on a class of statistical functionals.  Internat. J. Approx. Reason., 151:205–224.
  * (14) Cella, L. and Martin, R. (2022b).  Valid inferential models for prediction in supervised learning problems.  Internat. J. Approx. Reason., 150:1–18.
  * (15) Cella, L. and Martin, R. (2022c).  Validity, consonant plausibility measures, and conformal prediction.  Internat. J. Approx. Reason., 141:110–130.
  * Creasy, (1954) Creasy, M. A. (1954).  Symposium on interval estimation: Limits for the ratio of means.  J. Roy. Statist. Soc. Ser. B., 16:186–194.
  * Denœux, (2006) Denœux, T. (2006).  Constructing belief functions from sample data using multinomial confidence regions.  Internat. J. of Approx. Reason., 42(3):228–252.
  * Destercke et al., (2009) Destercke, S., Dubois, D., and Chojnacki, E. (2009).  A consonant approximation of the product of independent consonant random sets.  Internat. J. Uncertain. Fuzziness Knowledge-Based Systems, 17(6):773–792.
  * Dubois, (2006) Dubois, D. (2006).  Possibility theory and statistical reasoning.  Comput. Statist. Data Anal., 51(1):47–69.
  * Dubois and Prade, (1988) Dubois, D. and Prade, H. (1988).  Possibility Theory.  Plenum Press, New York.
  * Dufour, (1997) Dufour, J.-M. (1997).  Some impossibility theorems in econometrics with applications to structural and dynamic models.  Econometrica, 65(6):1365–1387.
  * Efron, (1979) Efron, B. (1979).  Bootstrap methods: Another look at the jackknife.  Ann. Statist., 7(1):1–26.
  * Efron and Tibshirani, (1993) Efron, B. and Tibshirani, R. J. (1993).  An Introduction to the Bootstrap.  Chapman & Hall, New York.
  * Faulkenberry, (1973) Faulkenberry, G. D. (1973).  A method of obtaining prediction intervals.  J. Amer. Statist. Assoc., 68:433–435.
  * Ferguson, (1973) Ferguson, T. S. (1973).  A Bayesian analysis of some nonparametric problems.  Ann. Statist., 1:209–230.
  * Fieller, (1954) Fieller, E. C. (1954).  Symposium on interval estimation: Some problems in interval estimation.  J. Roy. Statist. Soc. Ser. B., 16:175–185.
  * Fisher, (1935) Fisher, R. A. (1935).  The fiducial argument in statistical inference.  Ann. Eugenics, 6:391–398.
  * (28) Fisher, R. A. (1973a).  Statistical Methods and Scientific Inference.  Hafner Press, New York, 3rd edition.
  * (29) Fisher, R. A. (1973b).  Statistical Methods for Research Workers.  Hafner Publishing Co., New York.  Fourteenth edition—revised and enlarged.
  * Fraser, (1956) Fraser, D. A. S. (1956).  Sufficient statistics with nuisance parameters.  Ann. Math. Statist., 27:838–842.
  * Fraser, (2011) Fraser, D. A. S. (2011).  Is Bayes posterior just quick and dirty confidence?  Statist. Sci., 26(3):299–316.
  * Fraser et al., (2016) Fraser, D. A. S., Bédard, M., Wong, A., Lin, W., and Fraser, A. M. (2016).  Bayes, reproducibility and the quest for truth.  Statist. Sci., 31(4):578–590.
  * Fraser et al., (2018) Fraser, D. A. S., Reid, N., and Lin, W. (2018).  When should modes of inference disagree? Some simple but challenging examples.  Ann. Appl. Stat., 12(2):750–770.
  * Fraser et al., (1997) Fraser, D. A. S., Reid, N., and Wong, A. (1997).  Simple and accurate inference for the mean of a gamma model.  Canad. J. Statist., 25(1):91–99.
  * Fraser et al., (2009) Fraser, D. A. S., Wong, A., and Sun, Y. (2009).  Three enigmatic examples and inference from likelihood.  Canad. J. Statist., 37(2):161–181.
  * Ghosh and Kim, (2001) Ghosh, M. and Kim, Y.-H. (2001).  The Behrens-Fisher problem revisited: a Bayes-frequentist synthesis.  Canad. J. Statist., 29(1):5–17.
  * Gleser and Hwang, (1987) Gleser, L. J. and Hwang, J. T. (1987).  The nonexistence of $100(1-\alpha)\%$ confidence sets of finite expected diameter in errors-in-variables and related models.  Ann. Statist., 15(4):1351–1362.
  * Godambe, (1960) Godambe, V. P. (1960).  An optimum property of regular maximum likelihood estimation.  Ann. Math. Statist., 31:1208–1211.
  * Goodman, (1965) Goodman, L. A. (1965).  On simultaneous confidence intervals for multinomial proportions.  Technometrics, 7(2):247–254.
  * Grice and Bain, (1980) Grice, J. V. and Bain, L. J. (1980).  Inferences concerning the mean of the gamma distribution.  J. Amer. Statist. Assoc., 75(372):929–933.
  * Hamada et al., (2004) Hamada, M., Johnson, V., Moore, L. M., and Wendelberger, J. (2004).  Bayesian prediction intervals and their relationship to tolerance intervals.  Technometrics, 46(4):452–459.
  * Hannig and Xie, (2012) Hannig, J. and Xie, M.-g. (2012).  A note on Dempster–Shafer recombination of confidence distributions.  Electron. J. Stat., 6:1943–1966.
  * Hoff, (2023) Hoff, P. (2023).  Bayes-optimal prediction with frequentist coverage control.  Bernoulli, 29(2):901–928.
  * Hose, (2022) Hose, D. (2022).  Possibilistic Reasoning with Imprecise Probabilities: Statistical Inference and Dynamic Filtering.  PhD thesis, University of Stuttgart.
  * Hsu, (1938) Hsu, P. L. (1938).  Contributions to the theory of “Student’s” $t$-test as applied to the problem of two samples.  In Statistical Research Memoirs, pages 1–24. University College, London.
  * Huber, (1973) Huber, P. J. (1973).  The use of Choquet capacities in statistics.  Bull. Inst. Internat. Statist., 45(book 4):181–191.  With discussion by P. Gänssler, W. S. Hall, L. Schmetterer, Charles Stein, and G. N. Wilkinson and with reply by the author.
  * Huber, (1981) Huber, P. J. (1981).  Robust Statistics.  John Wiley & Sons Inc., New York.  Wiley Series in Probability and Mathematical Statistics.
  * (48) Jacob, P. E., Gong, R., Edlefsen, P. T., and Dempster, A. P. (2021a).  A Gibbs sampler for a class of random convex polytopes.  J. Amer. Statist. Assoc., 116(535):1181–1192.
  * (49) Jacob, P. E., Gong, R., Edlefsen, P. T., and Dempster, A. P. (2021b).  Rejoinder—A Gibbs sampler for a class of random convex polytopes.  J. Amer. Statist. Assoc., 116(535):1211–1214.
  * Jeffreys, (1940) Jeffreys, H. (1940).  Note on the Behrens-Fisher formula.  Ann. Eugenics, 10:48–51.
  * Jeffreys, (1946) Jeffreys, H. (1946).  An invariant form for the prior probability in estimation problems.  Proc. Roy. Soc. London Ser. A, 186:453–461.
  * Kalbfleisch and Sprott, (1970) Kalbfleisch, J. D. and Sprott, D. A. (1970).  Application of likelihood methods to models involving large numbers of parameters.  J. Roy. Statist. Soc. Ser. B, 32:175–208.
  * Kim and Cohen, (1998) Kim, S.-H. and Cohen, A. S. (1998).  On the Behrens-Fisher problem: A review.  Journal of Educational and Behavioral Statistics, 23(4):356–377.
  * Lawrence et al., (2009) Lawrence, E., Liu, C., Vander Weil, S., and Zhang, J. (2009).  A new method for multinomial inference using Dempster–Shafer theory.  https://www.osti.gov/biblio/956387.
  * Lehmann, (1975) Lehmann, E. L. (1975).  Nonparametrics: Statistical Methods Based on Ranks.  Holden-Day Inc., San Francisco, Calif.
  * Liu et al., (2014) Liu, R., Chakrabarti, A., Samanta, T., Ghosh, J. K., and Ghosh, M. (2014).  On divergence measures leading to Jeffreys and other reference priors.  Bayesian Anal., 9(2):331–369.
  * Maclaren, (2018) Maclaren, O. (2018).  Is profile likelihood a true likelihood? An argument in favor.  arXiv:1801.04369.
  * Manski, (2003) Manski, C. F. (2003).  Partial Identification of Probability Distributions.  Springer Series in Statistics. Springer-Verlag, New York.
  * Mardia and Jupp, (2000) Mardia, K. V. and Jupp, P. E. (2000).  Directional Statistics.  Wiley Series in Probability and Statistics. John Wiley & Sons, Ltd., Chichester.
  * Martin, (2015) Martin, R. (2015).  Plausibility functions and exact frequentist inference.  J. Amer. Statist. Assoc., 110(512):1552–1561.
  * Martin, (2018) Martin, R. (2018).  On an inferential model construction using generalized associations.  J. Statist. Plann. Inference, 195:105–115.
  * Martin, (2019) Martin, R. (2019).  False confidence, non-additive beliefs, and valid statistical inference.  Internat. J. Approx. Reason., 113:39–73.
  * Martin, (2021) Martin, R. (2021).  An imprecise-probabilistic characterization of frequentist statistical inference.  arXiv:2112.10904.
  * (64) Martin, R. (2022a).  Valid and efficient imprecise-probabilistic inference under partial priors, I. First results.  arXiv:2203.06703.
  * (65) Martin, R. (2022b).  Valid and efficient imprecise-probabilistic inference under partial priors, II. General framework.  arXiv:2211.14567.
  * (66) Martin, R. (2023a).  Fiducial inference viewed through a possibility-theoretic inferential model lens.  In Miranda, E., Montes, I., Quaeghebeur, E., and Vantaggi, B., editors, Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, volume 215 of Proceedings of Machine Learning Research, pages 299–310. PMLR.
  * (67) Martin, R. (2023b).  A possibility-theoretic solution to Basu’s Bayesian–frequentist via media.  Sankhya A, to appear, arXiv:2303.17425.
  * Martin and Lingham, (2016) Martin, R. and Lingham, R. T. (2016).  Prior-free probabilistic prediction of future observations.  Technometrics, 58(2):225–235.
  * Martin and Liu, (2013) Martin, R. and Liu, C. (2013).  Inferential models: a framework for prior-free posterior probabilistic inference.  J. Amer. Statist. Assoc., 108(501):301–313.
  * (70) Martin, R. and Liu, C. (2015a).  Conditional inferential models: combining information for prior-free probabilistic inference.  J. R. Stat. Soc. Ser. B. Stat. Methodol., 77(1):195–217.
  * (71) Martin, R. and Liu, C. (2015b).  Inferential Models, volume 147 of Monographs on Statistics and Applied Probability.  CRC Press, Boca Raton, FL.
  * (72) Martin, R. and Liu, C. (2015c).  Marginal inferential models: prior-free probabilistic inference on interest parameters.  J. Amer. Statist. Assoc., 110(512):1621–1631.
  * Martin and Syring, (2022) Martin, R. and Syring, N. (2022).  Direct Gibbs posterior inference on risk minimizers: construction, concentration, and calibration.  In Srinivasa Rao, A. S. R., Young, G. A., and Rao, C. R., editors, Handbook of Statistics: Advancements in Bayesian Methods and Implementation, volume 47, pages 1–41. Elsevier.
  * Miranda and de Cooman, (2014) Miranda, E. and de Cooman, G. (2014).  Lower previsions.  In Introduction to Imprecise Probabilities, Wiley Ser. Probab. Stat., pages 28–55. Wiley, Chichester.
  * Murphy and van der Vaart, (2000) Murphy, S. A. and van der Vaart, A. W. (2000).  On profile likelihood.  J. Amer. Statist. Assoc., 95(450):449–485.  With discussion.
  * Neyman, (1935) Neyman, J. (1935).  On a theorem concerning the concept of sufficient statistic.  Giorn. Ist. Ital. Att., 6:320–334.  in Italian.
  * Neyman and Scott, (1948) Neyman, J. and Scott, E. L. (1948).  Consistent estimates based on partially consistent observations.  Econometrica, 16:1–32.
  * Normand, (1999) Normand, S. (1999).  Meta-analysis: formulating, evaluating, combining, and reporting.  Stat. Med., 18(3):321–359.
  * Olshevsky, (1940) Olshevsky, L. (1940).  Two properties of sufficient statistics.  Ann. Math. Statistics, 11:104–106.
  * Owen, (1990) Owen, A. (1990).  Empirical likelihood ratio confidence regions.  Ann. Statist., 18(1):90–120.
  * Owen, (1988) Owen, A. B. (1988).  Empirical likelihood ratio confidence intervals for a single functional.  Biometrika, 75(2):237–249.
  * Owen, (2001) Owen, A. B. (2001).  Empirical Likelihood, volume 92 of Monographs on Statistics and Applied Probability.  Chapman & Hall/CRC Press, Boca Raton, FL.
  * Qin and Lawless, (1994) Qin, J. and Lawless, J. (1994).  Empirical likelihood and general estimating equations.  Ann. Statist., 22(1):300–325.
  * Rao, (1973) Rao, C. R. (1973).  Linear statistical inference and its applications.  Wiley Series in Probability and Mathematical Statistics. John Wiley & Sons, New York-London-Sydney, second edition.
  * Royall, (1997) Royall, R. M. (1997).  Statistical Evidence, volume 71 of Monographs on Statistics and Applied Probability.  Chapman & Hall, London.
  * Sandved, (1966) Sandved, E. (1966).  A principle for conditioning on an ancillary statistic.  Skand. Aktuarietidskr., 1966:39–47.
  * Scheffé, (1970) Scheffé, H. (1970).  Practical solutions of the Behrens–Fisher problem.  J. Amer. Statist. Assoc., 65:1501–1508.
  * Schweder and Hjort, (2013) Schweder, T. and Hjort, N. L. (2013).  Discussion: “Confidence distribution, the frequentist distribution estimator of a parameter: a review” [mr3047496].  Int. Stat. Rev., 81(1):56–68.
  * Shafer, (1976) Shafer, G. (1976).  A Mathematical Theory of Evidence.  Princeton University Press, Princeton, N.J.
  * Shafer and Vovk, (2008) Shafer, G. and Vovk, V. (2008).  A tutorial on conformal prediction.  J. Mach. Learn. Res., 9:371–421.
  * Shiue and Bain, (1990) Shiue, W. K. and Bain, L. J. (1990).  Simple approximate inference procedures for the mean of the gamma model.  J. Statist. Comput. Simulation, 34:67–73.
  * Stein, (1956) Stein, C. (1956).  Inadmissibility of the usual estimator for the mean of a multivariate normal distribution.  In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 1954–1955, vol. I, pages 197–206, Berkeley and Los Angeles. University of California Press.
  * Stein, (1959) Stein, C. (1959).  An example of wide discrepancy between fiducial and confidence intervals.  Ann. Math. Statist., 30:877–880.
  * Stigler, (1977) Stigler, S. M. (1977).  Do robust estimators work with real data?  Ann. Statist., 5(6):1055–1098.  With discussion and a reply by the author.
  * Syring and Martin, (2017) Syring, N. and Martin, R. (2017).  Gibbs posterior inference on the minimum clinically important difference.  J. Statist. Plann. Inference, 187:67–77.
  * Syring and Martin, (2019) Syring, N. and Martin, R. (2019).  Calibrating general posterior credible regions.  Biometrika, 106(2):479–486.
  * Syring and Martin, (2021) Syring, N. and Martin, R. (2021).  Stochastic optimization for numerical evaluation of imprecise probabilities.  In Cano, A., De Bock, J., Miranda, E., and Moral, S., editors, Proceedings of the Twelveth International Symposium on Imprecise Probability: Theories and Applications, volume 147 of Proceedings of Machine Learning Research, pages 289–298. PMLR.
  * Syring and Martin, (2023) Syring, N. and Martin, R. (2023).  Gibbs posterior concentration rates under sub-exponential type losses.  Bernoulli, 29(2):1080–1108.
  * Tang and Leng, (2010) Tang, C. Y. and Leng, C. (2010).  Penalized high-dimensional empirical likelihood.  Biometrika, 97(4):905–919.
  * Tibshirani, (1989) Tibshirani, R. (1989).  Noninformative priors for one parameter of many.  Biometrika, 76(3):604–608.
  * Troffaes and de Cooman, (2014) Troffaes, M. C. M. and de Cooman, G. (2014).  Lower Previsions.  Wiley Series in Probability and Statistics. John Wiley & Sons, Ltd., Chichester.
  * Troffaes et al., (2013) Troffaes, M. C. M., Miranda, E., and Destercke, S. (2013).  On the connection between probability boxes and possibility measures.  Inform. Sci., 224:88–108.
  * Vovk et al., (2005) Vovk, V., Gammerman, A., and Shafer, G. (2005).  Algorithmic Learning in a Random World.  Springer, New York.
  * Walley, (1991) Walley, P. (1991).  Statistical Reasoning with Imprecise Probabilities, volume 42 of Monographs on Statistics and Applied Probability.  Chapman & Hall Ltd., London.
  * Wang et al., (2012) Wang, C. M., Hannig, J., and Iyer, H. K. (2012).  Fiducial prediction intervals.  J. Statist. Plann. Inference, 142(7):1980–1990.
  * Wasserman, (2008) Wasserman, L. (2008).  Why isn’t everyone a Bayesian?  In Morris, C. and Tibshirani, R., editors, The Science of Bradley Efron, pages 260–261. Springer, New York.
  * Wasserman et al., (2020) Wasserman, L., Ramdas, A., and Balakrishnan, S. (2020).  Universal inference.  Proc. Natl. Acad. Sci. U.S.A., 117(29):16880–16890.
  * Wasserman, (1990) Wasserman, L. A. (1990).  Belief functions and statistical inference.  Canad. J. Statist., 18(3):183–196.
  * Welch, (1938) Welch, B. L. (1938).  The significance of the difference between two means when the population variances are unequal.  Biometrika, 29:350–362.
  * Welch, (1947) Welch, B. L. (1947).  The generalization of ‘Student’s’ problem when several different population variances are involved.  Biometrika, 34:28–35.
  * Wong, (1993) Wong, A. C. M. (1993).  A note on inference for the mean parameter of the gamma distribution.  Statist. Probab. Lett., 17(1):61–66.
  * Wu and Martin, (2022) Wu, P.-S. and Martin, R. (2022).  Generalized Bayes inference on a linear personalized minimum clinically important difference.  arXiv:2208.12565.
  * Zadeh, (1975) Zadeh, L. A. (1975).  The concept of a linguistic variable and its application to approximate reasoning. I.  Information Sci., 8:199–249.
  * Zadeh, (1978) Zadeh, L. A. (1978).  Fuzzy sets as a basis for a theory of possibility.  Fuzzy Sets and Systems, 1(1):3–28.
  * Zhou, (2022) Zhou, M. (2022).  emplik: Empirical Likelihood Ratio for Censored/Truncated Data.  R package version 1.2.