# Dempster-Shafer P-values: Thoughts on an Alternative Approach for
Multinomial Inference

Kentaro Hoffman, Kai Zhang, Tyler McCormick, Jan Hannig

In this paper, we demonstrate that a new measure of evidence we developed
called the Dempster-Shafer p-value which allow for insights and
interpretations which retain most of the structure of the p-value while
covering for some of the disadvantages that traditional p-values face.
Moreover, we show through classical large-sample bounds and simulations that
there exists a close connection between our form of DS hypothesis testing and
the classical frequentist testing paradigm. We also demonstrate how our
approach gives unique insights into the dimensionality of a hypothesis test,
as well as models the effects of adversarial attacks on multinomial data.
Finally, we demonstrate how these insights can be used to analyze text data
for public health through an analysis of the Population Health Metrics
Research Consortium dataset for verbal autopsies.

##  1 Introduction

Although p-values have historically played a ubiquitous role in many
statistical analyses, the past decade has seen a flurry of criticism and
scandals on their use and misuse. For example, in a highly cited Nature
article, the authors found in a meta-analysis across 4 highly cited
sociological and medical journals, an incredible 402 out of 791 $(51\%)$
articles erroneously conflate non-significance with no effect [1], a basic
misconception commonly covered in a variety of introductory statistics
textbooks [23, 13, 8] In another, in response to the landmark 2016 American
Statistical Association statement of propose on the proper use of p-values,
the New England Journal of Medicine announced a change in reporting standards
for p-values [12]. However, this new standard itself misinterpreted a p-value
that claims “A p-value of 0.05 carries a 5% risk of a false positive result”
[27]. This was despite the ASA statement of propose explicitly stating that
this is not correct.

In response to these issues, the past few years have seen a new movement in
statistics- one that seeks to end the uncritical use and misuse of the 0.05
p-value. This paper seeks to join this and demonstrate that a new measure of
evidence we developed called “Dempster-Shafer p-value” provides new insights
and interpretations which retain most of the structure of the p-value while
covering for some of the disadvantages that traditional p-values face. While
this is focused on p-values for multinomial inference, it is the author’s hope
that the useful properties, advantages, and disadvantages that arise from
incorporating Dempster-Shafer inference to the p-value will be thought-
provoking in the continued conversation on the future of statistical
hypothesis testing. Finally, we demonstrate the utility of the proposed
approach in the context of verbal autopsies, a strategy commonly used in
global health to determine the distribution of deaths by cause in settings
where deaths occur outside of hospitals and are not formally recorded.

##  2 Demspter-Shafer Inference

Dempster-Shafer (DS) Inference was developed in the 1960-70s by Arthur
Dempster [7] and Glen Shafer [5] as an alternative approach to prior-free
statistical inference. What makes DS distinctive is that for an event $A$, it
ascribes the triple, ($p(A)$, $p(A^{c})$, $1-p(A)-p(A^{c}))$, to quantify the
evidence for $A$, against $A$, and unknown rather than traditional inferential
approaches that denote a probability double $(P(A),1-P(A))$. Historically, DS
Inference has played an important role in the development of artificial
intelligence [31] and reliability engineering [24], leading to the development
of the field of imprecise probability.

Despite its popularity in other fields, DS Inference has had a smaller impact
on the statistical world in which it originated. It has been hypothesized that
it failed to take off due to technical difficulties in interpreting beliefs
and plausibilities [22], computational burden, and perhaps the biggest
problem: the lack of conventional long-run frequentist properties under
repeated sampling [19]. However, in the last several years, there has been
renewed interest from the statistical community in other forms of prior-free
forms of posterior inference which incorporate epistemic uncertainty. This has
come from closely related topics such as Generalized Fiducial Inference [11],
Confidence Distributions [30], Inferential Models [18], and faster
computational techniques using Gibbs sampling on polytopes [14]. We build on
this rising wave and demonstrate that a Dempster-Shafer inference procedure
known as Dirichlet-DS can be adapted to perform a wide variety of Dempster-
Shafer Hypothesis Tests which are computationally fast and model epistemic
uncertainty: in particular, we focus on the uncertainty that can result from
adversarial attacks. Moreover, we will show through classical large sample
bounds and simulations that there exists a close connection between our form
of DS hypothesis testing and the classical frequentist testing paradigm.
Finally, we will demonstrate how our approach gives unique insights into the
dimensionality of a hypothesis test, as well as models the effects of
adversarial attacks on multinomial data.

##  3 Multinomial Data Generation via Dirichlet-DSM

In a Multinomial Data generation scheme, we observe $n$, $k$-dimensional
multinomial observations $(n_{1},...n_{k})$ with unknown but fixed
probabilities $\mathcal{P}=(p_{1},...p_{k})$ such that $\sum_{i=1}^{k}p_{i}=1$

$n_{1},...,n_{k}\sim\text{ Multinomial }(1,p_{1},...,p_{k}).$

Each draw from the multinomial can be represented in component form as a sum
of binary vectors using $y_{ij}\in\\{0,1\\}$ for $i\in\\{1,...n\\}$ and
$j\in\\{1,...,k\\}$ such that $\sum_{j}y_{ij}=1$ and $\sum_{i}y_{ij}=n_{j}$.
We will interchangeably referring to count and component form as necessary.

To perform DS inference, it is first necessary to define the data generation
process that yielded $(n_{1},...,n_{k})$. This can be done through prior
insight into the data generation scheme, or it can be chosen to facilitate the
inference of the parameters. Consistent with the second reason, we will employ
the data generation process described in Lawrence et al. [15] as the
Dirichlet-DSM process. We first define an unknown permutation $\Pi$ of the $k$
classes as: $\Pi(\\{1,...,k\\})=\\{\pi(1),...,\pi(k)\\}$ and the inverse
permutation as $\Pi^{-1}(\\{\pi(1),...,\pi(k)\\})=\\{1,...,k\\}$. This allows
us to define $I_{\pi,a}$, as an interval of length $p_{a}$ whose location on
the unit interval is determined by:

$I_{\pi,a}=\begin{cases}[0,p_{a})&\text{ if }\pi(1)=a\\\
[\sum_{i=1}^{\tilde{a}-1}p_{\pi(i)},\sum_{i=1}^{\tilde{a}-1}p_{\pi(i)}+p_{a})&\text{
if }\pi(1)\neq a\end{cases}$

where $\tilde{a}=\pi^{-1}(a)$ for $a\in\\{1,...,j\\}$. Regardless of choice of
permutation $\Pi$, each interval $I_{\pi,a}$ will be of length $p_{a}$ and
combing together all the intervals yields $[0,1)$. Thus we can generate
multinomial draws through the data generating process:

$y_{ij}=a\textit{ iff }U_{i}\in I_{a}$

where $(U_{1},...U_{n})$ are $n$ independent, uniform random variables on
[0,1]. This data generating process holds two distinct advantages over related
processes such as  Simplex-DSM Dempster [6], interval-DSM in Hannig [11], and
Simplex-DS in Jacob et al. [14]. First, unlike interval-DSM, this inference
procedure for Dirichlet-DSM is not sensitive to the order of the categories.
Further, unlike the simplex-DSM and Simplex-DS, Dirichlet-DSM has a relatively
simple expression for its posterior estimation which requires no expensive
acceptance-rejection or Gibbs sampling, this makes the method more feasible
when combine with the subsequent repeated required for hypothesis testing.

###  3.1 A Recipe for Multinomial DS Inference using Dirichlet-DSM

Inferring $\mathcal{P}$ requires coming up with an inversion of the data
generating equation. We first assume that we have observed $n$ multinomial
draws generated by the Dirichlet-DSM scheme and the permutation that was used
for the draws, $\Pi$. For each multinomial draw $(y_{i1},...,y_{ij})$, let
$a_{i}\in\\{1,...,k\\}$ be the observed component,
$a_{i}=\sum_{j=1}^{k}jI(y_{ij}=1)$ for the $i$-th draw. Then, define
$\tilde{\Pi}_{i,\Pi,a_{i}}(\\{1,...,k\\})=(\tilde{\pi}(1),...\tilde{\pi}(k))$
as any permutation that has the observed category, $a_{i}$, first in its
order. In other words, this means that $\tilde{\pi}(a_{i})=1$. Now, let
$D_{\tilde{\Pi}_{i},a_{i}}$ be a function which takes a point on $[0,1)$, and
maps it back to $[0,1)$ but with the order of the intervals permuted by
$\tilde{\Pi}$. Being a well defined bijection determine by $\tilde{\Pi}$, $D$
also admits a well defined inverse $D^{-1}_{\tilde{\Pi}_{i},a_{i}}$ which
undoes the reordering. By applying $D^{-1}_{\tilde{\Pi}_{i},a_{i}}$ to
$U_{i}$, we can define an auxiliary variable:

$W_{i}=D_{\tilde{\Pi}_{i},a_{i}}(U_{i}).$

Due to the bijective nature of $D_{\tilde{\Pi}_{i},a_{i}}$, $W_{i}$ is
distributed $U[0,1)$. When conditioning $W_{i}$ on the observed data, $a_{i}$,
$W_{i}$ becomes:

$W_{i}|a_{i}\sim U(0,p_{a_{i}}).$

To perform a fiducial inversion, we start with an iid copy of $W_{i}$ which we
call $W_{i}^{*}$. Being iid, $W_{i}^{*}$ is also distributed $U(0,1)$ and we
would like to derive the distribution of $W_{i}^{*}$ conditional on this being
consistent with the observed data (This is refers to the ”Continuing to
believe” justification in DS inference). When $U_{i}^{*}$ is compatible with
$a_{i}$, this implies that $U_{i}^{*}\in I_{{\Pi}_{i},a_{i}}$ and by
construction $U_{i}^{*}\in I_{\tilde{\Pi}_{i},a_{i}}$. Applying $D$’s
bijective behavior, this yields the equivalent condition that:

$W_{i}^{*}\leq p_{a_{i}}.$

Thus, our posterior random set is defined by jointly generating $n$ auxiliary
variables $W_{i}^{*}$ such that each satisfies:

$W_{i}^{*}|\sum_{i}W_{i}^{*}\leq 1.$

As for the exact definition of this joint distribution, this constraint on the
auxiliary variables is identical to the constraints used in theorem 4.1 of
Lawrence et al. [15] so both though DS and Fiducial derivations, we arrive to
the same statement that:

######  Theorem 1.

Given data from a $k$ dimensional multinomial, $(n_{1},...,n_{k})$ the
posterior random set $\hat{\mathcal{P}}$ is distributed according to:

$\displaystyle\hat{\mathcal{P}}=(\hat{p}_{1},...,\hat{p}_{k})$
$\displaystyle:\hat{p}_{a}\geq Z_{a},a\in\\{1,...,k\\}$
$\displaystyle(Z_{0},Z_{1},...,Z_{k})\sim Dirichlet(1,n_{1},...,n_{k})$

For those that prefer a more geometric perspective, one can think of
$\hat{\mathcal{P}}$ as a polytope, specifically a simplex, with edges at:

$\bm{v}_{j}=(Z_{1},...Z_{j-1},Z_{j}+Z_{0},Z_{j+1},...Z_{k}),\,\forall
j\in\\{1,...k\\}.$

Any element in $\hat{\mathcal{P}}$ can be written as:

$\mathcal{\hat{P}}(z)=\\{z_{-0}+\bm{\theta}^{T}z_{0}|\bm{\theta}=(\theta_{1},...,\theta_{k}),\,\sum_{j=1}^{k}\theta_{j}=1,\,0\leq\theta_{j}\leq
1,\,\text{ for }j=1,...,k\\}$

or in vector form:

$\mathcal{\hat{P}}(z)=\begin{pmatrix}Z_{1}+\theta_{1}Z_{0}\\\ \vdots\\\
Z_{k}+\theta_{k}Z_{0}\end{pmatrix}.$

As the edges of the simplex is defined by a random variable, this of course
makes the location of the simplex random a random variable. However, the
behavior of this randomness is not without good properties as one can show
that as desired, the location of our posterior random set converges to the
true probabilities $\mathcal{P}$ in probability.

######  Theorem 2.

$\mathcal{\hat{P}}(\mathcal{Z})$ converges in probability to $\mathcal{P}$.

###### Proof.

It suffices to show that the edges of the simplex converge to $\mathcal{P}$.
By construction of $\bm{v}_{j}$, this occurs when $Z_{0}\rightarrow^{P}0$ and
$Z_{i}\rightarrow^{P}p_{i}$ for $i\in\\{1,...,k\\}$.

$\displaystyle\lim_{n\rightarrow\infty}P(|(Z_{0},Z_{1},...,Z_{k})-(0,p_{1},...,p_{k})|)$
$\displaystyle\leq\lim_{n\to\infty}P(|Z_{0}|\geq\frac{\epsilon}{2k})+\sum_{i=1}^{k}\lim_{n\to\infty}P(|Z_{0}-p_{i}|\geq\frac{\epsilon}{2k})$
$\displaystyle\leq\lim_{n\to\infty}\frac{2k(1-1/n)}{n(1+n)\epsilon}+\sum_{i=1}^{k}\lim_{n\to\infty}\frac{\frac{n_{i}}{n}(1-\frac{n_{i}}{n})}{n+1}\frac{2k}{\epsilon}$
$\displaystyle=\sum_{i=1}^{k}\lim_{n\to\infty}\frac{\frac{n_{i}}{n}(1-\frac{n_{i}}{n})}{n+1}\frac{2k}{\epsilon}$
$\displaystyle=\sum_{i=1}^{k}\lim_{n\to\infty}\frac{p_{i}(1-p_{i})}{n+1}\frac{2k}{\epsilon}=0$

∎

##  4 Point Estimation with DS Inference

Before we describe hypothesis testing, we must describe how we will be turing
our random polytopes $\mathcal{\hat{P}}(z)$, into point estimates. Recall that
$\mathcal{\hat{P}}(z)$ is a polytope that describes the region of feasible
parameter estimates. To emphasize this fact, we will denote
$\mathcal{\hat{P}}(z)$ using $\Delta(z)$ and $\mathcal{\hat{P}}(Z)$ using
$\Delta(Z)$. Using a test statistic $T(\mathcal{\hat{P}},\mathcal{P}_{0})$ and
fixed $(n_{1},...,n_{k})$, one can compute an upper, mean, and lower test
statistic for each $\Delta(z)$ as follows:

$\displaystyle T_{upper}(z)$
$\displaystyle=\sup_{\mathcal{\hat{P}}\in\Delta(z)}T(\mathcal{\hat{P}},\mathcal{P}_{Observed})$
(1) $\displaystyle T_{mean}(z)$
$\displaystyle=T(E_{\Delta(z)}\mathcal{\hat{P}},\mathcal{P}_{Observed})$ (2)
$\displaystyle T_{lower}(z)$
$\displaystyle=\inf_{\mathcal{\hat{P}}\in\Delta(z)}T(\mathcal{\hat{P}},\mathcal{P}_{Observed}).$
(3)

$E_{\Delta(z)}$ in this case represents taking the expectation with respect to
the uniform measure upon $\Delta(z)$.

By plugging in the random variable $\mathcal{Z}$, this turns the upper, mean
and lower statistics into upper, mean, and lower sampling distributions:

$\displaystyle T_{upper}$
$\displaystyle=\sup_{\mathcal{\hat{P}}\in\Delta(\mathcal{Z})}T(\mathcal{\hat{P}},\mathcal{P}_{0})$
(4) $\displaystyle T_{mean}$
$\displaystyle=T(E_{\Delta(\mathcal{Z})}\mathcal{\hat{P}},\mathcal{P}_{0})$
(5) $\displaystyle T_{lower}$
$\displaystyle=\inf_{\mathcal{\hat{P}}\in\Delta(\mathcal{Z})}T(\mathcal{\hat{P}},\mathcal{P}_{0}).$
(6)

Assuming that $T$ defines a valid distance metric, $T_{upper}(z)$ represents
the largest possible test statistic one could create on $\Delta(z)$,
$T_{lower}(z)$ represents the smallest, and $T_{mean}$ represents the average
case.

###  4.1 Comparison to other Frequentist point estimation

The creation of the upper, mean, and lower test statistics contrasts with the
frequentist hypothesis testing paradigm. Under a frequentist paradigm, one
derives a point estimate ,$\mathcal{\hat{P}}_{Freq}$, using the maximum
likelihood equation, and evaluates the asymptotic distribution of the point
estimate plugged into the test statistic:

$T_{Freq}=T(\mathcal{\hat{P}}_{Freq},\mathcal{P}_{0})$

where $\mathcal{P}_{0}$ is the parameter value for the null hypothesis. In the
case of a perason chi-squared hypothesis test, $\hat{\mathcal{P}}$ would be
the estimated proportions, ${\mathcal{P}}_{0}$ would be the proportions under
the null hypothesis, and
$T=\sum_{i}\frac{n_{i}\hat{p}_{i}-n_{i}p_{0i}}{n_{i}p_{0i}}$. It turns out,
one can connect the frequentist test statistic with that of the DS through the
choice of summary statistic used on the posterior random set.

######  Theorem 3.

For fixed observations $(n_{1},...,n_{k})$ from a multinomial distribution and
test statistic T, $T_{mean}$ is equal to
$T_{Freq}=T(\hat{\mathcal{P}}_{Freq},\mathcal{P}_{0})$ where
$\mathcal{\hat{P}}_{Freq}=(\frac{n_{1}+1/k}{n+k},...,\frac{n_{k}+1/k}{n+k})$

###### Proof.

Since:

$\displaystyle T_{mean}$
$\displaystyle=T(E_{\Delta(Z)}\hat{\mathcal{P}},\mathcal{P}_{0})$
$\displaystyle T_{Freq}$
$\displaystyle=T(\hat{\mathcal{P}}_{Freq},\mathcal{P}_{0})$

and $\mathcal{P}_{0}$ is fixed, it suffices to show that:

$E_{\Delta(Z)}\hat{\mathcal{P}}=\hat{P}_{Freq}.$

For each component of $i\in\\{1,...k\\}$, note that we can write:

$\displaystyle E_{\Delta(Z)}(\hat{p}_{i})$
$\displaystyle=E_{Z}E_{\Delta(z)}[(\hat{p}_{i})]$
$\displaystyle=E_{Z}E_{\Delta(z)}[z_{-0}+\theta_{i}z_{0}]$
$\displaystyle=E_{Z}(z_{-0})+\frac{1}{k}E_{Z}(z_{0})$
$\displaystyle=\frac{n_{i}+1/k}{n+1}$

∎

Note that at this point, $T_{mean}$ is the Bayes estimator under MSE of when
the $(p_{1},...,p_{k})$ has a prior distribution of Dirichlet($1/k,...,1/k)$
[16]. This illustrates that the frequentist approach of choosing an arbitrary
point estimate is equivalent to choosing an arbitrary way to summarize the
posterior random sets $\mathcal{\hat{P}}(Z)$. To the best of our knowledge, we
are unaware of other results which connect DS to frequentist via the choice of
summmarization.

##  5 Hypothesis Testing with DS Inference

With our upper, lower, and mean point estimates and their distributions
defined, we are ready to define how we perform a level $\alpha$-hypothesis
test with our DS approach.

######  Definition 5.1.

Level $\alpha$ two-sided DS Hypothesis Test
Let us have two competing hypotheses $H_{0}$ and $H_{1}$ about $k$-dimensional
parameter $\mathcal{P}_{0}\in\Delta^{k-1}$ such that:

$H_{0}:\mathcal{P}=\mathcal{P}_{0},H_{1}:\mathcal{P}\not=\mathcal{P}_{0}.$

Using $\mathcal{P}_{0}$ and a point estimate, $\mathcal{\hat{P}}_{observed}$
based on fixed counts $(n_{1},...,n_{k})$, we compute tail probabilities for
the upper and lower distributions:

$\displaystyle\pi_{upper}=P(T_{lower}$ $\displaystyle\geq
T(\hat{P}_{observed},\mathcal{P}_{0}))$ (7)
$\displaystyle\pi_{lower}=P(T_{upper}$ $\displaystyle\geq
T(\hat{P}_{observed},\mathcal{P}_{0})).$ (8)

We then conclude based on the tail probabilities:

$\begin{cases}\text{Reject }H_{0}&\text{ If }\pi_{upper}\leq\alpha\\\
\text{Accept }H_{0}&\text{ If }\pi_{lower}>\alpha\\\ \text{Unknown}&\text{ If
}\pi_{lower}\leq\alpha\text{ but }\pi_{upper}>\alpha\\\ \end{cases}$

######  Example 5.1.

As an example, let us perform a DS goodness-of-fit test using the chi-squared
test statistic. The data comes from a Multinomial (n,
$(\frac{1}{3},\frac{1}{3},\frac{1}{3})$ and the null hypothesis is that all
three classes are equally likely. The number of elements observed in each
class is $(n_{1},n_{2},n_{3})=(3,2,5)$. In such a case, our upper and lower
test statistics are based on the chi-squared statistic:

$T_{upper}(\mathcal{Z})=\sup_{(\hat{p}_{1},\hat{p}_{2},\hat{p}_{3})\in\Delta(\mathcal{Z})}\sum_{i=1}^{3}\frac{(10\hat{p}_{i}-10/3)^{2}}{10/3}$
$T_{lower}(\mathcal{Z})=\inf_{(\hat{p}_{1},\hat{p}_{2},\hat{p}_{3})\in\Delta(\mathcal{Z})}\sum_{i=1}^{3}\frac{(10\hat{p}_{i}-10/3)^{2}}{10/3}$

where elements of $\Delta(\mathcal{Z})$ are:

$\mathcal{\hat{P}({\mathcal{Z})}}=\begin{pmatrix}\hat{p}_{1}\\\ \hat{p}_{2}\\\
\hat{p}_{3}\\\ \end{pmatrix}=\begin{pmatrix}Z_{1}+\theta_{1}Z_{0}\\\
Z_{2}+\theta_{2}Z_{0}\\\ Z_{3}+\theta_{3}Z_{0}\end{pmatrix}$

for fixed $\theta_{1}+\theta_{2}+\theta_{3}=1$ and $0\leq\theta_{i}\leq 1$ for
$i\in\\{1,2,3\\}$ and:

$(Z_{0},Z_{1},Z_{2},Z_{3})\sim Dirichlet(1,3,2,5).$

In Figure 1, we plot out 100 realizations of posterior random sets as well as
the the null point, $\mathcal{P}_{0}=(\frac{1}{3},\frac{1}{3},\frac{1}{3})$
and point estimate
$(\frac{n_{1}+1}{n+3},\frac{n_{2}+1}{n+3},\frac{n_{3}+1}{n+3})=(\frac{4}{13},\frac{3}{13},\frac{6}{13})$.
Note how most of the polytopes are clustered around the point estimate which
is some distance from the null point.

Figure 1: A) 100 randomly chosen posterior random sets from a 3 dimensional
test of uniformity. Note how the polytopes are centered around the point
estimate (red). 100 was chosen for visibility reasons. B) 1000 simulations of
the upper and lower test statistic and a vertical line representing
$\mathcal{P}_{0}$. For this data, $\pi_{lower}$ is 0.12 and $\pi_{upper}$ is
0.034.

In the Figure 2, we have summarized 1000 posterior random sets into their
upper and lower chi-squared test distributions. As is, the lower tail
probability is 0.034 while the upper tail probability is 0.12, so at the
$\alpha=0.05$ level, we conclude that we lack the power to make a conclusion
either way. More precisely, if we performed a hypothesis test using our most
optimistic upper test statistic, we would reject while our conservative lower
test statistic would fail to reject. Having posterior estimates leading to
opposite conclusions is not a sign of reliability so we conclude that we
cannot make a conclusion either way.

Now, if we increase our sample size from n=10 to n=100 and observe
$(n_{1},n_{2},n_{3})=(30,20,50)$, we now have much more evidence that the null
hypothesis of each class being equally likely is incorrect. Note that our
posterior random sets are now defined by:

$(Z_{0},Z_{1},Z_{2},Z_{3})\sim Dirichlet(1,30,20,50).$

The width of each random set is $Z_{0}\sim Beta(1,100)$, rather than
$Z_{0}\sim Beta(1,10)$ as in the previous example. Correspondingly, in Figure
2, we can see in part A) that the polytopes are smaller and more clustered
around our point estimate. This corresponds with our intuition that a larger
sample size should result in more concentrated posterior estimates. Now none
of the 1000 posterior random sets are close to the null point. Thus,
$\pi_{lower}$ and $\pi_{upper}$ are both $<0.001$ meaning that we reject our
null hypothesis with confidence.

Figure 2: A) 100 randomly chosen posterior random sets. Note how the polytopes
are more tightly centered around the point estimate than in Figure 1. B) 1000
simulations of the upper and lower test statistic and a vertical line
representing $\mathcal{P}_{0}$. With the larger sample size, both
$\pi_{lower}$ and $\pi_{upper}$ are $<0.0001$.

As we saw in the previous example, increasing our sample size made us go from
an uncertain conclusion to a certain one. This turns out to be not only a
feature of the previous example, but holds for any continuous, bounded test
statistic.

######  Theorem 4.

Consider a k-dimensional DS hypothesis as described above based on a
continuous and bounded test statistics,
$T({\mathcal{\hat{P}}},\mathcal{P}_{0})$. As the sample size increases, the
probability of a DS hypothesis test returning an unknown result goes to 0 in
probability.

###### Proof.

We first desire to show that
$T_{upper}\overset{P}{\to}T(\mathcal{P},\mathcal{P}_{0})$. This requires that:

$\displaystyle\lim_{n\to\infty}P(|\sup_{\hat{p}\in\Delta(\mathcal{Z})}T(|\hat{P},\mathcal{P}_{0})-T(\mathcal{P},\mathcal{P}_{0}|\geq\epsilon)=0$

Since $\Delta(\mathcal{Z})\sim Dirichlet(1,n_{1},...,n_{k})$, we have by the
strong law that $\Delta(\mathcal{Z})$ converges to a degenerate distribution
centered at $\mathcal{P}$. If we let
$A_{n}=\\{\hat{p}:T(\hat{p},\mathcal{P}_{0})\geq
T(\mathcal{P},\mathcal{P}_{0})\\}$ $A_{n}$ converges point wise to
$\mathcal{P}$. Thus, by monotone convergence, we get that:

$\displaystyle\lim_{n\to\infty}P(|\sup_{\hat{p}\in\Delta(\mathcal{Z})}T(|\hat{P},\mathcal{P}_{0})-T(\mathcal{P},\mathcal{P}_{0})|\geq\epsilon)$
$\displaystyle=\lim_{n\to\infty}P(|\sup_{\hat{p}\in
A_{n}}T(|\hat{P},\mathcal{P}_{0})-T(\mathcal{P},\mathcal{P}_{0})|\geq\epsilon)$
$\displaystyle=P(|T(\mathcal{P},\mathcal{P}_{0})-T(\mathcal{P},\mathcal{P}_{0})|\geq\epsilon)=0$

By the same argument, one can show that
$T_{lower}\overset{P}{\to}T(\mathcal{P},\mathcal{P}_{0})$.

To finish the proof, recall that we return an “unknown” conclusion when
$\pi_{upper}\leq\alpha$ but $\pi_{lower}>\alpha$. However since $T_{upper}$
and $T_{lower}$ are converging in probability to
$T(\mathcal{P},\mathcal{P}_{0})$, $\pi_{upper}\rightarrow\pi_{lower}$. And it
becomes possible to return an ”unknown” result. ∎

##  6 Interpretations and Connections to Other Forms of Inference

Now that we have defined what we claim to be a DS hypothesis test, we show how
this test fits into the DS, Fiducial, and frequentist paradigm.

###  6.1 DS Interpretation

To see how this procedure fits into the DS framework, we will first abstract
this procedure in the spirit of Fraser’s structural inference [9] and
illustrate how our tail probabilities define proper DS upper and lower
probabilities.

First, let us write our data generation scheme in terms of a Fraser inspired
randomized structural equation:

$\mathcal{X}=G(\mathcal{P},\mathcal{U}|\pi).$

here $G$ is our deterministic data generating equation, $\mathcal{P}$ are our
parameters of interest, $\mathcal{U}$ is the random component and $\pi$ is an
unknown permutation. Unlike Fraser’s structural equation framework, we will
not be assuming a group structure and will allow for aspects of the data
generating equation to contain randomized components via $\pi$. We will
further expand upon the implications of this randomized data generating
equation in the Fiducial Inference interpretation section.

Using this, we can define the subset of dependent uniform random variables
$\mathcal{U}=(U_{1},...U_{n})$ that could have generated $(n_{1},...n_{n})$
for some $\mathcal{P}=(p_{1},...p_{k})$ as:

$\mathcal{R}_{x}=\\{\mathcal{U}\in[0,1]^{n}:\exists\mathcal{P}\text{ and
}\exists\pi\in\mathcal{S}(\\{1,...,k\\})\text{ st
}(n_{1},...n_{n})=G(\mathcal{P},\mathcal{\textbf{u}}|\pi)\\}.$

Here, $\mathcal{S}(\\{1,...,k\\})$ is the set of all permutations, $\pi$ on
the set $\\{1,...,k\\}$ which is. Given $\textbf{u}\in\mathcal{R}_{x}$ , there
exists a non-empty “feasible” set on the parameter space
$\mathcal{F}(\mathcal{U})$ that could have been used to generate the data:

$\mathcal{F}(\mathcal{U})=\\{\mathcal{P}:(n_{1},...n_{k})=G(\mathcal{P},\textbf{u}|\pi),\exists\pi\in\mathcal{S}_{\\{1,....,k\\}}\\}.$

In the case of Dirichlet-DSM, $\mathcal{F}(\mathcal{U})$ has a very simple
expression. $\mathcal{F}(\mathcal{U})$ is directly equal to our previously
described polytopes with Dirichlet vertices, which we have been calling
$\Delta(Z)$. Furthermore, as we can see from our definition of $\Delta(Z)$,
there is no dependence on the permutation $\pi$ that was used to generate the
data. The ability to decouple class randomization from inference is one of the
strengths of the method. We now introduce how these random sets yield valid DS
upper and lower probabilities. Consider a continuous test statistic
$T(\hat{\mathcal{P}},\mathcal{P}_{0})$ which defines a measurable map
$T(\cdot,\mathcal{P}_{0}):(\Delta^{k-1},\mathcal{M}_{\Delta^{k-1}})\rightarrow(\Omega,\mathcal{M}_{\Omega})$,
where $\mathcal{M}_{\Delta}^{k-1}$ is the usual Borel algebra on the $k-1$
dimensional simplex and $\Omega$ is a one-dimensional measure space, most
commonly $\mathbb{R}$ or $\mathbb{N}$. Given a measurable posterior random
set, $\Delta(Z)$, via continuity of $T$, $T(\Delta(Z))$ as well as $\sup
T(\Delta(Z))$ and $\inf T(\Delta(Z))$ are all continuous measurable sets in
$\mathcal{M}_{\Omega}$. To perform a hypothesis test, the user now provides a
measurable set $\Sigma\subset\mathcal{M}_{\Omega}$ that corresponds to the
hypothesis of interest.

For the previous example in Section 5, we tested if the parameters were
uniform, $(p_{1}=p_{2}=p_{3}=1/3)$ with chi-squared test statistic
$T(\mathcal{P},\mathcal{P}_{0})=\sqrt{\frac{\sum_{i=1}^{3}(np_{i}-n/3)^{2}}{n/3}}$.
With the test of uniformity, our corresponding set of interest is:
$\Sigma=\\{T(\mathcal{P},\mathcal{P}_{0})\leq
T(\mathcal{\hat{P}},\mathcal{P}_{0})_{\alpha\%}\\}$ where
$T(\mathcal{\hat{P}},\mathcal{P}_{0})_{\alpha\%}$ is the $\alpha$ quantile of
$T(\mathcal{\hat{P}},\mathcal{P}_{0})$.

DS theory presumes the existence of a mass function $m$ such that:

$m:2^{\Omega}\rightarrow[0,1]$

such that:

$m(\emptyset)=0$

and

$\sum_{S\in 2^{\Omega}}m(S)=1.$

Any probability measure can be written as a mass function by letting every
measurable set be equal to its probability and setting unmeasurable sets to 0:

$m(A)=\begin{cases}P(A)&\text{ if }A\in\mathcal{M}_{\Omega}\\\ 0&\text{ if
}A\not\in\mathcal{M}_{\Omega}\end{cases}.$

Moreover, for any set $S\subset 2^{\Omega}$ (as opposed to any measurable
set), the belief of a set S is defined as the mass of sets which are contained
within S:

$bel(S)=\sum_{A|A\subset S}m(A).$

The plausibility of S is defined as the mass of sets that intersect S:

$pl(S)=\sum_{A|A\cap S\neq\emptyset}m(A).$

Due to this construction:

$bel(A)\leq P(A)\leq pl(A).$

With the above definitions, we can now show that $\pi_{upper}$ and
$\pi_{lower}$ are valid belief and plausibility functions for the one-sided
hypothesis test previously described:

######  Theorem 5.

When $T$ defines a distance metric, $\pi_{upper}$ and $\pi_{lower}$ define
valid belief and plausibility functions for $S$ where
$S=[T(\mathcal{\hat{P}},\mathcal{P})_{\alpha\%}\infty)$.

###### Proof.

First, since T defines a distance metric, if for a given $Z$:

$T_{upper}(Z)\leq T(\mathcal{\hat{P}},\mathcal{P})_{\alpha\%}$

this implies that:

$S=[T_{upper}(Z)_{\alpha},\infty)\subset[T_{upper}(Z),\infty).$

Now, if we let $m$ be the counting measure:

$\displaystyle bel(S)$
$\displaystyle=\sum_{[T_{upper}(Z),\infty)|[T_{upper}(Z),\infty)\subset
S}m([T_{upper}(Z),\infty))$ $\displaystyle=\sum_{T_{upper}(Z)\leq
T(\mathcal{P},\hat{\mathcal{P}}_{\alpha\%})}m([T_{upper}(Z),\infty))$
$\displaystyle=P(T_{upper}\leq T(\mathcal{P},\mathcal{\hat{P}})_{\alpha\%,})$
$\displaystyle=\pi_{upper}$

Likewise using the same argument, we can show that $pl(S)=\pi_{lower}$. ∎

Based on this theorem, we can view the belief function as the probability we
observe a posterior set where every element have a $T$ value less than the
$\alpha\%$ cut-off. This belief function is what we have been referring to as
our lower p-value. On the other hand, the plausibility function is the
probability that we observe a posterior set where there exists at least one
element whose value $T$ is less than $\alpha$.

###  6.2 Fiducial Interpretation

In terms of Fiducial Inference, we will be showing that the previously defined
”feasible” set $\Delta(Z)$ defines what is known as a Generalized Fiducial
Quantity. Recall that given fixed latent parameters $z$ from a Dirichlet
distribution, our inference procedure results in a polytope $\Delta(z)$ of
parameter values for which there exist uniform random variables
$(U_{1},...U_{k})$ that could have generated the observed data. As above, we
write the set of parameters that could have feasibly generated
$(n_{1},...,n_{k})$ as:

$\mathcal{F}(\mathcal{U})=\\{\mathcal{P}\in\Delta^{k}:(n_{1},...n_{k})=G(\mathcal{P},\textbf{u}|\pi),\exists\pi\in\mathcal{S}_{\\{1,....,k\\}}\\}.$

However, unlike DS, which directly works with the random posterior sets,
$\mathcal{F}(\mathcal{U})$, in the Generalized Fiducial Scheme, when
$\mathcal{F}(\mathcal{U})$ contains multiple elements, one selects an element
according to some possible random rule V. Mathematically, we say that for any
measurable set $S\in\Delta^{k}$ of feasible parameters, one selects a possibly
random element $V(S)$ with support $\overline{S}$ where $\overline{S}$ in the
closure of S. In our case, our random set is $\mathcal{F}(\mathcal{U})$ from
which we choose the supremum, infimum, or average value of the polytope. As
supremum, infimum, and average values are all random rules that lie within the
closure, our upper, lower, and mean test distributions define valid Fiducial
Generalized Quantities.

##  7 The Advantages of DS Multinomial Hypothesis Testing

We would now like to pivot to demonstrating features of our DS hypothesis
testing procedure and the powerful properties though an investigation of DS’s
robust insight into testing.

###  7.1 The Unknown class can help control Type I/Type II Error Rates

Our DS inference procedure indicates if one should reject the null with
confidence, fail to reject the null with confidence, or if we lack the
information to conclude either way. If one restricts themselves to only the
samples for which DS has given a certain conclusion, we can demonstrate that
this extra confidence translates into a lower overall Type I/ Type II
misclassification rate. To demonstrate, we perform a 4 dimensional test:

$\displaystyle H_{0}:$ $\displaystyle p=(0.25,0.25,0.25,0.25)$ $\displaystyle
H_{1}:$ $\displaystyle p\neq(0.25,0.25,0.25,0.25)$

where $(X_{1},X_{2},X_{3},X_{4})\sim Multinomial(n,(0.3,0.3,0.3,0.1))$ and $n$
is varied from 10 to 100. For each n, 250 samples are drawn and both a DS and
a permutation-based chi-squared test are performed. From the chi-squared test,
we will calculate the percent of tests that are correctly rejected, (Freq
Percent Correct). And from the DS we will compute 2 values, first, the percent
of total tests we correctly reject (DS Total Correct) and the percent of
certain tests we correctly reject (DS Certain Correct). From the results of
the Figure 7.1, we can see that DS Certain Correct has the higher power,
followed by DS Total Correct and then Freq Percent Correct. Moreover, using DS
Certain Correct only comes with a small sacrifice in sample size, as only
between 0.05-0.1 of the samples were determined to be uncertain with the
number of uncertain tests intuitively going to 0 as sample size increase.

To confirm that this increase in accuracy is not coming at the expense of
preserving the 0.05 level, we perform the same simulation as before except
this time the null is true. The results of this null simulation in Figure 7.1
show that DS Certain Correct does indeed preserve a similar level to the
frequentist test with the slightly lower level of DS Total Correct explainable
by the uncertain classes.

###  7.2 The Unknown Class gives insight into the difficulty of a test

DS has a long history in the engineering community providing a way to
comprehend the level of epistemic uncertainty in a hypothesis test through the
inclusion of an “unknown” option. In figure 3, we demonstrate the utility of
this unknown class through a simulated 4-class Multinomial goodness-of-fit
test. The alternate hypothesis had probabilities
$(\frac{2}{6},\frac{1}{6},\frac{2}{6},\frac{1}{6})$. The test statistic used
was the chi-square statistic and the sample size, $n$, ranged from 4 to 256.
At each sample size, 1000 tests were performed and the upper and lower
p-values were generated using 100 polytopes. The null distribution of the
Frequentist test was simulated from 1000 draws from a multinomial distribution
with equal probabilities. No multiple testing correction was performed.

In the first plot in Figure 3, we can see that while both tests reach the
correct level of 0.05 as the sample size increases, they have very different
behavior at low sample sizes. In the frequentist test, when the dimension-to-
sample size ratio is one, the frequentist test rejects too few tests. This
inability to reject is a well-documented high-dimensional phenomenon [2] not
just for multinomials, but for a large class of hypothesis tests [29].
However, in the DS test, when the sample size is too small, there is a
substantial possibility that the hypothesis test will return with an
“Uncertain” conclusion. Unlike the frequentist test which fails to reject
nearly all, this unknown class is giving additional information to the user on
the difficulty of making a conclusion at the current sample size. We see this
additional information at work in Figure 4. Once again, at low sample sizes,
the frequentist test does not indicate if we are failing to reject the null
because the null is true or because we lack the sample size to make any
conclusions without performing a seperate power calculation. However, if we
look at the DS test, the large probability of a test returning an “Uncertain”
result indicates that our issue is in the sample size.

Figure 3: Results of Frequentist and DS hypothesis test of uniformity when the
null is true. At low sample sizes, the Frequentist test gives a deceptively
low rejection rate while the DS properly indicates the difficulty of this
problem by having a large probability for an “Uncertain” result.  Figure 4:
Results of Frequentist and DS hypothesis test of uniformity when the null is
false. Once again, with the frequentist test, at low sample sizes, it is
unclear if the alternate is correct or if the sample size is lacking, while
the DS makes this clear through the unknown class that the sample size is
lacking.

###  7.3 DS can model Adversarial Attacks on a Test

In addition to providing a novel characterization of power, our approach can
also employ a technique called “weakening” to evaluate the effect an
adversarial attack would have on a hypothesis. Weakening is a common practice
in reliability engineering when one gives additional weight for the unknown
class either to satisfy long-running frequentist properties [19] or to account
for potential uncertainties in the observed data [4]. For our DS model, we
will perform weakening by artificially increasing the width of our polytopes.
Formally:

######  Definition 7.1 ($\alpha$-Weakened Hypothesis Test).

An $\alpha$-Weakened Hypothesis test is the same as the original DS hypothesis
testing procedure except the Dirichlet parameters are drawn from:

$(Z_{0},...,Z_{k})\sim Dirichlet(1+\alpha,x_{1},...,x_{k})$

where $\alpha>0$.

Recalling that $Z_{0}$ directly controls the width of the random polytopes
that serves as our posterior estimates, one can see how this directly
increases the gap between the upper and lower test statistics. Since our DS
testing procedure returns an “unknown” when the upper and lower test
statistics disagree, this weakening directly leads to increasing the
possibility of a test returning an ‘unknown” result.

As a demonstration of this, in Figure 5, we added weakening to a 4-class
multinomial hypothesis test with the same alternate hypothesis as the previous
simulation. The sample size is 128 and once again, no multiple testing
correction was performed. From the results in the figure, we can see that
increasing the amount of weakening leads to a dramatic increase in the
probability of an uncertain result with a weakening of around 10-20 leading to
almost all tests resulting in an unknown result.

Figure 5: Effect of weakening on a DS test of uniformity when the null is
false. As the number of adversarial samples (represented by alpha) increases,
the conclusions become increasingly more muddled.

In addition to allowing for the addition of user-specified uncertainty, this
form of weakening comes with an easy-to-understand interpretation. For
demonstration purposes, let us consider a 3-dimensional hypothesis test in
which the observed counts were $(n_{1},n_{2},n_{3})$ with total count
$n^{*}=n_{1}+n_{2}+n_{3}$. Since:

$(Z_{0},Z_{1},Z_{2},Z_{3})\sim Dirichlet(1,n_{1},n_{2},n_{3})$

component wise, the $Z_{i}$’s have the distributions:

  * •

$Z_{0}\sim Beta(1,n+1)$

  * •

$Z_{1}\sim Beta(n_{1},n+1-n_{1})$

  * •

$Z_{2}\sim Beta(n_{2},n+1-n_{2})$

  * •

$Z_{3}\sim Beta(n_{3},n+1-n_{3})$

As described previously, $Z_{0}$ is then added to each of the other Zs to
define the edges of the polytope:

  * •

$(Z_{1}+Z_{0},Z_{2},Z_{3})$

  * •

$(Z_{1},Z_{2}+Z_{0},Z_{3})$

  * •

$(Z_{1},Z_{2},Z_{3}+Z_{0})$

So $Z_{0}$ controls the width of our polytope while $Z_{1},Z_{2},Z_{3}$
controls the location of the polytope.

Now, let us say that additional $\alpha$ data points has been added to our
dataset. We are unclear which of the 3 classes it has been added to, so we act
as though it was as likely to have been added to any of the 3 class. This
would result in observed counts of
$(n_{1}+\frac{\alpha}{3},n_{2}+\frac{\alpha}{3},n_{3}+\frac{\alpha}{3})$ and
$n=n_{1}+n_{2}+n_{3}+\alpha$. Component wise, our $Z_{i}$’s will become:

  * •

$Z_{0}\sim Beta(1+\alpha,n+1+\alpha)$

  * •

$Z_{1}\sim Beta(n_{1},n+1+\alpha-n_{1})$

  * •

$Z_{2}\sim Beta(n_{2},n+1+\alpha-n_{2})$

  * •

$Z_{3}\sim Beta(n_{3},n+1+\alpha-n_{3})$

Thus, our width, $Z_{0}$, has been increased proportional to the addition of
$\alpha$ datapoints. Therefore, $\alpha$\- weakening is analogous to the
effect of adding $\alpha$ datapoints to any of the classes. If we have
previously confidently rejected or failed to reject before weakening, but then
became unknown after $\alpha$-weakening, this indicates that $\alpha$ data
points are sufficient to contradict our previous results. Combine this with
the previous assertions that $\pi_{lower}$, $\pi_{upper}$ serve as a
reasonable bound on the behavior of $\pi_{freq}$ and we can also make
reasonable claims about the behavior of $\pi_{freq}$ under $\alpha$-weakening.

Returning to Figure 5, this simulation indicates that if 20 datapoints were
added to the original 128, it is possible to have the hypothesis reject (hence
why $\pi_{lower}\leq 0.05)$ and not reject (hence by $\pi_{upper}\geq 0.05$).
Stated another way, with 20 additional datapoints, you can make a hypothesis
test return whichever result you would like.

##  8 Using Dempster-Shafer Inference to study text data in Verbal Autopsies

In this section, we demonstrate Dempster-Schafer inference in the context of
verbal autopsies (VAs). In particular, in low-resource settings, there is a
lack of critical public health infrastructure to comprehensively record all
births and deaths [17, 26]. When deaths occur outside of hospitals or are not
routinely recorded, VAs are used as a cost-effective approach to learning
about the distribution of deaths by cause. VAs consist of a survey with a
surviving family member or caregiver. The typical VA survey contains a mix of
categorical (e.g. did the decedent have a fever?) and free response text
describing symptoms that a subject seemed to exhibit before passing away. The
symptoms are then tabulated and inputted into a specialized designed
classification algorithm such as Tariff 2.0 [25], InterVA [3], and InsilicoVA
[20].

As collecting data for VA surveys involves training personnel in countries
that may already be struggling to ensure adequate public health resources and
interviews may bring back potentially traumatic memories, there have been
recent developments in choosing more efficiently the types of questions to ask
during the interview [32]. Building off of this, we would like to use our DS
test to help determine which words in the free-response section may help us
discriminate various causes of death.

We take data from the Population Health Metrics Research Consortium (PHMRC)
which contains verbal autopsy and physical autopsy information from 7841
adults taken from 6 sites in developing nations around the world [21]. As this
contains information about verbal autopsy as well as the true cause of death
it is often considered a “Gold Standard” dataset that verbal autopsy
algorithms are validated on. In this example, we will focus our attention on
ability of the text portion of PHMRC to discriminate between potentially
similar causes of death, which in this case are the 8 different types of
cancers: Breast, Cervical, Colorectal, Esophageal, Leukemmia/Lymphomas, Lung,
Prostate, and Stomach. 854 deaths were from one of these cancers and had the
top “GS Level 1” in the cause of death assessment. To determine these causes
of death, there were a total of 624 uniquely identified words that were used
at least once.
We will be testing the DS multinomial test as a goodness-of-fit hypothesis
that the probability of the usage of a word across causes of death matches the
prevalence of the diseases. Words that reject the hypothesis indicate a
potential utility in discerning causes of death. For each word, a DS
Hypothesis test with 1000 replicates was performed. As a point of comparison,
a resampled chi-squared test with 1000 replicates was also performed. Both
test’s conclusions were made at the $\alpha=0.05$ level.

Figure 6: (Top Left): Chi squared test of word significance. The vertical grey
line represents the sample size of 50 which was used in [25]. (Top Right):
Histograms of the sample size of the words by Hypothesis test conclusion.
(Bottom Left): DS Hypothesis test of word significance.

Figure 6 A and C demonstrate the number of people who used a word as well as
it’s goodness-of-fit p-value under the chi-squared test and the DS test. We
can see in both that the distribution of word count is heavily right skewed,
This indicates that many words were used only by a small number of subjects
and accords with well known, empirical observations of word use in natural
language [33].

In both tests, we see an agreement on the words for which we have evidence to
reject the null hypothesis albeit. However, the two tests differ when it comes
to the words that the chi-square test failed to reject. While the frequentist
test categorizes anyone with a p-value above 0.05 in the “Fail to Reject”
cateogry, the DS test splits this group into two similarly sized groups. These
are the words for which Fail to Reject with confidence, and words for which we
lack evidence to conclude either way. The words for which we lack the evidence
to conclude either way includes some medical terms such as ”pneumonia” (used
31 times) and ”diabet[us] (used 26 times)”. While pneumonia and diabetus are
known to be associated with various forms of cancer or cancer treatment [28,
10], the low sample size could indicate an unfamiliarity with medical
terminology in the low-resource areas where VAs were performed. On the other
hand, words such as ”vomit” (used 28 times) and ”urin[e]” (used 23 times) were
rejected with confidence by the DS test. In total, by being able to separate
words which maybe helpful to discriminate between causes of death but may not
see sufficient usage due to educational or other resource related issue from
words for which we have some confidence are unrelated to the cause of death
gives the investigator a deeper insight into the data, as well as gives
potential inspiration for designing more focused interviews in the future.

Finally, we demonstrate the DS tests’ results as a variable prescreening
procedure. Words that were deemed significant were entered into a purpose-
built classification method for verbal autopsies called Tarrif [25]. On table
1 , we see that the Tariff algorithm, when trained on all 624 words, is able
to classify each verbal autospy into one of the 8 different types of cancer
with an accuracy of 31.5%. This compares favorably to the meager accuracy of
15. 97% according to the rule of thumb in the Tariff to use all words used at
least 50 times. This poor performance is corroborated by the results of the
previous goodness-of-fit tests, as many words which both the chi-squared and
DS test deemed significant were used by less than 50 people. This seems to
indicate that rule of thumb to ignore any words less than 50 times which was
presented in [25] is sub-optimal. Instead, we recommend that a word’s
inclusion should be a factor of both sample size and the strength of the
evidence, which are factors that are considered in the chi-squared and the DS
test. Finally, the best preforming models were those trained on the rejected
words by the chi-squared an DS test, albeit with the DS test gave the same
classification accuracy based on 114 words rather than the 149 of the chi-
squared test. The fact that fewer words were rejected by DS can be explained
by how the DS procedure is classifying each word under a trivariate rather
than binary scheme, and thus it is expected that some borderline words which
were rejected by chi-squared maybe classified as unknown by DS.

Scheme | Number of Words | Accuracy
---|---|---
Use all Words | 624 | 31.25 %
Use all Words that were used atleast 50 times | 60 | 15.97 %
Use all words there were rejected by chisq at $\alpha=0.05$ | 149 |  40.28 %
Use all words there were rejected by DS at $\alpha=0.05$ | 114 |  40.28 %
Use all words there were rejected or Unknown by DS at $\alpha=0.05$ | 362 | 35.42 %
Table 1: Classification Accuracy of the Tariff 2.0 algorithm given various

##  9 Conclusion

In this paper, we have demonstrated how to construct an alternative approach
to p-values for multinomial data that allows a test to return an ”unknown”
result. By allowing for this third category, we believe that this allows for
one to gain new insights into the data while simultaneously keeping much of
the intuition and implications that make p-value based hypothesis testing
appealing in the first place. It is the authors belief that such an approach
will provide an important voice in the continued conversation in the future
use of p-values in scientific endevors.

## References

  * Amrhein et al. [2019] V. Amrhein, S. Greenland, and B. McShane.  Scientists rise up against statistical significance, Mar 2019.  URL https://www.nature.com/articles/d41586-019-00857-9.
  * Balakrishnan and Wasserman [2018] S. Balakrishnan and L. Wasserman.  Hypothesis testing for high-dimensional multinomials: A selective review.  _The Annals of Applied Statistics_ , 12(2):727 – 749, 2018.  doi: 10.1214/18-AOAS1155SF.  URL https://doi.org/10.1214/18-AOAS1155SF.
  * Byass et al. [2019] P. Byass, L. Hussain-Alkhateeb, L. D’Ambruoso, S. Clark, J. Davies, E. Fottrell, J. Bird, C. Kabudula, S. Tollman, K. Kahn, L. ler, and M. Petzold.  An integrated approach to processing WHO-2016 verbal autopsy data: the InterVA-5 model.  _BMC Med_ , 17(1):102, May 2019.
  * de Campos and Benavoli [2011] C. de Campos and A. Benavoli.  Inference with multinomial data: why to weaken the prior strength.  In _International Joint Conference on Artificial Intelligence (IJCAI)_ , pages 2107–2112. AAAI Press, 2011.  (acc.rate 30
  * Dempster and Shafer [1976] A. Dempster and G. Shafer.  _A Mathematical Theory of Evidence_.  Limited paperback editions. Princeton University Press, 1976.  ISBN 9780691100425.  URL https://books.google.com/books?id=wug9DwAAQBAJ.
  * Dempster [1966] A. P. Dempster.  New Methods for Reasoning Towards Posterior Distributions Based on Sample Data.  _The Annals of Mathematical Statistics_ , 37(2):355 – 374, 1966.  doi: 10.1214/aoms/1177699517.  URL https://doi.org/10.1214/aoms/1177699517.
  * Dempster [1968] A. P. Dempster.  Upper and Lower Probabilities Generated by a Random Closed Interval.  _The Annals of Mathematical Statistics_ , 39(3):957 – 966, 1968.  doi: 10.1214/aoms/1177698328.  URL https://doi.org/10.1214/aoms/1177698328.
  * Diez et al. [2019] D. M. Diez, C. D. Barr, and M. Çetinkaya Rundel.  _OpenIntro statistics_.  OpenIntro, 2019.
  * Fraser [1968] D. A. S. Fraser.  _The Structure of Inference_.  John Wiley & Sons, 1968.
  * Giovannucci et al. [2010] E. Giovannucci, D. M. Harlan, M. C. Archer, R. M. Bergenstal, S. M. Gapstur, L. A. Habel, M. Pollak, J. G. Regensteiner, and D. Yee.  Diabetes and cancer.  _Diabetes Care_ , 33(7):1674–1685, July 2010\.  doi: 10.2337/dc10-0666.  URL https://doi.org/10.2337/dc10-0666.
  * Hannig [2009] J. Hannig.  On generalized fiducial inference.  _Statistica Sinica_ , 19(2):491–544, 2009.  ISSN 10170405, 19968507.  URL http://www.jstor.org/stable/24308841.
  * Harrington et al. [2019] D. Harrington, E. Al., A. A. the Department of Data Sciences, J. E. Goldstick, Y. G. Others, , and E. Braunwald.  New guidelines for statistical reporting in the journal: Nejm, Oct 2019\.  URL https://www.nejm.org/doi/full/10.1056/nejme1906559.
  * Illowsky et al. [2018] B. Illowsky, S. L. Dean, and B. Illowsky.  _Introductory statistics_.  OpenStax, Rice University, 2018.
  * Jacob et al. [2021] P. E. Jacob, R. Gong, P. T. Edlefsen, and A. P. Dempster.  A gibbs sampler for a class of random convex polytopes.  _Journal of the American Statistical Association_ , 116(535):1181–1192, 2021.  doi: 10.1080/01621459.2021.1881523.  URL https://doi.org/10.1080/01621459.2021.1881523.
  * Lawrence et al. [2009] E. Lawrence, C. Liu, S. V. Wiel, and J. Zhang.  A new method of multinomial inference using dempster-shafer theory.  _Unpublished Manuscript_ , 2009.
  * Lehmann and Casella [1998] E. L. Lehmann and G. Casella.  _Theory of Point Estimation_.  Springer-Verlag, New York, NY, USA, second edition, 1998.
  * Mahapatra et al. [2007] P. Mahapatra, K. Shibuya, A. D. Lopez, F. Coullare, F. C. Notzon, C. Rao, and S. Szreter.  Civil registration systems and vital statistics: successes and missed opportunities.  _The Lancet_ , 370(9599):1653–1663, 2007.
  * Martin and Liu [2015] R. Martin and C. Liu.  _Inferential models: reasoning with uncertainty_ , volume 145.  CRC Press, 2015.
  * Martin et al. [2010] R. Martin, J. Zhang, and C. Liu.  Dempster–Shafer Theory and Statistical Inference with Weak Beliefs.  _Statistical Science_ , 25(1):72 – 87, 2010.  doi: 10.1214/10-STS322.  URL https://doi.org/10.1214/10-STS322.
  * McCormick et al. [2016] T. H. McCormick, Z. R. Li, C. Calvert, A. C. Crampin, K. Kahn, and S. J. Clark.  Probabilistic cause-of-death assignment using verbal autopsies.  _Journal of the American Statistical Association_ , 111(515):1036–1049, 2016.  doi: 10.1080/01621459.2016.1152191.  URL https://doi.org/10.1080/01621459.2016.1152191.  PMID: 27990036.
  * Murray et al. [2011] C. J. Murray, A. D. Lopez, R. Black, R. Ahuja, S. M. Ali, A. Baqui, L. Dandona, E. Dantzer, V. Das, U. Dhingra, A. Dutta, W. Fawzi, A. D. Flaxman, S. Gómez, B. Hernández, R. Joshi, H. Kalter, A. Kumar, V. Kumar, R. Lozano, M. Lucero, S. Mehta, B. Neal, S. L. Ohno, R. Prasad, D. Praveen, Z. Premji, D. Ramírez-Villalobos, H. Remolador, I. Riley, M. Romero, M. Said, D. Sanvictores, S. Sazawal, and V. Tallo.  Population health metrics research consortium gold standard verbal autopsy validation study: design, implementation, and development of analysis datasets.  _Population Health Metrics_ , 9(1), Aug. 2011.  doi: 10.1186/1478-7954-9-27.  URL https://doi.org/10.1186/1478-7954-9-27.
  * Pearl [1988] J. Pearl.  On probability intervals.  _International Journal of Approximate Reasoning_ , 2(3):211–216, 1988.  ISSN 0888-613X.  doi: https://doi.org/10.1016/0888-613X(88)90117-X.  URL https://www.sciencedirect.com/science/article/pii/0888613X8890117X.
  * Rowntree [2018] D. Rowntree.  _Statistics without tears an introduction for non-mathematicians_.  Penguin Books, 2018.
  * Sentz and Ferson [2002] B. Sentz and S. Ferson.  Combination of Evidence in DempsterShafer Theory.  Technical report, Sandia National Laboratory, 2002.
  * Serina et al. [2015] P. Serina, I. Riley, A. Stewart, S. L. James, A. D. Flaxman, R. Lozano, B. Hernandez, M. D. Mooney, R. Luning, R. Black, R. Ahuja, N. Alam, S. S. Alam, S. M. Ali, C. Atkinson, A. H. Baqui, H. R. Chowdhury, L. Dandona, R. Dandona, E. Dantzer, G. L. Darmstadt, V. Das, U. Dhingra, A. Dutta, W. Fawzi, M. Freeman, S. Gomez, H. N. Gouda, R. Joshi, H. D. Kalter, A. Kumar, V. Kumar, M. Lucero, S. Maraga, S. Mehta, B. Neal, S. L. Ohno, D. Phillips, K. Pierce, R. Prasad, D. Praveen, Z. Premji, D. Ramirez-Villalobos, P. Rarau, H. Remolador, M. Romero, M. Said, D. Sanvictores, S. Sazawal, P. K. Streatfield, V. Tallo, A. Vadhatpour, M. Vano, C. J. L. Murray, and A. D. Lopez.  Improving performance of the tariff method for assigning causes of death to verbal autopsies.  _BMC Medicine_ , 13(1), Dec. 2015.  doi: 10.1186/s12916-015-0527-9.  URL https://doi.org/10.1186/s12916-015-0527-9.
  * Setel et al. [2007] P. W. Setel, S. B. Macfarlane, S. Szreter, L. Mikkelsen, P. Jha, S. Stout, and C. AbouZahr.  A scandal of invisibility: making everyone count by counting everyone.  _The Lancet_ , 370(9598):1569–1577, 2007.
  * Stone and G.W. [2016] S. P. Stone and G.W.  The nature of the p value: Nejm, Sep 2016.  URL https://www.nejm.org/doi/full/10.1056/NEJMc1612970.
  * Valvani et al. [2019] A. Valvani, A. Martin, A. Devarajan, and D. Chandy.  Postobstructive pneumonia in lung cancer.  _Ann Transl Med_ , 7(15):357, Aug 2019.
  * Wainwright [2019] M. J. Wainwright.  _High-Dimensional Statistics: A Non-Asymptotic Viewpoint_.  Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.  doi: 10.1017/9781108627771.
  * Xie and Singh [2013] M.-g. Xie and K. Singh.  Confidence distribution, the frequentist distribution estimator of a parameter: A review.  _International Statistical Review_ , 81(1):3–39, 2013.  doi: https://doi.org/10.1111/insr.12000.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/insr.12000.
  * Yager and Liu [2008] R. Yager and L. Liu.  _Classic Works of the Dempster-Shafer Theory of Belief Functions_.  Studies in Fuzziness and Soft Computing. Springer Berlin Heidelberg, 2008\.  ISBN 9783540447924.  URL https://books.google.com/books?id=sVZqCQAAQBAJ.
  * Yoshida et al. [2023] T. Yoshida, T. S. Fan, T. McCormick, Z. Wu, and Z. R. Li.  Bayesian active questionnaire design for cause-of-death assignment using verbal autopsies, 2023.
  * Zipf [1999] G. Zipf.  _The Psycho-Biology of Language: An Introduction to Dynamic Philology_.  Cognitive psychology. Routledge, 1999.  ISBN 9780415209762.  URL https://books.google.com/books?id=w1Z4Aq-5sWMC.