# On reconstruction of the Hilbert space from conceptual variables.

Inge S. Helland<EMAIL_ADDRESS>Department of Mathematics, University of
Oslo
P.O.Box 1053 Blindern, N-0316 Oslo, Norway

###### Abstract

In [1] the general problem of deriving the Hilbert space formulation in
quantum theory is discussed from the point of view of conceptual variables. A
conceptual variable is any variable defined by a person or by a group of
persons. These variables may be inaccessible, i.e., impossible to assign
numerical value to either by experiments or by other means, or accessible. As
one basic assumption, a group $K$ of actions on a space of an inaccessible
variable is defined, and accessible functions of this variable are introduced.
When such a function is permissible, a term that is precisely defined here, a
group of actions $G$ can be introduced on the range of the function. In
general in this paper only the existence of the group $G$ is assumed. By using
group representation theory on the groups defined on a class of accessible
variables, basic Hilbert space formalism is restored under the assumption that
the observator or the set of observators has two maximally accessible
variables in his (their) mind(s). Operators are connected to each variable,
and in the discrete case the possible values of the variables are given by the
eigenvalues of the operators. The entire approach may be connected to a
general epistemic interpretation of quantum theory. In this paper the main
results from [1] are made more precise and more general. It turns out that the
conditions of the main theorem there can be weakened in two essential ways: 1)
No measurements need to be assumed, so the result is also applicable to
general decision situations; 2) States can have arbitrary phase factors. Some
consequences of the main result are also briefly discussed here.

††preprint: AIP/123-QED

##  I Introduction

A fundamental problem in quantum theory is to derive the Hilbert space
formulation from other, more intuitive axioms and notions. In [1] this problem
was approached by using basic group theory together with the notion of
accessible and inaccessible conceptual variables. At the outset, a conceptual
variable in some given setting is any variable defined by a person or a group
of persons. In a typical quantum context the setting may be one where an
observer seeks some knowledge about a physical system. In [1] and [2] this is
called an epistemic process. The variable may then simply be a physical
variable as defined by the observer. Essentially different situations are
quantum models of cognition and decision (see [3]), where the variables may
have a psychological origin.

One limitation of the derivation in [1] is that, basically, an epistemic
prosess is assumed there, so that general applications to cognitions and
decisions are not covered. Here we will try to improve this, in particular, we
will make no assumption that there exists some measurement of the basic
accessible conceptual variable. This has important consequences: The whole
theory may be generalized to situations determined by cognition and decision;
see [3] and [18].

Another limitation was that the proof of the basic Theorem 4.3 in [1] was
based on the following simple version of quatum theory: Each (maximally
accessible; see below) state is given by a unit vector in a Hilbert space. In
reality the state is determined by a unit vector _with an arbitrary phase
factor_. This will be tried taken into account here.

##  II Background; the theory of [1]

A conceptual variable is called _inaccessible_ if it is impossible for an
agent to assign a numerical value to it. In a physical setting such a variable
could be a vector (position, momentum) or the total spin vector of a particle.
A psychological case may be a decision situation where the decision variable
is so comprehensive that the agent is unable to make a decision. As in [1], a
given inaccessible variable is called $\phi$, and is thought to vary in some
space $\Omega_{\phi}$. A group of transformations $K$ is supposed to be
defined on $\Omega_{\phi}$.

Accessible functions $\theta=\theta(\phi)$ were studied in [1] and [2], and
$\theta$ was assumed to vary on a space $\Omega_{\theta}$. In a physical
situation, $\theta$ can, for instance be the spin component in some given
direction; in a decision situation, it can be a simpler decision variable that
enables the agent to make a partial decision. If
$\phi=(\theta_{1},\theta_{2})$, where $\theta_{1}$ and $\theta_{2}$ are
accessible, but $\phi$ is inaccessible, we say that $\theta_{1}$ and
$\theta_{2}$ are _complementary_.

Two further notions are of interest:

Definition 1. The variable $\theta$ is maximally accessible if the following
holds: If $\theta=f(\xi)$ for a function $f$ which is not one-to-one, but has
a surjective inverse, then $\xi$ is inaccessible.

Definition 2. The variable $\theta$ is called permissible if the following
holds: $\theta(\phi_{1})=\theta(\phi_{2})$ implies
$\theta(k\phi_{1})=\theta(k\phi_{2})$ for all $k\in K$.

If $\theta$ is permissible, one can define a group $G$ of actions $g$ on
$\Omega_{\theta}$ such that

$(g\theta)(\phi)=\theta(k\phi);\ k\in K.$ (1)

The mapping from $K$ to $G$ defined by (1) is a group homomorphism. The left
and right invariant measures under $K$ on $\Omega_{\phi}$ and correspondingly
under $G$ on $\Omega_{\theta}$ may be defined under weak conditions.

The point of departure in [1] is group representation theory and the theory of
coherent states. Let $G$ be a group on $\Omega_{\theta}$, not necessarily
induced by $K$ as above. Assume that $G$ is transitive, has a trivial isotropy
group, and that $\theta$ is a maximally accessible variable. Introduce an
irreducible unitary representation $U(g)$ on a Hilbert space $\mathcal{H}$ and
coherent states $|\theta\rangle=U(g)|\theta_{0}\rangle$ whenever
$\theta=g\theta_{0}$, $|\theta_{0}\rangle$ is some fixed vector in
$\mathcal{H}$ and $\theta_{0}\in\Omega_{\theta}$ is fixed. From this, a
resolution of the identity

$I=\int|\theta\rangle\langle\theta|d\rho(\theta)$ (2)

is found, where $\rho$ is a left-invariant measure on $\Omega_{\theta}$. Using
this, an operator for the variable $\theta$ is defined as

$A=\int\theta|\theta\rangle\langle\theta|d\rho(\theta).$ (3)

If several accessible variables are to be discussed, it is useful to write the
resolution of the identity in equivalent form

$I=\int|\theta(\phi)\rangle\langle\theta(\phi)|d\mu(\phi),$ (4)

where $\mu$ is a left-invariant measure on $\Omega_{\phi}$ if such a measure
exists. The operator for $\theta$ can then be written

$A=\int\theta(\phi)|\theta(\phi)\rangle\langle\theta(\phi)|d\mu(\phi),$ (5)

If $G$ is not transitive at the outset, a model reduction is first made to an
orbit of $G$ on $\Omega_{\theta}$. This was motivated in [1] by a similar
procedure from statistics.

In order that the above development shall lead to a useful theory, several
assumptions must be made: 1) $G$ is transitive on $\Omega_{\theta}$, and
$\theta$ is maximally accessible; 2) The representation involved is
irreducible; 3) There can be established a one-to-one correspondence between
the coherent states $|\theta\rangle=U(g)|\theta_{0}\rangle$, the group
elements $g$ and the values $\theta=g\theta_{0}$ of the conceptual variable.
In Section IV below, the assumption 2) will be dropped, but instead it will be
assumed that _two_ related maximally accessible variables are present.

Some of the main results of [1] are the following: If $A$ has a discrete
spectrum, the possible values of $\theta$ are the eigenvalues of $A$, and $G$
is a subgroup of the group of permutations on these values. Furthermore,
$\theta$ is maximally accessible if and only if each eigenspace of $A$ is one-
dimensional.

Different (complementary) $\theta$ give different resolutions of the identity.
If each such $\theta$ is discrete and maximally accessible, each eigenvector
can be interpreted as connected to the question ‘What is the value of
$\theta$?’ together with a definite answer to it. I will also consider
operators also for non-maximally accessible $\theta$; then the eigenspaces of
$A$ have a similar interpretation. The generality of such interpretations for
all state vectors was discussed and also taken up as an open question in [4].

The purpose of this paper is to discuss some of this theory in more detail.
The remainder of this paper is organized as follows: In Section III the
assumptions behind theory in [1] are briefly reviewed. Section IV gives the
main theorem of [1] in a weaker and more precise form, and this theorem is
proved in Section V. In Section VI the properties of the constructed operators
are presented, and section VII treats the spin case. Section VIII discusses
the epistemic interpretation, and Section IX provides a brief final
discussion.

The result of Section IV implies a completely new approach to quantum theory.
This result can be seen as fundamental for the ideas promoted in [1]; the
discussion here could ideally replace parts of Chapter 4 in that book.

##  III Technical assumptions and basic theory

$\Omega_{\phi}$ and $\Omega_{\theta}$ are assumed to be equipped with
topologies, and all functions are assumed to be Borel measurable. The groups
are assumed to be locally compact. As is common, it is assumed that the group
operations $(g_{1},g_{2})\mapsto g_{1}g_{2}$, $(g_{1},g_{2})\mapsto
g_{2}g_{1}$ and $g\mapsto g^{-1}$ are continuous. Furthermore, it is assumed
that the action $(g,\theta)\mapsto g\theta$ is continuous. An additional
assumption, which ensures the existence of invariant measures on
$\Omega_{\theta}$, is the following: Every inverse image of compact sets under
the mapping $(g,\theta)\mapsto(g\theta,\theta)$ should be compact. This
ensures that a left-invariant measure $\rho$ on $\Omega_{\theta}$ exists.

Basic group representation theory was reviewed in [1]. It was assumed that the
groups $K$ and $G$ have representations that give square-integrable coherent
state systems (see [5]). It was assumed further that $G$ is transitive on
$\Omega_{\theta}$ and that the isotropy group is trivial. Then, we can fix
$\theta_{0}\in\Omega_{\theta}$, and represent every $\theta\in\Omega_{\theta}$
by $\theta=g\theta_{0}$, and from this there is a one-to-one correspondence
between $\theta$ and $g$. Let $U$ be a relevant irreducible unitary
representation of $G$ on a Hilbert space $\mathcal{H}$. Fix a vector in
$\mathcal{H}$ and call it $|\theta_{0}\rangle$. The coherent states are then
given by $U(g)|\theta_{0}\rangle$, and I define
$|\theta\rangle=U(g)|\theta_{0}\rangle$ when $\theta=g\theta_{0}$. From this,
the resolution of the identity (2) is proved in [1], and it is natural to
define the operator $A$ corresponding to the variable $\theta$ by (3).

In general the operator $A$ of (3) is unbounded, and is defined on the space
of $|u\rangle\in\mathcal{H}$ for which $\langle u|A|u\rangle$ converges.

By the spectral theorem, $A$ has a spectral decomposition. As is proved in
[1], if $A$ has a discrete spectrum, the possible values of $\theta$ are the
eigenvalues of the operator.

To arrive at quantum theory from this basis, we need a rule to calculate
probabilities, the Born rule. A recent review of various derivations of the
Born rule is given in [6]. In [1] the rule is derived from three assumptions:
1) a focused likelihood principle, that follows from the ordinary likelihood
principle of statistical inference; 2) an assumption of rationality as
formulated by the Dutch book principle. 3) The variables involved are
maximally accessible. In 2) it is not necessarily assumed that the actor(s)
involved is (are) perfectly rational; it is enough that they have ideals which
can be modelled in terms of an abstract perfectly rational being.

##  IV The main theorem

Theorem 4.3 in [1] was fundamental for that book. Here we want to prove a more
precise and in several respects weaker version of that result:

Theorem 1. Consider a situation where there are two maximally accessible
conceptual variables $\theta$ and $\xi$ in the mind of an actor or in the
minds of a communicating group of actors. Make the following assumptions:

(i) On one of these variables, $\theta$, there can be defined group actions
from a transitive group $G$ with a trivial isotropy group and with a left-
invariant measure $\rho$ on the space $\Omega_{\theta}$.

(ii) There exists a unitary irreducible representation $U(\cdot)$ of the group
$G$ defined on $\theta$ such that the coherent states $U(g)|\theta_{0}\rangle$
are in one-to-one correspondence with the values of $\theta$.

(iii) The two maximally accessible variables $\theta$ and $\xi$ can both be
seen as functions of an underlying inaccessible variable
$\phi\in\Omega_{\phi}$. There is a transformation $k$ acting on
$\Omega_{\phi}$ such that $\xi(\phi)=\theta(k\phi)$.

(iv) In the special case where $\phi=(\theta,\xi)$, the transformation $k^{2}$
is the identity.

Then there exists a Hilbert space $\mathcal{H}$ connected to the situation,
and to every accessible conceptual variable there can be associated a unique
symmetric operator on $\mathcal{H}$.

It is important that the theorem is no longer limited to an epistemic process;
the variables $\theta$ and $\eta$ can also be general decision variables,
variables connected to sets of prospects as defined in [18].

The condition (iii) seems to be crucial. Two variables $\theta$ an $\eta$
satisfying this condition are said to be _related_. If no such underlying
variable $\phi$ can be found the two variables are said to be _essentially
different_.

The condition (iv) holds in the case where $\theta$ is the position and $\xi$
is the momentum of a particle.

The condition (ii) is rather technical, but essential for the proof of Theorem
1. The following result may be helpful.

Proposition 1. Let $\rho$ be the left-invariant measure on $\Omega_{\theta}$,
and assume that

$\int|f(g\theta)-f(\theta)|d\rho(\theta)\neq 0$ (6)

for every $f\in L^{2}(\Omega,\rho)$ and every $g\in G$ different from the unit
element.

Then condition (ii) holds for the case where $\mathcal{H}=L^{2}(\Omega,\rho)$
and $U$ is the regular representation given by
$U(g)f(\theta)=f(g^{-1}\theta)$.

Proof. We need to prove that $U(g_{1})f(\theta)\neq U(g_{2})f(\theta)$
whenever $g_{1}\neq g_{2}$. In the $L^{2}$-space language and in terms of the
regular representation this means that

$\int|f(g_{2}^{-1}\theta)-f(g_{1}^{-1}\theta)|d\rho(\theta)\neq 0$ (7)

But this follows from (6), since by left invariance
$d\rho(\theta)=d\rho(g_{1}^{-1}\theta)$.

∎

A consequence of this is that one may drop assumption (ii) in the discrete
case.

##  V Proof of the main theorem.

###  V.1 Basic construction

In this subsection I will take as a point of departure Chapter 2 in Perelomov
[5], which discusses coherent states for arbitrary Lie groups. Let $G$ be a
transitive group acting on the space $\Omega_{\theta}$ associated with some
conceptual variable $\theta$ and $T(g)$ its unitary representation acting on
the Hilbert space $\mathcal{H}$. I will assume that $G$ has a trivial isotropy
group, so that the elements $g$ of $G$ are in one-to-one correspondence with
the values of $\theta$.

As in [5] (and in [1]) take a fixed vector $|\theta_{0}\rangle$ in
$\mathcal{H}$, and consider the set $\\{|\theta\rangle\\}$, where
$|\theta\rangle=T(g)|\theta_{0}\rangle$ with $g$ corresponding to the value
$\theta$. It is not difficult to see that two vectors $|\theta^{1}\rangle$ and
$|\theta^{2}\rangle$ correspond to the same state, i.e., differ by a phase
factor ($|\theta^{1}\rangle=\mathrm{exp}(i\alpha)|\theta^{2}\rangle$,
$|\mathrm{exp}(i\alpha)|=1$), only if
$T(g^{2,-1}g^{1})|\theta_{0}\rangle=\mathrm{exp}(i\alpha)|\theta_{0}\rangle$,
where $g^{1}$ corresponds to $\theta^{1}$ and $g^{2}$ corresponds to
$\theta^{2}$. Suppose $E=\\{e\\}$ is a subgroup of the group $G$, such that
its elements have the property

$T(e)|\theta_{0}\rangle=\mathrm{exp}[i\alpha(e)]|\theta_{0}\rangle.$ (8)

When the subgroup $E$ is maximal, it will be called the isotropy subgroup for
the state $|\theta_{0}\rangle$. More precisely, it is the isotropy subgroup of
the group $T(G)$ corresponding to this state.

The construction shows that the vectors $|\theta\rangle$ corresponding to a
value $\theta$ and thus to an element $g\in G$, for all the group elements $g$
belonging to a left coset class of $G$ with respect to the subgroup $E$,
differ only in a phase factor and so determine the same state. Choosing a
representative $g(x)$ in any equivalence class $x$, one gets a set of states
$\\{|\theta_{g(x)}\rangle\\}$, where $x\in X=G/E$. Again, using the
correspondence between $g$ and $\theta$, I will write these states as
$\\{|\theta(x)\rangle\\}$, or in a more concise form $\\{|x\rangle\\}$,
$|x\rangle\in\mathcal{H}$.

Definition 3. The system of states
$\\{|\theta\rangle=T(g)|\theta_{0}\rangle\\}$, where $g$ corresponds to
$\theta$ as above, is called the coherent-state system
$\\{T,|\theta_{0}\rangle\\}$. Let $E$ be the isotropy subgroup for the state
$|\theta_{0}\rangle$. Then the coherent state $|\theta(g)\rangle$ is
determined by a point $x=x(g)$ in the coset space $G/E$ corresponding to $g$
and to $|\theta(g)\rangle$ is defined by
$|\theta(g)\rangle=\mathrm{exp}(i\alpha)|x\rangle$,
$|\theta_{0}\rangle=|0\rangle$.

Remark. The states corresponding to the vector $|x\rangle$ may also be
considered as a one-dimensional subspace in $\mathcal{H}$, or as a projector
$P_{x}=|x\rangle\langle x|$, $\mathrm{dim}P_{x}=1$, in $\mathcal{H}$. Thus the
system of coherent states, as defined above, determines a set of one-
dimensional subspaces in $\mathcal{H}$, parametrized by points of the
homogeneous space $X=G/E$.

The general properties og the coherent state system was studies more closely
in [5], and I refer to that discussion here. The important consequence for me
is the following: Assume that the representation $T(\cdot)$ above is
irreducible, and that there is a measure on $G$ which is invariant under left
and right shifts. Assume also that convergence conditions are satisfied. Then
there exists a measure $\mu$ in $X$ such that we have a resolution of unity

$\int d\mu(x)|x\rangle\langle x|=I.$ (9)

(See equation (2.3.5) in [5].) The argument behind (9) is essentially the same
that I used in [1] to prove the resolution of the identity (where I only
assumed left invariance of the measure on $G$): Define an operator $B=\int
dx|x\rangle\langle x|$ and show by using Schur’s lemma that $B$ must be a
constant times the identity operator $I$.

Recall now two crucial assumptions behind the construction above:

1) $\\{T(g)\\}$ can be chosen as an irreducible representation of the group
$G$ on some Hilbert space $\mathcal{H}$.

2) There is a one-to-one correspondence between the elements $g\in G$, the
variable values $\theta$ and the coherent states $T(g)|\theta_{0}\rangle$.

If $G$ is abelian, it only possible to satisfy 1) if $\mathcal{H}$ is one-
dimensional, giving a trivial theory.

But, nevertheless, assuming in general 1) and 2), one can now, again under
convergence conditions, define a unique operator in $\mathcal{H}$
corresponding to the conceptual variable $\theta$:

$A^{\theta}=\int\theta(x)|x\rangle\langle x|d\mu(x).$ (10)

Properties of such operators are as studied in [1].

###  V.2 Two maximal conceptual variables

For many concrete examples, it is not possible to find a non-abelian group
satisfying 1) and 2) above, and then the theory of the previous subsection is
just trivial. For this case, study _two_ conceptual varables $\theta$ and
$\xi$.

Assume that the variables $\theta$ and $\xi$ are maximally accessible, and
suppose that there exists a transformation $k$ such that
$\xi(\phi)=\theta(k\phi)$. Let $G$ be a transitive group acting on $\theta$,
and let $H$ be the transitive group acting on $\xi$ defined by
$h\xi(\phi)=g^{1}\theta(k\phi)$ when $\xi(\phi)=\theta(k\phi)$, where
$g^{1}\in G^{1}$, an independent copy of $G$. This gives a isomorphism between
$G$ and $H$.

Let $N$ be the group acting on $\psi=(\theta,\xi)$ generated by $G$ and $H$
and a single element $j$ defined by $j\psi=(\xi,\theta)$ and $j\theta=\xi$.
For $g\in G$, define $gj\psi(\phi)=(g\theta(k\phi),\theta(\phi))$ when
$\xi=\theta(k\phi)$, and for $h\in H$ define
$hj\psi=(\xi(\phi),h\xi(k^{-1}\phi))$ when $\theta(\phi)=\xi(k^{-1}\phi)$.
Since $G$ and $H$ are transitive on the components, and since through $j$ one
can choose for a group element of $N$ to act first arbitrarily on the first
component and then arbitrarily on the second component, $N$ is transitive on
$\psi$. Also, $N$ is non-Abelian: $gj\neq jg$.

I want to fix some Hilbert space $\mathcal{H}$, and define a representation
$U(\cdot)$ of $G$ on this Hilbert space with the property that if we fix some
vector $|v_{0}\rangle\in\mathcal{H}$, then the vectors $U(g)|v_{0}\rangle$ are
in one-to-one correspondence with the group elements $g\in G$ and hence with
the values $g\theta_{0}$ of $\theta$ for some fixed $\theta_{0}$.

For each element $g\in G$ there is an element $h=jgj\in H$ and vise versa.
Note that $j\cdot j=e$, the unit element. Let $U(j)=J$ be some unitary
operator on $\mathcal{H}$ such that $J\cdot J=I$. Then for the representation
$U(\cdot)$ of $G$, there is a representation $V(\cdot)$ of $H$ given by
$V(jgj)=JU(g)J$. These are acting on the same Hilbert space $\mathcal{H}$ with
vectors $|v\rangle$.

Note that $J$ must satisfy $JU(jgj)=U(g)J$. By Schur’s Lemma this demands $J$
to be an isomorphism or the zero operator if the representation $U(\cdot)$ is
irreducible. In the reducible case a non-trivial operator $J$ exists, however:

In this case there exists at least one proper invariant subrepresentation
$U_{0}$ acting on some vector space $\mathcal{H}_{0}$, a subspace of
$\mathcal{H}$, and another proper invariant subrepresentation $U^{\prime}_{0}$
acting on a vector space $\mathcal{H}^{\prime}_{0}$. Fix
$|v_{0}\rangle\in\mathcal{H}_{0}$ and
$|v^{\prime}_{0}\rangle\in\mathcal{H}^{\prime}_{0}$, and then define
$J|v_{0}\rangle=|v^{\prime}_{0}\rangle$,
$J|v^{\prime}_{0}\rangle=|v_{0}\rangle$ and $J|v\rangle=|v\rangle$ for any
$|v\rangle\in\mathcal{H}$ which is orthogonal to $|v_{0}\rangle$ and
$|v^{\prime}_{0}\rangle$.

Now we can define a representation $W(\cdot)$ of the full group $N$ acting on
$\psi=(\theta,\xi)$ in the natural way: $W(g)=U(g)$ for $g\in G$, $W(h)=V(h)$
for $h\in H$, $W(j)=J$, and then on products from this.

If $U$ is irreducible, then also $V$ is an irreducible representation of $H$,
and we can define operators $A^{\theta}$ corresponding to $\theta$ and
$A^{\xi}$ corresponding to $\xi$ as in (10). If not, we need to show that the
representation $W$ of $N$ constructed above is irreducible on $\mathcal{H}$.

Lemma 1. $W(\cdot)$ as defined above is irreducible.

Proof. Assume that $W(\cdot)$ is reducible, which implies that both $U(\cdot)$
and $V(\cdot)$ are reducible, i.e., can be defined on a lowerdimensional space
$\mathcal{H}_{0}$, and that $E=W(j)$ also can be defined on this lower-
dimensional space. Let $R(\cdot)$ be the representation of $G$ restricted to
vectors $|u\rangle$ in $\mathcal{H}$ orthogonal to $\mathcal{H}_{0}$. Fix some
vector $|u_{0}\rangle$ in this orthogonal space; then consider the vectors in
this space given by $R(g)|u_{0}\rangle$. Note that the vectors orthogonal to
$\mathcal{H}_{0}$ together with the vectors in $\mathcal{H}_{0}$ span
$\mathcal{H}$, and the vectors $U(g)|u_{0}\rangle$ in $\mathcal{H}$ are in
one-to-one correspondence with $\theta$. Then the vectors $R(g)|u_{0}\rangle$.
are in one-to-one correspondence with a subvariable $\theta^{1}$. And define
the representation $S(\cdot)$ of $H$ by $S(jgj)=R(g)$ and vectors
$S(h)|v_{0}\rangle$, where $|v_{0}\rangle$ is a fixed vector of $\mathcal{H}$,
orthogonal to $\mathcal{H}_{0}$. These are in one-to-one correspondence with a
subparameter $\xi^{1}$ of $\xi$.

Given a value $\theta$, there is a unique element $g_{\theta}\in G$ such that
$\theta=g_{\theta}\theta_{0}$. (It is assumed that the isotropy group of $G$
is trivial.)

From this look at the fixed vector $S(jg_{\theta}j)|v_{0}\rangle$. By what has
been said above, this corresponds to a unique value $\xi^{1}$, which is
determined by $g_{\theta}$, and hence by $\theta$. But this means that a
specification of $\theta$ determines the vector $(\theta,\xi^{1})$, contrary
to the assumption that $\theta$ is maximally accessible. Thus $W(\cdot)$
cannot be reducible.

∎

This shows that there is a group $N$ acting on $(\theta,\xi)$ and an
irreducible representative $W$ of this group acting on some Hilbert space
$\mathcal{H}$. Hence the conclusion (9) holds if $G$ above is replaced by $N$
and the irreducible representation $W(\cdot)$. That is, the crucial assumption
1) is now satisfied. It is left to prove that the Hilbert space $\mathcal{H}$
can be chosen so that condition 2) holds in this case.

First one must establish the one-to-one correspondence between the values of
$(\theta,\xi)$ and the elements $n$ of the group $N$. Now these elements are
constructed from group elements $g$ acting on $\theta$, and since $G$ is
assumed to be transitive with a trivial isotropy group, there is a one-to-one
correspondence between $g$ and $\theta$. The group $H$ constructed in the
proof is also transitive on $\Omega_{\xi}$ with a trivial isotropy group, so
there is a one-to-one correspondence between the values $\xi$ and the group
elements $h\in H$. But $n\in N$ as acting on $(\theta,\xi)$ is given as
$n=(g,h)$, so the required one-to-one correspondence is established.

It is only left to prove that under suitable assumptions the vectors
$W(n)|\nu_{0}\rangle$ are in one-to-one correspondence with the group elements
$n$, and hence with $\zeta=(\theta,\xi)$. Once this is proved, by the
construction of subsection A one can define coherent states
$|\psi\rangle=W(n)|\psi_{0}\rangle\in\mathcal{H}$ and a left invariant
normalized measure $v(\psi)$ such that

$\int|\psi\rangle\langle\psi|\nu(d\psi)=I.$ (11)

As above, with a change of notation, $\psi$ are elements of the homogeneous
space $\Psi=N/M$, where $M$ is the isotropy subgroup of the group $W(N)$
corresponding to the initial state $|\psi_{0}\rangle$. In other words it is
the maximal sugroup $M=\\{m\\}$ such that

$W(m)|\psi_{0}\rangle=\mathrm{exp}[\alpha(m)]|\psi_{0}\rangle.$ (12)

Now $N$ is a group acting on $(\theta,\xi)$ constructed from the groups $G$
acting on $\theta$ and $H$ acting on $\xi$ and an element $j$ such that
$j(\theta,\xi)=(\xi,\theta)$.

We want to characterize the elements $z$ of $N/M$. As in (8) the elements of
$M$ are such that (12) holds.

Lemma 2. We can write $z=(x,y)$, where $x$ is an element of the homogeneous
space $X=G/E$ and $y$ is an element of the homogeneous space $Y=H/F$. Here $E$
is the isotropy subgroup of the group $U(G)$, and $F$ is the isotropy subgroup
of the group $V(H)$, both corresponding to the initial state
$|\psi_{0}\rangle\in\mathcal{H}$.

Proof. For $e\in E$ we have

$W(e)|\psi_{0}\rangle=\mathrm{exp}[i\alpha(e)]|\psi_{0}\rangle,$ (13)

while for $f\in F$ we have

$W(f)|\psi_{0}\rangle=\mathrm{exp}[i\alpha(f)]|\psi_{0}\rangle.$ (14)

Normalizing $\alpha(m)$ for each $m$ such that $\alpha(j)=0$, (12) follows
from (13) and (14).

Now the cosets $nM$ are found by first specializing $n$ to $g$ and $h$,
respectively $j$. We have $j(\theta,\xi)=(\xi,\theta)$, and the cosets $gM$
and $hM$ are given in terms of $x$ and $y$. It follows that $z=(x,y)$ if we in
addition allow $jM$ to interchange $x$ and $y$.

This last requirement imposes a limitation in the special case when the basic
inaccessible variable is just the pair $\phi=(\theta,\xi)$. Then $\xi=k\theta$
and at the same time, $j$ is defined in terms of $k$ such that $j^{2}$ is the
identity. Hence we must have in this case that the group element $k^{2}$ also
is the identity.

∎

###  V.3 The operators

By the symmetry of the situation, and since $G$ and $H$ act independently on
$\theta$ and $\xi$ in the construction of the group $N$, we conclude that the
measure $\nu$ in (11) can be written as $\nu(d\psi)=\rho(dx)\rho(dy)$ for some
marginal measure $\rho$. Then we can define

$P(x)=\int_{Y}|\psi\rangle\langle\psi|\rho(dy),$ (15)

and

$A^{\theta}=\int_{X}\theta(x)P(x)\rho(dx).$ (16)

Similarly:

$Q(y)=\int_{X}|\psi\rangle\langle\psi|\rho(dx),$ (17)

and

$A^{\xi}=\int_{Y}\xi(y)Q(y)\rho(dy).$ (18)

It follows from (11) that $A^{\theta}=I$ when $\theta$ is identically equal to
1.

This define the operators for every pair of maximally accessible conceptual
variables $\theta$ and $\xi$. The operators for variables that are not
maximal, are found by using the spectral theorem, taking as a point of
departure the operator for a corresponding maximal variable (cp. equations
(4.28) and (4.30) in [1]).

##  VI Properties of the operators

To continue the theory, we prove the important result (cp. Theorem 4.2 in
[1]):

Theorem 2. For any transformation $t$ of $\Omega_{\phi}$ and any unitary
representation $V$ on which $V(t)$ is defined, the operator
$V(t)^{\dagger}A^{\theta}V(t)$ is the operator corresponding to
$\theta^{\prime}(\phi)=\theta(t\phi)$.

Proof. By (16) we have

$V(t^{-1})A^{\theta}V(t)=\int_{X}\theta(x)P_{t}(x)\rho(dx),$ (19)

where

$P_{t}(x)=\int_{Y}|\psi(t^{-1}\zeta)\rangle\langle\psi(t^{-1}\zeta)|\rho(dy).$
(20)

Here $|\psi(t^{-1}\zeta)\rangle$ are the ket vector is constructed such that
they are in one-to-one correspondence with $t^{-1}\phi=t^{-1}(\theta,\xi)$ and
hence with $t^{-1}(g,h)$. Note that $t^{-1}\phi$ and $\theta=\theta(\phi)$,
$\xi=\xi(\phi)$ defines $t^{-1}\zeta$. Write the group elements $t^{-1}(g,h)$
as $(g^{\prime},h^{\prime})$, constituting groups $G^{\prime}$ and
$H^{\prime}$ acting on $\theta$ and $\xi$, respectively. Let $E^{\prime}$ be
the subgroup of $G^{\prime}$ constructed as in (8), and let $F^{\prime}$ be
the corrsponding subgroup of $H^{\prime}$. Write
$X^{\prime}=G^{\prime}/E^{\prime}$ and $Y^{\prime}=H^{\prime}/F^{\prime}$. The
elements of these cosets may be defined as $x^{\prime}=t^{-1}x$ and
$y^{\prime}=t^{-1}y$, respectively. By the left invariance of the measure
$\rho$, (19) and (20) can be written as

$V(t^{-1})A^{\theta}V(t)=\int_{X}\theta(x)P_{t}(x)\rho(dt^{-1}x),$ (21)

where

$P_{t}(x)=\int_{Y}|\psi(t^{-1}\zeta)\rangle\langle\psi(t^{-1}\zeta)|\rho(dt^{-1}y).$
(22)

The last integral may be written

$P_{t}(x)=\int_{Y}|\psi(\theta(t^{-1}\phi),\xi(\phi^{\prime})\rangle\langle\psi(\theta(t^{-1}\phi),\xi(\phi^{\prime})|\rho(dy^{\prime}),$
(23)

and we can write $P_{t}(x)=P(t^{-1}x)$. This is inserted into (21), and using
left-invariance of the measure again, this give that the operator
$V(t^{-1})A^{\theta}V(t)$ is associated with the conceptual variable
$\theta(tx)$, which also may be written as $\theta(t\phi)$. ∎

By using this result in the same way as Theorem 4.2 is used in [1], a rich
theory follows. I will limit me here to the case where $\theta$ is a discrete
conceptual variable. Then one can show:

1) The eigenvalues of $A^{\theta}$ coincide with the values of $\theta$.

2) The variable $\theta$ is maximally accessible if and only if the
eigenvalues of $A^{\theta}$ are non-degenerate.

3) For the maximal case the following holds in a given context: a) For a fixed
$\theta$ each question ‘What is the value of $\theta$?’ together with a sharp
answer ‘$\theta=u$’ can be associated with a normalized eigenvector of the
corresponding $A^{\theta}$. b) If there in the context are a set
$\\{\theta^{a};a\in\mathcal{A}\\}$ of maximally accessible conceptual
variables (these must by the results of Section IV be related to each other)
one can consider all ket vectors that are normalized eigenvectors of some
operator $A^{\theta^{a}}$. Then each of these may be associated with a unique
question-and-answer as above.

##  VII The spin case

Let $\phi$ be the total spin vector of a particle, and let
$\theta^{a}=\|\phi\|\mathrm{cos}(\phi,a)$ be the component of $\phi$ in the
direction $a$. If we have a coordinate system, the components
$\theta^{x},\theta^{y}$ and $\theta^{z}$ are of special interest. As noted in
[1], these components are not permissible if $K$ is the rotation group for
some fixed $\|\phi\|$.

Consider a Stern-Gerlach experiment with a beam of particles in the $y$
direction. Write $\phi=(\phi^{1},\theta^{y})$ where $\phi^{1}$ is the spin
component in the $xz$-plane. Let $K_{0}$ be the group of rotations of
$\phi^{1}$ in this plane for fixed $\|\phi^{1}\|$.

Proposition 2. Any component $\theta^{a}$ in the $xz$-plane is a permissible
function of $\phi^{1}$ under $K_{0}$.

Proof. It is sufficient to consider $\theta^{x}$. Let $k_{0}\in K_{0}$ and
$\theta^{x}(\phi^{1}_{1})=\theta^{x}(\phi^{1}_{2})$. The group element $k_{0}$
rotates the $x$-axis to a new direction $x_{0}$ and $\phi^{1}_{1}$ and
$\phi^{1}_{2}$ to new vectors $k_{0}\phi^{1}_{1}$ and $k_{0}\phi^{1}_{2}$. By
the geometry, the angle $(x,\phi^{1}_{1})$ must be equal to angle
$(x_{0},k_{0}\phi^{1}_{1})$, and angle $(x,\phi^{1}_{2})$ must be equal to
angle $(x_{0},k_{0}\phi^{1}_{2})$. Because the cosines of the unrotated angles
are equal by assumption, the cosines of the rotated angles must also be equal.
$\Box$

Unit vectors in a plane as quantum state vectors have been discussed
extensively in [8]. These vectors can be seen as elements of a real
2-dimensional Hilbert space, and they are uniquely determined by the angle
they form with the $x$-axis. Using this as a point of departure, density
matrices and quantum operators (matrices) were defined in [8]. The authors
noted that all real 2-dimensional matrices were generated by the unit matrix
and the two real Pauli spin matrices $\sigma_{1}$ and $\sigma_{3}$.

Spin coherent states and their connection to the group SU(2) are treated for
instance in [5] and [9]. I will follow parts of [5] without going into
details. It is crucial that any irreducible representation $D$ is given by a
nonnegative integer or half-integer $r$: $D(k)=D^{r}(k)$,
$\mathrm{dim}D^{r}=2r+1$. In the representation space
$\mathcal{H}=\mathcal{H}_{r}$ the canonical basis $|r;m\rangle$ exists, where
$m$ runs from $-r$ to $r$ in unit steps. The infinitesimal operators
$A^{\pm}=A^{x}\pm A^{y}$, $A^{0}=A^{z}$ of the group representation $V^{r}$
satisfy the commutation relations

$[A^{0},A^{\pm}]=\pm A^{\pm},\ \ [A^{-},A^{+}]=-2A^{0}.$ (24)

The operators $A^{x}$, $A^{y}$ and $A^{z}$ are related to infinitesimal
rotations around the $x$-axis, $y$-axis and $z$-axis, respectively, and may be
shown to be identical to the operators associated with $\theta^{x}$,
$\theta^{y}$ and $\theta^{z}$, as considered in the previous sections here. We
take $\bm{A}=(A^{x},A^{y},A^{z})$. The representation space vectors
$|r;m\rangle$ are eigenvectors for the operators $A^{0}$ and
$\bm{A}^{2}=(A^{x})^{2}+(A^{y})^{2}+(A^{z})^{2}$:

$A^{0}|r;m\rangle=m|r;m\rangle,\ \ \bm{A}^{2}|r;m\rangle=r(r+1)|r;m\rangle.$
(25)

The operator $\mathrm{exp}[i\omega(\bm{n}\cdot\bm{A})]$, $\|\bm{n}\|=1$,
describes the rotation by the angle $\omega$ around the axis directed along
$\bm{n}$. In [5] this was used to describe the coherent states
$D(k)|\phi_{0}\rangle$ in various ways. The ket vector $|\phi_{0}\rangle$ may
be taken as $|r;m\rangle$ for a fixed $m$; the simplest choice is $m=-r$.

The spin case is discussed more thoroughly in [1]. Among other things, the
discretization of spin components is motivated from a general principle of
model reduction.

##  VIII The epistemic interpretation

Consider a physical system, and an observator or a set communicating
observators on this system. The physical variables which can be measured in
this setting are examples of accessible conceptual variables, and are called
e-variables in [1] and [2].

The approach here should be compared to parts of various recent derivations of
quantum theory from a set of postulates (see, for instance, [12] and [13]). As
is stated in [14], there is a problem connecting these derivations to the many
different interpretations of quantum theory. By contrast, the derivation
presented here is tied to a particular interpretation: A general epistemic
interpretation. This is also elaborated on in [1].

A maximally accessible variable $\theta^{a}$ admits values $u_{j}^{a}$ that
are single eigenvalues of the operator $A^{a}$, uniquely determined from
$\theta^{a}$. Let $|a;j\rangle$ be the eigenvector associated with this
eigenvalue. Then $|a;j\rangle$ can be connected to the question ‘What is the
value of $\theta^{a}$?’ together with the sharp answer
‘$\theta^{a}=u_{j}^{a}$’.

A general ket vector $|v\rangle\in\mathcal{H}$ is always an eigenvector of
_some_ operator associated with a conceptual variable. It is natural to
conjecture that this operator always can be selected in such a way that the
accessible variable is maximally accessible. Then $|v\rangle$ is in a natural
way associated with a question-and-answer pair. It is of interest that Höhn
and Wever [11] recently derived quantum theory for sets of qubits from such
question-and-answer pairs; compare also the present derivation. Note that the
interpretation implied by such derivations is relevant for both the
preparation phase and the measurement phase of a physical system.

One may also consider linear combinations of ket vectors from this point of
view. Let the maximally accessible variable $\theta^{b}$ admits values
$u_{i}^{b}$ that are single eigenvalues of the operator $A^{b}$, uniquely
determined from $\theta^{b}$. Let $|b;i\rangle$ be the eigenvector associated
with this eigenvalue. Then one can write

$|b;i\rangle=(\sum_{j}|a;j\rangle\langle a;j|)|b;i\rangle=\sum_{j}\langle
a;j|b;i\rangle|a;j\rangle.$ (26)

This state can be interpreted in terms of a question ‘What is the value of
$\theta^{b}$?’ with the sharp answer ‘$\theta^{b}=u_{i}^{b}$’. But if one
tries to ask questions about $\theta^{a}$ for a system where the observer or
the set of observers is in this state, the answer is simply ‘I (we) do not
know’.

From a general point of view it may be considered of some value to have an
epistemic interpretation which is not necessarily tied to a strict Bayesian
view (see for instance [15] on this). Under an epistemic interpretation, one
may also discuss various “quantum paradoxes” like Schrödinger’s cat, Wigner’s
friend and the two-slit experiment.

Example 1. Schrödinger’s cat. The discussion of this example concerns the
state of the cat just before the sealed box is opened. Is it half dead and
half alive?

To an observer outside the box the answer is simply: “I do not know”. Any
accessible e-variable connected to this observer does not contain any
information about the status of life of the cat. But on the other hand – an
imagined observer inside the box, wearing a gas mask, will of course know the
answer. The interpretation of quantum mechanics is epistemic, not ontological,
and it is connected to the observer. Both observers agree on the death status
of the cat once the box is opened.

Example 2. Wigner’s friend. Was the state of the system only determined when
Wigner learned the result of the experiment, or was it determined at some
previous point?

My answer to this is that at each point in time a quantum state is connected
to Wigner’s friend as an observer and another to Wigner, depending on the
knowledge that they have at that time. The superposition given by formal
quantum mechanics corresponds to a ‘do not know’ epistemic state. The states
of the two observers agree once Wigner learns the result of the experiment.

Example 3. The two-slit experiment. This is an experiment where all real and
imagined observers communicate at each point of time, so there is always an
objective state.

Look first at the situation when we do not know which slit the particle goes
through. This is a ‘do not know’ situation. Any statement to the effect that
the particles somehow pass through both slits is meaningless. The interference
pattern can be explained by the fact that the particles are (nearly) in an
eigenstate in the component of momentum in the direction perpendicular to the
slits in the plane of the slits. If an observer finds out which slit the
particles goes through, the state changes into an eigenstate for position in
that direction. In either case the state is an epistemic state for each of the
communicating observers, and thus also an ontological state.

##  IX Discussion

The derivation above was partly limited to accessible conceptual variables
taking a finite number of values. But the notion of conceptual variables is
much more general, and can thus be considered in more general settings.

This notion of conceptual variables also has links to other interpretations of
quantum theory. Take for instance the classical Bohm interpretation,
constructed from particle trajectories plus a pilot wave. These constructions
may be seen as conceptual variables, but at least the full trajectory must be
inaccessible. Or consider the many worlds interpretation, where the different
worlds must be considered as conceptual variables, but only one world is
accessible at each point in time.

The arguments developed here are based on focusing (choice of $a$) and
symmetry. These concepts are not confined to the microscopic world. This is
consistent with the fact that quantum theory has recently been applied to
cognitive models and to certain social and economic models (see [3] and [16]
and references there). In fact it has been claimed that the quantum structure
is ubiquitous, see [17].

Further philosophical consequences of these results are discussed in Chapter 6
of [1]. Chapter 1 of op. cit. has an informal approach to quantum mechanics
that is very similar to the general arguments introduced more formally here.

## References

[1] I. S. Helland, “Epistemic Processes. A Basis for Statistics and Quantum
Theory.” Revised version. Springer, Berlin (2021).

[2] I. S. Helland, “Symmetry in a space of conceptual variables.” Journal of
Mathematical Physics 60 (5) 052101 (2019). Erratum, J. Math. Phys. 61 (1)
019901 (2020).

[3] J. R. Busemeyer, and P. D. Buza, “Quantum Models for Cognition and
Decision.” Cambridge University Press, Cambridge (2012).

[4] I. S. Helland, “When is a set of questions to nature together with sharp
answers to those questions in one-to-one correspondence with a set of quantum
states?” arXiv: 1909.08834 [quant-ph] (2019).

[5] A. Perelomov, “Generalized Coherent States and Their Applications.”
Springer, Berlin (1986).

[6] L. Vaidman, “Derivations of the Born rule.” phil-sci-archive.pitt.edu
(2019).

[7] I. S. Helland, “Steps Towards a Unified Basis for Scientific Models and
Methods.” World Scientific, Singapore (2010).

[8] H. Bergeron, E. M. F. Curado, J.-P. Gazeau, and L. M. C. S. Rodrigues, “A
baby Majorana quantum formalism.” arXiv: 1701.04026v3 [quant-ph] (2017).

[9] J.-P. Gazeau, “Coherent States in Quantum Physics.” Wiley-VCH, Weinberg
(2009).

[10] Z.-Q. Ma, “Group Theory for Physicists.” World Scientific, Singapore
(2007).

[11] P. A. Höhn and C. S. P. Wever, “Quantum theory from questions” Physical
Review A 95, 012102 (2017)

[12] L. Hardy, “Quantum theory from reasonable axioms.” arXiv: 01010112v4
[quant-ph] (2001).

[13] G. Chiribella, G. M. D’Ariano, and P. Perinotti, “Quantum from
principles.” In: “Quantum Theory: Informational Foundation and Foils.” G.
Chiribella and P. W. Spekkens, [Eds.] pp. 171-221. Springer, Berlin (2016),

[14] G. Chiribella, A. Cabello, M. Kleinmann and M. P. Müller, “General
Bayesian theories and the emergence of the exclusitivity principle.” arXiv:
1901.11412v2 [quant-ph] (2019).

[15] C. A. Fuchs, “QBism, the perimeter of quantum Bayesianism.” arXiv:
1003.5209 [quant-ph] (2010).

[16] E. Haven and A. Khrennikov, “Quantum Social Sciences.” Cambridge Press,
Cambridge (2013).

[17] A. Khrennikov, “Ubiquitous quantum structure: from psychology to
finances.” Springer, Berlin (2010).

[18] V.I. Yukalov and D. Sornette, D. Mathematical structure of quantum
decision theory. Adv. Compl. Syst., 13, 659–698 (2010).


# Farrington-Manning in the Extreme Case

William N. Anderson111Carpinteria, California, USA. Email:
<EMAIL_ADDRESS>LinkedIn: william-anderson-46384b7

(May 6, 2022)

###### Abstract

The Farrington-Manning method is a common method for evaluating equivalence
and non-inferiority of independent proportions. It is implemented in various
software, in particular SAS® PROC FREQ, and the R® function
farrington.manning(), which is part of the DescrTab2 package. The equations
for the estimated proportions can create numerical issues in case both sample
proportions equal 1, and neither of these packages will yield an acceptable
solution in this special case. In this note we demonstrate a closed form
solution for the situation in this and other extreme cases.

We assume that the reader has available the original paper of Farrington and
Manning [1], and we repeat only a minimal amount from the paper. Some text is
quoted verbatim from the paper.

##  1 The setting

We consider a comparative binomial trial involving two groups of sizes
$N_{1},N_{2}$ in the predetermined ratio $\theta=N_{2}/N_{1}$, and independent
response variables $r_{1}\sim\text{Bi}(N_{1},p_{1})$ and
$r_{2}\sim\text{Bi}(N_{2},p_{2})$. The binomial probabilities $p_{1}$ and
$p_{2}$ are estimated by $\hat{p}_{1}=r_{1}/N_{1}$, $\hat{p}_{2}=r_{2}/N_{2}$.
The true difference is $s_{T}=p_{1}-p_{2}$, estimated by
$s=\hat{p}_{1}-\hat{p}_{2}$. In the equivalence or non-inferiority setting the
null hypothesis is $p_{1}-p_{2}=s_{0}$, where $s_{0}$ is a predetermined
value, often called the non-inferiority margin.

The Farrington-Manning method is to compute maximum likelihood estimates
$\tilde{p}_{1D}$ and $\tilde{p}_{2D}$ for $p_{1}$ and $p_{2}$, under the
constraint that $\tilde{p}_{1D}-\tilde{p}_{2D}=s_{0}$. These estimates will
typically be different from $\hat{p}_{1}$ and $\hat{p}_{2}$. The method is
described in the original paper [1], and in many other places, including
section 11.2.3 of Rothmann et al. [6]. This method is implemented in many
statistical software packages, including SAS® PROC FREQ, and the R® function
farrington.manning(), which is part of the DescrTab2 package. 222The
farrington.manning() function was written by Kevin Kunzmann, and the DescrTab2
package is maintained by Jan Meis. As of the date of this note, Version 2.1.9
of the package, dated January 20, 2022, is the most current.  333Thanks to Jan
Meis for many helpful comments relating to the package, and to the issues
described here.

There is no restriction on the sign of $s_{0}$, but, of course, $s_{0}$ must
have a value suitable for the difference of proportions. Reading between the
lines, it appears that Farrington and Manning had in mind positive $s_{0}$. In
practice, one can arrange this choice by choosing the groups appropriately.
The farrington.manning() function allows for negative $s_{0}$; SAS PROC FREQ
does not.

Then the test statistic is

$z_{D}=\hat{p}_{1}-\hat{p}_{2}-s_{0}.$ (1)

For large $N$, $z_{D}$ is approximately Normally distributed. Under the null
hypothesis, the variance of $z_{D}$ is estimated by:

$\displaystyle\hat{v}_{0}$
$\displaystyle=\tilde{p}_{1D}\tilde{q}_{1D}/N_{1}+\tilde{p}_{2D}\tilde{q}_{2D}/N_{2}$
$\displaystyle=\left[\tilde{p}_{1D}\tilde{q}_{1D}+\tilde{p}_{2D}\tilde{q}_{2D}\theta\right]/N_{1}$
(2)

where $\tilde{p}_{1D}$ and $\tilde{p}_{2D}$ are maximum likelihood estimators
for $p_{1}$ and $p_{2}$ under the null hypothesis and
$\tilde{q}_{1D}=1-\tilde{p}_{1D}$, $\tilde{q}_{2D}=1-\tilde{p}_{2D}$. The null
hypothesis may then be tested by referring the statistic:

$z=(\hat{p}_{1}-\hat{p}_{2}-s_{0})/\sqrt{\hat{v}_{0}}$ (3)

to the standard normal distribution.

In the general case, the estimates $\tilde{p}_{1D}$ and $\tilde{p}_{2D}$ are
obtained by solving a cubic equation, as will be seen below.

However, this procedure has potential numerical issues when both
$\hat{p}_{1}=1$ and $\hat{p}_{2}=1$. This situation may seem unlikely in an
actual trial, but it can occur when one is doing a simulation with assumed
probabilities $p_{1}$ and $p_{2}$ both close to 1. In this situation the SAS
PROC FREQ simply refuses to compute the risk difference, and the R function
farrington.manning() may cause an error condition.

##  2 Computing the maximum likelihood estimators

The estimate $\tilde{p}_{1D}$ is obtained by solving the maximum likelihood
equation

$ax^{3}+bx^{2}+cx+d=0$ (4)

with

$\displaystyle a$ $\displaystyle=1+\theta$ $\displaystyle b$
$\displaystyle=-[1+\theta+\hat{p}_{1}+\theta\hat{p}_{2}+s_{0}(\theta+2)]$
$\displaystyle c$
$\displaystyle=s_{0}^{2}+s_{0}(2\hat{p}_{1}+\theta+1)+\hat{p}_{1}+\theta\hat{p}_{2}$
$\displaystyle d$ $\displaystyle=-\hat{p}_{1}s_{0}(1+s_{0})$

Any method for solving a cubic equation could in principle be used to compute
$\tilde{p}_{1D}$, but one must be careful to choose the root that maximizes
the underlying likelihood function. Some considerations on this point are
given in section 5 below.

In the special case of interest here, both $\hat{p}_{1}=1$ and
$\hat{p}_{2}=1$. The coefficients of the cubic equation (4) take a simpler
form, and the equation can be factored.

$\displaystyle ax^{3}+bx^{2}+cx+d$ $\displaystyle=(1+\theta)x^{3}$
$\displaystyle\quad-[2+2\theta+s_{0}(\theta+2)]x^{2}$
$\displaystyle\quad+[s_{0}^{2}+s_{0}(3+\theta)+1+\theta]x$
$\displaystyle\quad-s_{0}(1+s_{0})$
$\displaystyle=(x-1)(x-(1+s_{0}))((1+\theta)x-s_{0}).$ (5)

The cubic has three real roots, and the choices for $\tilde{p}_{1D}$ and
$\tilde{p}_{2D}$ are given in table 9 below.

$\begin{bmatrix}\tilde{p}_{1D}\\\
\tilde{p}_{2D}\end{bmatrix}=\begin{bmatrix}1&1+s_{0}&s_{0}/(1+\theta)\\\
1-s_{0}&1&-s_{0}\theta/(1+\theta)\end{bmatrix}$ (6)

  * •

If $s_{0}>0$, only $\tilde{p}_{1D}=1$ and $\tilde{p}_{2D}=1-s_{0}$ give valid
proportions for $\tilde{p}_{1D}$ and $\tilde{p}_{2D}$.

  * •

If $s_{0}<0$, only $\tilde{p}_{1D}=1+s_{0}$ and $\tilde{p}_{2D}=1$ give valid
proportions for $\tilde{p}_{1D}$ and $\tilde{p}_{2D}$.

An alternative method for finding the estimates is to note that the likelihood
function is monotonic increasing in $\tilde{p}_{1D}$ and $\tilde{p}_{2D}$.
Accordingly the maximum likelihood will be obtained when these two are as
large as possible. That is when one of the $\tilde{p}_{iD}$ is 1, and the
other is $1-|s_{0}|$.

##  3 Hypothesis tests and confidence intervals

Using the general variance equation (1), the variance then becomes

$\hat{v}_{0}=\begin{cases}s_{0}(1-s_{0})/N_{2}&s_{0}>0\\\
-s_{0}(1+s_{0})/N_{1}&s_{0}<0\end{cases}$ (7)

Using this variance and the test statistic of (3) we have for $s_{0}>0$

$\displaystyle z$
$\displaystyle=(\hat{p}_{1}-\hat{p}_{2}-s_{0})/\sqrt{\hat{v}_{0}}$
$\displaystyle=-s_{0}/\sqrt{s_{0}(1-s_{0})/N_{2}}$
$\displaystyle=-\sqrt{N_{2}s_{0}/(1-s_{0})}.$

Similarly, for $s_{0}<0$

$\displaystyle z$
$\displaystyle=(\hat{p}_{1}-\hat{p}_{2}-s_{0})/\sqrt{\hat{v}_{0}}$
$\displaystyle=-s_{0}/\sqrt{-s_{0}(1+s_{0})/N_{1}}$
$\displaystyle=\sqrt{-N_{1}s_{0}/(1+s_{0})}.$

The two-sided confidence limits for the difference are obtained by inversion
of the two-sided hypothesis test For the general Farrington-Manning case the
computations require an iterative solution, because there is no simple formula
relating the $z$-statistic to $s_{0}$. For the special case here limits are
simply obtained by setting $z$ to the critical values, and solving for
$s_{0}$.

The confidence interval is

$\left(\dfrac{-z_{\alpha}^{2}}{N_{1}+z_{\alpha}^{2}},\dfrac{z_{\alpha}^{2}}{N_{2}+z_{\alpha}^{2}}\right)$

As a practical matter, if one really had data this extreme, exact methods
would likely be used for evaluating hypothesis tests and computing confidence
limits. The intention here is simply to give reasonable limits that can be
used to let a simulation proceed.

##  4 Other extreme cases

A number of other extreme cases can be considered in the same manner. Here we
give only minimal details, leaving the rest to the reader.

Suppose $\hat{p}_{1}=0$ and $\hat{p}_{2}=0$.

The maximum likelihood equation (4) factors

$\displaystyle ax^{3}+bx^{2}+cx+d$ $\displaystyle=(1+\theta)x^{3}$
$\displaystyle\quad-[1+\theta+s_{0}(\theta+2)]x^{2}$
$\displaystyle\quad+[s_{0}^{2}+s_{0}(\theta+1)]x$ $\displaystyle\quad-0$
$\displaystyle=x(x-s_{0})((1+\theta)x-(1+\theta+s_{0})).$ (8)

The possibilities for $\tilde{p}_{1D}$ and $\tilde{p}_{2D}$ are given in table
(9) below. The appropriate choice depends on the sign of $s_{0}$.

$\begin{bmatrix}\tilde{p}_{1D}\\\
\tilde{p}_{2D}\end{bmatrix}=\begin{bmatrix}0&(1+\theta+s_{0})/(1+\theta)&s_{0}\\\
-s_{0}&(1+\theta-s_{0}\theta)/(1+\theta)&0\end{bmatrix}$ (9)

  * •

If $s_{0}>0$, only $\tilde{p}_{1D}=s_{0}$ and $\tilde{p}_{2D}=0$ give valid
proportions for $\tilde{p}_{1D}$ and $\tilde{p}_{2D}$.

  * •

If $s_{0}<0$, only $\tilde{p}_{1D}=0$ and $\tilde{p}_{2D}=-s_{0}$ give valid
proportions for $\tilde{p}_{1D}$ and $\tilde{p}_{2D}$.

The remaining computations are highly similar to those above.

Suppose $\hat{p}_{1}=0$ and $\hat{p}_{2}=1$. The maximum likelihood equation
(4) factors

$\displaystyle ax^{3}+bx^{2}+cx+d$ $\displaystyle=(1+\theta)x^{3}$
$\displaystyle\quad-[1+2\theta+s_{0}(\theta+2)]x^{2}$
$\displaystyle\quad+[s_{0}^{2}+s_{0}(\theta+1)+\theta]x$
$\displaystyle\quad-0$
$\displaystyle=x(x-(1+s_{0}))((1+\theta)x-(\theta+s_{0})).$ (10)

The possibilities for $\tilde{p}_{1D}$ and $\tilde{p}_{2D}$ are given in table
(11) below. The appropriate choice depends on the sign of $\theta+s_{0}$.

$\begin{bmatrix}\tilde{p}_{1D}\\\
\tilde{p}_{2D}\end{bmatrix}=\begin{bmatrix}0&(\theta+s_{0})/(1+\theta)&1+s_{0}\\\
-s_{0}&(\theta-s_{0}\theta)/(1+\theta)&1\end{bmatrix}$ (11)

  * •

If $\theta+s_{0}>0$, the choice $\tilde{p}_{1D}=(\theta+s_{0})/(1+\theta)$ and
$\tilde{p}_{2D}=(\theta-s_{0}\theta)/(1+\theta)$ gives the maximum likelihood
estimator. (If additionally $s_{0}<0$, all three choices give valid
proportions, but the other two choices give minima).

  * •

If $\theta+s_{0}=0$, the choice $\tilde{p}_{1D}=0$ and $\tilde{p}_{2D}=-s_{0}$
is a double root, and gives the maximum.

  * •

If $\theta+s_{0}<0$, the choice $\tilde{p}_{1D}=0$ and $\tilde{p}_{2D}=-s_{0}$
gives the maximum. (The choice $\tilde{p}_{1D}=1+s_{0}$ and $\tilde{p}_{2D}=1$
also gives valid proportions, but this is a minimum.)

The remaining computations proceed as above. The upper confidence limit for
the difference will be 1; we have not found a simple form for the lower
confidence limit.

Suppose $\hat{p}_{1}=1$ and $\hat{p}_{2}=0$. The maximum likelihood equation
(4) factors

$\displaystyle ax^{3}+bx^{2}+cx+d$ $\displaystyle=(1+\theta)x^{3}$
$\displaystyle\quad-[2+\theta+s_{0}(\theta+2)]x^{2}$
$\displaystyle\quad+[s_{0}^{2}+s_{0}(\theta+3)+1]x$ $\displaystyle\quad-
s_{0}(1+s_{0})$ $\displaystyle=(x-1)(x-s_{0})((1+\theta)x-(1+s_{0})).$

The possibilities for $\tilde{p}_{1D}$ and $\tilde{p}_{2D}$ are given in table
(12) below. The appropriate choice depends on the sign of $1-s_{0}\theta$.

$\begin{bmatrix}\tilde{p}_{1D}\\\
\tilde{p}_{2D}\end{bmatrix}=\begin{bmatrix}s_{0}&(1+s_{0})/(1+\theta)&1\\\
0&(1-s_{0}\theta)/(1+\theta)&1-s_{0}\end{bmatrix}$ (12)

  * •

If $1-s_{0}\theta>0$, the choice $\tilde{p}_{1D}=(1+s_{0})/(1+\theta)$ and
$\tilde{p}_{2D}=(1-s_{0}\theta)/(1+\theta)$ gives the maximum likelihood
estimator. (If additionally $s_{0}>0$, all three choices give valid
proportions, but the other two choices give minima).

  * •

If $1-s_{0}\theta=0$, the choice $\tilde{p}_{1D}=s_{0}$ and $\tilde{p}_{2D}=0$
is a double root, and gives the maximum.

  * •

If $1-s_{0}\theta<0$, the choice $\tilde{p}_{1D}=0$ and
$\tilde{p}_{2D}=-s_{0}$ gives the maximum. (The choice $\tilde{p}_{1D}=1$ and
$\tilde{p}_{2D}=1-s_{0}$ also gives valid proportions, but this is a minimum.)

The remaining computations proceed as above. The lower confidence limit for
the difference will be -1; we have not found a simple form for the upper
confidence limit.

Suppose $s_{0}=0$. This is in effect a standard superiority test, but there
seems to be no mathematical reason not to compute using the methods above. In
this case the special form of the equation becomes.

$\displaystyle ax^{3}+bx^{2}+cx+d$ $\displaystyle=(1+\theta)x^{3}$
$\displaystyle\quad-[1+\theta+\hat{p}_{1}+\theta\hat{p}_{2}]x^{2}$
$\displaystyle\quad+[\hat{p}_{1}+\theta\hat{p}_{2}]x$ $\displaystyle\quad-0$
$\displaystyle=x(x-1)((1+\theta)x-(\hat{p}_{1}+\theta\hat{p}_{2})).$

The desired solution is
$x=(\hat{p}_{1}+\theta\hat{p}_{2})/(1+\theta)=(r_{1}+r_{2})/(N_{1}+N_{2})$.
The resulting statistical test is algebraically identical to the chi-squared
test without the continuity correction.

##  5 Solution Considerations

The confidence intervals above are computed by inverting the two-sided
hypothesis test. We note that the intervals using the farrington.manning()
function are obtained in this manner, while those in SAS PROC FREQ are not.
Various other methods of computing confidence intervals are discussed in the
textbooks of Newcombe [4] and Rothmann et al [6]. In reading any of the
references it is vital to check on the definition of $s_{0}$; in some cases
the sign is reversed from the original paper.

The Farrington-Manning paper [1] states without proof that the cubic equation
(4) has a unique solution in $(s_{0},1)$.

  * •

For $s_{0}>0$, the interval $(s_{0},1)$ makes sense, since
$\tilde{p}_{2D}=\tilde{p}_{1D}-s_{0}$ must be a proportion.

  * •

But if $s_{0}<0$, the appropriate interval is $(0,1+s_{0})$, because
$\tilde{p}_{2D}>\tilde{p}_{1D}$. This situation is why it seems that the
authors had in mind $s_{0}>0$.

The paper [1] uses the cubic formula to solve (4). The particular form of the
cubic formula is the trigonometric form, which is appropriate for the case
when the cubic has three real roots. Only one of the three roots is given by
the formula used in the paper. It appears that the this form of the cubic
formula is appropriate, and that the chosen root is the correct one, even in
the extreme cases. However, the paper [1] does not give any proofs in this
regard. In this note we simply accept the situation, and have made no attempt
to present proofs. This use of the cubic formula is implemented in both SAS
PROC FREQ, and the R function farrington.manning(). It is straightforward to
program the formula in Excel.

Numerical problems with the formulas occur in two different situations:

  * •

The cubic formula involves computing an arc cosine. If there is a double root,
the numeric argument for the arc cosine function may be exactly 1. But
roundoff can cause the computed argument to be slightly greater than 1. One
can fix the problem using the formulas for $\tilde{p}_{1D}$ and
$\tilde{p}_{1D}$ above. Alternatively one could check the argument of the arc
cosine function, and set arguments microscopically greater than 1 to exactly
1.

An example would be when $\hat{p}_{1}=0$, $\hat{p}_{2}=1$, $\theta=2$, and
$s_{0}=0.5$. In computation for the farrington.manning() function roundoff
causes the argument of the acos() function to be microscopically greater than
1, which produces an error condition. One could fix the situation by checking
the argument before the acos() function is used. Excel does not generate an
error for this example, nor does SAS.

  * •

Inverting the computation to find the confidence limits involves solving a
non-linear equation over a defined range. Too small a range can result in
missing the root. This situation can arise in the case $\hat{p}_{1}=0$ and
$\hat{p}_{2}=1$, or vice versa. In these cases one confidence limit is
obvious, and it may be reasonable to write special code to compute the other
limit. As mentioned above, if real data produced this situation one would
likely use exact methods, and the formulas here would be irrelevant.

  * •

Jan Meis [3] has modified the farrington.manning() function to incorporate
both suggestions in this paragraph. Those suggestions might appear in a later
version of the DescrTab2 package.

In the examples above the closed form values for $\tilde{p}_{1D}$ and
$\tilde{p}_{2D}$ were obtained by factoring the cubic equation (4). One could
speculate as to whether such factoring can always be done. For an example to
the contrary, suppose that $\hat{p}_{1}=1/3$, $\hat{p}_{2}=1/2$, $s_{0}=1/2$,
and $\theta=2/3$. Then the equation becomes

$\displaystyle(5/3)x^{3}-(11/3)x^{2}+(25/12)x-(3/12)$ $\displaystyle=0$
$\displaystyle 20x^{3}-44x^{2}+25x-3$ $\displaystyle=0$ (13)

Any rational root of the form $m/n$ must have $m\mid 3$ and $n\mid 20$. There
are only a finite number of cases to check, and none yield a rational root.
Since (5) is a cubic, it must therefore be irreducible over the rationals, and
the full complications of cubic fields enter into the solution.

## References

  * [1] CP Farrington and G. Manning.  Test statistics and sample size formulae for comparative binomial trials with null hypothesis of non-zero risk difference or non-unity relative risk.  Statistics in Medicine, 9:1447–54, 1990.
  * [2] Jan Meis, Lukas Baumann, Maximilian Pilz, and Lukas Sauer.  DescrTab2: Publication Quality Descriptive Statistics Tables, 2022\.  R package version 2.1.9.
  * [3] Jan Mies.  Personal Communication, 2022.
  * [4] Robert Newcombe.  Confidence Intervals for Proportions and Related Measures of Effect Size.  CRC Press, Boca Raton, 2013.
  * [5] R Core Team.  R: A Language and Environment for Statistical Computing.  R Foundation for Statistical Computing, Vienna, Austria, 2021.
  * [6] Mark Rothmann, Brian Wiens, and Ivan Chan.  Design and Analysis of Non-Inferiority Trials.  CRC Press, Boca Raton, 2012.
  * [7] SAS Institute Inc., Cary, NC.  SAS/STAT® 15.1 User’s Guide, 2018.