acoustics actuators admsci adolescents aerospace agriculture agriengineering
agronomy ai algorithms allergies analytica animals antibiotics antibodies
antioxidants appliedchem applmech applmicrobiol applnano applsci arts asi
atmosphere atoms audiolres automation axioms batteries bdcc behavsci beverages
biochem bioengineering biologics biology biomechanics biomedicines
biomedinformatics biomimetics biomolecules biophysica biosensors biotech birds
bloods brainsci buildings businesses cancers carbon cardiogenetics catalysts
cells ceramics challenges chemengineering chemistry chemosensors chemproc
children civileng cleantechnol climate clinpract clockssleep cmd coatings
colloids compounds computation computers condensedmatter conservation
constrmater cosmetics crops cryptography crystals curroncol cyber dairy data
dentistry dermato dermatopathology designs diabetology diagnostics digital
disabilities diseases diversity dna drones dynamics earth ebj ecologies
econometrics economies education ejihpe electricity electrochem electronicmat
electronics encyclopedia endocrines energies eng engproc entropy environments
environsciproc epidemiologia epigenomes fermentation fibers fire fishes fluids
foods forecasting forensicsci forests fractalfract fuels futureinternet
futuretransp futurephys galaxies games gases gastroent gastrointestdisord gels
genealogy genes geographies geohazards geomatics geosciences geotechnics
geriatrics healthcare hearts hemato heritage highthroughput histories
horticulturae humanities hydrogen hydrology hygiene idr ijerph ijfs ijgi ijms
ijns ijtpp immuno informatics information infrastructures inorganics insects
instruments inventions iot j jcdd jcm jcp jcs jdb jfb jfmk jimaging
jintelligence jlpea jmmp jmp jmse jne jnt jof joitmc jor journalmedia jox jpm
jrfm jsan jtaer jzbg land languages laws life liquids literature livers
logistics lubricants machines macromol magnetism magnetochemistry make
marinedrugs materials materproc mathematics mca medicina medicines medsci
membranes metabolites metals metrology microarrays microbiolres micromachines
microorganisms minerals mining modelling molbank molecules mps mti
nanoenergyadv nanomanufacturing nanomaterials ncrna network neuroglia
neurolint neurosci nitrogen notspecified nursrep nutrients obesities oceans
ohbm onco optics oral organics osteology oxygen parasites parasitologia
particles pathogens pathophysiology pediatrrep pharmaceuticals pharmaceutics
pharmacy philosophies photochem photonics physics physiolsci plants plasma
pollutants polymers polysaccharides proceedings processes prosthesis proteomes
psych psychiatryint publications quantumrep quaternary qubs radiation
reactions recycling religions remotesensing reports reprodmed resources risks
robotics safety sci scipharm sensors separations sexes signals sinusitis
smartcities sna societies socsci soilsystems solids sports stats stresses
surfaces surgeries suschem sustainability symmetry systems taxonomy
technologies telecom textiles thermo tourismhosp toxics toxins transplantology
traumas tropicalmed universe urbansci uro vaccines vehicles vetsci vibration
viruses vision water wevj women world


# Interpretive Blindness

Nicholas Asher111CNRS, IRIT and Julie Hunter222Linagora GSO

###### Abstract

We model here an epistemic bias we call interpretive blindness (IB). IB is a
special problem for learning from testimony, in which one acquires information
only from text or conversation. We show that IB follows from a co-dependence
between background beliefs and interpretation in a Bayesian setting and the
nature of contemporary testimony. We argue that a particular characteristic
contemporary testimony, argumentative completeness, can preclude learning in
hierarchical Bayesian settings, even in the presence of constraints that are
designed to promote good epistemic practices.

##  1 INTRODUCTION

In this paper, we describe and analyze an as far as we know theoretically un-
examined kind of bias, which we call interpretive blindness (IB). IB is
exemplified by humans (and perhaps soon by sophisticated machine learning
algorithms) whose beliefs are guided and shaped by testimony. When learning
through testimony—perhaps the primary way that most people acquire information
nowadays—an agent acquires beliefs through conversations with other agents, or
from books, newspapers or social networks, and so on. Typically, such people
lack direct access to the phenomena described via that testimony. Typically
too, humans only pay attention to a restricted set of bodies of testimony from
a limited number of sources for their information—which makes sense in terms
of an agent’s limited resources and attention span. Our paper is about the
strategic consequences of opinion diffusion through testimony and the
distortions on learning and information that can result.

IB results from this restriction to few sources of testimony and a natural co-
dependence between beliefs and interpretation (Asher and Paul, 2018). Relying
on testimony $T$ from a restricted set of sources to update one’s beliefs
leads to the mutual reinforcement of our confidence in the source and our
belief in $T$; this creates a bias that can preclude learning when an agent
tries to exploit new data that are incompatible with or simply distinct from
$T$. Agents who are interpretively blind will discount any evidence that
challenges their beliefs. We use Wolpert’s 2018 extended Bayesian framework to
prove our results.

While IB is problematic for a standard Bayesian framework, it also poses
problems for hierarchical Bayesian approaches (Gelman et al., 2013), because
testimony from sources on social media like Facebook, 24/7 media outlets and
web interest groups is often argumentatively complete, a notion we analyze
precisely in Section 4; in an argumentatively complete body of testimony $T$,
the authors of that testimony can respond to and argue with any doubts raised
by other data or arguments in a body $T^{\prime}$ that might threaten $T$’s
credibility. A skillful climate denier, for example, will always find a way to
undercut the most scientifically careful argument. Argumentatively complete
testimony thus can undermine higher order constraints and good epistemic
practices that should guide first order learning.

Our paper starts in Section LABEL:sec:testimony by discussing testimony. We
then introduce the codependence of belief and interpretation and apply it to
the situation of testimony and the sources that support it. In Section 3 we
formally show how IB can result in ordinary Bayesian learning. Section 4 shows
how IB is reinforced in a hierarchical Bayesian learning setting. Section 6
develops a game theoretic setting to investigate the complexity of IB. We
provide results as to whether it is possible to free agents from interpretive
bias in several epistemic settings.

##  2 Testimony and sources

IB arises in learning because of a co-dependence between beliefs and the
interpretation of evidence, in particular its reliability. In this case, we
are talking of the interpretation of written or linguistically conveyed
information. Others have already noted a co-dependence of beliefs and
linguistic interpretation(Asher and Paul, 2018). Consider this exchange. R:
“Why hasn’t the senator commented on the story that he received undisclosed
gifts from supporters?” A: “The Senator has declared every gift that he has
received.” As Asher and Paul (2018) argue, one could interpret A’s response
either as an answer to R’s question or as an evasion and it depends upon one’s
beliefs about the honesty of the senator. Those beliefs get confirmed as the
exchange and its interpretation continue: the interpretation that relies on
the honesty of the Senator gets confirmed as A refuses to engage with R, and
this in turn confirms the honesty of the belief; mutatis mutandis for the
other interpretation.

An analogous co-dependence occurs with interpretation, belief and learning: in
updating our beliefs with new evidence; our beliefs color how we interpret
that evidence, in particular how trustworthy we find it. The updated evidence
in turn conditionally updates our beliefs.

Let us look this codependence in learning by testimony. A body of testimony
$T$ is a collection of information conveyed by one or more sources like The
New York Times, Fox News, CNN, Facebook, 4Chan, a particular individual or set
of individuals. The sources may “promote” or vouch for $T$ or cast doubt on
$T$. Such bodies are also dynamic; they evolve over time as they are updated
with new facts and events. In other words, $T$ comes in “stages”, where stages
might be defined by times or even conversational turns, and each stage $T_{i}$
is the body of evidence accumulated up to stage $i$.
$T=\\{T_{1},T_{2},...,T_{n},...\\}$ is the collection of all the stages of a
dynamic body of evidence. Dynamic bodies of testimony are ubiquitous in our
communicative landscape; on-line, 24/7 news sources as well as particular
groups on social media provide evolving, updated coverage of new events. Let
${\mathcal{T}}$ be a collection of bodies of testimony about some phenomenon
$P$. We will assume that $\hat{f}$ does not have independent access to $P$ and
uses evidence from bodies of testimony in ${\mathcal{T}}$ together with
background beliefs to update probabilities about hypotheses about $P$, some of
which are hypotheses about marginal probabilities of events described in
${\mathcal{T}}$.

Learning from testimony $T$ with source $s$ requires a learner $\hat{f}$ to
judge $T$ as credible, a judgment that will depend on $s$’s evaluation of $T$
(whether $s$ promotes or challenges $T$), as well as $\hat{f}$’s antecedent
hypotheses about $s$. Let ${\mathcal{H}}$ be a set of evaluation hypotheses,
where each $h\in{\mathcal{H}}$ gives the evaluation of a set ${\mathcal{T}}$
of bodies of testimony $T$ relative to a source $s$. $h\in{\mathcal{H}}$
defines a conditional probability $P(T|h)$ for $T\in{\mathcal{T}}$, which we
will sometimes write as $h(T)$, where $h(T)=0$ means $T$ is untrustworthy
according to $h$, and $h(T)=1$ means $T$ is trustworthy ($s$ fully endorses
$T$). Following Wolpert’s 2018 extended Bayesian framework, our learner
$\hat{f}$ updates his belief in $T$ relative to ${\mathcal{H}}$.

Our learner $\hat{f}$ will have a probability distribution over his evaluation
hypotheses ${\mathcal{H}}$. Given the co-dependence of beliefs and evidence,
this distribution is updated relative to the stages of $T$ as it develops.
This is intuitive; the testimony $T$ should serve as evidence upon which
$\hat{f}$ updates his beliefs. But the co-dependence tells us that $\hat{f}$
updates his confidence in $T$ via these updated beliefs.

Most if not all of us acquire new information from a restricted set of bodies
of evidence that push a particular point of view. This is reasonable given the
balance rational agents need to find between exploiting already acquired data
and gathering more data. In addition, attending to a particular body of
evidence can give a sense of community, as has been amply documented in the
scholarly literature and the press. But this trade off can lead to a problem
in learning: when we rely on testimony to learn and we restrict the testimony
we pay attention to, the confirming evidence for the evaluation hypothesis and
what it supports threaten to collapse into one. We now turn to see how
iterated Bayesian updating in learning from testimony can ultimately lead to a
situation where only evaluation hypotheses supporting our restricted evidence
are credible and this leads to IB.

##  3 IB in a first order Bayesian setting

To formalize IB and its consequences, we first present a simple experiment to
show how the codependence of interpretation and belief leads to bias
hardening. To illustrate, suppose that $\hat{f}$ considers a consistent
dynamic body of testimony $T=\\{T_{1},T_{2},...,T_{n},...\\}$ and has two
evaluation hypotheses $h_{1},h_{2}$, where the prior probabilities assigned to
$h_{1}$ and $h_{2}$ by $\hat{f}$ are:

$P(h_{1})=.6,\ \ P(h_{2})=.4$ (1)

and the evaluation hypotheses assign probabilities to $T$ as it evolves
through stages $T_{i}$ as follows:

$P(T_{i}|h_{1})=.8,\ \ P(T_{i}|h_{2})=.2\mbox{ for all i }$ (2)

We can now calculate the probability of $T_{1}$ using the general rule for
marginal probabilities in 3. Let ${\mathcal{B}}$ be $\hat{f}$’s background
beliefs; and let the set of all $h_{i}$, the alternative hypotheses that are
consistent with or assigned non-zero conditional probability relative to
${\mathcal{B}}$ (Lampinen and Vehtari, 2001; Tenenbaum et al., 2006; L
Griffiths et al., 2008; Tenenbaum et al., 2011), be the set of evaluation
hypotheses $h_{i}$ (so $\\{h_{1},h_{2}\\}$, in our example).

$P(x)=\sum_{i=1}^{i=k}P(x|h_{i},{\mathcal{B}}).P(h_{i},{\mathcal{B}})$ (3)

Then using (1), (2), and (3), we have:

$P(T_{1})=P(T_{1}|h_{1}).P(h_{1})+P(T_{1}|h_{2}).P(h_{2})=.56.$ (4)

This is our estimation of our belief in the body of evidence $T$ based on what
we have so far. We will continue to update the probability of $T$ given new
stages $T_{i}$ below by distinguishing prior probabilities $P_{prior}$ and
updated probabilities $P_{post}$. Now suppose there is a new conversational
turn in $T$, a new stage of evidence $T_{2}$. Given our assumptions,
$P(T_{2}|h_{1})=.8$, while $P(T_{2}|h_{2})=.2$, $T_{2}$ is supported by
$h_{1}$ but not by $h_{2}$—$h_{1}$ and $h_{2}$ are consistent with their roles
on $T_{1}$. Given the dependence of beliefs and interpretation of evidence,
$T_{2}$ also leads us to re-evaluate our evaluation hypotheses by adapting
Bayes’ formula to our evidential hypotheses:

$P(h_{i}|T_{n+1})=\frac{P(T_{n+1}|h_{i})P_{prior}(h_{i})}{P_{post}(T_{n})}$
(5)

Given $T_{2}$, whose initial probability we set to what the posterior
calculated for $T_{1}$—i.e., $P_{post}(T_{1})=P_{prior}(T_{2})$, we can update
our confidence in $h_{1}$ as follows:

$P(h_{1}|T_{2})=\frac{P(T_{2}|h_{1}).P_{prior}(h_{1})}{P_{post}(T_{1})}\approx.86.$
(6)

Thus, we have posterior probabilities for our evaluation hypotheses as well as
for stages of bodies of evidence. The similarly updated probability for
$h_{2}$ now drops to roughly $.14$. Using the updated values for $h_{1}$ and
$h_{2}$, we see that $T_{2}$, which includes $T_{1}$, is now even more
believable: $P_{post}(T_{2})=.74$. Now suppose that a new bit of evidence,
$T_{3}$, is added to $T$. As before, we set
$P_{post}(T_{2})=P_{prior}(T_{3})$. Given our assumptions about our source
functions, $P(T_{3}|h_{1})=.8$, we have $P(h_{1}|T_{3})=.96$, while
$P(h_{2}|T_{3})\approx 0.04$, and confidence in $T_{3}$ is also updated:
$P_{new}(T_{3})=.776\approx.78$. Updating $h_{1}$’s probability conditional on
new evidence $T_{4}$ now yields a value of $.989\approx.99$, while
$P(h_{2}|T_{4})=0.008\approx 0.01$. By the time we get to $T_{5}$, the
probability of $h_{1}$ will have gone to $1$, while $P(h_{2})=0$, and
$P(T_{5})=.8$. In sum, as $n$ increases, the updated probabilities of $h_{1}$
go to $1$ and $P(T_{n})\rightarrow P(T|h_{1})$, that is, to the strength of
$h_{1}$’s support for $T$.

Our co-dependence of belief and evidence suggests a loopy structure (cyclic
graph) for updating. However, by exploiting stages, we can disentangle such
structures; and efficient approximations are possible in disentangled
structures (Murphy et al., 2013). Proposition 1 below shows a convergence
under certain assumptions. Let ${\sf P}_{n}(h_{i})$ be the probability of
$h_{i}$ after conditionalizing on $T_{n}$ and ${\sf P}_{n}(T)$ the value of
$T$ after n conditional updates as defined above. Generalizing our discussion
above, we have:

###### Proposition 1

Suppose testimony $T=\\{T_{1},T_{2},...,T_{n},...\\}$, and suppose $\hat{f}$’s
evaluation hypotheses ${\mathcal{H}}_{\hat{f}}$, with a probability
distribution and with $h_{1}\in{\mathcal{H}}$ such that $P(h_{1})\neq 0$, and
$P(T_{n}|h_{1})>.5$ and is monotone increasing for all $n$, while
$P(T_{n}|h_{j})<.5$ and is monotone decreasing for all $n$ and for all
$h_{j}\in{\mathcal{H}},h_{j}\neq h_{1}$. Then:

$\mbox{As }n\rightarrow\infty,\ {\sf P}_{n}(T)\rightarrow
limsup(P(T_{n}|h_{1})),$ ${\sf P}_{n}(h_{1})\rightarrow 1\mbox{ and }{\sf
P}_{n}(h_{j})\rightarrow 0\mbox{ for }j\neq 1$

Given the calculations above and using standard updating rules for the
probabilities $P$ assigned by $\hat{f}$, if $P(T_{i}|h_{1})$ is monotonic
increasing with respect to $i$ and $P(T_{i}|h_{j})$ for any $j\neq 1$ is
monotonic decreasing, then the updates of $P(T_{i})$, $P(h_{1}|T_{i})$ and
$P(h_{j}|T_{i})$ will follow the pattern of our experiment above and converge
to the support of $h_{1},1,$ and $0$ respectively. $\Box$

We now introduce three important properties of evaluation hypotheses.

###### Definition 1

An evaluation hypothesis $h$ for a set of bodies of testimony ${\mathcal{T}}$
is consistent iff for $T,T^{\prime}\in{\mathcal{T}}$, if $T\cup T^{\prime}$ is
inconsistent, then $P(T|h)=1-P(T^{\prime}|h)$. An evaluation hypothesis $h$ is
probability-wise model complete (PWMC for $T$ iff: for any putative piece of
evidence $\phi$ if for no stage $T_{i}$ $T_{i}\models\phi$ ($\phi$ is not
predicted or included in any stage of $T$), then $P(\phi|h)=1-P(T|h)$.

###### Definition 2

An evaluation hypothesis $h\in{\mathcal{H}}$ with a probability distribution
$P$ makes $T$ potentially trustworthy ($h\models T$), if for all $n$,
$P(T_{n}|h)>.5$ and as $n\rightarrow\infty$, $P(T_{n}|h)\rightarrow 1$.

We take consistency to be a basic requirement of evaluation hypotheses. PWMC
hypotheses generalize consistent hypotheses. But what is their rationale? As
$T$ evolves through its stages, it is natural to assume that $T_{i+1}$ to
provide a more complete coverage of the facts than $T_{i}$. And as $T$ becomes
more complete, an evidential hypothsis may assume elements $\phi$ that are not
mentioned in any $T_{i}$ are in some way incompatible with $T$. The PWMC
condition codifies this in terms an operation akin to the operation of
negation as failure in Prolog; if $h$ makes $T$ probability wise model
complete, then if $T$ doesn’t mention $\phi$ then $h$ supports $\neg\phi$ to
the extent that $h$ supports $T$.

In addition, an agent plausibly has, among the many evaluation hypotheses that
she countenances, an evaluation hypothesis $h$ for which the conditional
probability of $T$ given $h$ increases as $T$ evolves. Up to now we have taken
an evaluation hypothesis to confer a fixed probability on a body of evidence
$T$ it supports or discounts. But the support for $T$ might increase (or
decrease) as $T$ gets more extended with more and more stages. A potentially
trustworthy evaluation hypothesis is a kind of “soundness” or accuracy
assumption about a body of evidence. For an agent who remains wedded to a body
of testimony such a soundness assumption seems a rational requirement. If
potential trustworthiness is a soundness requirement then PWMCness is a kind
of completeness requirement. Together they furnish the rational justification
for an agent to restrict his learning to a particular body or particular
bodies of evidence, something that we’ve argued most people in fact do and do
so with rational justification.

###### Proposition 2

Let ${\mathcal{T}}$ be a set of consistent bodies of testimony and let
${\mathcal{H}}$ be a set of evaluation hypotheses for ${\mathcal{T}}$, with
$h_{1}\in{\mathcal{H}}$ and $h_{1}\models T$, for some $T\in{\mathcal{T}}$.
Let the priors on $h_{i}\in{\mathcal{H}},h_{i}\neq h_{1}$ be as in Proposition
1 and let the probabilities of $h_{i},h_{1}\in{\mathcal{H}}$ be updated on
$T$. Then:

$\mbox{As }n\rightarrow\infty,\ {\sf P}_{n}(T)\rightarrow 1.$ (1)

Suppose in addition, $h_{1}$ is PWMC for $T$ and $T\not\models T^{\prime}$.
Then:

$\mbox{As }n\rightarrow\infty,{\sf P}_{n}(T^{\prime})\rightarrow 0$ (2)

To show (1), note that $P(T_{1}|h_{1})>.5$ and since as $n\rightarrow\infty$,
${\sf P}_{n}(h_{1})\rightarrow 1$, after a certain point $P(T_{n}|h_{1})$ is
monotone increasing. Then by Proposition 1, ${\sf P}_{n}(T)\rightarrow
P(T_{n}|h_{1})$. Since $h_{1}$ makes $T$ potentially trustworthy, as
$n\rightarrow\infty,{\sf P}_{n}(T)=1$. To show (2), suppose $h_{1}$ is PWMC
for $T$. Given that $T\not\models T^{\prime}$,
$h_{i}(T^{\prime}_{i})=1-h_{i}(T_{i})$ for each $i$, and the expected
probability of $T^{\prime}$ will decrease strictly monotonically over n, as
${\sf P}_{n}(h_{1})\rightarrow 1$. So as $n\rightarrow\infty,\ {\sf
P}_{n}(T^{\prime})=0.$ $\Box$

Note that our agent may have many evaluation hypotheses and the result of
Proposition 2. Crucially $\hat{f}$ has updated his beliefs only on $T$. But
this matches our intuitions about what agents actually do. As long as the co-
dependence between background beliefs and bodies of evidence holds and certain
bodies of evidence are supported more than others, belief in some bodies of
evidence $T\in{\mathcal{T}}$ will be strengthened, while belief in bodies of
evidence in conflict with $T$ or just different from $T$ will be weakened.
Importantly, this can happen merely by  $T_{i}$ repeating content already in
$T_{k}$ for $i>k$. Such repetitions of content are commonplace on social media
sites and news sites that broadcast continuously. In addition, the assumption
of a dogmatic evaluation hypothesis is actually mild; it reflects an agent’s
mistrust of bodies of evidence other than the ones he relies on—a rather
common situation.

Proposition 2 impacts the marginalization of new data, because if its
assumptions are met, as ${\sf P}_{n}(T^{\prime})\rightarrow 0$, $\hat{f}$
discounts evidence from $T^{\prime}$, despite the presence of evaluation
hypotheses supporting $T^{\prime}$.

###### Proposition 3

Suppose evidence $\phi$ such that $T^{\prime}\models\phi$, $T\not\models\phi$
and $T,T^{\prime}$, and $\hat{f}$’s evaluation hypotheses are as in
Proposition 2 and $\hat{f}$ conforms to Bayesian learning. Then:
hspace*.5in As $n\rightarrow\infty,\ {\sf P}_{n}(\phi)\rightarrow 0$.

Since $\hat{f}$ conforms to Bayesian learning, the marginal probability for
$\phi$ is based on Equation 3 and the set of hypotheses $h_{i}$ in Equation 3
is the set ${\mathcal{H}}$ that for $\hat{f}$ pronounce on testimony that
mentions or asserts $\phi$. By Proposition 1, as $n\rightarrow\infty$, ${\sf
P}_{n}(h_{1})\rightarrow 1$. By Proposition 2, ${\sf
P}_{n}((T^{\prime}h_{1})\rightarrow 0$. But for all other $h_{k}$ such that
$h_{k}(T^{\prime})\neq 0$, by Proposition 1 again, as $n\rightarrow\infty$,
${\sf P}_{n}(h_{k})\rightarrow 0$. But then ${\sf
P}_{n}(\phi|h_{i},{\mathcal{B}})\rightarrow 0$ for all relevant $h_{i}$. Given
Equation 3, the result follows. $\Box$

In this situation, $\hat{f}$ assigns no credence to $\phi$. The prior beliefs
of $\hat{f}$ may so limit the alternative hypotheses $h_{i}$ such that even an
actual fact $\phi$ will have a marginal probability of $0$; $\hat{f}$ will
discount $\phi$ completely.

Now consider general learning in this situation, defined in Wolpert’s 2018
extended Bayesian framework via Bayes’s formula below.

$P(h|x,{\mathcal{B}})=\frac{P(x|h,{\mathcal{B}}).P(h|{\mathcal{B}})}{\sum_{i=1}^{i=k}P(x|h_{i},{\mathcal{B}}).P(h_{i}|{\mathcal{B}})}$
(7)

To learn a hypothesis $h$, $\hat{f}$’s estimation of $h$ at some stage should
be closer to the objective or ideal assignment (posterior) $h_{p}$ to $h$,
than his prior probability for $h$. Similarly for marginal probabilities:
${\sf P}_{n}(x)$ should track $x_{p}$, the posterior of $x$, given a random
sampling of $X$. We consider loss functions ${\mathcal{L}}({\sf
P}_{n}(h),h_{p})$ and ${\mathcal{L}}({\sf P}_{n}(x),x_{p})$. The greater
divergence between the ideal posterior probability and the Bayesian subjective
estimation of that probability, the worse will be the score for $\hat{f}$’s
learning. We say that $\hat{f}$ cannot learn $h$ if additional evidence does
not eventually decrease loss; i.e. we cannot show
$lim_{n\rightarrow\infty}{\mathcal{L}}({\sf
P}_{n}(h),h_{p})<{\mathcal{L}}({\sf P}_{0}(h),h_{p})$.

###### Proposition 4

Suppose $\hat{f}$ is a Bayesian learner with evaluation hypotheses and
testimony $T$, $T^{\prime}$ as in Proposition 2 and all evidence $e$
confirming $h$ is such that $T^{\prime}\models e$. Then $\hat{f}$ is incapable
of learning $h$.

Consider $e$ such that $T^{\prime}\models e$ and e confirms $h$. So the true
posterior $P_{p}(h|e)>P(h)$, with $P(h)$ the prior on $h$. Suppose $\hat{f}$’s
evaluation hypotheses and probabilities have been updated via $T$ as in
Proposition 2. By Proposition 3, as $n\rightarrow\infty,\ {\sf
P}_{n}(e)\rightarrow 0$. In the limit, Bayesian learning as specified by
equation (4) simply isn’t defined when ${\sf P}_{n}(e)=0$. So assuming $e$ is
discounted as evidence in updating, we set $P(h|e,T_{n})=P(h|T_{n})$. But this
is just $E_{0}(h)$ or $P(h)$, $\hat{f}$’s prior on $h$. It follows that as
$lim_{n\rightarrow\infty}\ {\mathcal{L}}({\sf
P}_{n}(h),h_{p})\not<{\mathcal{L}}({\sf P}_{0}(h),h_{p})$. $\Box$

Proposition 4 is a formal statement of IB in a first order setting. It shows
that under certain conditions, $\hat{f}$ will be incapable of learning any
hypothesis that involves a dependence on testimony not in $T$, upon which
$\hat{f}$ has formed his beliefs. $\hat{f}$ is interpretively blind to any
possibilities outside of $T$.

##  4 IB in hierarchical Bayesian learning

It’s not unreasonable to rule out new evidence from unreliable testimony,
provided the assignment of one’s evaluation hypotheses to the testimony is
reasonable. But nothing in our discussion above forces the evaluation
hypotheses to be be reasonable. Without any constraints, $\hat{f}$’s
evaluation hypotheses may rule out evidence that is completely grounded in
reality and comes from testimony that an ideal rational agent would trust.

To solve this problem, we need to correct the background beliefs
${\mathcal{B}}$. Ideally, a rational agent should control for the biases in
testimony by consulting several different bodies of testimony. However,
${\mathcal{B}}$ cannot be corrected itself by evidence, because that evidence
is already discounted if it conflicts with ${\mathcal{B}}$. Very clearly,
background beliefs can be a source of bad epistemic biases, and they can
prevent straightforward corrections to improve one’s beliefs as Bayesian
learning would have us do.

Hierarchical Bayesian models were designed to address this problem (Gelman et
al., 2013). In hierarchical Bayesian models, a Bayesian learning model like
the one we have discussed in Section 3 has certain parameters; the one
parameter we have is our evaluation hypotheses providing the reliability of
testimony. At a second level of the hierarchy, we could have a Bayesian
learning model concerning evaluation hypotheses, in which we could detail
factors that would allow us to estimate reliably the accuracy of an evaluation
hypothesis. Abstractly, we would have evaluation hypotheses about evaluation
hypotheses that would discuss factors like the consistency or the predictive
accuracy of a testimony source, or the extent to which testimony from other
sources agrees with its content. One could also require a longer or more
thorough exploration of the data about the phenomenon before the agent’s
restricting himself to a small subset for exploitation (once again an
application of the work in Cesa-Bianchi and Lugosi (2006)). All of these ideas
and more have been proposed.

Simply requiring evaluation hypotheses that obey exogenous constraints,
however, begs the question of why $\hat{f}$ should accept them. In fact, the
interdependence of testimony, new information and background beliefs can make
the resort to higher order parameters to resolve IB a failure because a body
of dynamic testimony $T$, when directed by a conversational agent for the
purposes of persuading and keeping his audience, can react to and attack not
only a conflicting body of testimony $T^{\prime}$ but also sources supporting
it. This behavior provides arguments for or against not only first order
evaluation hypotheses, as we’ve seen with the notion of consistency, but also
for higher order functions and in fact sequences of evaluation hypotheses.

To formalize this picture, we assume a hierarchy of sets of evaluation
hypotheses where,

$h^{n+1}:h^{n}\rightarrow[0,1],\mbox{ for }h^{n+1}\in{\mathcal{H}}^{n+1}.$

Higher order parameters at level $n+1$ are related to probabilities to
evaluation hypotheses at level $n$ via a notion of rationality.

###### Definition 3

A set of sets of evaluation hypotheses
${\mathcal{H}}=\\{\mathcal{H}^{1},\mathcal{H}^{2},...,\mathcal{H}^{n}\\}$ is
rational iff for all $m<n$, $h^{m}_{k}\in{\mathcal{H}}^{m}$,
$P(h^{m}_{k})=\lambda\sum_{h_{j}\in{\mathcal{H}}^{m+1}}P(h_{j}^{m+1}).P(h^{m}_{k}|h_{j}^{m+1})$
for some normalizing factor $\lambda$.

A rational set of sets of evaluation hypotheses is thus one in which the
probability of the evaluation hypotheses at one level reflects what the higher
levels say about it. Henceforth, we assume that agents’ sets of evaluation
hypotheses are rational.

Given rational
${\mathcal{H}}=\\{\mathcal{H}^{1},\mathcal{H}^{2},...,\mathcal{H}^{n}\\}$, we
define a ${\mathcal{H}}^{n}$ sequence
$\sigma\in\prod_{i=1}^{n}\mathcal{H}^{i}$ of consistent evaluation hypotheses
to support $T$ ($\sigma\mbox{$\mid$ \kern-5.0pt$\approx$}T$) (or that make $T$
potentially trustworthy—$\sigma\models T$) iff the ${\mathcal{H}}^{1}$ element
$h^{1}_{\sigma}$ of $\sigma$ is such that $P(T_{j}|h^{1}_{\sigma})$ is
eventually monotone increasing with respect to $j$ (converges to $1$ as
$j\rightarrow\infty$) and every element of $\sigma$ has non-0 probability
given ${\mathcal{H}}$. Conversely, we say that $T\mbox{$\mid$
\kern-5.0pt$\approx$}\sigma$ iff for each element $h^{i}_{\sigma}$ of $\sigma$
$P(h^{i}_{\sigma}|T_{j})$ is eventually monotone increasing for all stages
$T_{j}$. We note that $\sigma\mbox{$\mid$ \kern-5.0pt$\approx$}T\rightarrow
T\mbox{$\mid$ \kern-5.0pt$\approx$}\sigma$.

Let $\sigma^{2}$ to be the subsequence of $\sigma$ such that
$\sigma^{2}=\sigma\upharpoonright(\prod_{i=2}^{n}\mathcal{H}^{i}).$ For
$h^{1}\in\mathcal{H}^{1},\sigma^{2}(h^{1})$ signifies the support $h_{1}$
receives from the higher order functions in $\sigma$ via Definition 3.

###### Definition 4

Given
${\mathcal{H}}=\\{\mathcal{H}^{1},\mathcal{H}^{2},...,\mathcal{H}^{n}\\}$, we
say that an ${\mathcal{H}}^{n}$ sequence $\sigma$ undercuts $T$ iff for any
$h^{1}\in\mathcal{H}^{1}$ if $P(T|h^{1})>.5,\sigma^{2}(h^{1})\leq
1-P(T|h^{1})$.

###### Definition 5

$\phi$ disagrees with $T^{\prime}$ just in case $P(T|\phi)<P(T).$

###### Definition 6

$T$ attacks $T^{\prime}$ iff there is a
${\mathcal{H}}=\\{\mathcal{H}^{1},\mathcal{H}^{2},...,\mathcal{H}^{n}\\}$
with: (i) ${\mathcal{H}}^{m}\mbox{ sequences }\sigma$ for $m<n$ such that: if
$\sigma\models T$ $P(T^{\prime}|h^{1}_{\sigma})=1-P(T|h^{1}_{\sigma})$ and
$\sigma$ undercuts $T^{\prime}$, and (ii) for any ${\mathcal{H}}^{m}$ sequence
$\sigma$, $m<n$ if $\sigma\mbox{$\mid$ \kern-5.0pt$\approx$}T^{\prime}$,
$\exists h^{m+1}\in{\mathcal{H}}^{m+1}$ such that $(P(h^{m+1}|T)>.5$ and
$h^{m+1}(\sigma)=0)$.

###### Definition 7

$T$ is argumentatively complete iff:
(i) $(T^{\prime}\models\phi\mbox{ and \em
Disagree}(\phi,T))\rightarrow\mbox{\em Attack}(T,T^{\prime})$; (ii) If
$T_{n}\not\models\phi$ but $P(T_{n}|\phi)\geq P(T_{n})$, then
$T_{n+1}\models\phi$. (iii) for any T undercutting ${\mathcal{H}}^{m}$
sequence $\sigma$, $\forall h^{m+1}\in{\mathcal{H}}^{m+1}$ such that
$(P(h^{m+1}|T)>.5,h^{m+1}(\sigma)=0)$ (iv) $\exists{\mathcal{H}}^{n}\mbox{
sequence }\sigma$ such that $h_{\sigma}^{1}\models T.$

###### Proposition 5

If $T$ is argumentatively complete, then $T$ supports a hypothesis $h$ that is
PWMC for $T$.

Assume that $T$ is argumentatively complete. Then
$\exists{\mathcal{H}}^{n}\mbox{ sequence }\sigma$ such that
$h_{\sigma}^{1}\models T$. Now assume $T_{n}\not\models\phi$ for some $\phi$
for all stages $n$. But then $P(T_{n}|\phi)<P(T_{n})$ for each stage $T_{n}$
of $T$. But then $T$ and $\phi$ disagree and so $T$ attacks $\phi$. By the
definition of attack, $P(\phi|h^{1}_{\sigma_{1}})=1-P(T|h^{1}_{\sigma_{1}})$.
So $h_{\sigma}^{1}$ is PWMC. $\Box$

###### Proposition 6

Let $T$ be argumentatively complete with a rational set of evaluation
hypotheses ${\mathcal{H}}$ with $\sum_{h^{1}\in{\cal H}^{1}}P(h^{1})\neq 0$
and probabilities updated on $T$.

$\mbox{\em As }n\rightarrow\infty,\ {\sf P}_{n}(T)\rightarrow 1.$ (1)

In addition suppose there is a $T^{\prime}\not\subseteq T$.

$\mbox{\em As }n\rightarrow\infty,\ {\sf P}_{n}(T^{\prime})\rightarrow 0.$ (2)

We first show (1). Since $T$ is argumentatively complete,
$\exists\sigma\in{\mathcal{H}}$ such that $h^{1}_{\sigma}\models T$. We need
to show that for some such $h^{1}_{\sigma}$, $P(h^{1}_{\sigma})\neq 0$
relative to ${\mathcal{H}}$. Suppose that $P(h^{1}_{\sigma})=0$, for all
$h^{1}_{\sigma}$ such that $h^{1}_{\sigma}\models T$. By rationality, for each
such $h^{1}_{\sigma}$,
$P(h^{1}_{\sigma})=\lambda\sum_{h_{j}\in{\mathcal{H}}^{2}}P(h_{j}^{2}).P(h^{1}_{\sigma}|h_{j}^{2})=0$.
Thus, all the non-0 probability mass of ${\cal H}$ falls on $T$ undercutting
sequences $\sigma_{i}$. But for each such $T$ undercutting $\sigma_{i}$ of
length $m$, since $T$ is argumentatively complete, there is an evaluation
hypothesis $h^{m+1}$ supported by $T$ such that $P(\sigma_{i}|h^{m+1})=0$.
Since ${\cal H}$ has only finitely many levels, at some level $k$ all T
undercutting sequences $\sigma_{j}$ get $0$ probability. This, together with
the fact that $\sum_{h^{1}\in{\cal H}^{1}}P(h^{1})\neq 0$, contradicts the
assumption that $P(h^{1}_{\sigma})=0$. Since $T$ is argumentatively complete,
any sequence supporting any $h^{1}$ where $P(T|h^{1})<P(T)$ will eventually
get probability $0$; so $\sum_{\\{h^{1}:P(T|h^{1})\geq
P(T)\\}}P(h^{1})=\sum_{h^{1}\in{\cal H}^{1}}P(h^{1})$. Moreover, as ${\sf
P}_{n}$ gets updated, as $n\rightarrow\infty,\\{h^{1}:{\sf
P}_{n}(T|h^{1})\geq{\sf P}_{n}(T)\\}\rightarrow\\{h^{1}:h^{1}\models T\\}$.
The conditions on first order evaluation hypotheses in ${\mathcal{H}}$ of
Proposition 1 are now met. By Propositions 1 and 2, as ${n\rightarrow\infty},\
{\sf P}_{n}(h^{1}_{\sigma})\rightarrow 1,{\sf P}_{n}(h^{1}_{i})\rightarrow 0$
for $i\neq 1$. By Proposition 2, ${\sf P}_{n}(T)\rightarrow 1$.

To show (2), by Proposition 5, $h^{1}_{\sigma}$ is also PWMC for $T$. As
${n\rightarrow\infty},since\ {\sf P}_{n}(h^{1}_{\sigma})\rightarrow 1$, ${\sf
P}_{n}(T^{\prime})\rightarrow 0.$ $\Box$

###### Proposition 7

Suppose $T$ is argumentatively complete. Let $\hat{f}$ be a hierarchical
Bayesian learner whose evaluation hypotheses are rational and are updated on
$T$. If $T^{\prime}\subsetneq T$ such confirms a hypothesis $h$ that $T$ does
not, then $\hat{f}$ is incapable of learning $h$.

Claim 2 of Proposition 6 shows that ${\sf P}_{n}(T^{\prime})\rightarrow 0$.
Then apply Proposition 4. $\Box$

Argumentatively complete testimony thus collapses the case of higher order
Bayesian frameworks to our first order setting. What is troubling about IB is
that our learner $\hat{f}$ may hold onto an argumentatively complete $T$
regardless of how inadequate it is in the eyes of others or standard epistemic
criteria; an argumentatively complete theory will always eventually find a
reply to any attack or any doubt $\hat{f}$ might acquire.

Argumentatively complete testimony isn’t just an abstract concept; many social
media and news sites already approximate this condition. Outlets like NewsMax
or One Amercan News Network that have a particular political bias will attack
the credibility of stories from other bodies of testimony that have gone
against a narrative they were and are promoting; darker conspiracy spinning
websites like those promoting QAnon will attack arguments against their
theories once they become aware of them.333See Stuart A. Thompson, “Three
Weeks Inside a Pro-Trump QAnon Chat Room” NY Times, Jan 26, 2021). In
anecdotal support of our claims, consider Michelle Goldberg’s “It’s Marjorie
Taylor Greene’s Party Now” New York Times, 2/2/2021) description of a group in
IB: “American conservatism — particularly its evangelical strain — has
fostered derangement in its ranks for decades, insisting that no source of
information outside its own self-reinforcing ideological bubble is
trustworthy.”

A crucial component of argumentatively complete testimony $T$ is that it
promotes evaluation hypotheses that both make $T$ eventually trustworthy but
also PWMC for $T$. Sources like the New York Times embody this in their slogan
all the news that’s fit to print, but there’s a commercial reason for this
outcome; news sites and social media are out to capture market share and so
they naturally promote themselves as accurate and complete at least in a
certain domain. The nature of contemporary testimony leads agents naturally to
a situation where IB occurs.

How general are the results in Propositions 6 and 7? Wolpert (2018) argues
that PAC, Statistical Physics Framework, VC, and supervised Bayesian learning
are four different instantiations of his extended Bayesian formalism, which we
use. Thus our results should hold for other frameworks.

##  5 Comparisons to Prior Work

IB is an epistemological bias that is clearly related to confirmation bias
(Lord et al., 1979; Nickerson, 1998; Oswald and Grosjean, 2004), in which
agents interpret new evidence in a way that confirms their beliefs, and to the
framing biases of Tversky and Kahneman (1975, 1985). People tend to see in the
evidence what they believe. These forms of bias, however, concern how beliefs
and bias influence interpretation, painting only part of the picture of IB
(see also Asher and Paul (2018)). Further, unlike much of the psychological
literature which finds epistemologically exogenous justifications for this
bias (Dardenne and Leyens, 1995), we show how IB is a natural outcome of
Bayesian updating, rational resource management and the belief interpretation
co-dependence.

IB is a concrete application of the work on bandits in, determining optimal
allocation of resources to the exploration and exploitation of sources Whittle
(1980); Lai and Robbins (1985); Banks and Sundaram (1994); Burnetas and
Katehakis (1997); Auer et al. (2002); Cesa-Bianchi and Lugosi (2006); Garivier
and Cappé (2011). It is also related to work on generalization in machine
learning. Epistemic biases affect generalization and learning capacity in ways
that are still not fully understood (Lampinen and Vehtari, 2001; Zhang et al.,
2016; Kawaguchi et al., 2017; Neyshabur et al., 2017). Zhang et al. (2016)
show that standard techniques in machine learning for promoting good epistemic
biases and generalization—training error minimization, regularization
techniques like weight decay or dropout, or complexity measures used to
minimize generalization error (the difference between training error and test
error)—do not necessarily lead to good generalization and test performance.
Argumentatively complete testimony $T$ incorporates an adversarial attack
mechanism against any good epistemic practices that might discount $T$. It’s
this mechanism that guarantees IB.

The argumentation literature (Amgoud and Demolombe, 2014; Dung, 1995) is also
relevant to IB. If testimony $T$ is argumentatively complete, then $T$ always
provides a counterargument to an attack against $T$–much like an acceptable
argument in Dung (1995). In addition, however, an argumentatively complete $T$
also supports higher order evaluation hypotheses that support hypotheses that
support $T$. There are also important connections to the literature on trust
(Castelfranchi and Falcone, 2010); in our set up learning agents trust certain
sources over others, and our higher order setting invokes a hierarchy of
reasons. Nevertheless, the argumentation and trust-based work of which we are
aware is complementary to our approach. An argumentation framework takes a
possibly inconsistent belief base and imposes a static constraint on inference
in such a setting. Similarly, trust is typically modeled in some sort of
static modal framework. By contrast, ME learning games and the whole Bayesian
framework are dynamic, with beliefs evolving under evidence and game
strategies evolving under agent interaction. It is this dynamic evolution that
is crucial to our approach and, we think, to modeling agents and learning. In
sum, we are not looking at the problem of consistency, but rather the problems
of entrenchment and bias.

##  6 The complexity of IB

IB is a result about learning. IB is a suboptimal but natural outcome of the
way contemporary bodies of evidence are set up and how humans interpret them.
Given our set up, everything turns on what body of evidence on which to update
and with which evidential hypotheses.

If IB is suboptimal, its effects are still more worrisome, because agents in
the grip of IB are often unwilling or incapable of changing their beliefs so
as to be able to learn. Of course, our learner might just be happy with $T$;
perhaps he needs no more accurate or more truthful body of testimony. He may
not be interested in learning anything beyond what $T$ presents him with. In
this section, however, we assume a learner who might be interested in learning
but has difficulting escaping his IB prison. We assume a rational learner
$\hat{f}$ who updates according to his evidential hypotheses; so if he has an
evidential hypothesis that confers a high probability on some $T$, he will
update on $T$. We’ve seen that $\hat{f}$ can get IB when he unduly restricts
the bodies of evidence which serve as the basis of update or when he attends
to an argumentatively complete testimony. So key to removing IB is to get
$\hat{f}$ to change his hypotheses and consider other evidence that that to
which he is wedded.

Anecdotally, we have a lot of evidence that IB is hard to escape444See
Thompson, cited in note 2. In general, however, we lack a precise analysis of
its difficulty. In this section, we introduce a game theoretic method that
shows IB is not only hard to defeat but it can even be hard to detect (leading
to self-deception). We will see that the choice of epistemic paradigms is
important.

To motivate our approach, consider how an actual conversation might go between
our learner $\hat{f}$ in the grip of IB and a person $E$ who wants to correct
his problem. $E$ might question $\hat{f}$’s reasons for believing some
proposition $\phi$; she might try getting $\hat{f}$ to consider different
bodies of evidence $T^{\prime}$ that might disconfirm $\phi$. $\hat{f}$ might
accept $T^{\prime}$ or he might argue against it—by providing, for example,
reasons why $T^{\prime}$ is not trustworthy or why the arguments supporting
$T^{\prime}$ are faulty. $E$ might attack those arguments or provide new
evaluation hypotheses for consideration. Our ME games formalize this
interaction.

In an ME learning game $\mathcal{G}=((V_{0}\cup
V_{1})^{\infty},\mathit{Win})$, the two players, our investigator $E$ and our
Bayesian learner $\hat{f}$, construct a larger ”conversation” by consecutively
playing finite strings from the vocabulary $V_{0}$ and $V_{1}$ respectively.
$\mathit{Win}$ specifies the winning condition of $E$. The vocabularies
$V_{i}$ of an ME learning game $\mathcal{G}$ consists of sequences of
evaluation hypotheses (with some abuse of notation, we’ll take a single
$h^{n}_{j}$ to be a one place sequence) and a predicate ACCEPT. ACCEPT means
that $\hat{f}$ accepts the last suggestion by $E$ and confers upon it a non
zero probability mass. Our ME learning games are subject to several
constraints.

  1. A.

Knowledge first (Williamson, 2002): this is a constraint from formal
epistemology; $\hat{f}$ only adds a sequence $\sigma$ to
$\prod^{n}_{1}\mathcal{H}^{i}_{\hat{f}}$ for
$\mathcal{H}^{i}_{\hat{f}}\in{\mathcal{H}}_{\hat{f}}$ if he has no argument
that attacks $\sigma$—in other words no evaluation hypothesis
$h^{n+1}\in{\mathcal{H}}^{n+1}_{\hat{f}}$ such that $h^{n+1}(\sigma)=0$.

  2. B.

The Jury in an ME learning game is epistemologically competent; i.e. it
sanctions only evaluation hypotheses that advance learning.

  3. C.

$E$ may only add sequences of evaluation hypotheses sanctioned by the Jury. We
assume this to be a finite set ${\mathcal{H}}_{J}$.

  4. D.

Both players must only propose consistent and rational sequences.

  5. E.

$\hat{f}$ has learned from some body of evidence $T$, which is common
knowledge.

  6. F.

$\hat{f}$ may only refuse a proposal of $E$ in the higher order setting, if he
has a reason to do so.

We define a sequence $\sigma\in\prod^{n}_{1}{\mathcal{H}}^{i}$ to be positive
if for each element $h^{m+1}$ and $h^{m}$ of $\sigma$ $h^{m+1}(h^{m})>>0$. A
sequence $\sigma$ nullifies a sequence $\sigma_{1}$, if for all $m$ and for
$h^{m}_{1}$ of $\sigma_{1}$, $h^{m+1}$ of $\sigma$ is such that
$h^{m+1}(h_{1}^{m})=0$. We can have two sequences each one nullifying the
other. This formally represents an $n$ round argument, with each round $j+1$
offering a counterargument to the argument of round $j$. We will say that a
hypothesis $h^{1}$ is $T$ positive if $h^{1}$ is positive and $P(T|h^{1})=1$

We now define the moves of a game $\mathcal{G}$, in which we suppose a body of
evidence $T$ that $\hat{f}$ has attended to and a body of evidence
$T^{\prime}$ inconsistent with $T$. $E$ plays first, then $\hat{f}$ then
replies. The game ends if $\hat{f}$ plays ACCEPT, which implies that he adds a
hypothesis $h^{1}_{*}$ to ${\mathcal{H}}^{1}_{\hat{f}}$, with a non-0
probability mass and with with high $P(T^{\prime}|h^{1}_{*})$, where
$T^{\prime}\cup T$ is inconsistent.

  1. (m1)

$E$ proposes $T^{\prime}$-positive $h^{1}\in{\mathcal{H}}^{1}_{J}$ to be added
to ${\mathcal{H}}^{1}_{\hat{f}}$.

  2. (m2)

Suppose at round $k\geq 1$ of $\rho$ in $\mathcal{G}$ $E$ has proposed a
$T^{\prime}$ positive $h^{1}$. At $k+1$ $\hat{f}$ may play ACCEPT.

  3. (m3)

Suppose at round $k$ of $\rho$ in $\mathcal{G}$ $E$ has proposed a a
$T^{\prime}$ positive $h^{1}$. At $k+1$ $\hat{f}$ may play a nullifying
$h^{2}\in{\mathcal{H}}^{1}_{\hat{f}}$ such that $h^{2}(h^{1})=0$, if there
exists such $h^{2}\in{\mathcal{H}}^{2}_{\hat{f}}$.

  4. (m4)

Suppose $E$ has proposed a positive sequence $\sigma$ of length $m$ and with
$h^{1}_{\sigma}$ $T^{\prime}$ positive at round $k$ of $\rho$ in
$\mathcal{G}$. At round $k+1$ $\hat{f}$ may respond with sequence of length
$m+1$ nullifying $\sigma$.

  5. (m5)

Suppose at round $k$ of $\rho$ of $\mathcal{G}$, $\hat{f}$ has proposed an
m-length sequence $\sigma$ nullifying a positive $\sigma_{*}$ proposed by $E$,
with $T^{\prime}$ positive $h^{1}_{\sigma_{*}}$. $E$ may respond at round
$k+1$ of $\rho$ with a positive $m+1$ length sequence
$h_{*}^{m+1}.\sigma_{*}$, with $h^{m+1}_{1}(h^{m})\neq 1$ for $h^{m}$ in
$\sigma$.

  6. (m6)

Suppose at round $k$ of $\rho$ in $\mathcal{G}$, $E$ has proposed a positive
sequence $\sigma$ of length $m$ and with $h^{1}_{\sigma}$ $T^{\prime}$
positive. At round $k+1$ $\hat{f}$ may play ACCEPT, which implies that he adds
$\sigma$ to $\prod^{n}_{1}\mathcal{H}^{i}_{\hat{f}}$.

We note that if move (m6) occurs $\hat{f}$, assigns $h^{1}_{*}$ and
$T^{\prime}$ a non-0 probability mass and updates with evidence $T^{\prime}$.
In which case the game ends.

Suppose that in an ME learning game $\mathcal{G}$, $E$’s winning condition is
simply to discover that $\hat{f}$ is interpretively blind, if he is. Call this
condition $IB$. We establish the complexity of $E$’s attempt to achieve $IB$.
The first order case with a finite ${\cal H}$ where the game is restricted to
moves m1,m2,m3, is rather trivial. More interesting is the case of an ME
learning game $\mathcal{G}=((V_{0}\cup V_{1})^{\infty},\mathit{Win})$ with
$\mathit{Win}=IB$ and in which $E$ and $\hat{f}$ play higher order evaluation
hypotheses.

###### Proposition 8

Suppose an ME learning game $\mathcal{G}=((V_{0}\cup
V_{1})^{\infty},\mathit{Win})$ with $\mathit{Win}=IB$ in which $\hat{f}$ plays
moves described in (m4)- (m7). Then $\hat{f}$ is not interpretively blind iff
play stops at some finite ordinal $n$.

Suppose that in the play of $\mathcal{G}$, $\hat{f}$ accepts at some level $n$
to add the sequence of evaluation hypotheses proposed by $E$. Then by the
construction of the sequence and the requirement of coherence (constraint D),
this confers upon some evaluation hypothesis $s*_{1}$ a non zero probability
such that $P(T^{\prime}|h^{1}_{*})=1$, where $T^{\prime}$ is incompatible with
the body of evidence $T$. By accepting, $\hat{f}$ will have an evaluation
hypothesis $h^{1}_{*}$ with non zero probability such that
$P(T^{\prime}|h^{1}_{*})=1$, where $T^{\prime}$ is incompatible with the body
of evidence $T$, which $\hat{f}$ has proposed as a source of learning
(constraint E). Now when $\hat{f}$ updates his belief in $T$ he must do so
with respect to $h^{1}_{*}$, and he must now update his confidence in his
evaluation hypotheses with respect not only to $T$ but also $T^{\prime}$. In
that case, $P(h^{1}_{*}|T_{n},T^{\prime}_{n})\not\rightarrow 0$ and ${\sf
P}_{n}(T^{\prime})\not\rightarrow 0$. As a result, $\hat{f}$ will be able to
learn from $T^{\prime}$, and so he is not interpretively blind with respect to
$T$.

If there is no stopping point at any finite ordinal, then $E$’s is never able
to get $\hat{f}$ to accept a $T^{\prime}$ positive hypothesis. In which case,
$\hat{f}$ continues to only update on $T$ and by Propositions 6 7, $\hat{f}$
is interpretively blind. $\Box$

Suppose $E$’s winning condition for an ME learning $\mathcal{G}$, is to get
$\hat{f}$ to accept a $T^{\prime}$ positive evaluation hypothesis. Call this
winning condition for $E$ ${\mathcal{P}}$ (for persuasion).

###### Corollary 1

Suppose that in an ME learning game $\mathcal{G}$ with
$\mathit{Win}={\mathcal{P}}$. The complexity of $\mathit{Win}$ is an R.E. set.
If $\mathit{Win}=IB$ then $\mathit{Win}$ is co-r.e.

If ${\mathcal{P}}$ is the set of all finite plays, IB is its complement to a
countable union of closed sets of plays in $(V_{0}\cup V_{1})^{\omega}$ that
never pass through an ACCEPT move. $\Box$

###### Proposition 9

Suppose an ME learning game $\mathcal{G}$ with $\mathit{Win}={\mathcal{P}}$
and $\hat{f}$ as described in Proposition 6. Then $E$ has no winning strategy
in $\mathcal{G}$.

Proposition 7 implies $\hat{f}$’s evaluation hypotheses are updated on an
argumentatively complete body of evidence $T$. When implemented via an ME game
$\mathcal{G}$, the sequence of evaluation hypotheses in Proposition 6 provide
a winning strategy for $\hat{f}$. Suppose $E$ proposes an $h^{1}$ supporting
$e$ that is inconsistent with $T$. Even if $E$ generates a suitable sequence
of higher order $T^{\prime}$ positive evaluation hypotheses
$h^{1},h^{2},h^{3},\ldots$, given Constraint A above, $\hat{f}$ will only
accept an evaluation hypothesis if he has no argument against it. But as $T$
will eventually supply such an argument, $\hat{f}$ can always counter $E$’s
proposals. So she has no winning strategy. $\Box$

Not only is IB computationally complex (Corollary 1 shows it is not computable
but $\Pi_{1}$), Proposition 9 shows formally that even if $E$ has rationally
compelling arguments to show that $\hat{f}$ is better off (his payoff or
reward is higher) in accepting her proposed sequence of evaluation hypotheses,
$\hat{f}$ can rationally resort to $T$ to counter her argument. Extracting
someone from higher order IB is thus impossible by purely epistemic means.
There is no way of getting someone, even a rational agent, out of higher order
IB by purely epistemic arguments, given our assumptions. This pessimistic is
borne out empirically: some people in the grip of right wing conspiracy
theories in the US were dying of Covid19 in December of 2020 and January 2021
but continued to refuse to believe that it was that disease that was killing
them—despite all the evidence and arguments they were given, they refused to
let go of an obviously faulty but argumentatively complete $T$.

Of course, people sometimes do change their minds and do escape the grip of
argumentatively complete theories, many times for epistemically exogenous
reasons.555For instance, the satisfaction they derived from belonging to a
particular community supported by a particular body of testimony might and
does wane. But by challenging one of our assumptions, rational agents can of
course also reject IB. The weak link in our argument is assumption $A$, the
”knowledge first” assumption. Perhaps $\hat{f}$ should accept evaluation
hypotheses even if $T$ attacks them. More likely, $\hat{f}$ should not accept
all attacks equally; he should be skeptical of any body of evidence $T$ that
promotes PWMC for $T$ and $T$ eventually trustworthy evaluation hypotheses
while attacking any point of view at variance with it.

We now explore the play between $E$ and $\hat{f}$ in an ME learning game
$\mathcal{G}$ where $\mathit{Win}={\mathcal{P}}$ before $\hat{f}$ has accepted
enough of the argumentatively complete $T$ to close off learning from
alternative bodies of evidence. Suppose $T$ is argumentatively complete but
comes in stages; if $T^{\prime}_{i}$ attacks $T_{i}$, then $T_{i+1}$ but not
$T_{i}$ attacks $T^{\prime}_{i}$. That is, an argumentatively complete $T$
reacts to attacks but does not forsee all attacks in advance. Suppose a set of
consistent first order evaluation hypotheses
${\mathcal{H}}^{1}=\\{h^{1}_{1},h^{1}_{2},...\\}$, with
$P(h^{1}_{1})=.6,P(h^{1}_{2})=.4$, and
$P(T_{i}|(h_{1})=1=P(T^{\prime}_{i}|h_{2})$. Now suppose $T^{\prime}_{1}\cup
T_{1}$ is inconsistent and $E$ proposes $h^{1}_{2}$ since $h^{1}_{2}\models
T^{\prime}_{i}$. Since the $h^{1}_{i}$ are consistent,
$P(T_{1}|h^{1}_{2})=0=P(T^{\prime}_{2}|h^{1}_{1})$. At this point, $\hat{f}$
could accept $E$’s proposal under constraint (A), $\mathcal{G}$ ends and $E$
wins. $\hat{f}$ will continue to update over stages $T$ and $T^{\prime}$ with
the marginal probabilities $P(T_{i})=.6$ and $P(T^{\prime}_{i})=.4$ remaining
stationary.

On the other hand, $\hat{f}$ may decide to wait to see what the next stage
$T_{2}$ of $T$ brings. As $T$ is argumentatively complete, $T_{2}$ will attack
$T^{\prime}_{2}$, and add a nullifying $h^{2}\in{\mathcal{H}^{2}}$ supported
by $T_{2}$. Should $\hat{f}$ accept $h^{2}$, the probability of $h^{1}_{2}$
will go to $0$ in ${\mathcal{H}}$. But now suppose we have a constraint,
Discount,that discounts any nullifying sequence from $T$. It would be
unreasonable for $\hat{f}$ to wipe out alternatives in the face of this level
of uncertainty; at this stage, $P(T_{2})=.6$ and $P(T^{\prime}_{2})=.4$.
Summarizing:

###### Proposition 10

Suppose an ME learning game $\mathcal{G}$ with constraint $A$ replaced by
Discount and with $\mathit{Win}={\mathcal{P}}$ and $\hat{f}$ as described in
Proposition 6. $E$ then has a winning strategy in $G$, and IB does not arise
for $\hat{f}$.

##  7 Conclusions

Interpretive blindness results from a dynamic, iterative process whereby a
learner’s background beliefs and biases lead her to update her beliefs based
on a body of testimony $T$, and then biases inherent in $T$ come back to
reinforce her beliefs and her trust in $T$’s source(s), further biasing her
towards these sources for future updates. We have introduced and formally
characterized IB. We have shown that IB can prevent learning even in higher
order Bayesian frameworks for learning from argumentatively complete
testimony, despite the presence of constraints designed to promote good
epistemic practices. We also shown that IB is computationally complex as a
co-r.e. set via a game theoretic analysis, and that an agent may rationally
remain in IB in the face of epistemic arguments. Our game theoretic analysis
can also be extended to cases where the agent falls out of IB but then is a
recidivist and becomse a prisoner once more. We leave that for future work.

Investigating IB alas is not just an academic enterprise. IB really does
happen, with sometimes tragic or dangerous results. We think a careful formal
analysis is urgent for society. Finally, we note that while we have focused on
IB as a problem for learning from testimony, the problem it raises for
learning extends to any case in which we do not have unmediated access to
ground truth and our data is “theory laden” Hanson (1958).

## References

  * (1)
  * Amgoud and Demolombe (2014) Leila Amgoud and Robert Demolombe. 2014.  An argumentation-based approach for reasoning about trust in information sources.  _Argument and Computation_ 5:2-3 (2014), 191–215.
  * Asher and Paul (2018) Nicholas Asher and Soumya Paul. 2018.  Strategic conversation under imperfect information: epistemic Message Exchange games.  _Logic, Language and Information_ 27.4 (2018), 343–385.
  * Auer et al. (2002) Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. 2002\.  Finite-time analysis of the multiarmed bandit problem.  _Machine learning_ 47, 2-3 (2002), 235–256.
  * Banks and Sundaram (1994) Jeffrey S Banks and Rangarajan K Sundaram. 1994.  Switching costs and the Gittins index.  _Econometrica: Journal of the Econometric Society_ (1994), 687–694.
  * Burnetas and Katehakis (1997) Apostolos N Burnetas and Michael N Katehakis. 1997.  Optimal adaptive policies for Markov decision processes.  _Mathematics of Operations Research_ 22, 1 (1997), 222–255.
  * Castelfranchi and Falcone (2010) Christiano Castelfranchi and Rino Falcone. 2010.  _Trust theory: A socio-cognitive and computational model_. Vol. 18.  John Wiley & Sons.
  * Cesa-Bianchi and Lugosi (2006) Nicolo Cesa-Bianchi and Gábor Lugosi. 2006.  _Prediction, learning, and games_.  Cambridge university press.
  * Dardenne and Leyens (1995) Benoit Dardenne and Jacques-Philippe Leyens. 1995.  Confirmation Bias as a Social Skill.  _Personality and Social Psychology Bulletin_ 21.11 (1995), 1229–1239.
  * Dung (1995) Phan Minh Dung. 1995\.  On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games.  _Artificial intelligence_ 77, 2 (1995), 321–357.
  * Garivier and Cappé (2011) Aurélien Garivier and Olivier Cappé. 2011.  The KL-UCB Algorithm for Bounded Stochastic Bandits and Beyond.. In _COLT_. 359–376.
  * Gelman et al. (2013) Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. 2013\.  _Bayesian data analysis_.  CRC press.
  * Hanson (1958) Norwood Russell Hanson. 1958\.  _Patterns of discovery: An inquiry into the conceptual foundations of science_. Vol. 251.  Cambridge University Press.
  * Kawaguchi et al. (2017) Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. 2017.  Generalization in deep learning.  _arXiv preprint arXiv:1710.05468_ (2017).
  * L Griffiths et al. (2008) Thomas L Griffiths, Charles Kemp, and Joshua B Tenenbaum. 2008\.  Bayesian models of cognition.  In _Cambridge Handbooks in Psychology_ , Ron Sun (Ed.). Cambridge University Press, 59–100.
  * Lai and Robbins (1985) Tze Leung Lai and Herbert Robbins. 1985.  Asymptotically efficient adaptive allocation rules.  _Advances in applied mathematics_ 6, 1 (1985), 4–22.
  * Lampinen and Vehtari (2001) Jouko Lampinen and Aki Vehtari. 2001.  Bayesian approach for neural networks—review and case studies.  _Neural networks_ 14, 3 (2001), 257–274.
  * Lord et al. (1979) Charles G. Lord, Lee Ross, and Mark R. Lepper. 1979\.  Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence.  _Journal of Personality and Social Psychology_ 37.11 (1979), 2098–3009.
  * Murphy et al. (2013) Kevin Murphy, Yair Weiss, and Michael I Jordan. 2013.  Loopy belief propagation for approximate inference: An empirical study.  _arXiv preprint arXiv:1301.6725_ (2013).
  * Neyshabur et al. (2017) Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. 2017.  Exploring generalization in deep learning. In _Advances in Neural Information Processing Systems_. 5947–5956.
  * Nickerson (1998) Raymond S Nickerson. 1998\.  Confirmation bias: A ubiquitous phenomenon in many guises.  _Review of General Psychology_ 2.2 (1998), 175–220.
  * Oswald and Grosjean (2004) Margit E. Oswald and Stefan Grosjean. 2004.  Confirmation bias.  In _Cognitive illusions: A handbook on fallacies and biases in thinking, judgement and memory_ , Rüdiger F. Pohl (Ed.). Hove, UK: Pyschology Press, 79–96.
  * Tenenbaum et al. (2006) Joshua B Tenenbaum, Thomas L Griffiths, and Charles Kemp. 2006\.  Theory-based Bayesian models of inductive learning and reasoning.  _Trends in cognitive sciences_ 10, 7 (2006), 309–318.
  * Tenenbaum et al. (2011) Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. 2011.  How to grow a mind: Statistics, structure, and abstraction.  _science_ 331, 6022 (2011), 1279–1285.
  * Tversky and Kahneman (1975) Amos Tversky and Daniel Kahneman. 1975.  Judgment under uncertainty: Heuristics and biases.  In _Utility, probability, and human decision making_. Springer, 141–162.
  * Tversky and Kahneman (1985) Amos Tversky and Daniel Kahneman. 1985.  The framing of decisions and the psychology of choice.  In _Environmental Impact Assessment, Technology Assessment, and Risk Analysis_. Springer, 107–129.
  * Whittle (1980) Peter Whittle. 1980\.  Multi-armed bandits and the Gittins index.  _Journal of the Royal Statistical Society. Series B (Methodological)_ (1980), 143–149.
  * Williamson (2002) Timothy Williamson. 2002\.  _Knowledge and its Limits_.  Oxford University Press.
  * Wolpert (2018) David H Wolpert. 2018\.  The relationship between PAC, the statistical physics framework, the Bayesian framework, and the VC framework.  In _The mathematics of generalization_. CRC Press, 117–214.
  * Zhang et al. (2016) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016.  Understanding deep learning requires rethinking generalization.  _arXiv preprint arXiv:1611.03530_ (2016).