key: cord-0634742-rk7xm24g
authors: Asher, Nicholas; Hunter, Julie
title: Interpretive Blindness
date: 2021-10-19
journal: nan
DOI: nan
sha: 9949d0bcfe1814246c3fdcab21e9d0363eb0a119
doc_id: 634742
cord_uid: rk7xm24g

We model here an epistemic bias we call textit{interpretive blindness} (IB). IB is a special problem for learning from testimony, in which one acquires information only from text or conversation. We show that IB follows from a co-dependence between background beliefs and interpretation in a Bayesian setting and the nature of contemporary testimony. We argue that a particular characteristic contemporary testimony, textit{argumentative completeness}, can preclude learning in hierarchical Bayesian settings, even in the presence of constraints that are designed to promote good epistemic practices.

In this paper, we describe and analyze an as far as we know theoretically un-examined kind of bias, which we call interpretive blindness (IB). IB is exemplified by humans (and perhaps soon by sophisticated machine learning algorithms) whose beliefs are guided and shaped by testimony. When learning through testimony-perhaps the primary way that most people acquire information nowadays-an agent acquires beliefs through conversations with other agents, or from books, newspapers or social networks, and so on. Typically, such people lack direct access to the phenomena described via that testimony. Typically too, humans only pay attention to a restricted set of bodies of testimony from a limited number of sources for their information-which makes sense in terms of an agent's limited resources and attention span. Our paper is about the strategic consequences of opinion diffusion through testimony and the distortions on learning and information that can result.

IB results from this restriction to few sources of testimony and a natural co-dependence between beliefs and interpretation (Asher and Paul, 2018) . Relying on testimony T from a restricted set of sources to update one's beliefs leads to the mutual reinforcement of our confidence in the source and our belief in T ; this creates a bias that can preclude learning when an agent tries to exploit new data that are incompatible with or simply distinct from T . Agents who are interpretively blind will discount any evidence that challenges their beliefs. We use Wolpert's 2018 extended Bayesian framework to prove our results.

While IB is problematic for a standard Bayesian framework, it also poses problems for hierarchical Bayesian approaches (Gelman et al., 2013) , because testimony from sources on social media like Facebook, 24/7 media outlets and web interest groups is often argumentatively complete, a notion we analyze precisely in Section 4; in an argumentatively complete body of testimony T , the authors of that testimony can respond to and argue with any doubts raised by other data or arguments in a body T ′ that might threaten T 's credibility. A skillful climate denier, for example, will always find a way to undercut the most scientifically careful argument. Argumentatively complete testimony thus can undermine higher order constraints and good epistemic practices that should guide first order learning.

Our paper starts in Section ?? by discussing testimony. We then introduce the codependence of belief and interpretation and apply it to the situation of testimony and the sources that support it. In Section 3 we formally show how IB can result in ordinary Bayesian learning. Section 4 shows how IB is reinforced in a hierarchical Bayesian learning setting. Section 6 develops a game theoretic setting to investigate the complexity of IB. We provide results as to whether it is possible to free agents from interpretive bias in several epistemic settings.

IB arises in learning because of a co-dependence between beliefs and the interpretation of evidence, in particular its reliability. In this case, we are talking of the interpretation of written or linguistically conveyed information. Others have already noted a co-dependence of beliefs and linguistic interpretation (Asher and Paul, 2018) . Consider this exchange. R: "Why hasn't the senator commented on the story that he received undisclosed gifts from supporters?" A: "The Senator has declared every gift that he has received." As Asher and Paul (2018) argue, one could interpret A's response either as an answer to R's question or as an evasion and it depends upon one's beliefs about the honesty of the senator. Those beliefs get confirmed as the exchange and its interpretation continue: the interpretation that relies on the honesty of the Senator gets confirmed as A refuses to engage with R, and this in turn confirms the honesty of the belief; mutatis mutandis for the other interpretation.

An analogous co-dependence occurs with interpretation, belief and learning: in updating our beliefs with new evidence; our beliefs color how we interpret that evidence, in particular how trustworthy we find it. The updated evidence in turn conditionally updates our beliefs.

Let us look this codependence in learning by testimony. A body of testimony T is a collection of information conveyed by one or more sources like The New York Times, Fox News, CNN, Facebook, 4Chan, a particular individual or set of individuals. The sources may "promote" or vouch for T or cast doubt on T . Such bodies are also dynamic; they evolve over time as they are updated with new facts and events. In other words, T comes in "stages", where stages might be defined by times or even conversational turns, and each stage T i is the body of evidence accumulated up to stage i. T = {T 1 , T 2 , ..., T n , ...} is the collection of all the stages of a dynamic body of evidence. Dynamic bodies of testimony are ubiquitous in our communicative landscape; on-line, 24/7 news sources as well as particular groups on social media provide evolving, updated coverage of new events. Let T be a collection of bodies of testimony about some phenomenon P. We will assume thatf does not have independent access to P and uses evidence from bodies of testimony in T together with background beliefs to update probabilities about hypotheses about P, some of which are hypotheses about marginal probabilities of events described in T .

Learning from testimony T with source s requires a learnerf to judge T as credible, a judgment that will depend on s's evaluation of T (whether s promotes or challenges T ), as well asf 's antecedent hypotheses about s. Let H be a set of evaluation hypotheses, where each h ∈ H gives the evaluation of a set T of bodies of testimony T relative to a source s. h ∈ H defines a conditional probability P(T |h) for T ∈ T , which we will sometimes write as h(T ), where h(T ) = 0 means T is untrustworthy according to h, and h(T ) = 1 means T is trustworthy (s fully endorses T ). Following Wolpert's 2018 extended Bayesian framework, our learnerf updates his belief in T relative to H .

Our learnerf will have a probability distribution over his evaluation hypotheses H .

Given the co-dependence of beliefs and evidence, this distribution is updated relative to the stages of T as it develops. This is intuitive; the testimony T should serve as evidence upon whichf updates his beliefs. But the co-dependence tells us thatf updates his confidence in T via these updated beliefs.

Most if not all of us acquire new information from a restricted set of bodies of evidence that push a particular point of view. This is reasonable given the balance rational agents need to find between exploiting already acquired data and gathering more data. In addition, attending to a particular body of evidence can give a sense of community, as has been amply documented in the scholarly literature and the press. But this trade off can lead to a problem in learning: when we rely on testimony to learn and we restrict the testimony we pay attention to, the confirming evidence for the evaluation hypothesis and what it supports threaten to collapse into one. We now turn to see how iterated Bayesian updating in learning from testimony can ultimately lead to a situation where only evaluation hypotheses supporting our restricted evidence are credible and this leads to IB.

To formalize IB and its consequences, we first present a simple experiment to show how the codependence of interpretation and belief leads to bias hardening. To illustrate, suppose thatf considers a consistent dynamic body of testimony T = {T 1 , T 2 , ..., T n , ...} and has two evaluation hypotheses h 1 , h 2 , where the prior probabilities assigned to h 1 and h 2 byf are:

and the evaluation hypotheses assign probabilities to T as it evolves through stages T i as follows:

We can now calculate the probability of T 1 using the general rule for marginal probabilities in 3. Let B bef 's background beliefs; and let the set of all h i , the alternative hypotheses that are consistent with or assigned non-zero conditional probability relative to B (Lampinen and Vehtari, 2001; Tenenbaum et al., 2006; L Griffiths et al., 2008; Tenenbaum et al., 2011) , be the set of evaluation hypotheses h i (so {h 1 , h 2 }, in our example).

Then using (1), (2), and (3), we have: P(T 1 ) = P(T 1 |h 1 ).P(h 1 ) + P(T 1 |h 2 ).P(h 2 ) = .56.

This is our estimation of our belief in the body of evidence T based on what we have so far. We will continue to update the probability of T given new stages T i below by distinguishing prior probabilities P prior and updated probabilities P post . Now suppose there is a new conversational turn in T , a new stage of evidence T 2 . Given our assumptions, P(T 2 |h 1 ) = .8, while P(T 2 |h 2 ) = .2, T 2 is supported by h 1 but not by h 2 -h 1 and h 2 are consistent with their roles on T 1 . Given the dependence of beliefs and interpretation of evidence, T 2 also leads us to re-evaluate our evaluation hypotheses by adapting Bayes' formula to our evidential hypotheses:

Given T 2 , whose initial probability we set to what the posterior calculated for T 1 -i.e., P post (T 1 ) = P prior (T 2 ), we can update our confidence in h 1 as follows:

Thus, we have posterior probabilities for our evaluation hypotheses as well as for stages of bodies of evidence. The similarly updated probability for h 2 now drops to roughly .14. Using the updated values for h 1 and h 2 , we see that T 2 , which includes T 1 , is now even more believable: P post (T 2 ) = .74. Now suppose that a new bit of evidence, T 3 , is added to T . As before, we set P post (T 2 ) = P prior (T 3 ). Given our assumptions about our source functions, P(T 3 |h 1 ) = .8, we have P(h 1 |T 3 ) = .96, while P(h 2 |T 3 ) ≈ 0.04, and confidence in T 3 is also updated: P new (T 3 ) = .776 ≈ .78. Updating h 1 's probability conditional on new evidence T 4 now yields a value of .989 ≈ .99, while P(h 2 |T 4 ) = 0.008 ≈ 0.01. By the time we get to T 5 , the probability of h 1 will have gone to 1, while P(h 2 ) = 0, and P(T 5 ) = .8. In sum, as n increases, the updated probabilities of h 1 go to 1 and P(T n ) → P(T |h 1 ), that is, to the strength of h 1 's support for T . Our co-dependence of belief and evidence suggests a loopy structure (cyclic graph) for updating. However, by exploiting stages, we can disentangle such structures; and efficient approximations are possible in disentangled structures (Murphy et al., 2013) . Proposition 1 below shows a convergence under certain assumptions. Let P n (h i ) be the probability of h i after conditionalizing on T n and P n (T ) the value of T after n conditional updates as defined above. Generalizing our discussion above, we have:

Proposition 1 Suppose testimony T = {T 1 , T 2 , ..., T n , ...}, and supposef 's evaluation hypotheses Hf , with a probability distribution and with h 1 ∈ H such that P(h 1 ) = 0, and P(T n |h 1 ) > .5 and is monotone increasing for all n, while P(T n |h j ) < .5 and is monotone decreasing for all n and for all h j ∈ H , h j = h 1 . Then:

As n → ∞, P n (T ) → limsup(P(T n |h 1 )), P n (h 1 ) → 1 and P n (h j ) → 0 for j = 1 Given the calculations above and using standard updating rules for the probabilities P assigned byf , if P(T i |h 1 ) is monotonic increasing with respect to i and P(T i |h j ) for any j = 1 is monotonic decreasing, then the updates of P(T i ), P(h 1 |T i ) and P(h j |T i ) will follow the pattern of our experiment above and converge to the support of h 1 , 1, and 0 respectively.

We now introduce three important properties of evaluation hypotheses.

Definition 2 An evaluation hypothesis h ∈ H with a probability distribution P makes T potentially trustworthy (h |= T ), if for all n, P(T n |h) > .5 and as n → ∞, P(T n |h) → 1.

We take consistency to be a basic requirement of evaluation hypotheses. PWMC hypotheses generalize consistent hypotheses. But what is their rationale? As T evolves through its stages, it is natural to assume that T i+1 to provide a more complete coverage of the facts than T i . And as T becomes more complete, an evidential hypothsis may assume elements φ that are not mentioned in any T i are in some way incompatible with T . The PWMC condition codifies this in terms an operation akin to the operation of negation as failure in Prolog; if h makes T probability wise model complete, then if T doesn't mention φ then h supports ¬φ to the extent that h supports T . In addition, an agent plausibly has, among the many evaluation hypotheses that she countenances, an evaluation hypothesis h for which the conditional probability of T given h increases as T evolves. Up to now we have taken an evaluation hypothesis to confer a fixed probability on a body of evidence T it supports or discounts. But the support for T might increase (or decrease) as T gets more extended with more and more stages. A potentially trustworthy evaluation hypothesis is a kind of "soundness" or accuracy assumption about a body of evidence. For an agent who remains wedded to a body of testimony such a soundness assumption seems a rational requirement. If potential trustworthiness is a soundness requirement then PWMCness is a kind of completeness requirement. Together they furnish the rational justification for an agent to restrict his learning to a particular body or particular bodies of evidence, something that we've argued most people in fact do and do so with rational justification.

Proposition 2 Let T be a set of consistent bodies of testimony and let H be a set of evaluation hypotheses for T , with h 1 ∈ H and h 1 |= T , for some T ∈ T . Let the priors on h i ∈ H , h i = h 1 be as in Proposition 1 and let the probabilities of h i , h 1 ∈ H be updated on T . Then:

As n → ∞, P n (T ) → 1.

(1)

Suppose in addition, h 1 is PWMC for T and T |= T ′ . Then:

To show (1), note that P(T 1 |h 1 ) > .5 and since as n → ∞, P n (h 1 ) → 1, after a certain point P(T n |h 1 ) is monotone increasing. Then by Proposition 1, P n (T ) → P(T n |h 1 ). Since h 1 makes T potentially trustworthy, as n → ∞,

for each i, and the expected probability of T ′ will decrease strictly monotonically over n, as P n (h 1 ) → 1. So as n → ∞, P n (T ′ ) = 0. Note that our agent may have many evaluation hypotheses and the result of Proposition 2. Cruciallyf has updated his beliefs only on T . But this matches our intuitions about what agents actually do. As long as the co-dependence between background beliefs and bodies of evidence holds and certain bodies of evidence are supported more than others, belief in some bodies of evidence T ∈ T will be strengthened, while belief in bodies of evidence in conflict with T or just different from T will be weakened. Importantly, this can happen merely by T i repeating content already in T k for i > k. Such repetitions of content are commonplace on social media sites and news sites that broadcast continuously. In addition, the assumption of a dogmatic evaluation hypothesis is actually mild; it reflects an agent's mistrust of bodies of evidence other than the ones he relies on-a rather common situation.

Proposition 2 impacts the marginalization of new data, because if its assumptions are met, as P n (T ′ ) → 0,f discounts evidence from T ′ , despite the presence of evaluation hypotheses supporting T ′ .

Proposition 3 Suppose evidence φ such that T ′ |= φ, T |= φ and T, T ′ , andf 's evaluation hypotheses are as in Proposition 2 andf conforms to Bayesian learning. Then: hspace*.5in As n → ∞, P n (φ) → 0.

Sincef conforms to Bayesian learning, the marginal probability for φ is based on Equation 3 and the set of hypotheses h i in Equation 3 is the set H that forf pronounce on testimony that mentions or asserts φ. By Proposition 1, as n → ∞, P n (h 1 ) → 1. By

, the result follows.

In this situation,f assigns no credence to φ. The prior beliefs off may so limit the alternative hypotheses h i such that even an actual fact φ will have a marginal probability of 0;f will discount φ completely. Now consider general learning in this situation, defined in Wolpert's 2018 extended Bayesian framework via Bayes's formula below.

To learn a hypothesis h,f 's estimation of h at some stage should be closer to the objective or ideal assignment (posterior) h p to h, than his prior probability for h. Similarly for marginal probabilities: P n (x) should track x p , the posterior of x, given a random sampling of X . We consider loss functions L(P n (h), h p ) and L(P n (x), x p ). The greater divergence between the ideal posterior probability and the Bayesian subjective estimation of that probability, the worse will be the score forf 's learning. We say thatf cannot learn h if additional evidence does not eventually decrease loss; i.e. we cannot

Consider e such that T ′ |= e and e confirms h. So the true posterior P p (h|e) > P(h), with P(h) the prior on h. Supposef 's evaluation hypotheses and probabilities have been updated via T as in Proposition 2. By Proposition 3, as n → ∞, P n (e) → 0. In the limit, Bayesian learning as specified by equation (4) simply isn't defined when P n (e) = 0.

So assuming e is discounted as evidence in updating, we set P(h|e, T n ) = P(h|T n ). But

Proposition 4 is a formal statement of IB in a first order setting. It shows that under certain conditions,f will be incapable of learning any hypothesis that involves a dependence on testimony not in T , upon whichf has formed his beliefs.f is interpretively blind to any possibilities outside of T .

It's not unreasonable to rule out new evidence from unreliable testimony, provided the assignment of one's evaluation hypotheses to the testimony is reasonable. But nothing in our discussion above forces the evaluation hypotheses to be be reasonable. Without any constraints,f 's evaluation hypotheses may rule out evidence that is completely grounded in reality and comes from testimony that an ideal rational agent would trust.

To solve this problem, we need to correct the background beliefs B. Ideally, a rational agent should control for the biases in testimony by consulting several different bodies of testimony. However, B cannot be corrected itself by evidence, because that evidence is already discounted if it conflicts with B. Very clearly, background beliefs can be a source of bad epistemic biases, and they can prevent straightforward corrections to improve one's beliefs as Bayesian learning would have us do.

Hierarchical Bayesian models were designed to address this problem (Gelman et al., 2013) . In hierarchical Bayesian models, a Bayesian learning model like the one we have discussed in Section 3 has certain parameters; the one parameter we have is our evaluation hypotheses providing the reliability of testimony. At a second level of the hierarchy, we could have a Bayesian learning model concerning evaluation hypotheses, in which we could detail factors that would allow us to estimate reliably the accuracy of an evaluation hypothesis. Abstractly, we would have evaluation hypotheses about evaluation hypotheses that would discuss factors like the consistency or the predictive accuracy of a testimony source, or the extent to which testimony from other sources agrees with its content. One could also require a longer or more thorough exploration of the data about the phenomenon before the agent's restricting himself to a small subset for exploitation (once again an application of the work in Cesa-Bianchi and Lugosi (2006) ). All of these ideas and more have been proposed.

Simply requiring evaluation hypotheses that obey exogenous constraints, however, begs the question of whyf should accept them. In fact, the interdependence of tes-timony, new information and background beliefs can make the resort to higher order parameters to resolve IB a failure because a body of dynamic testimony T , when directed by a conversational agent for the purposes of persuading and keeping his audience, can react to and attack not only a conflicting body of testimony T ′ but also sources supporting it. This behavior provides arguments for or against not only first order evaluation hypotheses, as we've seen with the notion of consistency, but also for higher order functions and in fact sequences of evaluation hypotheses.

To formalize this picture, we assume a hierarchy of sets of evaluation hypotheses where,

Higher order parameters at level n + 1 are related to probabilities to evaluation hypotheses at level n via a notion of rationality.

for some normalizing factor λ.

A rational set of sets of evaluation hypotheses is thus one in which the probability of the evaluation hypotheses at one level reflects what the higher levels say about it. Henceforth, we assume that agents' sets of evaluation hypotheses are rational.

Given rational H = {H 1 , H 2 , ..., H n }, we define a H n sequence σ ∈ ∏ n i=1 H i of consistent evaluation hypotheses to support T (σ| ≈T ) (or that make T potentially trustworthyσ |= T ) iff the H 1 element h 1 σ of σ is such that P(T j |h 1 σ ) is eventually monotone increasing with respect to j (converges to 1 as j → ∞) and every element of σ has non-0 probability given H . Conversely, we say that T | ≈σ iff for each element h i σ of σ P(h i σ |T j ) is eventually monotone increasing for all stages T j . We note that σ| ≈T → T | ≈σ.

Let σ 2 to be the subsequence of σ such that σ 2 = σ ↾ (∏ n i=2 H i ). For h 1 ∈ H 1 , σ 2 (h 1 ) signifies the support h 1 receives from the higher order functions in σ via Definition 3.

Definition 4 Given H = {H 1 , H 2 , ..., H n }, we say that an H n sequence σ undercuts T iff for any h 1 ∈ H 1 if P(T |h 1 ) > .5, σ 2 (h 1 ) ≤ 1 − P(T |h 1 ).

Definition 5 φ disagrees with T ′ just in case P(T |φ) < P(T ). Definition 7 T is argumentatively complete iff: (i) (T ′ |= φ and Disagree(φ, T )) → Attack(T, T ′ ); (ii) If T n |= φ but P(T n |φ) ≥ P(T n ), then T n+1 |= φ. (iii) for any T undercutting H m sequence σ, ∀h m+1 ∈ H m+1 such that (P(h m+1 |T ) > .5, h m+1 (σ) = 0) (iv) ∃H n sequence σ such that h 1 σ |= T.

Assume that T is argumentatively complete. Then ∃H n sequence σ such that h 1 σ |= T . Now assume T n |= φ for some φ for all stages n. But then P(T n |φ) < P(T n ) for each stage T n of T . But then T and φ disagree and so T attacks φ. By the definition of attack, P(φ|h 1 σ 1 ) = 1 − P(T |h 1 σ 1 ). So h 1 σ is PWMC.

Proposition 6 Let T be argumentatively complete with a rational set of evaluation hypotheses H with ∑ h 1 ∈H 1 P(h 1 ) = 0 and probabilities updated on T .

As n → ∞, P n (T ) → 1.

(1)

In addition suppose there is a T ′ ⊆ T .

As n → ∞, P n (T ′ ) → 0.

(2)

We first show (1). Since T is argumentatively complete, ∃σ ∈ H such that h 1 σ |= T . We need to show that for some such h 1 σ , P(h 1 σ ) = 0 relative to H . Suppose that

Thus, all the non-0 probability mass of H falls on T undercutting sequences σ i . But for each such T undercutting σ i of length m, since T is argumentatively complete, there is an evaluation hypothesis h m+1 supported by T such that P(σ i |h m+1 ) = 0. Since H has only finitely many levels, at some level k all T undercutting sequences σ j get 0 probability. This, together with the fact that ∑ h 1 ∈H 1 P(h 1 ) = 0, contradicts the assumption that P(h 1 σ ) = 0. Since T is argumentatively complete, any sequence supporting any h 1 where P(T |h 1 ) < P(T ) will eventually get probability 0; so ∑ {h 1 :P(T |h 1 )≥P(T )} P(h 1 ) = ∑ h 1 ∈H 1 P(h 1 ). Moreover, as P n gets updated, as n → ∞, {h 1 : P n (T |h 1 ) ≥ P n (T )} → {h 1 : h 1 |= T }. The conditions on first order evaluation hypotheses in H of Proposition 1 are now met. By Propositions 1 and 2, as n → ∞, P n (h 1 σ ) → 1, P n (h 1 i ) → 0 for i = 1. By Proposition 2, P n (T ) → 1. To show (2), by Proposition 5, h 1 σ is also PWMC for T . As n → ∞, since P n (h 1 σ ) → 1, P n (T ′ ) → 0.

Claim 2 of Proposition 6 shows that P n (T ′ ) → 0. Then apply Proposition 4.

Argumentatively complete testimony thus collapses the case of higher order Bayesian frameworks to our first order setting. What is troubling about IB is that our learnerf may hold onto an argumentatively complete T regardless of how inadequate it is in the eyes of others or standard epistemic criteria; an argumentatively complete theory will always eventually find a reply to any attack or any doubtf might acquire.

Argumentatively complete testimony isn't just an abstract concept; many social media and news sites already approximate this condition. Outlets like NewsMax or One Amercan News Network that have a particular political bias will attack the credibility of stories from other bodies of testimony that have gone against a narrative they were and are promoting; darker conspiracy spinning websites like those promoting QAnon will attack arguments against their theories once they become aware of them. 1 In anecdotal support of our claims, consider Michelle Goldberg's "It's Marjorie Taylor Greene's Party Now" New York Times, 2/2/2021) description of a group in IB: "American conservatism -particularly its evangelical strain -has fostered derangement in its ranks for decades, insisting that no source of information outside its own self-reinforcing ideological bubble is trustworthy."

A crucial component of argumentatively complete testimony T is that it promotes evaluation hypotheses that both make T eventually trustworthy but also PWMC for T . Sources like the New York Times embody this in their slogan all the news that's fit to print, but there's a commercial reason for this outcome; news sites and social media are out to capture market share and so they naturally promote themselves as accurate and complete at least in a certain domain. The nature of contemporary testimony leads agents naturally to a situation where IB occurs.

How general are the results in Propositions 6 and 7? Wolpert (2018) argues that PAC, Statistical Physics Framework, VC, and supervised Bayesian learning are four different instantiations of his extended Bayesian formalism, which we use. Thus our results should hold for other frameworks.

IB is an epistemological bias that is clearly related to confirmation bias (Lord et al., 1979; Nickerson, 1998; Oswald and Grosjean, 2004) , in which agents interpret new evidence in a way that confirms their beliefs, and to the framing biases of Kahneman (1975, 1985) . People tend to see in the evidence what they believe. These forms of bias, however, concern how beliefs and bias influence interpretation, painting only part of the picture of IB (see also Asher and Paul (2018) ). Further, unlike much of the psychological literature which finds epistemologically exogenous justifications for this bias (Dardenne and Leyens, 1995) , we show how IB is a natural outcome of Bayesian updating, rational resource management and the belief interpretation co-dependence.

IB is a concrete application of the work on bandits in, determining optimal allocation of resources to the exploration and exploitation of sources Whittle (1980) ; Lai and Robbins (1985) ; Banks and Sundaram (1994) ; Burnetas and Katehakis (1997); Auer et al. (2002) ; Cesa-Bianchi and Lugosi (2006) ; Garivier and Cappé (2011) . It is also related to work on generalization in machine learning. Epistemic biases affect generalization and learning capacity in ways that are still not fully understood (Lampinen and Vehtari, 2001; Zhang et al., 2016; Kawaguchi et al., 2017; Neyshabur et al., 2017) . Zhang et al. (2016) show that standard techniques in machine learning for promoting good epistemic biases and generalization-training error minimization, regularization techniques like weight decay or dropout, or complexity measures used to minimize generalization error (the difference between training error and test error)-do not necessarily lead to good generalization and test performance. Argumentatively complete testimony T incorporates an adversarial attack mechanism against any good epistemic practices that might discount T . It's this mechanism that guarantees IB.

The argumentation literature (Amgoud and Demolombe, 2014; Dung, 1995) is also relevant to IB. If testimony T is argumentatively complete, then T always provides a counterargument to an attack against T -much like an acceptable argument in Dung (1995) . In addition, however, an argumentatively complete T also supports higher order evaluation hypotheses that support hypotheses that support T . There are also important connections to the literature on trust (Castelfranchi and Falcone, 2010) ; in our set up learning agents trust certain sources over others, and our higher order setting invokes a hierarchy of reasons. Nevertheless, the argumentation and trust-based work of which we are aware is complementary to our approach. An argumentation framework takes a possibly inconsistent belief base and imposes a static constraint on inference in such a setting. Similarly, trust is typically modeled in some sort of static modal framework. By contrast, ME learning games and the whole Bayesian framework are dynamic, with beliefs evolving under evidence and game strategies evolving under agent interaction. It is this dynamic evolution that is crucial to our approach and, we think, to modeling agents and learning. In sum, we are not looking at the problem of consistency, but rather the problems of entrenchment and bias.

IB is a result about learning. IB is a suboptimal but natural outcome of the way contemporary bodies of evidence are set up and how humans interpret them. Given our set up, everything turns on what body of evidence on which to update and with which evidential hypotheses.

If IB is suboptimal, its effects are still more worrisome, because agents in the grip of IB are often unwilling or incapable of changing their beliefs so as to be able to learn. Of course, our learner might just be happy with T ; perhaps he needs no more accurate or more truthful body of testimony. He may not be interested in learning anything beyond what T presents him with. In this section, however, we assume a learner who might be interested in learning but has difficulting escaping his IB prison. We assume a rational learnerf who updates according to his evidential hypotheses; so if he has an evidential hypothesis that confers a high probability on some T , he will update on T . We've seen thatf can get IB when he unduly restricts the bodies of evidence which serve as the basis of update or when he attends to an argumentatively complete testimony. So key to removing IB is to getf to change his hypotheses and consider other evidence that that to which he is wedded.

Anecdotally, we have a lot of evidence that IB is hard to escape 2 In general, however, we lack a precise analysis of its difficulty. In this section, we introduce a game theoretic method that shows IB is not only hard to defeat but it can even be hard to detect (leading to self-deception). We will see that the choice of epistemic paradigms is important.

To motivate our approach, consider how an actual conversation might go between our learnerf in the grip of IB and a person E who wants to correct his problem. E might questionf 's reasons for believing some proposition φ; she might try gettingf to consider different bodies of evidence T ′ that might disconfirm φ.f might accept T ′ or he might argue against it-by providing, for example, reasons why T ′ is not trustworthy or why the arguments supporting T ′ are faulty. E might attack those arguments or provide new evaluation hypotheses for consideration. Our ME games formalize this interaction.

In an ME learning game G = ((V 0 ∪V 1 ) ∞ , Win), the two players, our investigator E and our Bayesian learnerf , construct a larger "conversation" by consecutively playing finite strings from the vocabulary V 0 and V 1 respectively. Win specifies the winning condition of E. The vocabularies V i of an ME learning game G consists of sequences of evaluation hypotheses (with some abuse of notation, we'll take a single h n j to be a one place sequence) and a predicate ACCEPT. ACCEPT means thatf accepts the last suggestion by E and confers upon it a non zero probability mass. Our ME learning games are subject to several constraints.

A. Knowledge first (Williamson, 2002) : this is a constraint from formal epistemology;f only adds a sequence σ to ∏ n 1 H î f for H î f ∈ Hf if he has no argument that attacks σ-in other words no evaluation hypothesis h n+1 ∈ H n+1 f such that h n+1 (σ) = 0.

B. The Jury in an ME learning game is epistemologically competent; i.e. it sanctions only evaluation hypotheses that advance learning.

C. E may only add sequences of evaluation hypotheses sanctioned by the Jury. We assume this to be a finite set H J .

D. Both players must only propose consistent and rational sequences.

2 See Thompson, cited in note 2.

E.f has learned from some body of evidence T , which is common knowledge.

F.f may only refuse a proposal of E in the higher order setting, if he has a reason to do so.

We define a sequence σ ∈ ∏ n 1 H i to be positive if for each element h m+1 and h m of σ h m+1 (h m ) >> 0. A sequence σ nullifies a sequence σ 1 , if for all m and for h m 1 of σ 1 , h m+1 of σ is such that h m+1 (h m 1 ) = 0. We can have two sequences each one nullifying the other. This formally represents an n round argument, with each round j + 1 offering a counterargument to the argument of round j. We will say that a hypothesis h 1 is T positive if h 1 is positive and P(T |h 1 ) = 1

We now define the moves of a game G, in which we suppose a body of evidence T thatf has attended to and a body of evidence T ′ inconsistent with T . E plays first, thenf then replies. The game ends iff plays ACCEPT, which implies that he adds a hypothesis h 1 * to H 1 f , with a non-0 probability mass and with with high P( We note that if move (m6) occursf , assigns h 1 * and T ′ a non-0 probability mass and updates with evidence T ′ . In which case the game ends.

Suppose that in an ME learning game G, E's winning condition is simply to discover thatf is interpretively blind, if he is. Call this condition IB. We establish the complexity of E's attempt to achieve IB. The first order case with a finite H where the game is restricted to moves m1,m2,m3, is rather trivial. More interesting is the case of an ME learning game G = ((V 0 ∪V 1 ) ∞ , Win) with Win = IB and in which E andf play higher order evaluation hypotheses.

Proposition 8 Suppose an ME learning game G = ((V 0 ∪V 1 ) ∞ , Win) with Win = IB in whichf plays moves described in (m4)-(m7). Thenf is not interpretively blind iff play stops at some finite ordinal n.

Suppose that in the play of G,f accepts at some level n to add the sequence of evaluation hypotheses proposed by E. Then by the construction of the sequence and the requirement of coherence (constraint D), this confers upon some evaluation hypothesis s * 1 a non zero probability such that P(T ′ |h 1 * ) = 1, where T ′ is incompatible with the body of evidence T . By accepting,f will have an evaluation hypothesis h 1 * with non zero probability such that P(T ′ |h 1 * ) = 1, where T ′ is incompatible with the body of evidence T , whichf has proposed as a source of learning (constraint E). Now whenf updates his belief in T he must do so with respect to h 1 * , and he must now update his confidence in his evaluation hypotheses with respect not only to T but also T ′ . In that case, P(h 1 * |T n , T ′ n ) → 0 and P n (T ′ ) → 0. As a result,f will be able to learn from T ′ , and so he is not interpretively blind with respect to T .

If there is no stopping point at any finite ordinal, then E's is never able to getf to accept a T ′ positive hypothesis. In which case,f continues to only update on T and by Propositions 6 7,f is interpretively blind.

Suppose E's winning condition for an ME learning G, is to getf to accept a T ′ positive evaluation hypothesis. Call this winning condition for E P (for persuasion).

Corollary 1 Suppose that in an ME learning game G with Win = P . The complexity of Win is an R.E. set. If Win = IB then Win is co-r.e.

If P is the set of all finite plays, IB is its complement to a countable union of closed sets of plays in (V 0 ∪V 1 ) ω that never pass through an ACCEPT move.

Proposition 9 Suppose an ME learning game G with Win = P andf as described in Proposition 6. Then E has no winning strategy in G.

Proposition 7 impliesf 's evaluation hypotheses are updated on an argumentatively complete body of evidence T . When implemented via an ME game G, the sequence of evaluation hypotheses in Proposition 6 provide a winning strategy forf . Suppose E proposes an h 1 supporting e that is inconsistent with T . Even if E generates a suitable sequence of higher order T ′ positive evaluation hypotheses h 1 , h 2 , h 3 , . . ., given Constraint A above, f will only accept an evaluation hypothesis if he has no argument against it. But as T will eventually supply such an argument,f can always counter E's proposals. So she has no winning strategy.

Not only is IB computationally complex (Corollary 1 shows it is not computable but Π 1 ), Proposition 9 shows formally that even if E has rationally compelling arguments to show thatf is better off (his payoff or reward is higher) in accepting her proposed sequence of evaluation hypotheses,f can rationally resort to T to counter her argument. Extracting someone from higher order IB is thus impossible by purely epistemic means. There is no way of getting someone, even a rational agent, out of higher order IB by purely epistemic arguments, given our assumptions. This pessimistic is borne out empirically: some people in the grip of right wing conspiracy theories in the US were dying of Covid19 in December of 2020 and January 2021 but continued to refuse to believe that it was that disease that was killing them-despite all the evidence and arguments they were given, they refused to let go of an obviously faulty but argumentatively complete T .

Of course, people sometimes do change their minds and do escape the grip of argumentatively complete theories, many times for epistemically exogenous reasons. 3 But by challenging one of our assumptions, rational agents can of course also reject IB. The weak link in our argument is assumption A, the "knowledge first" assumption. Perhapŝ f should accept evaluation hypotheses even if T attacks them. More likely,f should not accept all attacks equally; he should be skeptical of any body of evidence T that promotes PWMC for T and T eventually trustworthy evaluation hypotheses while attacking any point of view at variance with it.

We now explore the play between E andf in an ME learning game G where Win = P beforef has accepted enough of the argumentatively complete T to close off learning from alternative bodies of evidence. Suppose T is argumentatively complete but comes in stages; if T ′ i attacks T i , then T i+1 but not T i attacks T ′ i . That is, an argumentatively complete T reacts to attacks but does not forsee all attacks in advance. Suppose a set of consistent first order evaluation hypotheses H 1 = {h 1 1 , h 1 2 , ...}, with P(h 1 1 ) = .6, P(h 1 2 ) = .4, and P(T i |(h 1 ) = 1 = P(T ′ i |h 2 ). Now suppose T ′ 1 ∪ T 1 is inconsistent and E proposes h 1 2 since h 1 2 |= T ′ i . Since the h 1 i are consistent, P(T 1 |h 1 2 ) = 0 = P(T ′ 2 |h 1 1 ). At this point,f could accept E's proposal under constraint (A), G ends and E wins.f will continue to update over stages T and T ′ with the marginal probabilities P(T i ) = .6 and P(T ′ i ) = .4 remaining stationary.

On the other hand,f may decide to wait to see what the next stage T 2 of T brings. As T is argumentatively complete, T 2 will attack T ′ 2 , and add a nullifying h 2 ∈ H 2 supported by T 2 . Shouldf accept h 2 , the probability of h 1 2 will go to 0 in H . But now suppose we have a constraint, Discount,that discounts any nullifying sequence from T . It would be unreasonable forf to wipe out alternatives in the face of this level of uncertainty; at this stage, P(T 2 ) = .6 and P(T ′ 2 ) = .4. Summarizing:

Proposition 10 Suppose an ME learning game G with constraint A replaced by Dis-count and with Win = P andf as described in Proposition 6. E then has a winning strategy in G, and IB does not arise forf .

Interpretive blindness results from a dynamic, iterative process whereby a learner's background beliefs and biases lead her to update her beliefs based on a body of testimony T , and then biases inherent in T come back to reinforce her beliefs and her trust in T 's source(s), further biasing her towards these sources for future updates. We have introduced and formally characterized IB. We have shown that IB can prevent learning even in higher order Bayesian frameworks for learning from argumentatively complete testimony, despite the presence of constraints designed to promote good epistemic practices. We also shown that IB is computationally complex as a co-r.e. set via a game theoretic analysis, and that an agent may rationally remain in IB in the face of epistemic arguments. Our game theoretic analysis can also be extended to cases where the agent falls out of IB but then is a recidivist and becomse a prisoner once more. We leave that for future work. Investigating IB alas is not just an academic enterprise. IB really does happen, with sometimes tragic or dangerous results. We think a careful formal analysis is urgent for society. Finally, we note that while we have focused on IB as a problem for learning from testimony, the problem it raises for learning extends to any case in which we do not have unmediated access to ground truth and our data is "theory laden" Hanson (1958) .

An argumentation-based approach for reasoning about trust in information sources

Strategic conversation under imperfect information: epistemic Message Exchange games. Logic, Language and Information

Finite-time analysis of the multiarmed bandit problem

Switching costs and the Gittins index

Optimal adaptive policies for Markov decision processes

Trust theory: A socio-cognitive and computational model

Prediction, learning, and games

Confirmation Bias as a Social Skill

On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games

The KL-UCB Algorithm for Bounded Stochastic Bandits and Beyond

Bayesian data analysis

Patterns of discovery: An inquiry into the conceptual foundations of science

Bayesian models of cognition

Asymptotically efficient adaptive allocation rules

Bayesian approach for neural networks-review and case studies

Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence

Loopy belief propagation for approximate inference: An empirical study

Exploring generalization in deep learning

Confirmation bias: A ubiquitous phenomenon in many guises

Confirmation bias

Theory-based Bayesian models of inductive learning and reasoning

How to grow a mind: Statistics, structure, and abstraction

Judgment under uncertainty: Heuristics and biases. In Utility, probability, and human decision making

The framing of decisions and the psychology of choice

Multi-armed bandits and the Gittins index

Knowledge and its Limits

The relationship between PAC, the statistical physics framework, the Bayesian framework, and the VC framework

Understanding deep learning requires rethinking generalization