Interpret Bayesian probability as frequentist probability?

Question

It is usually said that the Bayesian probability is a subjective concept, quantifying one's degree of belief in something, while the frequentist probability is the the fraction of certain outcomes when observation is conducted many times (either in space or in time).

My simple questions are:

1. If we consider statistics as a science, instead of a philosophy or even a psychology, isn't it necessary that a statistical prediction (or the outcome of a statistical calculation) be tested in the real world (at least in principle), as physical sciences usually do?

2. If statistical calculations need to be tested (at least in principle), is the method of test the same as physical sciences, namely, through experiments and observations?

My thinking is that if the answer to both questions are yes, then the Bayesian probability can be interpreted as frequentist, and ultimately the two are not so different.

You may stop reading here and give your comments.

For example, suppose a geologist says that "there is a 0.9 probability that dinosaur was extinguished by a meteorite impact some 70 million years ago", a Bayesian may interpret the number 0.9 as the "degree of confidence" of this geologist.

To me, this "degree of confidence" is too "void" to understand. I would prefer to understand it as something like the following:

Given the current knowledge of the world, especially of the paleontological and geological discoveries, if there is a God who can create the universe for many times, then among the Earths (within these universes) that have been created and share the same features as our present Earth, dinosaurs are killed by meteorite impact in around 90% of them.

Well, of course, apparently there is not a God who is willing to do such kind of "experiment" for us (we may observe many extraterrestrial worlds instead, though that's also extremely difficult if not impossible in the imaginable future). But without such kind of "thought experiment", I can hardly understand what does the number 0.9 mean, and, how to prove that it is correct.

If the number 0.9 is not testable, even in principle or in imagination, or it is not meant to be tested, then I think it should not be spoken out in the first place.

But if you agree that "degree of confidence" also needs to be tested, then isn't it equivalent to the frequentist understanding of probability?

Note that in this example, the prior knowledge of paleontological and geological discoveries simply helps selecting out a subsample of the (hypothetical) universes in the "frequentist" approach to probability. I mean, if I were a frequentist (I am not sure whether I am, since I don't quite understand yet), I would not mind making use of prior knowledge, since using prior knowledge is not in contradictory with my "belief" in the frequentist approach.

Thanks in advance if you could point out errors or blind spots in my understanding of the two approaches!

Note: I asked this question at mathoverflow but was closed. They believe my question is too philosophical, so I ask it here.

Answer 1

1. If we consider statistics as a science, instead of a philosophy or even a psychology, isn't it necessary that a statistical prediction (or the outcome of a statistical calculation) be tested in the real world (at least in principle), as physical sciences usually do?

Ultimately every scheme has some basis in experience, so tests are helpful in principle and necessary in the beginning of construction of such scheme (people played hazardous games and only then the probability theory came around).

However, presently the theory of probability (and statistics) is not a natural science. It is much closer to mathematics and logic and perhaps should be placed next to them. These schemes do not depend on tests for us to believe in their validity.

The probability calculation takes input data and assumptions, and produces probabilities.

What is usually tested here, is not the correctness of the calculation itself, but the validity of the initial data/assumptions.

1. If statistical calculations need to be tested (at least in principle), is the method of test the same as physical sciences, namely, through experiments and observations?

Yes, but again, in doing the test, most often we question the assumptions (hypotheses), not the calculation itself.

Answer 2

First of all, I know this is more than an year old. However, I can’t leave this without making some observations, which may or may not answer the question (which the poster may or may not care about anymore).

The reason to make this post is two fold. One is to address your direct questions, and the other to address your thought experiment about dinosaurs.

1. If we consider statistics as a science, instead of a philosophy or even a psychology, isn't it necessary that a statistical prediction (or the outcome of a statistical calculation) be tested in the real world (at least in principle), as physical sciences usually do?
2. If statistical calculations need to be tested (at least in principle), is the method of test the same as physical sciences, namely, through experiments and observations?

I think it is a fundamental mistake to equate statistics with other “hard” sciences. I say statistics is a kind of “observational” science. It is more like Geology than Chemistry. For a Geological theory one can make hypotheses but only look for clues in order to deduce the correctness of the hypotheses and so on, unlike a Chemistry theory which can be readily tested in laboratory. (I am talking about “classical” Chemistry here, not the blurry Quantum Chemistry and the like). The theory of plate tectonics is a great example of how observational science works. One can never experimentally recreate it, and simulation is not the same thing.

Further, as modern Theoretical Physics has shown, things are kind of probabilistic in nature, at least at the sub atomic level, contrary to what we experience at macroscopic level. The experience humans have at the scale where we can manipulate object teaches us that things are fully observable and deterministic. Although the reality is not quite so, it is so close enough that we can ignore the discrepancies (not that we can perceive it anyway).

In some sense, the Frequentist approach believes in a deterministic world. Per that theory, all one needs to do is observe/measure enough number of times to get close to the “true” parameter defining the event. This kind of conflicts with the nature of the…err…nature as espoused by modern Theoretical Physics, so people might be a bit jittery in accepting it. It gets more puzzling if you imagine that every event that could ever happen has a predestined parameter that defines it.

To put it very loosely, the world is filled with randomness and the purpose of statistics is to quantify that randomness, which effectively precludes physical experimenting to verify because that would be contradictory to the idea of randomness.

For example, suppose a geologist says that "there is a 0.9 probability that dinosaur was extinguished by a meteorite impact some 70 million years ago", a Bayesian may interpret the number 0.9 as the "degree of confidence" of this geologist.[…] Given the current knowledge of the world, especially of the paleontological and geological discoveries, if there is a God who can create the universe for many times, then among the Earths (within these universes) that have been created and share the same features as our present Earth, dinosaurs are killed by meteorite impact in around 90% of them.

This was the real motivation to make this post. I think this formulation is quite wrong, not to mention simplistic and even a bit dishonest. Bringing something clearly imaginary like God into the discussion of science… :-)

Formulating a Matrix-esque scenario complicates the problem and diverges into all kinds of tangents. I would look at the number 0.9 this way. Lets us go back to observational science part. Let us say there are quite a few geological, astronomical, meteorological… events that could have occurred. Let us say each event leaves behind a telltale sign, a "residue". Let us say geologists gather a total of a million such residue. The number 0.9 could be interpreted as 900,000 of the residue pointing to a meteorite strike. Elementary probability theory suffices if you ask me.

Answer 3

First, to address the original question, I don't see the distinction between question 1 and question 2, and the answer would be: yes, statistical predictions need to be tested against reality. This is routine in statistics, and there are methods for doing some checking with existent data (for example, hold-out test sets, cross-validation, permutation tests, etc), as well as the experimental test of gathering new data.

This applies to both frequentist and Bayesian methods. If the OP means "let's do a bunch of experiments using frequentist and Bayesian techniques to see which performs better in the real world", then things get tricky because Bayesian methods are more flexible because of prior probabilities.

For example, if you use uniform priors in linear regression, you will tend to get answers very close to frequentist linear regression. Uniform priors are arguably not sensible, though. And if you have principled priors, you will get better results.

The OP believes that prior knowledge is not incompatible with frequentism. This is a core part of the (vociferous and long-running) Bayesian-frequentist debate: frequentist procedures assume no prior knowledge. A frequentist no doubt uses prior knowledge to form a hypothesis or to choose a procedure but the procedure itself doesn't incorporate any prior knowledge.

To address the comments:

Feller (1950!) is mentioned but the field of statistics has moved on since then. It would be informative to read de Finetti (1995) to see Bayesian statistics as a continuous version of logic. (Prior = Axiom, etc.) And there are now three Bayesian schools regarding priors: the traditional "subjective", as well as "objective" and "empirical".

Jaynes may be strident, but he was essentially replying in kind. Statisticians lost positions over being Bayesians, and the dismissive attitude of many frequentists was over-the-top, even if Feller managed to keep farther from vociferous than Jaynes.

I'm not sure what behavioral economics "proves" about the frequentist/Bayesian debate.

One of the comments mentions quantum mechanics, which reminds me that many (frequentists) seem to believe in a force of nature called Chance. They speak of things being caused by Chance. Perhaps they are two steps ahead, to the degree that quantum mechanics might affect the macro world. But if we look only one step ahead, Bayesians attribute to "uncertainty" what frequentists attribute to Chance, and that makes a lot more sense to me: we get distributions for things not because of some mysterious force (Chance) but because of things like measurement error or because our model is leaving out variables and our results are integrating over them -- creating distributions rather than points.

Answer 4

Bayes Law, the principle that Bayesian statistics admits and classical statistics avoids is a true statement about what you are calling frequentist probability. So the distinction is moot.

The real question is whether predictions from psychology can ever really represent probabilities, or whether they are so "meta-fuzzy" that one dares not use them. That is not a question for statistics, it is a question for each domain of application to answer for itself.

In general, it probably only makes sense to begin to rely upon psychological statements of probability once one has 'tuned' the psychology extensively. So yes, all of that outcome testing is in order. But it is in order before using the measure, to confirm that you know the uncertainty in the predictions given by the source is not so high as to warn you off further math.

At that point the definitions converge, and you can do real statistics without arguing interpretation. But at that point you already know most of your answers.

Answer 5

I debated leaving this posting alone, but I didn't like any of the answers. Of course, my liking something must be the defining methodology to determine truth!

My difficulty is that the conclusion doesn't follow from the premises in the questions.

My simple questions are:

1) If we consider statistics as a science, instead of a philosophy or even a psychology, isn't it necessary that a statistical prediction (or the outcome of a statistical calculation) be tested in the real world (at least in principle), as physical sciences usually do?

2)If statistical calculations need to be tested (at least in principle), is the method of test the same as physical sciences, namely, through experiments and observations?

My thinking is that if the answer to both questions are yes, then the Bayesian probability can be interpreted as frequentist, and ultimately the two are not so different.

The answer to 1 is yes depending on how you mean "as physical sciences do?" Statistics are often tested by numerical simulation which is a physical simulation in that it is performed using pseudorandom numbers. They are also tested through validation in specific studies, regardless of whether hard or soft.

The answer to 2 may be no, again depending on what you man about experiment and observation. Many sciences have to use observational studies and cannot perform experiments. The public might get angry if meteorologists created a category five hurricane just to see if a hypothesis about it was false. They may get even angrier if it is to test the sufficiency of building materials in Miami. Finally, some statistics don't really need to be "tested," because if their assumptions are met in fact and if their properties are understood, then a test doesn't serve any purpose except to waste resources.

What does not follow is that Bayesian probability can be interpreted as Frequentist and they are not different. This is not remotely true unless you have a sample size that is approaching infinity. In that case, their predictions will match. Subjective beliefs will converge to objective realities and bad objective models will vanish so that the final models will produce identical predictions.

Consider the cookie story in https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval

What is important here is to note that in Winstein's first table, the probabilities sum vertically. This is a Frequentist table. In his fourth table, the same data creates probabilities but they are summed horizontally.

Consider the case in Winstein's first table and assume the null hypothesis is that the type of cookie jar is class A. There is a 1% chance of observing a cookie with no chips given that the null is true. Now consider Winstein's fourth table. Bayesian methods have no null hypothesis to condition on. As a result, the Bayesian probability that the class of cookie jar is A given that no chips were in the cookie that was drawn is 1.9%.

These are very different probability statements. The former is Pr(data|model) whereas the latter is Pr(model|data). This is not at all the same thing. The Frequentist is, in a sense, trying to minimize the maximum possible damage that could be done from having no prior information of using a model to make a wrong choice. The Bayesian is trying to minimize the average damage, using all available information, from using a model to make a wrong choice. The Bayesian can do this because the data is choosing the model rather than the other way around as the Frequentist is.

Consider how the two would solve the simple linear regression problem ax+by+cz+d+e=0, where e is some error term, (x,y,z) are random variables and (a,b,c,d) are unknown constants. The Frequentist would assert that the model is the true model, using Fisher's "no effect hypothesis," would assert that a=b=c=0 as the null. If that is rejected, then the estimated parameters would be used. This is not similar to the Bayesian method.

The Bayesian method would calculate the probability that each of the separate following models is true:

1. ax+d+e=0
2. by+d+e=0
3. cz+d+e=0
4. d+e=0
5. ax+by+d+e=0
6. ax+cz+d+e=0
7. by+cz+d+e=0
8. ax+by+cz+d+e=0

A probability weighting would be applied to each one. Now depending upon the cost or utility function, one would either be chosen, or some or all would be averaged together. This can only be accomplished because a prior distribution over the parameter and model space exists. This prior knowledge allows the researcher to let the data dominate the solution. Indeed, if averaging is used, then the researcher truly averages the risk of loss over the entire set of possible outcomes.

There is another set of issues that make them different. Bayesian probabilities are coherent and Frequentist probabilities are not. Fair gambles can be placed on Bayesian predictions because of this. Frequentist probabilities and statistics cannot be gambled on because a crafty opponent can set up a bookie for a sure loss in all possible outcomes. Conversely, a Bayesian cannot discuss statistical power. There is also no guaranteed coverage against false positives. This is due to the fact that the Bayesian model, where Pr(model|data), is driven by only and exactly this one sample.

Power and guarantees against false positives come from the seeing the sample with respect to the sample space, implying infinite repetition. This is only credible under Pr(data|model). It is a valid model that inherits sample properties from this potentially infinite set of possible outcomes.

Because Bayesian methods work in the parameter space, things such as testing logical assertions or gambling on the results of an outcome are reasonable things to do. Because Frequentist methods work in the parameter space, things such as guaranteeing a minimum protection against false positives and a guaranteed level of power are possible.

Bayesian and Frequentist methods use the same data to answer very different types of questions. Bayesian probabilities cannot be interpreted as Frequencies. Conversely, classical Frequencies cannot be interpreted as probabilities as they are worst-case distributions and not actual distributions.

Answer 6

If the number 0.9 is not testable, even in principle or in imagination, or it is not meant to be tested, then I think it should not be spoken out in the first place.

There may be some truth to this statement from a practical point of view. The German press currently tells us something like the following

Auf der zweiten Seite von Steinmeiers Prüfberichts steht, die Gesamtplagiatswahrscheinlichkeit liege bei 63 Prozent.

What does this 63%, i.e. the number 0.63 means in this context? It means they are man hunting again in Germany, which always feels strange to me when I see the havoc caused by people like Silvio Berlusconi at the same time.

But back to your question about the meaning of such numbers, this a number automatically generated by some more or less carefully designed procedure. The same is probably also true for probabilities reported by the weather forecast. The point about such automatically generated numbers is that you can try to calibrate the translation of these numbers into probabilities such that the deviation from actual frequentist probabilities is minimized. As such a calibration doesn't really modify the underlying procedure, the reported numbers can't be true testable frequentist probabilities. All we have done by the calibration is to convert the output of the procedure into a "measurement scale" which we know how to interpret (more or less intuitively).

Answer 7

I guess the simple answer maybe: Bayesian and Frequentist define 'probability' in two different ways, so you can't simply do the interpret. In the simplest case, you observe the fact that dinosaurs exterminated, a frequentist can only give probability by saying 'ok, if infinite identical Earth undergoes some events, and became the one we see today, then some fraction of it is caused by comet hitting, while some are caused by some other reasons.' You see, there's no definition of probability for a single event for frequentist, you have to generate a 'infinite identical situations' to calculate the probability. While for Bayesian, there is definition for probability on single event. And I personally recommend this xkcd comic:

Answer 8

I think you're confusing statistics with the various arts of prediction, such as science, which use statistics.

Statistics is a math. It's a collections of axioms and proofs. In mathematics, not only is empirical data superfluous, it can be dangerously misleading. A famous example is Fermat's conjecture that 2^2^n+1 is prime for all n.

Of course, statistics isn't just any math, it's the most earthy. Developed by gamblers, thieves and gambler-thieves (bankers), statistics is the formal language for reasoning under uncertainty, and thus the language of science.

Just like one could use logic to "prove" the existence of a benevolent sky-beard, one could use statistics to compute the "probability" of a dinosaur ending meteorite. The validity of the conclusion depends on the validity of the assumptions.

The difference between Bayesian and frequentist methods is one of assumptions and less so one of machinery (no frequentist denies the truth of Bayes' theorem). Usually problems lend themselves to one method or the other, but with sufficient data, both approaches usually converge to the same solution.

Statisticians tend to be pragmatic about their choice.

I hope that helps.

Answer 9

The following is taken from my manuscript on confidence distributions - Johnson, Geoffrey S. "Decision Making in Drug Development via Confidence Distributions." arXiv preprint arXiv:2005.04721 (2020).

In the Bayesian framework the population-level parameter of interest is considered an unrealized or unobservable realization of a random variable that depends on the observed data. This premise has cause and effect reversed. In order to overcome this the Bayesian approach reinterprets probability as measuring the subjective belief of the experimenter. Another interpretation is that the unknown fixed parameter, say theta, was randomly selected from a known collection or prevalence of theta's (prior distribution) and the observed data is used to subset this collection, forming the posterior. The unknown fixed true theta is now imagined to have instead been randomly selected from the posterior. Every time the prior or posterior is updated the sampling frame from where we obtained our unknown fixed true theta under investigation must be changed. A third interpretation is that all values of theta are true simultaneously. The truth exists in a superposition depending on the evidence observed (think Schrodinger's cat). Ascribing any of these interpretations to the posterior allows one to make philosophical probability statements about hypotheses given the data. While the p-value is typically not interpreted in the same manner, it does show us the plausibility of a hypothesis given the data - the ex-post sampling probability of the observed result or something more extreme if the hypothesis is true. This does not reverse cause and effect.

To the Bayesian, probability is axiomatic and measures the experimenter. To the frequentist, probability measures the experiment and must be verifiable. The Bayesian interpretation of probability as a measure of belief is unfalsifiable. Only if there exists a real-life mechanism by which we can sample values of theta can a probability distribution for theta be verified. In such settings probability statements about theta would have a purely frequentist interpretation (see the second interpretation of the posterior above). This may be a reason why frequentist inference is ubiquitous in the scientific literature.

The interpretation of frequentist inference is straight forward for non-statisticians by citing confidence levels, e.g. 'We are 15.9% confident that theta is less than or equal to theta_0.' Of course to fully appreciate this statement of confidence one must more fully define the p-value as a frequency probability of the experiment if the hypothesis is true (think of the proof by contradiction structure a prosecutor uses in a court room setting, innocent until proven guilty). A Bayesian approach may make it easy for some to interpret a posterior probability, e.g. 'There is 17.4% Bayesian belief probability that theta is less than or equal to theta_0.' Of course to fully appreciate this statement one must fully define Bayesian belief and make it clear this is not a verifiable statement about the actual parameter, the hypothesis, nor the experiment. If the prior distribution is chosen in such a way that the posterior is dominated by the likelihood or is proportional to the likelihood, Bayesian belief is more objectively viewed as confidence based on frequency probability of the experiment. In short, for those who subscribe to the frequentist interpretation of probability, the confidence distribution summarizes all probability statements about the experiment one can make. It is a matter of correct interpretation given the definition of probability and what constitutes a random variable. The posterior remains an incredibly useful tool and can be interpreted as a valid asymptotic confidence distribution. The frequentist framework can easily incorporate historical data through a fixed-effect meta-analysis.

Answer 10

Both frequentist and subjective probabilities are untestable. Testing that even a coin would land on heads 50% of the time if thrown an infinite number of times is impossible which is the standard frequentist interpretation. You can change the frequentist definition to mean a certain number of trials but then you would have to define how close to 50% you would want those number of trials to be, and would be limited to a single experiment. This makes it hard to come up with a single number that represents a probability.

With that being said, subjective probabilities are much worse for primarily one reason: the correct answer in all cases is 0 or 1, so the test itself would be useless. In the case of single case probabilities such as the probability of dinosaurs existing, dinosaurs either exist or do not. Anything that isn’t 0 or 1 should by definition be wrong and yet, still, the field of subjective probability thrives. I suppose this is only because it’s a useful tool for practical reasons especially in the cases of decision theory and computing. However, a tool being useful does not mean it has any basis in reality. Subjective probability is a good example of an illusion, which despite being practical, has no basis in reality, especially in cases of evaluating the existence of agents.