"Frequentists vs Bayesians"

Steve Hsu:

Frequentists vs Bayesians, by Steve Hsu: Noted Berkeley statistician David Freedman recently passed away. I recommend the essay below if you are interested in the argument between frequentists (objectivists) and Bayesians (subjectivists). I never knew Freedman, but based on his writings I think I would have liked him very much -- he was clearly an independent thinker :-)

In everyday life I tend to be sympathetic to the Bayesian point of view, but as a physicist I am willing to entertain the possibility of true quantum randomness.

I wish I understood better some of the foundational questions mentioned below. In the limit of infinite data will two Bayesians always agree, regardless of priors? Are exceptions contrived?

Some issues in the foundation of statistics Abstract: After sketching the conflict between objectivists and subjectivists on the foundations of statistics, this paper discusses an issue facing statisticians of both schools, namely, model validation. Statistical models originate in the study of games of chance, and have been successfully applied in the physical and life sciences. However, there are basic problems in applying the models to social phenomena; some of the difficulties will be pointed out. Hooke’s law will be contrasted with regression models for salary discrimination, the latter being a fairly typical application in the social sciences.

...The subjectivist position seems to be internally consistent, and fairly immune to logical attack from the outside. Perhaps as a result, scholars of that school have been quite energetic in pointing out the flaws in the objectivist position. From an applied perspective, however, the subjectivist position is not free of difficulties. What are subjective degrees of belief, where do they come from, and why can they be quantified? No convincing answers have been produced. At a more practical level, a Bayesian’s opinion may be of great interest to himself, and he is surely free to develop it in any way that pleases him; but why should the results carry any weight for others? To answer the last question, Bayesians often cite theorems showing "inter-subjective agreement:" under certain circumstances, as more and more data become available, two Bayesians will come to agree: the data swamp the prior. Of course, other theorems show that the prior swamps the data, even when the size of the data set grows without bounds-- particularly in complex, high-dimensional situations. (For a review, see Diaconis and Freedman, 1986.) Theorems do not settle the issue, especially for those who are not Bayesians to start with.

My own experience suggests that neither decision-makers nor their statisticians do in fact have prior probabilities. A large part of Bayesian statistics is about what you would do if you had a prior.7 For the rest, statisticians make up priors that are mathematically convenient or attractive. Once used, priors become familiar; therefore, they come to be accepted as "natural" and are liable to be used again; such priors may eventually generate their own technical literature. ...

It is often urged that to be rational is to be Bayesian. Indeed, there are elaborate axiom systems about preference orderings, acts, consequences, and states of nature, whose conclusion is-- that you are a Bayesian. The empirical evidence shows, fairly clearly, that those axioms do not describe human behavior at all well. The theory is not descriptive; people do not have stable, coherent prior probabilities.

Now the argument shifts to the "normative:" if you were rational, you would obey the axioms, and be a Bayesian. This, however, assumes what must be proved. Why would a rational person obey those axioms? The axioms represent decision problems in schematic and highly stylized ways. Therefore, as I see it, the theory addresses only limited aspects of rationality. Some Bayesians have tried to win this argument on the cheap: to be rational is, by definition, to obey their axioms. ...

How do we learn from experience? What makes us think that the future will be like the past? With contemporary modeling techniques, such questions are easily answered-- in form if not in substance.

·The objectivist invents a regression model for the data, and assumes the error terms to be independent and identically distributed; "iid" is the conventional abbreviation. It is this assumption of iid-ness that enables us to predict data we have not seen from a training sample-- without doing the hard work of validating the model.

·The classical subjectivist invents a regression model for the data, assumes iid errors, and then makes up a prior for unknown parameters.

·The radical subjectivist adopts an exchangeable or partially exchangeable prior, and calls you irrational or incoherent (or both) for not following suit.

In our days, serious arguments have been made from data. Beautiful, delicate theorems have been proved; although the connection with data analysis often remains to be established. And an enormous amount of fiction has been produced, masquerading as rigorous science. [!!!]

Comments

[Mike D]

I think everyone is a closet Bayesian, at least on social science topics where they have some personal anecdotes and/or ideology that they overweight as data points.

Even on a natural science topics totally outside any personal experience (How many atoms are in the Sun) everyone has some kind of prior belief (I'm 100% sure it's greater than the number of atoms in my body, and I'm 100% sure that number is greater than Avogadro's number.)

This parallels my belief that everyone has an implicit and unarticulated social science model in their head. Otherwise, they'd have no basis to interact with other human beings. Didn't we already run through all this stuff about how observation without pre-existing theory was a pipe-dream back with David Hume?

The advantage of Bayesianism and explicit modeling is that it forces you to own up to your biases instead of leaving you totally blind to your confirmation bias.

[Barkley Rosser]

Diaconis and Freedman is indeed the relevant source for the problems with Bayesianism, or more properly, the limits of Bayesian analysis. So, if you have an infinite dimensional die and your support is discontinuous, Bayes' Theorem fails and there is no necessary convergence. Can end up oscillating between competing priors located in different, discontinuously disconnected sections of the support.

It is my understanding that Bayesian approaches have all but displaced classical ones in much of the testing of new drugs and certain other areas of medicine. In economics, Bayesian methods are creeping in through certain kinds of time-series methods such as state space analysis.

[Eric L. Prentis]

It is useful to read the philosopher Soren Kierkegaard (1813-1855) to understand how human beings distinguish between objective and subjective truths. Objective truths are external to the individual and answer the questions of “what, who, where and when” of thought and action. Subjective truths relate to one’s selfhood, are internal to the individual and have no objective criteria for judgment. Subjective truths are beliefs and values about concepts and answer the questions of “how and why.” All objective truths ultimately become subjective truths to individuals in social institutions because objective truths get prioritized through the filter of subjective truths.

[Bruce Wilder]

". . .the prior swamps the data, even when the size of the data set grows without bounds-- particularly in complex, high-dimensional situations."

Politics!

"Theorems do not settle the issue, especially for those who are not Bayesians to start with."

Republicans!

[Bruce Wilder]

Mike D: "Didn't we already run through all this stuff about how observation without pre-existing theory was a pipe-dream back with David Hume?"

Indeed.

Is this like a dispute among artists arguing over whether landscapes are more meritorious than portraits, photographs better than fantasy comics? An argument among poets over the proper use of a metaphor for contingency? Are we to understand there to be probability in the past? Do we think reality must be a plural?

[robertdfeinman]

I recently wrote an essay where I compared econometrics to signal processing, a branch of information theory. Bayesian analysis is considered as one of the techniques (knowing something about the signal a priori).

For the wonky here's the link:

Econometrics as Signal Processing

[Bruce Wilder]

Bernanke: "The likely duration of the financial turmoil is difficult to judge, and thus the uncertainty surrounding the economic outlook is unusually large."

How radical a Bayesian?

[Mark]

I'm often skeptical of Bayesian approaches, but there are circumstances where their appeal is undeniable. For example, if you've seen one 20-year old southern pine plantation, you've got a pretty good idea what the next one is gonna look like. Classical fequentist methods deny the utility of this prior evidence. Taken to extremes, the dogma of either camp is just tedious.

[FairEconomist]

Bayesian beliefs do *not* converge. The theorems show that if Bayesian A has a prior belief level in a proposition of P(a) and Bayesian B has a prior of P(b) then with enough evidence P(a)-P(b) becomes arbitrarily small. But - the ratio of A's betting ratio P(a)/P(~a) to B's betting ration P(b)/P(~b) is a constant. So the ratio of what A will pay as insurance against the proposition to what B will pay never changes. In a sense disagreements between Bayesian never go away; it's just that the disagreements become about extremely rare possibilities.

Bayesian statistics isn't without problems, but frequentism is a total basket case. Frequentism has no philosophical basis; it's just an arbitrary set of hurdles for arbitrarily chosen "null hypotheses".

[Richard H. Serlin]

This whole issue is really easy to misinterpret, but basically Bayesian statistics is often better because it utilizes more data, not just the data from the survey or gathering in question, but at least some of the other data available in the world (called the prior or a priori data).

Using more data is always better -- if it's used properly, but that's the key; it's often hard to use it close to properly, and Bayesians can often use it far from properly. Good Frequentists (and there are far too few of them) also consider prior data, but do so informally, or less formally.

An extremely valuable thing about Bayesian statistics is that it often provides an entire distribution rather than just point estimates and little more, like p-values. A distribution is extremely valuable because it's often easy to be mislead if you only look at relatively small aspects of the data (like mean and variance only) instead of examining the data from many angles like you can with a whole distribution or percentiles.

Looking at the whole distribution and/or percentiles is a fantastic way to not get mislead by data and statistics. Generally, interpreting statistics and data well, whether using Bayesian or Frequentist tools, or both, depends on good intuition and high level thinking, things that are grossly under-rewarded in economics and finance academia today.

[donna]

Bayesian analysis is a tool that works great in some situations, like tracking objects and traffic control. It's always about what tool is appropriate to a situation, not what ideology is correct.

Sometimes my viewpoint is objective, sometimes it is subjective. So what? It depends on the situation.

"A frequentist is a person whose long-run ambition is to be wrong 5% of the time.

A Bayesian is one who, vaguely expecting a horse, and catching a glimpse of a donkey, strongly believes he has seen a mule."

[Barkley Rosser]

FairEconomist,

Asymptotic approach is effectively convergence, for practical purposes. I was speaking loosely, but not wildly inaccurately. Lots of so-called "convergence theorems" are actually infinite asymptotic ones, meaning one never "gets there" in finite time. This only becomes a big deal when someone claims that asymptotic infinite convergence is the same thing as being there now, as for example people who argue that rational expectations hold because under certain conditions people with adaptive expectations will learn to asymptotically approach having rational expectations, hah.

The failure of the theorem means a "convergence" to a cycle of bouncing around from one prior to another that are far apart. You are just being overly picky with language here to for no discernible point.

[Jrossi]

Excellent link, Prof. Thoma. This kind of thing is one of the reasons why your blog is such valuable service.

[Richard]

For large sample sized, bayesian and frequentist approaches converge; for small sample sizes it's a crap shoot no matter what you do.

The primary objection to bayesian approaches is that it's a vehicle for gaming the conclusion. The bayesian will respond that the frequentist's approach is just a bayesian prior of a uniform distribution.

There's an annecdote I've heard about the space program before Challenger--that, given the complexity of space flight, estimates were made that several missions would result in complete loss of life. The estimates were then assumed to lessen as time went by, because we get better at making things. But an actuary would have predict one disaster every hundred flights or so--which is what happened.

Frequentists believe that the discipline of equally weighing data keeps you honest. Bayesians believe that not using prior information is to constrain yourself unnecessarily. Like everything in this sloppy world each has their place, and they depend mostly on facts and circumstances.

[Bill Jefferys]

In parameter estimation, under mild conditions Bayesian procedures converge (almost surely) to the same result, independent of the prior.

It is true that the betting ratios don't change, but if you are estimating a parameter, that is not relevant. But this fact does affect estimates of other things, e.g., variances.

The fact that betting ratios don't change is of some relevance in Bayesian decision theory. What it means is that if the loss functions of two decision-makers are the same, but their priors are different, then as the data come in, the point at which one decision-maker will decide on the final (i.e., in the limit) decision will differ from the decision point for the other decision-maker. So, even though with enough data they would make the same decision, there can be points in the data-collection process where their decisions would be different.

Herman Rubin (Purdue) has long advocated considering the prior and loss together, since changes in the prior can be compensated for by changes in the loss function.

But as Abraham Wald has taught us, under mild conditions admissible frequentist decision rules are Bayes rules, and vice versa. I regard this as a significant unifying principle between Bayesian and frequentist theory. In particular, an admissible frequentist decision rule is just as arbitrary as a Bayesian prior.

But the common frequentist decision-making methodologies, p-values and hypothesis testing, are incompatible with frequentist decision theory, and in my view, should be scrapped.