26
Nov

Statistical zealots

Tweet about this on Twitter36Share on Facebook76Share on Google+3Share on LinkedIn1Email this to someone

Yesterday my data sharing policy went a little bit viral. It hit the front page of Hacker News and was a trending repo on Github. I was reading the comments on Hacker News and came across this gem:

So, while I can imagine there are good Frequentists Statisticians out there, I insist that frequentism itself is bogus.

This is the extension of a long standing debate about the relative merits of frequentist and Bayesian statistical methods. It is interesting that I largely only see one side of the debate being played out these days. The Bayesian zealots have it in for the frequentists in a big way. The Hacker News comments are one example, but here are a few more. Interestingly, even the "popular geek press" is getting in the game.

I think it probably deserves a longer post but here are my thoughts on statistical zealotry:

  1. User effect >>>>>>>>>>>>>>>>> Philosophy effect. The person doing the statistics probably matters more than the statistical philosophy. I would prefer Andrew Gelman analyzed my data than a lot of frequentists. Similarly, I'd prefer that John Storey analyzed my data than a lot of Bayesians. 
  2. I agree with Noahpinion that this is likely mostly a philosophy battle than a real practical applications battle.
  3. I like Rob Kass's idea that we should move away from frequentist vs. Bayesian to pragmatism. I think most real applied statisticians have already done this, if for no other reason than being pragmatic helps you get things done.
  4. Papers like this one that claim total victory for one side or the other all have one thing in common: they rarely use real data to verify their claims. The real world is messy and one approach never wins all the time.

My final thought on this matter is: never trust people with an agenda bearing extreme counterexamples.

  • fonnesbeck

    2 things:

    1. I'll bet the person behind the HN comment is not a practicing statistician.

    2. Most of the folks I know who are Bayesians are so for pragmatic reasons (it makes your life easier much of the time) than for philosophical ones.

    • Jake

      HN user "loup-vaillant" admitted to being just a fanboy but! he has read "half of E.T. Jaynes' Probability Theory: the Logic of Science so far.".

      There's a lot of people like him out there that have jumped on the "Bayesian" bandwagon but who don't actually know anything.

    • IfAllYouHaveIsABayesHammer

      With respect to point one. He isn't a statistician, he is a programmer. He is a typical LessWrong poster: A status-striving, white (or jewish), male, computer programmer with an over-inflated ego and high level of epistemic arrogance. In his and other's cases: it is the arrogance to believe they speak as experts in the field of statistics. While for Yudkowsky it is not only statistics, but various other fields, like AI and physics. For lukeprog, it is philosophy. Pick a random LessWrong member, and you can find many of them speaking with authority on subjects that they aren't even experts in. One way to classify LessWrong is not as frauds, but as a roleplaying endeavour where community members act out roles as autistic little professors. They love to prattle on about the "outside view" over there, but it makes me laugh when they don't realize that most individuals outside of LessWrong see them as a joke, and that they are essentially engaging in internet role playing. The CMU stats guys think they are delusional, while various philosophers (including people over at NewAPPS and Leiter's blog) think they are insane. It makes me wonder what the AI and psychology community thinks of them.

      • http://itschancy.wordpress.com/ Corey Yanofsky

        But tell us how you really feel...

  • Matt.0

    I definitely agree that that this is a philosophical battle. For example, Joppa et al. (2013 - Science) have shown that most ecologists select methods software guided primarily by concerns of fashion (in other words, whatever everybody else uses). The
    recent expansion of readily available statistical software has greatly
    increased the number of shirts on the rack. Because software enables researchers to make use of methods without the statistical knowledge of how to implement them from the ground up, many echo the position so memorably articulated by Jim Clark that we are "handing guns to children." So yes; the person doing the statistics probably matters more than the statistical philosophy.

    Here is a good level-headed clinical analysis of both methods - "Case Study Comparing Bayesian and Frequentist Approaches for Multiple Treatment Comparisons"
    http://www.ncbi.nlm.nih.gov/books/NBK132729/

  • http://www.facebook.com/people/Stephen-Henderson/621809902 Stephen Henderson

    The last word is surely Bradley Efrons' last words in his recent Science article (Bayes' Theorem in the 21st Century):

    "My own practice is to use Bayesian analysis in the presence of genuine prior information; to use empirical Bayes methods in the parallel cases situation; and otherwise to be cautious when invoking uninformative priors. In the last case, Bayesian calculations cannot be uncritically accepted and should be checked by other methods, which usually means frequentistically."

    Can't say fairer than that.

    • Brendon J. Brewer

      Pity the paragraph is nonsense.

      Inference "not in the presence of genuine prior information" is a will-o'-the-wisp.

  • Entsophy

    This is a nice summary of conventional wisdom and is unobjectionable if in the future, statistical applications are just more of the same old blah blah we see today.

    If on the other hand, present day statistics represents an initial, and not terribly powerful, advance and there are major advances still to be had, it makes a huge difference goinf forward whether you're frequentist or baysain.

    Frequentism isn’t a disaster because practicing frequentists are worse than bayesians. I don't know that they are since I've only hired frequentists myself. Rather it’s a disaster because it limits us to the current muddled paradigm and its shabby results.

    Similarly, Epicycles weren’t bad because they did work. They did work in fact and served well for a 1000 years. It’s just that anyone dogmatically devoted to them would have missed out on all the physics of the last 300 years.

  • Brendon J. Brewer

    Hello. Zealot here. I don't mind if someone is a frequentist. It just means you only ever do calculations I'm not interested in. If you're a good frequentist, you probably did these irrelevant calculations correctly.

    • jtleek

      A compelling and rational argument :-).

  • Steven

    "Non-academics who don't know anything about an academic field beyond wikipedia articles and blogs have dumb and outspoken opinions on it" it hardly a new thing.

    I guess in statistics we are largely protected from the sort of cranks who usually latch onto mathematics/physics ("quantum physics is a lie"/"I have proved the Collatz Conjecture using elementary algebra!"/etc), because our discipline isn't glamorous enough to attract attention, but the Bayesian vs frequentism thing is just close enough to philosophy to give the cranks a way in. Anyone who has strong opinions on this matter yet has never actually done statistics at a non-trivial level should probably be ignored on principle.

  • Rasmus Arnling Bååth

    From a practical pragmatic pedagogical perspective it is a battle that the Bayesian side is loosing quite heavily right now, at least in my field (psychology). I'm going to teach a beginners course in statistics next year and I tried to find one introductory PsychStats book that was at least a little bit Bayesian (Even though it is great, I don't consider Kruschkes "Doing Bayesian Data Analysis" an introductory statistics book). I looked trough 40+ books and all were heavily old school NHST. A Bayesian perspective on statistics is extremely useful in Psych as it makes it easy to grok and use hierarchical models, latent variable analysis, non-normal distributions, etc, that is, useful stuff if you do Psych research. Right now, most undergraduate psychology students are denied learning this useful perspective ...

  • Ken Rice

    Hi Jeff, think I agree with everything except #3 - I think principles are useful when one meets new situations, at least. Anyway, here's a comment;

    > the relative merits of frequentist and Bayesian statistical methods

    I think you really mean default frequentist and default Bayesian methods? Many (most?) standard analyses have sane justifications under both paradigms, even if they're not well known.

    If frequentist or Bayesian take care to provide an analysis that answers the same question as the opposing camp, often the "battle" you mention vanishes. See e.g. two-sided p-values, robust standard errors, outlier-robustness, some conditional likelihoods... I could go on. Figuring out quite what the "same question" is teaches one a lot about what the method actually provides, and what it doesn't.

  • http://itschancy.wordpress.com/ Corey Yanofsky

    I like Noah Smith's stuff usually, but in this case his outsider view is uninformed. His justification for his opinion is this:

    "Why do I think this? Basically, because Bayesian inference has been around for a while - several decades, in fact - and people still do Frequentist inference. If Bayesian inference was clearly and obviously better, Frequentist inference would be a thing of the past. The fact that both still coexist strongly hints that either the difference is a matter of taste, or else the two methods are of different utility in different situations."

    This reasoning fails because in general, Bayesian inference, unlike frequentist inference (which has no canonical "in-general" form), requires integrations over high-dimensional probability distributions of arbitrary form. The knowledge and computational resources to accomplish this have been ramping from close to zero up since the '50s; awareness of their existence only really penetrated the Bayesian statistics community 25 years ago. (You'd think a macro commentator like Noah would have more appreciation for technology shocks.)

    Entsophy's got the right perspective on philosophical foundations: they matter because they guide research and development. Personally, I think that the fact that there are three possible philosophical foundations for Bayesian statistics (Savage/de Finetti, Cox/Jaynes, decision theory) is the strong hint. On the other hand, adequate philosophical foundations for frequentist inference were lacking prior to the work of Deborah Mayo. (For me, it is still an open question whether these foundations are adequate; I'm exploring that question on my blog.)