Statistical illiteracy may lead to parents panicking about Autism.

I just was doing my morning reading of a few news sources and stumbled across this Huffington Post article talking about research correlating babies cries to autism. It suggests that the sound of a babies cries may predict their future risk for autism. As the parent of a young son, this obviously caught my attention in a very lizard-brain, caveman sort of way. I couldn't find a link to the research paper in the article so I did some searching and found out this result is also being covered by Time, Science Daily, Medical Daily, and a bunch of other news outlets.

Now thoroughly freaked, I looked online and found the pdf of the original research article. I started looking at the statistics and took a deep breath. Based on the analysis they present in the article there is absolutely no statistical evidence that a babies' cries can predict autism. Here are the flaws with the study:

  1. Small sample size. The authors only recruited 21 at risk infants and 18 healthy infants. Then, because of data processing issues, only ended up analyzing 7 high autistic risk versus 5 low autistic-risk in one analysis and 10 versus 6 in another. That is no where near a representative sample and barely qualifies as a pilot study.
  2. Major and unavoidable confounding. The way the authors determined high autistic risk versus low risk was based on whether an older sibling had autism. Leaving aside the quality of this metric for measuring risk of autism, there is a major confounding factor: the families of the high risk children all had an older sibling with autism and the families of the low risk children did not! It would not be surprising at all if children with one autistic older sibling might get a different kind of attention and hence cry differently regardless of their potential future risk of autism.
  3. No correction for multiple testing. This is one of the oldest problems in statistical analysis. It is also one that is a consistent culprit of false positives in epidemiology studies. XKCD even did a cartoon about it! They tested 9 variables measuring the way babies cry and tested each one with a statistical hypothesis test. They did not correct for multiple testing. So I gathered resulting p-values and did the correction for them. It turns out that after adjusting for multiple comparisons, nothing is significant at the usual P < 0.05 level, which would probably have prevented publication.

Taken together, these problems mean that the statistical analysis of these data do not show any connection between crying and autism.

The problem here exists on two levels. First, there was a failing in the statistical evaluation of this manuscript at the peer review level. Most statistical referees would have spotted these flaws and pointed them out for such a highly controversial paper. A second problem is that news agencies report on this result and despite paying lip-service to potential limitations, are not statistically literate enough to point out the major flaws in the analysis that reduce the probability of a true positive. Should journalists have some minimal in statistics that allows them to determine whether a result is likely to be a false positive to save us parents a lot of panic?

 

This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.
  • Keith O’Rourke

    Dr. Spiegelhalter and colleague have developed an all purpose vaccine or defence strategy named after John Humphrys for this type of harmful information virus – available from Significance October2012.

    So far no RCT evidence of its effectiveness and suspected side-effects (decreased readership)

  • http://www.facebook.com/profile.php?id=418748 Vadim Zipunnikov

    decoding adjustments with simpler (but improper) representation

    adjustedPvaluesPain = pvaluesPain*8
    adjustedPvaluesNoPain = pvaluesNoPain*9

  • Pingback: Habits of Mind That Drive Science | The CSUPERB Blog

  • Steven Salzberg

    Jeff, Great job re-analyzing this paper and pointing out the flaws. Too bad you can't get equal publication credit for showing there's nothing here, much less articles in the NY Times pointing out your "startling" finding. Headline: "autism researchers try but fail to predict autism from the pitch of babies' crying."

    There ought to be a publication venue for this kind of analysis. Well at least we have your blog.

  • Pingback: Weekly roundup November 26 – December 2 « lingwhatics

  • WatchinIt

    To counter the proliferation of bogus science headline, encourage competent grad students to work in the media.

    For example -

    The Washington Examiner seeks a data editor for its special reporting
    team. Must possess significant data analysis skills, be conversant with
    public policy issues and personalities across the ideological spectrum,
    and be able to locate and utilize documents, data sources and digital
    investigative tools quickly and confidently. Compensation commensurate
    with qualifications and experience. Email resume and clips to:
    mark.tapscott@gmail.com. DO NOT CALL.

  • Pingback: cheap edu backlinks

  • Pingback: cheap edu backlinks

  • Pingback: buy edu backlinks

  • Pingback: buy edu backlinks