30
Apr

Reproducibility and reciprocity

Tweet about this on Twitter46Share on Facebook14Share on Google+7Share on LinkedIn0Email this to someone

One element about the entire discussion about reproducible research that I haven't seen talked about very much is the potential for the lack of reciprocity. I think even if scientists were not concerned about the possibility of getting scooped by others by making their data/code available this issue would be sufficient to give people pause about making their work reproducible.

What do I mean by reciprocity? Consider the following (made up) scenario:

  1. I conduct a study (say, a randomized controlled trial, for concreteness) that I register at clinicaltrials.gov beforehand and specify details about the study like the design, purpose, and primary and secondary outcomes.
  2. I rigorously conduct the study, ensuring safety and privacy of subjects, collect the data, and analyze the data.
  3. I publish the results for the primary and secondary outcomes in the peer-reviewed literature where I describe how the study was conducted and the statistical methods that were used. For the sake of concreteness, let's say the results were "significant" by whatever definition of significant you care to use and that the paper was highly influential.
  4. Along with publishing the paper I make the analytic dataset and computer code available so that others can look at what I did and, if they want, reproduce the result.

So far so good right? It seems this would be a great result for any study. Now consider the following possible scenarios:

  1. Someone obtains the data and the code from the web site where it is hosted, analyzes it, and then publishes a note claiming that the intervention negatively affected a different outcome not described in the original study (i.e. not one of the primary or secondary outcomes).
  2. A second person obtains the data, analyzes it, and then publishes a note on the web claiming that the intervention was ineffective for the primary outcome in a the subset of participants that were male.
  3. A third person obtains the data, analyzes the data, and then publishes a note on the web saying that the study is flawed and that the original results of the paper are incorrect. No code, data, or details of their methods are given.

Now, how should one react to the follow-up note claiming the study was flawed? It's easy to imagine a spectrum of possible responses ranging from accusations of fraud to staunch defenses of the original study. Because the original study was influential, there is likely to be a kerfuffle either way.

But what's the problem with the three follow-up scenarios described? The one thing that they have in common is that none of the three responding people were subjected to the same standards to which the original investigator (me) was subjected. I was required to register my trial and state the outcomes in advance. In an ideal world you might argue I should have stated my hypotheses in advance too. That's fine, but the point is that the people analyzing the data subsequently were not required to do any of this. Why should they be held to a lower standard of scrutiny?

The first person analyzed a different outcome that was not a primary or secondary outcome. How many outcomes did they test before the came to that one negatively significant one? The second person examined a subset of the participants. Was the study designed (or powered) to look at this subset? Probably not. The third person claims fraud, but does not provide any details of what they did.

I think it's easy to take care of the third person--just require that they make their work reproducible too. That way we can all see what they did and verify that there was in fact fraud. But the first two people are a little more difficult. If there are no barriers to obtaining the data, then they can just get the data and run a bunch of analyses. If the results don't go their way, they can just move on and no one would be the wiser. If they did, they can try to publish something.

What I think a good reproducibility policy should have is a type of "viral" clause. For example, the GNU General Public License (GPL) is an open source software license that requires, among other things, that anyone who writes their own software, but links to or integrates software covered under the GPL, must publish their software under the GPL too. This "viral" requirement ensures that people cannot make use of the efforts of the open source community without also giving back to that community. There have been numerous heated discussions in the software community regarding the pros and cons of such a clause, with (large) commercial software developers often coming down against it. Open source developers have largely beens skeptical of the arguments of large commercial developers, claiming that those companies simply want to "steal" open source software and/or maintain their dominance.

I think it is important that if we are going to make reproducibility the norm in science, that we have analogous "viral" clauses to ensure that everyone is held to the same standard. This is particularly important in policy-relevant or in politically sensitive subject areas where there are often parties involved who have essentially no interest (and are in fact paid to have no interest) in holding themselves to the same standard of scientific conduct.

Richard Stallman was right to assume that without the copyleft clause in the GPL that large commercial interests would simply usurp the work of the free software community and essentially crush it before it got started. Reproducibility needs its own version of copyleft or else scientists will be left to defend themselves against unscrupulous individuals who are not held to the same standard.

  • http://twitter.com/tslumley Thomas Lumley

    I think this would require a prior agreement from people who accessed your data, which sort of defeats the openness -- in contrast to the GPL, you're trying to restrict use, not just distribution. Also, if you start having contracts limiting what can be done with the data, you can easily imagine what sort of contracts some people would write.

    • Roger Peng

      So I admittedly left the implementation of this "copyleft" a bit vague. I didn't really imagine that there would be actual contracts. Rather, I wanted to emphasize that there should be a mutual (perhaps unspoken) agreement at least amongst scientists, that everyone should be held to the same standard. Right now, I don't believe we have that agreement.

      • Yaroslav Halchenko

        "mutual agreement"... and more of a "healthy atmosphere". In my opinion, no licensing or any other legal enforcement would help here. It is more of what should be considered adequate if someone expresses his opinion on your data while performing some secondary analysis of it.

        Bringing a simplified life example: If someone comes out and claims that I am wrong, I would just consider himself a joke unless he could argument his statement. In the examples you provide the only reasonable argumentation could be the code performing such secondary analysis. There is simply no valid excuse nowadays to not share the code under some reasonable open (so we could at least see the code) license. If no such arguments follow -- I would just quote L.Torvalds "Talk is cheap. Show me the code" -- that person is a joke and just earned you another citation.

        • Roger Peng

          The problem is that it rarely matters what you think of a person but rather what everyone else thinks. And it may be difficult or someone not intimately familiar with your field to understand the details with or without code. You may think a person is a joke but if others think he is "serious" then you are in for a world of headaches.

    • Matt Shotwell

      But, in each of the above 'uses' that we might like to regulate, the data are re-published. Maybe this is the feature that distinguishes any use from the restricted use. Indeed, to publish a derivative work is to distribute it. Also, we might specify an "Acceptance Not Required for Having Copies" clause (GPL section 9), that give the right to download and execute data and code without having accepted the license.

  • Markus Spatz

    Absolutely agree. Open access and reproducibility do open this particular can of worms, and I have not really seen the issue properly discussed yet. I foresee floods of youtube-quality comments appended to papers, and the most snappy put-downs garnering thousands of "likes" regardless of their scientific merit.

  • Ken

    I have seen a situation where a meta-analysis included one trial which was 60% of the data, and the results in that trial and the meta-analysis were different. This sort of makes you wonder what has happened. Turned out that the trial used a baseline correction but the meta-analysis didn't, as individual data wasn't available for the other trials. The noncorrected results could be determine from the corrected trial because there were sufficient results described. However the corrected results were what was specified in the protocol for that trial, so they should be used. Probably the only option was to combine both.

    It is easy to imagine situations where results are changed by a new analysis and it is very difficult to argue against the new analysis. Using an improved missing data algorithm or baseline corrections. The only way to resolve this is to improve judgement of what is acceptable in analysis. It would be very easy for groups to misuse a re-analysis to back their beliefs. For example subgroup analysis for adverse events from vaccination.