18
Dec

The value of re-analysis

Tweet about this on Twitter21Share on Facebook1Share on Google+9Share on LinkedIn1Email this to someone

I just saw this really nice post over on John Cook's blog. He talks about how it is a valuable exercise to re-type code for examples you find in a book or on a blog. I completely agree that this is a good way to learn through osmosis, learn about debugging, and often pick up the reasons for particular coding tricks (this is how I learned about vectorized calculations in Matlab, by re-typing and running my advisors code back in my youth).

In a more statistical version of this idea, Gary King has proposed reproducing the analysis in a published paper as a way to get a paper of your own.  You can figure out the parts that a person did well and the parts that you would do differently, maybe finding enough insight to come up with your own new paper. But I think this level of replication involves actually two levels of thinking:

  1. Can you actually reproduce the code used to perform the analysis?
  2. Can you solve the "paper as puzzle" exercise proposed by Ethan Perlstein over at his site. Given the results in the paper, can you come up with the story?

Both of these things require a bit more "higher level thinking" than just re-running the analysis if you have the code. But I think even the seemingly "low-level" task of just retyping and running the code that is used to perform a data analysis can be very enlightening. The problem is that this code, in many cases, does not exist. But that is starting to change. If you check out Rpubs or RunMyCode or even the right parts of Figshare you can find data analyses you can run through and reproduce.

The only downside is there is currently no measure of quality on these published analyses. It would be great if people could focus their time re-typing only good data analyses, rather than one at random. Or, as a guy once (almost) said, "Data analysis practice doesn't make perfect, perfect data analysis practice makes perfect."

  • Pingback: Software Tips | Pearltrees

  • BillyJoeBob

    Per usual, practice does not make perfect. Practice makes permanent.

    One must practice doing the right things, and doing them well, lest one simply create bad habits that must later be unlearned (with great difficulty) to progress.

    But it's also worth noting that the perfect is the enemy of the good, and all that is really required is to practice being good (getting better). This is why good coaches, colleagues, and mentors are so valuable -- they can see mistakes that we cannot, and warn us.