I used peer-review for the data analysis course I just finished. As I mentioned in the post-mortem podcast I knew in advance that it was likely to be the most controversial component of the class. So it wasn’t surprising that based on feedback in the discussion boards and on this blog, the peer review process is by far the thing students were most concerned about.
But to evaluate complete data analysis projects at scale there is no other alternative that is economically feasible. To give you an idea, I have our local students perform 3 data analyses in an 8 week term here at Johns Hopkins. There are generally 10-15 students in that class and I estimate that I spend around an hour reading each analysis, digesting what was done, and writing up comments. That means I usually spend almost an entire weekend grading just for 10-15 data analyses. If you extrapolate that out to the 5,000 or so people who turned in data analysis assignments, it is clearly not possible for me to do all the grading.
Another alternative would be to pay trained data analysts to grade all the assignments. Of course that would be expensive - you couldn’t farm it out to the mechanical turk. If you want to get a better/more consistent grading scheme than peer review you’d need to hire highly trained data analysts to do that and that would be very expensive. While Johns Hopkins has been incredibly supportive in terms of technical support and giving me the flexibility to pursue the class, it is definitely something I did on my own time and with a lot of my own resources. It isn’t clear that it make sense for Hopkins to pour huge resources into really high-quality grading. At the same time, I’m not sure Coursera could afford to do this for all of the classes where peer review is needed, as they are just a startup.
So I think that at least for the moment, peer review is the best option for grading. This has big implications for the value of the Coursera statements of accomplishment in classe where peer review is necessary. I think that it would benefit Coursera hugely to do some research on how to ensure/maintain quality in peer review (Coursera - if you are reading this and you have some $$ you want to send my way to support some students/postdocs I have some ideas on how to do that). The good news is that the amazing Coursera platform collects so much data that it is possible to do that kind of research.