18
Jan

Comparing online and in-class outcomes

Tweet about this on Twitter13Share on Facebook0Share on Google+4Share on LinkedIn3Email this to someone

My colleague John McGready has just published a study he conducted comparing the outcomes of students in the online and in-class versions of his Statistical Reasoning in Public Health class that he teaches here in the fall. In this class the online and in-class portions are taught concurrently, so it's basically one big class where some people are not in the building. Everything is the same for both groups--quizzes, tests, homework, instructor, lecture notes. From the article:

The on-campus version employs twice-weekly 90 minute live lectures. Online students view pre-recorded narrated versions of the same materials. Narrated lecture slides are made available to on-campus students.

The on-campus section has 5 weekly office hour sessions. Online students communicate with the course instructor asynchronously via email and a course bulletin board. The instructor communicates with online students in real time via weekly one-hour online sessions. Exams and quizzes are multiple choice. In 2005, on-campus students took timed quizzes and exams on paper in monitored classrooms. Online students took quizzes via a web-based interface with the same time limits. Final exams for the online students were taken on paper with a proctor.

So how did the two groups fair in their final grades? Pretty much the same. First off, the two groups of students were not the same. Online students were 8 years older on average, more likely to have an MD degree, and more likely to be male. Final exam scores between online and in-class groups differed by -1.2 (out of 100, online group was lower) and after adjusting for student characteristics they differed by -1.5. In both cases, the difference was not statistically significant.

This was not a controlled trial and so there are possibly some problems with unmeasured confounding given that the populations appeared fairly different. It would be interesting to think about a study design that might allow a measure of control or perhaps get a better measure of the difference between online and on-campus learning. But the logistics and demographics of the students would seem to make this kind of experiment challenging.

Here's the best I can think of right now: Take a large class (where all students are on-campus) and get a classroom that can fit roughly half the number of students in the class. Then randomize half the students to be in-class and the other half to be online up until the midterm. After the midterm cross everyone over so that the online group comes into the classroom and the in-class group goes online to take the final. It's not perfect--One issue is that course material tends to get harder as the term goes on and it may be that the "easier" material is better learned online and the harder material is better learned on-campus (or vice versa). Any thoughts?

  • Andrew Wheeler

    How about some pre-class test material with which you can match students on, presumably controlling for baseline differences? Of course that is problematic in that the pre-class test material would be difficult to effectively measure, but it is feasible for instances in which randomization is not possible (like your colleagues).

    Your design seems like a good idea to me, but what is the motivation for the cross-over at all? I.e. why not randomly assign students to either take the entire class in-person or online?

    I would be most concerned about self-selection artifacts for the online students (above and beyond just demographic differences). Are they more proactive, engaged "go-getters" than in class students? Are online students taking as a optional elective whereas local students is it a mandatory class? These seem to be really difficult confounds to address if they are present.

    Interesting problem!

    • Roger Peng

      I think the idea of the crossover is mostly practical. Students who paid to be on campus might expect to spend a little time in the classroom. Crossing people over would at least allow them that time as opposed to spending the entire class online.

      Your comment about self-selection is exactly the problem with comparing online vs. on-campus, which makes the observational setting a bit challenging to interpret. On the other hand, it may be difficult to generalize findings from an experimental study conducted on-campus.

      • Keith O’Rourke

        Pratical but non-interpretable (without untestable assumptions) due to possible carry-over effects. Stephen Senn has written most dramatically about this.
        Unless you have physics or strong biological rationale to rule out carry-over, it is not logically better than an observational study - unmeasured confounding just being replaced with unknown carry-over effect (very poorly informed by study data)
        The challenge is administrative - an informed administration would find a way for an appropriate randomized study...

  • jimmy

    would this be another possible issue with a crossover - if the coursework built on previous topics? and so, later material would be much harder if one did not learn the previous material well. if there are differences in learning between types of format, would this not be a problem?

  • Ken Beath

    The factors that would be most important would be number of courses attended, and work commitments. It is not easy having a job and enrolled for two subjects.