What they’re saying about “blended learning”: “Perhaps the most reasonable explanation is that no one watched the video or did the textbook reading . . .”

template_cell

Someone writes in:

I was wondering if you had a chance to see the commentary by the Stockwells on blended learning strategies that was recently published in Cell and which also received quite a nice write up by Columbia. It’s also currently featured on Columbia’s webpage.

In fact, I was a student in Prof. Stockwell’s Biochemistry class last year, and a participant in this study, which was why I was so surprised that it ended up in Cell and received the attention that it did.

I was part of the textbook group, for which he assigned over 30 pages of dense textbook reading (which would probably have taken multiple hours to fully digest, and was more than 2-3 times more than what he’d assign for a typical class), so I’m sure the video was much more tailored to the material he covered in class and ultimately quizzed everyone on. Moreover, in his interview Stockwell claims that he’ll “use video lectures and assign them in advance,” rather than relying exclusively on a textbook, yet it was surprising that in their commentary they write:

We also compared the exam scores of students in the textbook versus video preparation groups but found no statistically significant difference in this relatively modest sample size, despite the trend toward higher scores in the group that received the video assignment.

Perhaps the most reasonable explanation is that no one watched the video or did the textbook reading for a class that wasn’t going to be covered on any of the exams? What’s even more confusing to me is that they admit the sample size of the textbook/video groups were “modest,” but are readily able to draw conclusions about which of the 4 arms provides the most effect model for learning, when each arm had half as many participants as these two larger groups! I’m not sure if that’s just confirmation bias, or if the results are truly significant? I’m also not sure if the figure in the paper is mislabeled since Group 2 and 3 and panel A are different than what’s used in panel D (see above).

Do you have any thoughts on the statistical power of such a study?

I know the above seems a little bit like I have an axe to grind, but it seemed to me like the conclusions of this experiment were quite reaching, especially for such a short study with so few participants, and I was wondering what someone else with more expertise on experimental design than I have thought.

I had not heard about this study and don’t really have the time to look at it, but I’m posting it here in case any of you have any comments.

As to why Cell chose to publish it: This seems clear enough. Everybody knows that teaching is important and it’s hard to get students to learn, we try lots of teaching strategies but there are not a lot of controlled trials of teaching methods, so when there is such a study, and when it gives positive results with that magic “p less than .05,” then, yeah, I’m not surprised it gets published in a top journal.

13 thoughts on “What they’re saying about “blended learning”: “Perhaps the most reasonable explanation is that no one watched the video or did the textbook reading . . .”

  1. So just to be clear: you can experiment on 111 students, split into 4 treatment arms (N=27, 28, 28 28), investigate a bunch of permutations of comparisons possible with the data (each group independently, then groups 1+2, 1+3 2+4, 3+4), make no mention of multiple hypothesis testing corrections, conclude that these comparisons “suggests that textbook preparation for a science class is less satisfactory and engaging for students compared to assignments that involve watching a video”, and get it published in Cell. Oh, and it appears the experiment lasted exactly 1 class-day*.

    Also this, from the conclusion: “These results also illustrate the feasibility of using the clinical trial methodology in educational intervention evaluations.” Yeah, no one had ever done that before. Unless they mean they were giving the students drugs. In which case, I’d think more than 111 of the 172 enrolled students would have agreed to participate.

    Someone: I’m sorry you didn’t get to see these apparently very entertaining videos that would have made you very excited to go to class, and only got extra reading instead (in Psychology, I think they code that as “-1” for treatment level). But I’m happy to confirm your suspicions that we did not learn much from this.

    *This claim cannot currently be verified from reading the paper, but seems like it is true. The authors write: “Unfortunately, we were only able to examine the effects of teaching style in one class” which at first I thought meant “one course” but I now think means actually one day of class. Maybe Someone can confirm.

  2. I’m someone who did an entire MSc in Statistics over the internet over three years, using “blended learning” in the sense that I had only lecture notes, a mailing list, and graded homework as learning tools (plus the final exams). So a sample size of 1, but lots of (subjective) data. In the final year, they added video recordings of the lectures (the course is taught physically as well as online, simultaneously, so lectures are given to students but the distance students couldn’t see them). The video lectures added a lot more information than the notes themselves. One also learns by watching the professor do the derivations, because he walks you through the reasoning in a way that lecture notes can’t. Importantly, I learnt to mentally model the professor’s approach to attacking a derivation; I could only do this because he was spelling out aloud how he was thinking about it. This is not conclusive of course, but this was the only course I got a distinction in (Time series and modeling dependent data).

    So, assuming that the student is diligent (which I was; I did every exercise or HW ever assigned), lecture notes + video is superior to lecture notes alone. I then introduced this to my intro stats class; I add video lectures to my normal lectures. Several students have told me that the video lectures greatly help them over and above the lecture notes and exercises. Actually having a person walk you through the argument has a great impact on learning. Reading a book or the like silently leads to poorer outcomes.

  3. The consistency in Panel B doesn’t bother anyone?

    The results are consistent with the Freeman et al meta analysis of active learning http://www.pnas.org/content/111/23/8410.abstract . It’s really just a replication study, and in that sense the fact that the passive groups, whether lecture or video, did worse than the problem solving groups basically confirms what has been found over and over about STEM learning.

    Typical students won’t read or watch unless they have good reason to and it pays off in better grades, I think that’s the very common. Most textbooks are awful for someone who doesn’t know a lot to just sit and read on their own. I think the reason you would assign video lectures in advance is that they are no worse than assigning the textbook and in either case not relying on lectures for information transfer and thereby opening up the classroom time for problem solving or other active learning is what is going to get the results.

    • re: Panel B – “The students were stratified for randomization by gender and prior exam performance (low: lower third versus high: upper two-thirds) into each of the four arms to ensure equal representation of these students in each study arm”

      • Oh okay. The image says that “randomization” ensured that, it should have said stratification, but if it was stratified why report p values on the differences?

Leave a Reply

Your email address will not be published. Required fields are marked *