Usually I don’t post answers to questions right away, but Mark Liberman was kind enough to answer my question yesterday so I think I should reciprocate. Mark asks:
I’ve been playing around with data from Coursera transaction logs, for an economics course and a modern poetry course so far.
For the Modern Poetry course, where there’s quite a bit of activity in the forums, the instructor (Al Filreis) is interested in what the factors are that lead to discussion threads being longer or shorter. For example, he wonders whether his own (fairly frequent) interventions have the effect of prolonging discussion or cutting it off.
Some background explorations are here with the relevant stuff mostly at the end, including this.
With respect to Al’s specific question, my thought was to look at each of his comments, each one being the nth in some sequence, and to look at the empirical probability of continuing (at all, or perhaps for at least 1,2,3,… additional turns) in those cases compared to the same numbers for nth comments contributed by others.
More generally, I wondered about doing some kind of regression to predict the probability of continuation as a function of whatever factors (current length of the thread, size of the current comment, some function of the words in the current comment, etc.) But this is presumably a well-trodden path in survival analysis, about which I know very little. Any suggestions for reading material?
My reply: I think just about any analysis you do will be useful. (I say this partly because Mark’s analyses on Language Log always look pretty excellent to me, also because more generally it’s a good idea to play with the data rather than sitting in the corner trying to come up with the ideal analysis.)
But, to be more specific, my inclination, given the way the question was framed, is to set it up as an observational study. In this case, the experimental items are class periods, or segments of class periods, the “treatment group” are those periods where Al intervenes, and the “control group” are those periods where Al doesn’t intervene. To me, the natural way to proceed is to put together a bunch of treatment and control cases from your data, then get pre-treatment background variables (time of day, day of week, time during the semester, whether there is an exam coming up, number of discussions the previous day, etc.) and compare various outcomes of interest (basically, whatever flows from the treatment). The treatment itself can be considered as a continuous variable. Chapter 9 of my book with Jennifer should give the basic idea. Any “survival analysis” aspect of the problem will come up naturally in terms of the pre-treatment variables used in the regression, or in the coding of the post-treatment outcomes.
Also, if you have enough data, you can try to uncover treatment interactions, so that the question is not, “Does the prof’s interventions prolong discussion or cut it off?” but, rather, “Under what circumstances do the prof’s interventions prolong discussion” etc.
This sounds like fun! I totally agree with Andrew’s basic conceptual outlook, but a couple suggestions for funs:
1 – One graph might be something like an x-axis of “number of comments since first professor comment” that goes from -t to t, and a y-axis with “fraction of papers with at least N more comments” – sort of treating it like an event study, using only the threads with a professor comment in them. Similarly, you could have “number of comments” on the x-axis, and then draw two lines, one for “number of more comments given N’th comment was a student” and one for “number of more comments given Nth comment was from professor.” Neither is perfect, but both give you something to see. You might have to trim the data differently for both of these and think about the sample selection (how many threads have how many comments).
2 – Go to the IRB. Get a waiver (you can always experiment on students’ education!). Then: I expect there are two kinds of professor comments: 1) responding directly to a question or necessarily shutting down a topic; 2) Chiming in with a quick thought. I think the important pedagogical question is probably about (2), about whether or not in those marginal cases the teacher should jump in or not . Do (1) always (every thread may need some moderation), but, when the professor has a comment that he thinks falls into (2), then he presses a button that randomly determines whether or not he posts. You take the threads on which the professor has pressed the button, recording what N’th comment “would have been the professors or was” and then you trace out (something like above in “event time”) the effect of the comment on probability of more comments – so you have a treatment group event study and a control group placebo test.
A professor’s comment cutting off a discussion could be a good thing if it occurs because he has answered the question of the original poster. I doubt that the length of a forum thread is a good measure of quality — think of Godwin’s Law.
It seems to me that the professor is interested in the causal effect of his intervention, so we have to worry about (or at least think about) the length that threads he intervenes in would counterfactually have had had he not intervened, and vice versa. If the factors that influence thread length also influence his decision to intervene, then we’ll observe so-called “spurious correlation”.
In addition to investigating the return on frequency of intervention, I would be interested in knowing the effect of “intervention quality.”
would be interesting to break things down by length of comment and whether or not the comment was or included a question. Also I agree with @Corey that the prof should record the times when he or she was about to comment, but the trial-randomizing computer on her or his desk said “nope, you ain’t commenting here”.