While visiting the education school at the University of Pennsylvania a couple months ago, I had a long conversation with Bob Boruch, a prominent researcher in the field of evidence-based education. We shared Fred Mosteller stories and talked about a lot of other things too.
Boruch sent me an article about teaching randomized controlled trials to education students, which gave me the following idea which connects to my longstanding embarrassment (and subject of my next column on ethics, forthcoming in Chance magazine) about the lack of systematic measurement, sampling, or experimentation in our own teaching efforts.
Anyway, here’s my idea for experimentation in statistics teaching, an idea that I think could work particularly well in classes with education students.
Each class could, as part of the course, design an educational experiment to be performed on next year’s class. Easier said than done, I know, but perhaps ed school students would be particularly motivated to do this.
Why not do a similar experiment across multiple schools for a single year (i.e. a class you teach at Columbia and an equivalent class taught at NYU). That way the students see the results in real time and can relate better no?
I have taken to randomly selecting six students each week in my ninety-student data analysis class, and have them come in and spend 15–30 minutes explaining to me what they did on the last problem set and answer my questions about it. It’s extremely informative for me, but also a complete pain.
Cosma: Not to pick on you – you are not the worst in this regard, but your learning objectives for your undergraduate course
“you should be able to (1) select appropriate methods, (2) use statistical software to implement them, (3) critically evaluate the resulting statistical models, and (4) communicate the results of their analyses to collaborators and to non-statisticians.”
After 25+ years and a Phd from a reasonable school (Oxford, Stats Dept) my batting average here is quite low on all but 2 and my current posterior 95% interval for the number of individuals with a high batting average on all 4 in North America is 6 to 12.
Are we not setting up unreasonable expectations?
I would definitely love to introduce such methods into my approach to teaching, but I can see it causing a big problem. Suppose I randomly assign half of my class to treatment A, and the other half to treatment B. Now, it turns out that, at the end of the semester, the grades are substantially (not “just” statistically) higher for those students receiving treatment A. If I was a student receiving treatment B, I’d appeal my grade. And guess what? I’d win.
Solution 1. Standardize grades within treatment.
Solution 2. Keep grades private.
Solution 3. Console students that at least they were not in a cancer treatment clinical trial.
Apparently solution 1 was proposed by Valen Johnson for Duke University and abandoned due to pressure from student groups.
Guess what, like everyone one else no one really wants to be fairly evaluated
– not really!
(Now they work here http://www.mdanderson.org/education-and-research/departments-programs-and-labs/departments-and-divisions/division-of-quantitative-sciences/index.html )
Better to say “no one who got a lucky break really wants to be fairly evaluated”. I’m sure those on the losing treatement arm would like to standardize grades within treatment.
Solution 4: build a model and use predicted grades
that would go over well :) and really test our beliefs in our methods