Skip to content
 

Some quick disorganized tips on classroom teaching

Below are a bunch of little things I typically mention at some point when I’m teaching my class on how to teach. But my new approach is to minimize lecturing, and certainly not to waste students’ time by standing in front of a group of them, telling them things they could’ve read at their own pace.

Anyway, here I am preparing my course on statistical computing and graphics and thinking of points to mention during the week on classroom teaching. My old approach would be to organize these points in outline format and then “cover” them in class. Instead, though, I’ll stick them here and then I can assign this to students to read ahead of time, freeing up class time for actual discussion.

Working in pairs:

This is the biggie, and there are lots of reasons to do it. When students are working in pairs, they seem less likely to drift off, also with two students there is more of a chance that one of them is interested in the topic. Students learn from teaching each other, and they can work together toward solutions. It doesn’t always work for students to do homeworks pairs or groups—I have a horrible suspicion that they’ll often just split up the task, with one student doing problem #1, another doing problem #2, and so forth—but having them work in pairs during class seems like a no-lose proposition.

The board:

Students don’t pay attention all the time nor do they have perfect memories; hence, use the blackboard as a storage device. For example, if you are doing a classroom activity (such as the candy weighing), outline the instructions on the board at the same time as you explain them to the class. For another example, when you’re putting lots of stuff on the board, organize it a bit: start at the top-left and continue across and down, and organize the board into columns with clear labels. In both cases, the idea is that if a student is lost, he or she can look up at the board and have a chance to see what’s up.

Another trick is to load up the board with relevant material before the beginning of class period, so that it’s all ready for you when you need it.

The projector:

It’s becoming standard to use beamer (powerpoint) slide presentations in classroom teaching as well as with research lectures. I think this is generally a good idea, and I have just a few suggestions:
– Minimize the number of words on the slides. If you know what you’re talking about, you can pretty much just jump from graph to graph.
– The trouble with this strategy is that, without seeing the words on the screen, it can be hard to remember what to say. This suggests that what we really need is a script (or, realistically, a set of notes) to go along with the slide show. Logistically this is a bit of a mess—it’s hard enough to keep a set of slides updated without having to keep the script aligned at the same time—and as a result I’ve tended to err on the side of keeping too many words on my slides (see here, for example). But maybe it’s time for me to bite the bullet and move to a slides-and-script format.

Another intriguing possibility is to go with the script and ditch the slides entirely. Indeed, you don’t even need a script; all you need are some notes or just an idea of what you want to be talking about. I discovered this gradually over the past few years when giving talks (see here for some examples). I got into the habit of giving a little introduction and riffing a bit before getting to the first slide. I started making these ad libs longer and longer, until at one point I gave a talk that started with 20 minutes of me talking off the cuff. It seemed to work well, and the next step was to give an entire talk with no slides at all. The audience was surprised at first but it went just fine. Most of the time I come prepared with a beamer file full of more slides than I’ll ever be able to use, but it’s reassuring to know that I don’t really need any of them.

Finally, assuming you do use slides in your classes, there’s the question of whether to make the slides available to the students. I’m always getting requests for the slides but I really don’t like it when students print them out. I fear that students are using the slides as a substitute for the textbook, also that if the slides are available, students will think they don’t need to pay attention during class because they can always read the slides later.

It’s funny: Students are eager to sign up for a course to get that extra insight they’ll obtain from attending classes, beyond whatever they’d get by simply reading the textbook and going through the homework problems on their own. But once they’re in class, they have a tendency to drift off, and I need to pull all sorts of stunts to keep them focused.

The board and the projector, together:

Just cos your classroom has a projector, that don’t mean you should throw away your blackboard (or whiteboard, if you want to go that stinky route). Some examples:
– I think it works better to write out an equation or mathematical derivation in real time rather than to point at different segments of an already-displayed formula.
– It can help to mix things up a little. After a few minutes of staring at slides it can be refreshing to see some blackboard action.
– You can do some fun stuff by projecting onto the blackboard. For example, project x and y axes and some data onto the board, then have a pair of students come up and draw the regression line with chalk. Different students can draw their lines, then you click onto the next slide which projects the actual line.

Handouts:

Paper handouts can be a great way to increase the effective “working memory” for the class. Just remember not to overload a handout. Putting something on paper is not the same thing as having it be read. You should figure out ahead of time what you’re going to be using in class and then refer to it as it arises.

I like to give out roughly two-thirds as many handouts as there are people in the audience. This gives the handouts a certain scarcity value, also it enforces students discussing in pairs since they’re sharing the handouts already. I found that when I’d give a handout to every person in the room, many people would just stick the handout in their notebook. The advantage of not possessing something is that you’re more motivated to consume it right away.

Live computer demonstrations:

These can go well. It perhaps goes without saying that you should try the demo at home first and work out the bugs, then prepare all the code as a script which you can execute on-screen, one paragraph of code at a time. Give out the script as a handout and then the students can follow along and make notes. And you should decide ahead of time how fast you want to go. It can be fine to do a demo fast to show how things work in real life, or it can be fine to go slowly and explain each line of code. But before you start you should have an idea of which of these you want to do.

Multiple screens:

When doing computing, I like to have four windows open at once: the R text editor, the R console, an R graphics window (actually nowadays I’ll usually do this as a refreshable pdf or png window rather than bothering with the within-R graphics window), and a text editor for whatever article or document I’m writing.

But it doesn’t work to display 4 windows on a projected screen: there’s just not enough resolution, and, even if resolution were not a problem, the people in the back of the room won’t be able to read it all. So I’m reluctantly forced to go back and forth between windows. That’s one reason it can help to have some of the material in handout form.

What I’d really like is multiple screens in the classroom so I can project different windows on to different screens and show all of them at once. But I never seem to be in rooms with that technology.

Jitts:

That’s “just in time teaching”; see here for details. I do this with all my classes now.

Peer instruction:

This is something where students work together in pairs on hard problems. It’s an idea from physics teaching that seems great to me but I’ve never succeeded in implementing true “peer instruction” in my classes. I have them work in pairs, yes, but the problems I give them don’t look quite like the “Concept Tests” that are used in the physics examples I’ve seen. The problem, perhaps, is that intro physics is just taught at a higher level than intro statistics. In my intro statistics classes, it’s hard enough to get the students to learn about the basics, without worrying about getting them into more advanced concepts. So when I have students work in pairs, it’s typically on more standard problems.

Drills:

In addition to these pair or small-group activities, I like the idea of quick drills that I shoot out to the whole class and students do, individually, right away. I want them to be able to handle basic skills such as sqrt(p*(1-p)/n) or log(a*x^(2/3)) instantly.

Getting their attention:

You want your students to stay awake and interested, to enter the classroom full of anticipation and to leave each class period with a brainful of ideas to discuss. Like a good movie, your class should be a springboard for lots of talk.

But you don’t want to get attention for the wrong things. An extreme example is the Columbia physics professor who likes to talk about his marathon-fit body and at one point felt the need to strip to his underwear in front of his class. This got everyone talking—but not about physics. At a more humble level, I sometimes worry that I’ll do goofy things in class to get a laugh, but then the students remember the goofiness and not the points I was trying to convey. Most statistics instructors probably go too far in the other direction, with a deadpan demeanor that puts the students to sleep.

It’s ok to be “a bit of a character” to the extent that this motivates the students to pay attention to you. But, again, I generally recommend that you structure the course so that you talk less and the students talk more.

Walking around the classroom:

Or wheeling around, if that’s your persuasion. Whatever. My point here is that you want your students to spend a lot of the class time working on problems in pairs. While they’re doing this, you (and your teaching assistants, if this is a large so-called lecture class with hundreds of students) should

Teaching tips in general:

As I explained in my book with Deb Nolan, I’m not a naturally good teacher and I struggle to get students to participate in class. Over the decades I’ve collected lots of tricks because I need all the help I can get. If you’re a naturally good teacher or if your classes already work then maybe you do without these ideas.

Preparation:

It’s not clear how much time should be spent preparing the course ahead of time. I think it’s definitely a good idea to write the final exam and all the homeworks before the class begins (even though I don’t always do this!) because then it gives you a clearer sense of where you’re heading. Beyond that, it depends. I’m often a disorganized teacher and I think it helps me a lot to organize the entire class before the semester begins.

Other instructors are more naturally organized and can do just fine with a one-page syllabus that says which chapters are covered which weeks. These high-quality instructors can then just go into each class, quickly get a sense of where the students are stuck, and adapt the class accordingly. For them, too much preparation might well backfire.

My problem is that I’m not so good at individualized instruction; even in a small class, it’s hard for me to keep track of where each student is getting stuck, and what the students’ interests and strengths are. I’d like to do better on this, but for now I’ve given up on trying to adapt my courses for individuals. Instead I’ve thrown a lot of effort into detailed planning of my courses, with the hope that these teaching materials will be useful for other instructors.

Students won’t (in general) reach your level of understanding:

You don’t teach students facts or even techniques, you teach them the skills needed to solve problems (including the skills needed to find the solution on their own). And there’s no point in presenting things they’re not supposed to learn; for example, if a mathematical derivation is important, put it on the exam with positive probability. And if students aren’t gonna get it anyway (my stock example here is the sampling distribution of the sample mean), just don’t cover it. That’s much better, I think, than wasting everyone’s time and diluting everyone’s trust level with a fake-o in-class derivation.

The road to a B:

You want a plan by which a student can get by and attain partial mastery of the material. See discussion here.

Evaluation:

What, if anything, did the students actually learn during the semester?

You still might want to evaluate what your students are actually learning, but we don’t usually do this. I don’t even do it, even though I talk about it. Creating a pre-test and post-test is work! And it requires some hard decisions. Whereas not testing at all is easy. And even when educators try to do such evaluations, they’re often sloppy, with threats to validity you could drive a truck through. At the very least, this is all worth thinking about.

Relevance of this advice to settings outside college classrooms:

Teaching of advanced material happens all over, not just in university coursework, and much of the above advice holds more generally. The details will change with the goals—if you’re giving a talk on your latest research, you won’t want the audience to be spending most of the hour working in pairs on small practice problems—but the general principles apply.

Anyway, it was pretty goofy that I used to teach a course on teaching and stand up and say all these things. It makes a lot more sense to write it here and reserve class time for more productive purposes.

One more thing

I can also add to this post between now and the beginning of class. So if you have any ideas, please share them in the comments.

100 Comments

  1. Do you just use the built-in R script editor for teaching? That surprises me a little because it lacks syntax highlighting and other more advanced features. It would seem useful to students to be able to visually distinguish function names from other syntax. At least I’ve encountered confusion with that (granted it is during teaching of R novices, whereas I’m sure you’re only doing this in more advanced courses).

    • Phil says:

      I don’t know any experienced R user who types stuff into the command line. Everyone uses an editor.

    • Sam Clifford says:

      At no point in my subject do we use the R GUI, it’s always RStudio. It’s rare that I’ll fire RStudio up during a lecture but I try to point out when I’m using square brackets for indexing, the names of functions, data frames, etc. “So let’s make an object called rats.lm and assign it the value of this *L*inear *M*odel where the weight is regressed on the age and these variables live in my rats.data data frame”. Telling the students what your doing makes it seem less like voodoo.

  2. question says:

    I always wish my professors had spent more time on the history and original publications behind each concept and discuss the points of controversy and why they find the evidence/logic convincing or not. I feel like teaching a high-level anonymized version of things may have done me more harm than good, by the end I found alot of what I believed really came down to “someone wrote words in a book” and/or “the professor says so”. This may mean the class has to be more limited in scope though.

    • Michael Lew says:

      I always include some historical detail in my statistics lectures. Some students like it, but many say that it is an unwelcome distraction. However, I think that it is important that the students see that choices among statistical methods and philosophies are often not much more than matters of taste or opinion.

      Making explicit the human aspects of the historical development of statistics reduces the tendency of the introductory-level students to assume that stats is nothing more than an approved set of recipes and rules for inference. I don’t know how to achieve that end in any other way.

      • question says:

        Michael,

        I was referring to my education in general, but I think the historical aspect of statistics is even more important than most fields. Ignoring the history is like taking the most controversial topic and making it appear as dry as possible.

  3. Phil says:

    I vastly prefer chalkboards to whiteboards. I know I’m in the minority — I know this makes me sound like an egotistical jackass but it’s what I think — most people want things to look good even if they’re less useful, whereas I value utility more. I have a chalkboard in my office, and about five colors of chalks on the tray. (I bought several boxes with multiple colors). You can shade by turning the chalk on its side; you can pick up a chalk, use it, and put it down again without having to pull a cap on and off. You can make a dark line by pushing hard, or a subtle line by doing it gently.

    As for whiteboards: There are probably a lot of colors of pens, but all anyone seems to have is about two or three. You can’t tell that one is going to run out until it runs out. Even the ones that don’t stink have an unpleasant smell. You have to re-cap them. You can make different widths of line, but not different intensities. If you leave something on the board for a long time it becomes very hard to erase, even if you use a solvent.

    Whiteboards do look better, and they don’t produce the mess that chalk dust does. Other than that, I give the advantage to chalkboards in every way. Whenever I find myself in a conference room with a whiteboard, and there are three pens, only one of which still works, and I have to use a sprayer with solvent in order to erase what is already on the board, I find myself getting way more frustrated than the situation actually warrants. Not because the situation is really so dire — OK, I only have one pen, I can deal — but because it is such an egregious example of people choosing something that is supposed to be cool even though it is deficient in almost every way to the thing it replaces. Indeed, I think this is the best example I know.

    • I’ve become a great fan of using a document camera in class — it’s much better than a chalkboard or whiteboard. I write and draw on paper with nice markers, in whatever colors I want, and the page is magically projected on the screen. (I might agree that chalkboards are better than whiteboards, except that when I’ve used chalkboards, in a class in which I also used a computer, I always ended up with dust all over my keyboard.) In general, I’ve been pushing the use of powerpoint slides to as close to zero as possible, using them only for images or graphs — students grasp words (and especially equations) vastly better if I write them out by hand than if they see them pre-made on a slide, which should come as a shock to no one.

      • If you’ve got a 5 megapixel document camera or something, then fine. But I used to find the low-res VGA crap just blurry, limited, and awful, like looking at a chalkboard through out of focus binoculars.

        The amount of material you can see on a wall sized chalkboard all at once is just vastly larger than the tiny zoom of the document camera.

      • Robin Morris says:

        The lecture hall I most prefer has two document cameras. That way I write on one, and when the sheet is full, I move it over to the other one, so there’s always the page before available to refer to/for those who need more time to keep up.

        At the end of the class I then have a set of notes which I scan and put on the course website.

        I used to get comments along the lines of “please prepare slides in advance”, ie powerpoint, but those have gone away in the last few years, as, I hope, the students have recognized that powerpoint is about the worst medium for this type of material — it helps to see the math developed, not just put down in its completed state.

      • Phil says:

        One of the things I like about the chalkboard is the ease of erasing. Indeed, it’s not all that unusual for me to step to my chalkboard even when I’m alone in my office.

        I don’t teach, though; most of the time my chalkboard (or our meeting room whiteboard) gets used is when a small group is working on something together. For this, I think a chalkboard is the way to go. For teaching, when you have things planned out and you won’t have to erase, I can believe a document camera would be useful.

        I haven’t found chalk dust to be a problem. I don’t store anything important on the cabinet below the chalkboard, because some dust does fall down there, but dust production is almost miraculously low. Obviously every piece of chalk ends up completely pulverized (except for the little unusable chalk nubbins that I eventually throw away), so there must be dust…and yet, I only clean the eraser very rarely, and I clean out the tray even less. It really is a puzzle.

      • Martha Smith says:

        I’m also a fan of the doc cam. I often use it as a cross between a power point and a blackboard: I make up lecture notes that I put online for students to download and bring to class. But the notes have gaps in them (for example, just the start of a calculation; or part of a drawing; or places for reasons to be supplied; or questions to be answered). I fill these in during class, but by asking students how to fill them in. This gets the class involved, but students don’t get bogged down taking lots of notes as they might with a blackboard lecture.
        I keep a zippered “pencil bag” with different colored markers to write with — and colored chalk as well, in case there’s a blackboard that I use for questions that come up that I haven’t prepared anything for; and put in any other small stuff that I’m likely to need in class. Helps keep me organized, since that doesn’t come naturally to me.

        • Simon says:

          Tablets make a good cross between the blackboard and a PowerPoint on a laptop/desktop. You can annotate slides etc. with a decent wireless setup you can wander round the theatre with the tablet and annotate as you go along.

    • I think some of those are solvable problems, but no-one actually takes any initiative. I bought myself a 12 pack of 6 packs of low-odor small multi-colored pens and carried them around in my backpack, and left some on my desk. Whenever I wanted to use a whiteboard I could whip them out and do whatever I wanted.

      The low odor markers are the only ones I’ll use. Uncap a “bold” marker and within minutes I’m too dizzy to do math, or much of anything. When they first installed them in my middle school back in the 80’s I used to just jump up in the middle of class and walk over to the windows and open them all wide, open the door wide… the teacher found it disturbing the first few times but when I explained to him that I simply couldn’t function with that level of VOC he stopped complianing, and learned to actually open the windows and things himself.

      The mess of chalk dust especially in an electronics heavy office is a serious downside. in a special classroom I much prefer chalk.

      The big thing that people need to realize though is that watching someone create a whole wall of information in front of your eyes is TOTALLY different from watching either pre-made slides, or one tiny window into that wall worth of information (document camera).

  4. Bill Jefferys says:

    With iPhone in hand, I will snap pictures of the whiteboard/blackboard and post them on the class blog.

  5. Rahul says:

    I found it helpful to ask someone else what they’d want to quiz students on after a certain course. Doing the teaching & evaluation by yourself sometimes severely biases the class towards idiosyncratic material that only the instructor cares about.

    It’s hard to make the jump from what one loves to teach to material that’s actually going to prove useful to the students.

    About projectors, I think the traditional overhead slide projector (non powerpoint) is an underrated tool.

  6. Rahul says:

    Powerpoint seems particularly good when using it as a medium to project material not made in powerpoint itself. e.g. In engineering courses, the ability to show students sketches, photos, CAD drawings, complex schematics, charts etc. is great.

    But most of the graphics created in PPT itself suck. So also, if you use powerpoint to project bullet points it becomes lame.

    With equations etc. I think the main limitation of projection still is that resolutions remain so low that you can’t yet cram anything close to a blackboard / whiteboard’s worth of information density into a slide without it becoming illegible.

    • Jim says:

      I use a tablet and its pen and Microsoft OneNote. I can have any color and thickness of pen I want and can embed all sorts of objects – images, videos etc. I also record my voice and post the notes after the class. We do pair work in class so students still need to attend class. This talk of chalkboard versus whiteboard vs document camera seems so old-fashioned. With the tablet I am still creating notes on the fly and therefore responding to class questions and most students are still transcribing the notes into their own notebooks – an important learning discipline. I do lose the ability to refer back and compare simultaneously to what is currently projected overhead. I have to scroll back. You can do this better if you have a classroom full of boards but we have few rooms with that much wall space.

  7. Jack says:

    As a fairly recent student, I’d like to toss in some comments on working in pairs.

    You are completely correct that if you assign homework to pairs, they will split the work. Maybe they’d cooperate with a very difficult problem set with only a few problems, but for introductory stats? People take the option that requires less work.

    More importantly – for at least some of your students, maybe a lot of them, working in pairs is miserable. If Alice is interested in the material and Bob is hung over and never cared much in the first place, Alice is going to be under a lot of pressure to just do the work herself. Or she’ll feel social pressure to feign indifference. Bob doesn’t even have to express any dislike for the material, he could just want to talk. Again, a much bigger problem in an introductory class, and one that just makes things worse for the people who might be encouraged to study statistics further. Even if Bob is interested in the work, he might struggle with it in a way that Alice doesn’t know how to help with. Teaching stuff is hard, and if you just throw someone into it without the necessary skills it’s frustrating for everyone involved.

    If both of Alice and Bob are interested and get the material, it isn’t much different than if they were working alone or splitting the work. If neither Alice nor Bob is interested, they’ll just talk about other stuff and end up doing less work than they would have alone. And all of that assumes the students are sociable – some people find social interaction really god damn stressful.

    This is why the lack of testing in teaching methods drives me nuts. It seems like you’re laboring under a huge misconception that just isn’t visible to you, because any time the professor is watching the students are going to change their behavior. You have absolutely no idea whether it’s working unless you look a the results.

    Apologies if this is too much of me venting. I’m sure working in pairs can help – but it requires a particular set of circumstances and people, which in my experience was pretty rare. And it’s the sort of thing that would have convinced me to skip a class and figure out the homework assignments from the textbook.

    • Rahul says:

      OTOH, maybe it is good training for the real world? In small doses.

      On real projects it’s quite typical to find a mix of motivations and abilities. Learning to cope with whatever you get there is somewhat crucial to success.

      • Jack says:

        I figured this would be a response. The lack of data is really hurting, here – I don’t like arguing from personal impressions and experience, but I feel like it’s all I have to go on. If anyone knows of more objective sources, please share.

        When I first started working in an office, my boss took me around and introduced me to everyone I’d be working with. For the first projects I was assigned to, there was a pretty clear idea of what needed to be done. I got direct instruction from my boss or just people who were more experienced than me, and if I had questions there was basically always a ready answer. Over time I got to do more collaborative work, where I was actively participating in problem-solving. But leading up to that point there was a period of at least several months of less difficult collaboration, small talk, and listening to stories about possums. All of that taught me something about people’s skills and motivations, while laying down some basis for a social relationship.

        Contrast that to group work in a college course. You know next to nothing about the people your working with, the only time you interact is during the group work itself. Maybe you have a friend or two in the class, but most people won’t. This might not hold in an upper-level course in a small department, but I think it’s typical of an introductory or intermediate class.

        The group work might help prepare you for the real world, a bit. But collaboration is hard, full stop. There’s a reason that I didn’t jump right into my job as an equal partner, and it’s not just background knowledge. The ability to manage people on a project has a high market value because it’s scarce and valuable. And if you require that students try and handle it at the same time as they learn new material, particularly in a more technical course like statistics, it’s going to hurt their ability to learn the subject they signed up to study.

        It might be better for a business course, where the interpersonal side is a large part (maybe most) of what students are there to learn. But that doesn’t make sense for math or statistics.

        • I agree with this post. I’m in the interesting position of being a professor but also a student (in an MSc program in statistics). I have experience in collaborative work, both in academia and in industry (preparing patent applications in a law office). Doing simulated consulting in the MSc program didn’t do much to help me learn how to collaborate, it was mostly a mess. This had partly to do with the fact that the collaborators usually didn’t know each other (personal relationships really matter), but the most important problem was that there was no boss who made the final call. That was a very unrealistic situation and it hurt our efforts at joint work a lot. I found these simulated consulting settings not very helpful; the same goes for group work. I learnt the most when I was struggling with a problem alone. In retrospect, and knowing what real-world collaboration is like, my conclusion is that one should do away with all this collaborative work stuff during the course-work period when one is focused on learning things. Instead, it may be useful to give some lectures on what it’s like out there in the real world, with perhaps some case studies. There are good books on running a lab (e.g., Lab Dynamics, At the Helm) that provide that kind of insight. Also, one learns to run and work in a collaborative team only by hard experience. Some people will never learn to do this no matter how much you try to train them or teach this, and some people learn from their mistakes. For learning statistics, studying and working alone (in the context of formal course-work that is examined through timed exams; I’m not talking about self-study without feedback) has been the most effective way for me. The only thing that really helped me improve a lot was detailed feedback on homework and project submissions.

    • Mikhail Shubin says:

      Being a student, I always hated the pair works. Teachers should recognize the existence of introverts in the classrooms.

      • Andrew says:

        MIkhail:

        If they want to do statistics, introverts need to learn to communicate and work with others. Nobody said that college was going to be easy.

        • James Cunningham says:

          Eh. If introverts wish to do much of *anything* they need to learn to communicate and work with others. But I’d question whether highly frequent pair work furthers the goal of instilling good communication skills, and I doubt very much that it helps teach the course material itself any better. Pairing off to solve a classroom problem is artificial and every instance of doing so incurs fixed costs in assembly and ensuring that everyone understands the problem.

          The value of group work doesn’t even come into play except for slightly longer projects, where different strengths can be sorted out and tasks divided thereby. What you describe always seemed to bring class to a screeching halt.

          • Andrew says:

            James:

            I admit I’m no expert on the research here but it was my impression that there have been some studies supporting the idea that in-class work in pairs or small groups is effective. It certainly feels like it’s worked in my classes. It does not bring my classes to a screeching halt, nor does it incur any extra costs. When students work individually they also need to understand the problem.

            As noted above, I’ve often had difficulty getting class participation, and pair activities have been a way to keep students involved. Some instructors can keep students involved by other means, and that’s fine with me too.

            • Anonymous says:

              > I admit I’m no expert on the research here but it was my impression that there have been some studies supporting the idea that in-class work in pairs or small groups is effective. It certainly feels like it’s worked in my classes.

              Well, if you’re no expert on the research then I’m worse than no expert: I only have the experience of what seemed to work (and not work) during my time as a student. The tiny amount of teaching I did was at the graduate level, and the same rules may not apply there anyway.

              > It does not bring my classes to a screeching halt, nor does it incur any extra costs. When students work individually they also need to understand the problem.

              I’m unsure how bringing the class to some degree of halt is avoidable. Students have to pair off somehow, which depending on circumstances could mean figuring out who to group with, moving around, exchanging pleasantries — even if this takes twenty seconds, add it all up and eventually we’re talking about *real* time spent in a context switch. And although a student working by herself also has to understand the problem, she’s not held up by the lowest common denominator of her group.

              But maybe that’s the point, I dunno. My courses never seemed to be improved by frequent in-class group work (although outside-class group work is another story).

        • Bill Jefferys says:

          Mikhail is right. Statistics in particular is a field where the statistician must work closely with non-statisticians in order to accomplish the research goals of the non-statistician. A prominent statistician once told me that the reason s/he loves statistics is that it gives the statistician a chance to work in everyone else’s back yards. That’s the upside. For an introvert, the downside is that you are going to have to learn to collaborate with others, even others in very different fields, if you are to be a successful statistician.

        • Mikhail Shubin says:

          > If they want to do statistics, introverts need to learn to communicate and work with others.

          Introvertion is not equal to the lack of communication skills. Even if it was, I dont see how pair work(or group works) would change the situation. Pair work just make the class more boring and frustrating. Nothing more. I can believe that the pair works are liked by students _on_average_ and are more effective _on_average_. I just want to point out that some student are left behind. This is my personal experience.

          Of course, it depends on the implementation. Creative pair works are OK. Long ones are OK. However, simple (and most common one) “Turn to your nearest neighbor and discuss the solution of the problem 73.b” is simply frustrating. Solving equation is like falling in love, to understand it you should do it yourself.

          • Andrew says:

            Mikhail:

            Could be. There’s variation, and I recognize that what works for my class in general might not work for all students.

          • Rahul says:

            No matter what one tries some students will always be left behind.

            • Mikhail Shubin says:

              Although it is technically correct, It is a harmful philosophy to embark upon. It is just a weaker version of “We all gonna die anyway, why bother?”.

              I mean, one can be a much better teacher if he/she wold optimize the classroom taking everyone everyone, not just for the majority group. You could make the group works more creative. Or make them optional. Or just ask students if they like it or not!

              • Andrew says:

                Mikhail:

                You have to remember where I’m coming from, which is the traditional class where I stand there and lecture and the students do not participate. I think they get a lot more out of the class period when it is structured to enforce participation. I’m sure that a better teacher than I could do this in other ways, but for me, pair activities do the trick. I do not think it would work to make them optional. When I do pair activities and some students sit alone, I force them to work in pairs, because when they sit alone, they just sit and stare, they do not seem so focused. Again, I’m sure that a better teacher could find another way to get students focused but for now I will go with, and recommend, what works for me.

        • Phillip M. says:

          So yeah, no disagreement here, but I wonder how communication really figures into a 3-4 hr/wk course. While some of the best roads to learning come through teaching others a concept, or for others a course (and holding oneself accountable to striving to do this as best as one can), still others are definitely your ‘birds’ in the classroom. They may have more inclination towards the theoretician than the applied practitioner. And they have often (in my own experience) been the ‘lone wolves’in the classroom (and to maintain interest, aren’t strangers to needing a bit of ‘extra inspiration’, though not generally handholding). These students don’t occur frequently, but there are infamous such examples of this behavior.

  8. As a TA, I used to walk through about 2 or 3 example problems in detail in class (Engineering Dynamics, probability/stats, Statics, computer programming, classes like that). I’d force the students to provide next steps, answer questions about alternative possibilities, check for conceptual errors, check the units/dimension of equations, describe the principles that were being illustrated. Consider alternative problems and whether they would be more or less easy… etc

    Students used to give me terrible reviews, they’d all say things like “we just want him to do as many example problems as quickly as possible, that are just like the homework”. Many of the professors had already taken the position that they might as well give the students what they want… midterms would be homework problems word-for-word from the previous homeworks with the numerical values changed… or things like that. However, there were always 5 or 6 students who actually came to my class through the whole semester, and they would say things like “really knowledgeable and helpful, forced us to really learn the material… etc”

    It sounds to me like Andrew’s classes run a lot more like my TA classes did, with interaction and thinking, and not emphasis on shoving material on the board as fast as possible. How do you find the students respond? Do you have a “reputation” that encourages students to self-select into your section or what?

    • Phillip M. says:

      And this leads us to ‘the statistics of education’ given the differences in student attributes brought to a particular course. The fact is, there are so many things we do not know. This problem isn’t just at the collegiate level. It is ubiquitous. This is quite a problem – and thus hacks abound, slurping money from educational funding with absolutely no value given in return (I might point everyone to the rather pricey EVAAS system by SAS).

      The NCES hasn’t exactly been spurning enough complex research in matters being discussed here. The psychological community has done some work – much of which doesn’t give me the feel of ‘dots connected’. The ASA itself has a section on Statistics in Education – but where is the section on the Statistics of Education?

      I proposed this section to the ASA, but my reply was that they didn’t intend to add/support a section like this unless enough interest were garnered, and perhaps I would need to chair those efforts. So while this would be a nice opportunity, I know there are far better, more effective and credible people to do this work. While I may be empassioned about education in the United States, I am no expert, not even in the zip code.

      My question is, why wouldn’t the ASA see value in this without a petition, given how tightly it likes to be woven in with Washington DC?

  9. ezra abrams says:

    Why is this such a craft, anecdotal thing ?

    given how much money parents like me spend on tuition, the number of classes, and how long people have been teaching, you would think that there would be known things that work, not anecdotal evidence from one guy
    not that that one guy isn’t smart and right; it is just, with hundreds of billions a year spend on teaching,over dozens of years for the modern research university, over 10s of thousands of class rooms and millions of teachers, you would think there would be standards for what is known to work.

    [a snide person might note that professor’s salaries are tied to the craft nature of teaching]

    or maybe lecturing is such an idiosyncratic thing that it depends on the lecturer ?

    • Anonymous says:

      Universities are not institutions of higher learning, they’re government pork for the over-educated?

      There’s a certain aspect of that cynical view that isn’t entirely wrong. In general, the move towards terribly under-paid second class citizens (aka Adjunct Professors / Lecturers) has improved the quality of teaching (because these people tend to be the ones who actually LIKE teaching, why else would they accept effectively poverty level wages?). The fact that some of the university profs like Andrew actually take their “public good” component seriously and disseminate useful information, and try hard to teach well… doesn’t take away from the fact that the vast majority of university professors really dislike teaching and would much rather write grants to study women’s ovulation and choice of shirt color, or the use of stochastic PDEs to describe impossibly large computing problems requiring “peta-scale computing” that ultimately mostly solves problems of no interest to anyone outside of a tiny niche group.

      Cynical, but based on a lot of observation of Academia.

      • Anonymous says:

        “why else would they accept effectively poverty level wages?”

        Because if you’re 30 years old and lived your entire live in sheltered and protected environments disconnected from life’s great forces, the real world looks mighty scary. By that point they’re absolutely desperate not to leave.

      • Rahul says:

        …and maybe roads are not meant for travel but are state pork for those afflicted by wanderlust?

        • Anonymous says:

          Is this a sarcastic way of saying we should build 10x more roads than we need? hey maybe we could hire all those PhD’s and grad students working the counter at Barns and Nobel to build them!

        • A LOT of road and travel infrastructure projects have a LOT of pork in them. Some recent examples:

          SF Bay bridge replacement: Caltrans wanted a precast concrete causeway for around $300M if I remember correctly, politicians wanted a “unique single tower suspension bridge” for around $1 Billion, something like 20 years later it’s open but it cost around $6.4 Billion and there are major questions about corrosion issues.

          California high speed rail corridor: supposed to connect SoCal and the SF Bay with high speed rail. Note we already have very reliable commuter airplane flights via Southwest etc. The initial funding back in 2008 was $10 Billion, now estimates of total cost are around $68 Billion, if it gets done for less than half of that I’d be surprised given the history of this kind of thing.

          Gravina Island Bridge in Alaska: commonly called the “bridge to nowhere”, funding was cut, then maybe reinstated in 2011, in any case, it seems to be largely pork and if we’re lucky it won’t get built.

          So, roads and travel pork are a very very real thing.

          • Kansai airport: total cost around $20 Billion, given a “Civil Engineering Monument of the Millenium” prize or something. Built on a man-made island. Soil settlement on this island was like an order of magnitude larger than predicted by geotech engineers (50cm in 1994 alone!). It’s now down to less than 10cm per year. Note that just 10cm total settlement in a typical building in say Manhattan would be a catastrophe! The airport is basically built on a series of hydraulic jacks to constantly push it up, and allow them to add more foundation materials to keep it level.

            Huge debt costing hundreds of millions of dollars a year in interest alone, and high landing fees (second highest in the world) make it fairly impractical to actually run large quantities of airplanes through it without subsidies. The govt subsidizes the airport to the tune of tens of millions of dollars per year. I guess more recently it may be somewhat more successful. You can read about it on wikipedia for example.

          • Rahul says:

            That’s precisely the point though. The fact that there’s a lot of pork in road projects doesn’t say that there’s zero point in building any roads.

            • I think you and I are making the same point from different points of view then.

              You: There’s a LOT of pork but plenty of road projects are still useful.

              Me: some road projects are useful, but there’s a LOT of pork (especially among “new” construction projects rather than maintenance).

              In my opinion the same things apply to universities, and teaching. There’s no question that I’ve learned a lot of useful stuff at universities. They do serve some important education functions. But it’s not clear that the rate at which we’re currently eating up economic resources on Universities is a good thing, especially the rate at which we’re expanding student services and building “new” facilities to try to capture specialized grants, instead of “maintaining” the overall quality of education, replacing deteriorating facilities from the 1960’s that house research that “isn’t sexy enough” and soforth.

    • Isn’t your insight the essential truth behind all the Coursera/Udacity/MOOC stuff? Basically once you’ve recorded the Feynman Lectures on Physics once, you can play them to intro audiences for decades (choose your own version of the “ultimate intro course” if you aren’t a Feynman fan)…

      Finding out what is the right mix of accompanying supporting material and soforth is what those groups are all trying to figure out now.

      People have even started to think along the lines of “well maybe we don’t really need COURSES”, this may be the reason why online courses do so poorly at retaining students. Students really only need a fraction of the material and once they’ve got that, or figured out that they aren’t going to get it, the students drop out.

      There is something to be said for just oodles of short, single-topic lessons out there being combined by people who are building their own “course”.

      That’s putting a lot of self-reliance pressure on students, but it resembles what grad school is more or less like. All the best grad level courses generally involve grabbing some papers and reading them carefully in discussion, and then trying to apply the lessons to your own areas of interest. PhD level grad students generally have the self-motivation. It isn’t necessarily a bad thing to think that we might just force that kind of self motivation onto undergrads, perhaps slowly, with a couple of decades of transition time. Going from taking 5 courses a semester to 4 and a self-motivated seminar, to 3 courses and 2 self motivated seminars, to 2 courses, a group seminar and 2 self-motivated seminars… etc.

      A bigger question is, what is the point of universities? Is it to teach people a lot of things, or is it to certify minimal levels of knowledge and signal to employers.

      As a society, we lie to students about this stuff ALL THE TIME. See popular articles about STEM shortages, and then articles by STEM workers about the vast surplus of STEM workers and the lack of jobs in those fields for highly qualified graduates…

      I don’t have all the answers, but I have a LOT of questions about this whole situation.

      • Anonymous says:

        The Feynman lectures are legends to those who never learn physics from them. Those that did had an equally legendary preferred the standard issue physics textbooks.

        What about a hybrid college course. Consider a typical mundane college were there are students bright enough to have gotten into MIT but didn’t for whatever reason. They may only be small percentage of the student population, in absolute terms there are quite a few.

        If the college wanted to run a Bayesian Stat course for those students they could use Gelman’s lectures, homework, book and so on. A TA (or equivalently, adjunct prof) could then grade homework, answer questions and go over problems in class.

        The mundane college gets a cheap top-notch class. Gelman gets some extra pay, and greater reach. The students get a course better than they were likely to get.

        “A bigger question is, what is the point of universities?”

        None. Every half way intelligent answer give to this question is really answering “what is the point of thinking or learning?” as if they were the same question.

        • Yes, fine, someone points out that Feynman lectures are not that great for intro students, but the essential point still stands. Once you have a “good” way to teach {intro,intermediate,advanced} {Physics, Chemistry, Econ, Biology, Political Science, Philosophy, Gerontology… whatever} which is a relatively “standard” set of ideas (ie. a course that people can agree “covers” much of the “important material”) and the “live” materials have been recorded and can be played back, then the question is really just what do you need to add to those recordings to get a decent level of learning?

          I don’t think the answer to “what is the point of universities?” is “none”. Perhaps for you it was none, but for many the university gives some needed structure, some needed interactions, some access to tools… whatever. It does do something, but as you point out it isn’t typically what people think it is. It certainly isn’t to “create more qualified STEM graduates to fill the gaps in the economy” or whatever baloney is the current theory of university role for society.

          • BTW: I did learn a lot of my physics from the feynman lectures books.

            The key to doing that is to have a fairly sophisticated mathematical foundation before you start, a lot of motivation, and a willingness to question things and look them up elsewhere when something seems not quite right. The average intro student is learning a lot of basic concepts (eg. Calculus) at the same time as they’re learning physics. I think that contributed pretty heavily to the poor results from the original Feynman lectures (ie. the actual Caltech classes). Feynman can show you numerical integration on the board, but if you only sort of understand what an integral is… it’s floating right over your head.

          • Rahul says:

            But then why have the live materials even at all? Might as well study from the textbooks? If you are motivated enough.

            It’s not a given that a recording is equal to the real deal. Nor that just because a recording exists everyone will conscientiously listen to it. Nor that Feynman’s level of pedagogy is the right match, for say, a community college student. Et cetra.

        • Rahul says:

          And yet, universities continue to exist. Maybe thrive. When the data doesn’t agree with your hypothesis one must stop and ponder.

          I’ll agree with the universities-have-no-point crowd when MIT and Caltech classrooms (and coffers) start looking empty. I can live with “there’s no point in running mediocre universities” or “highly motivated self learners don’t need universities” but to say there’s no point in them entirely is without evidence.

          • Anonymous says:

            If the Government stopped subsiding universities and made IQ tests legal for job hiring, any university without a big enough endowment would disappear faster than you can say “free market’.

            • Rahul says:

              IQ tests for hiring are totally legal in many other nations. Have Universities disappeared there?

              If Universities are pointless from a student perspective how do the universities (with fat endowments) still get students to pay fat fees?

              And how come most adults don’t passionately hate their alma mater, this pointless leach of an institution, a drag on society and a waste of their time & money?

              • Anonymous says:

                Being heavily subsidized by the government is the bigger issue obviously.

                They get students to pay fat fees because it’s still a ticket to a higher paying job. If they had the option of getting that higher paying job, without paying the fees do you really thing most wouldn’t? Well, for most students in most majors that is a real possibility since their major isn’t relevant to the work they wind up doing.

                Well lots of people do passionately hate their alma mater. You might even say those who have student loan debt they can’t possibly pay back are downright bitter.

              • Rahul says:

                Indeed. But where is that wrong or unusual? Students pay fat fees for commercial pilot lessons because that too is a ticket to a higher paying job.

                Your counter-factual is like asking if students had the option to become pilots without any training would they pay fat fees? Of course not.

                No one is compelling industry to hire university grads (for most part). If a University Education is absolutely irrelevant to, say, designing microchips we should see Intel hiring high school dropouts. Is it?

                PS. Criticizing some crappy majors or colleges is a lot different from saying “What is the point of universities?” None!”

              • Anonymous says:

                I take it you make your livelihood from universities (I.e. bilking taxpayers). Just remember farming is far more essential in every sense than universities. That didn’t stop farmers dropping from 90% of the population to less than 1%.

                Good luck with that.

              • Rahul says:

                @anon

                Bad guess. I don’t.

              • Anonymous says:

                In that case, stop defending those petty, trifling, cesspools of flimflam, unearned egos, pompous falsehoods built on nothing but guile and their ability to bamboozle idiot politicians into swindling taxpayers; whose chief institutional purpose is to provide ugly old professors a fresh batch of coeds to hit on each year without the hassle of having to spend their days doing anything useful in exchange for the vast quantities the food, shelter and other resources that they piss down the drain each year.

                Then join the revolution!

              • Rahul says:

                @anon

                You do realize that this blog is run by academics? I guess swimming in cesspools is fun!

              • Anonymous says:

                And there were honorable soldiers in the Wehrmacht circa 1940’s. That didn’t stop us from crushing them into dust.

      • Radford Neal says:

        “What is the point of universities?”

        I think the point is that going to university is a socially acceptable way of not doing any productive work for four years. For some people, this provides them with the opportunity to learn lots of stuff that is of interest to themselves, and possibly future employers. For other people, it’s an opportunity to party. For yet others, particularly those for which is was the ONLY socially acceptable path, it’s four years of misery.

        The big question is whether MOOCs and other such innovations will lead to it being socially acceptable to spend four years learning stuff without actually registering in an official program of study.

        • Radford: I applaud your candor and agree with a lot of what you say. Student motivation is a big big factor. I am a big supporter of the CA city college system, especially for the nontraditional students, there is a lot of motivation and a lot of learning that goes on relative to either the party or misery track at big universities.

        • Anonymous says:

          But it’s not socially acceptable. The sight of many our brightest coasting so long on taxpayer subsidies they reach 30 (middle age) without ever having had a real job, or very often, any other of life’s great responsibilities, is disgusting.

          Alexander the Great conquered a big chunk of the known world before before most college students get a bachelor degree. Einstein published three revolutionary papers well before most Ph.D.s finish their dissertations (even before most pass qualifying exams).

          The military regularly gives more responsibility to 19-20 year olds than most academics will have their entire lives. The fact that many of those academics look down on those same 19-20 years as dullards makes the whole thing sublimely absurd.

          Academics may find it’s socially acceptable in their closed academic environment, but huge chunks of the rest of America either hates it with a passion, or thinks it’s a temporary necessary evil and can’t wait until it’s gone.

          • Anonymous says:

            “huge chunks of the rest of America either hates it with a passion, or thinks it’s a temporary necessary evil and can’t wait until it’s gone.”

            Your statement might contain the truth, but it depends on what you mean by “it” and “can’t wait until it’s gone”

            The VAST majority of America currently wants their kids to go to college, they save enormous amounts of money for this purpose, they push their children to take on huge loans, they do it all in the name of “getting ahead” because education through undergrad has clear economic returns for most majors.

            But most people would rather not have the huge loans, the bloated universities with declining quality of teaching, the administrators who suck down a greater and greater fraction of the resources.

            I think most people also still have a vague “warm fuzzy” feeling about university professors. Few know what academia, especially academic science is like these days. The backstabbing and scrabble for rapidly declining funds hasn’t hit home. The non-replicability problem isn’t on most people’s radar. People still believe “we don’t have enough STEM graduates” even though ALL the real actual evidence suggests we have WAY TOO MANY.

            So if you think there is a big rabble just waiting for the right opportunity to get their pitchforks… no I don’t think so. But if you think there’s a lot of enthusiasm for anything that resembles a more sane economic way to get some of the economic returns to education… and therefore a lot of general enthusiasm for the ideas of MOOCs or e-textbooks or other cheaper and still effective methods that might be experimented with in the near future, then yes I think that’s probably true.

          • Rahul says:

            I wonder if @Anonymous’ awe of the precociousness of Alexander & Einstein gets actually expressed by him patronizing 20 year old physicians and 22 year old surgeons? And indeed since the military gives 19 year olds huge responsibility perhaps @Anonymous will feel most comfortable being operated upon by a 19 year old during a medical emergency?

            How old Alexander was when he conquered the world is about as irrelevant a fact to this debate as can get.

            • Anonymous says:

              My father in law was a surgeon. He was from a foreign country and like most of the world he went straight from high school to medical school. The same used to be true for lawyers (some states incidentally still allow “apprentership” lawyers with no degree).

              So he was surgeon before his mid 20’s and practiced for 15 years as a surgeon before coming to the US. The gratuitous requirement for a bachelor degree before going to medical school, law school, psychiatrist is absolutely ridiculous.

              Note there is no pre-med major. You can get a BA in Art history and become a surgeon.

              It’s not irrelevant to the debate. It shows that people can perform a high level at a much younger age than is common now. Einstein’s experience wasn’t unusual. It used to be the norm. The idea now that researchers can’t do their own thing in a major way until they’re 40 is the aberration.

              • Rahul says:

                Whew. Something we can finally agree on. Yes, I think the pre-med degree is gratuitous too.

                My point it there are a lot of such particular flaws in the system that ought to be pointed out and corrected.

                It’s just the broad statements like “Universities have no point” or “Professors are good-for-nothing leaches” etc. that rile me up.

                I’m not a passionate defender of the status quo but neither am I convinced that MOOCs are so damn good and revolutionary that they are going to completely supplant the traditional university model. I think of them as competition and supplements but not as replacements.

              • hjk says:

                Agree with Rahul on pretty much everything in this thread.

                I’ll throw in my own two cents and say that academia is not a business, obsession with ‘productivity’ is one of the major problems today, and war analogies are pointless.

                Academia, like art, is its own thing. How to best support endeavours like these is a difficult question but I see no answers here.

              • Anonymous says:

                “Academia, like art, is its own thing”

                When you give decisive arguements like that it’s a nice retorical flush to end with “QED”. How about this:

                “Academia, like art, is its own thing – QED”

                Academia is more like a paint brush. It’s either doing the job or not. I stand by my original comment that any half way decent argement for universities is really an arguemnt for thinking/learning, but they’re not the same. Just like art isn’t the same as a paint brush.

              • hjk says:

                How about – academia, like an artist collective, is it own thing. Its (supposed, anyway) aims are not those of a business and its a great way to meet, collaborate with and learn from other people.

              • Anonymous says:

                Starbucks is a great place to meet and collaborate with people.

                Academia is trillion dollar boondoggle which massively fails, most students, most academics, the taxpayer, and is easily the single biggest hindrance to genuine progress in every field with with I’m intimately familiar.

              • Rahul: I agree, “Anonymous” goes over the top with “Universities have no point” type rhetoric. There is however a very long list of problems with the current US academic system.

                To argue like hjk that “academia is not a business” is to simply be wrong. perhaps “academia ought not to be a business” which requires some further explication, but in the modern day, academia *is* one of the fastest growing industries. The number of administrators of academic institutions has skyrocketed. http://chronicle.com/article/Administrator-Hiring-Drove-28-/144519/

                From the standpoint of economics, Universities are basically in the business of getting grants to do research, and charging enormous tuitions to get student loan money. They attract research money by convincing the people on the grant review boards to fund their projects. Who are those people? Oh yes, other professors at university institutions. There is a *definite* aspect of regulatory capture going on there. There is basically NO input from the general population about prioritizing research.

                They attract student loan money not by offering the top level of education but more recently by offering things like single dormrooms, fancy spa-like student recreation facilities, and lots of student services that require all those administrators. The economics seems pretty clear: you could pay a crapload for no-party, or you could pay a crapload for party… the only thing you can’t do is “not pay a crapload”.

              • Rahul says:

                @anon

                Big words. So what fields are you “intimately familiar” with?

              • Anonymous says:

                Rahul,

                Oh, the usual fields: rationals, algebriac numbers, real numbers, complex numbers, p-adic numbers.

                (that’s an old math joke)

    • Anonymous says:

      There’s an enormous amount of research in education/teaching. Entire journals are filled with “p-value<.05" type studies. The problem is everyone with a clue has figured out by now that anecdotes+judgment work better than following those studies.

      • question says:

        Origin of NHST: Lindquist, E.F. (1940) Statistical Analysis In Educational Research.

        It wouldn’t be surprising if educational research is the most unreliable field of all, I would guess the longer that method of inference is relied upon the more difficult it is to accomplish anything.

        As a simple model to demonstrate my line of thinking, lets say a field starts out with N=100 previous claims with false positive rate of p0(false)=0.05. Researcher’s randomly sample n=3 supporting claims from the literature before embarking on a new study. After NHST is introduced, researchers simply measure their own opinion by p-hacking: if they find at least n=3 pieces of supporting background info they will then achieve a positive result, if they cannot then they will not perform/publish the study (ie, 100% file-drawer effect). If at least one of those background findings are false then their claim will also be false since it is based on at least one false premise. I’m sure this can be solved analytically but we can get an approximation (and it will be easier to expand on the model) with the following R code:
        #
        N=100
        n=3
        p0=.05
        dat<-c(rep(0,p0*N),rep(1,N-p0*N))
        t=1:3000

        pr=matrix(nrow=length(t))
        for(i in t){
        samp<-sample(dat,n,replace=F)
        dat<-c(dat,ifelse(any(samp==0),0,1))
        pr[i]<-length(which(dat==0))/length(dat)
        }

        plot(t,pr, xlab="# of Studies", ylab="p(false)", ylim=c(0,1))
        #
        http://s28.postimg.org/gmysxz6ct/pfalse.png

        If this worst case scenario were the case, by the time there are 3000 papers on a topic, ~90% will be false. I don't know about educational research but in biomed the best information available indicates that the replication rate is 10-30%:
        http://www.newscientist.com/article/mg21528826.000-is-medical-science-built-on-shaky-foundations.html#.U_JqPWPaZuQ

        Assuming each of these replications indicate a false claim was made and the conditions used above, the model would predict that each new claim was supported by 900-3000 previous claims. To test this model we would need to count the number of supporting references found in the introduction of a paper, following the refs to that paper and counting, etc. Then perform replications of every study in the chain at intervals from some arbitrary initial time to the most recent and check the fit.

    • mpledger says:

      or maybe learning is such an idiosyncratic thing that it depends on the learner ?

      ~~~~~~~~~
      I think the tone of this commnent just reflects the change in responsibiites for learning that have evolved over time, probably as a reflection of the increasing cost of college, but also that an undergrad education is now a long way off from the forefront of knowledge – nowadays undergrad can just be taught as if it was a high school course out of a textbook rather than “last week I did this in the lab”.

      In the old days it was entirely a students responsibility to learn the material and he should be grateful that the great man came to drop a few pearls of wisdom. Nowadays it’s the lecturer’s job to fill up students heads with wisdom whether the students wants it or not.

      • I’m always very suspicious of the “in the old days” school of argument.

        I don’t think it would be too hard to argue that a bachelor’s degree used to be essentially like a masters degree and a masters degree used to be essentially like a PhD and a PhD used to be essentially like a PhD + Postdoc. Except that “like” is about the methods of learning and teaching, not about the quantity and quality of information. After all, an undergraduate in physics today knows more than Einstein did in 1900 about how the world works (for example, the exact size of atoms, the nature of certain quantum events, the numbers and types of particles, the constancy of the speed of light… etc) By 1905 Einstein had discovered clues to a lot of these things, and by 1920’s or 1930’s physicists knew most of what an undergrad today would learn, but that’s an enormous amount of information that an undergrad is expected to understand in 3 or 4 years. It’s only possible by a process of distillation, doing it in the lab took a productive lifetime.

        The same is true for a lot of fields: the plate tectonics theory of geology was only widely accepted in the 1950s or 1960’s!

        Your point might be that “back in the day” you were buying “access to guys who were doing the experiemnts” and “discussions about the latest stuff”. But it’s pointless to discuss “the latest stuff” if you don’t have the background to understand it.

        There is, undoubtedly, a proper balance between “learning a bunch of distilled stuff” and “learning cutting edge stuff and the techniques used to discover it”, and this balance would tend to change through time pretty naturally I’d say.

    • Martha Smith says:

      I think teaching is at least to some extent an idiosyncratic thing that depends on the teacher, but not entirely.

      Anecdotal evidence that there is also an idiosyncratic aspect to students’ preferences: Once I was on jury duty for two weeks. A colleague was willing and able to teach one of my classes the entire time. When I got course evaluations at the end of the semester, six students compared and contrasted my teaching and the colleague’s. Half of them said they were glad they had me instead of him, and the other half said they wish the had him instead of me.

      • Rahul says:

        Most students are so bored filling end of course evaluations (at a time when exams, projects & upcoming vacations beckon) that I wouldn’t be surprised if you were looking at random noise.

    • See:

      http://www.hup.harvard.edu/catalog.php?isbn=9780674013254

      This is a book worth reading for beginning teachers and goes beyond anecdote (iIRC).

      In my (limited; 10 years on the job) experience, there is no one standard approach that will work, it depends on the subject being taught, and on student motivation and ability.

      • Andrew says:

        Shravan:

        You link to a book called “What the Best College Teachers Do.” I agree that this is a good book to read, but there is an issue, addressed in my post above, that what works for mediocre teachers such as myself is not necessarily what works for the best teachers. In particular, I use all sorts of tricks to get my students involved, and I suspect that the best teachers can get the students involved in more direct ways, without gimmicks.

        • Rahul says:

          Are “mediocre” & “best” invariant attributes? Can a mediocre teacher be trained to become a great teacher?

          • Andrew says:

            Rahul:

            Teaching quality is neither invariant nor unidimensional. My point was that “doing what the best teachers do” will not necessarily make someone a better teacher. To some extent the opposite could be true: that to become a better teacher you might need to do some things that the best teachers would never bother to do.

            • Agreed; but that’s the same situation as when some statistician delivers an average risk value. It’s only an abstract indicator, your mileage (individual risk) may (will almost certainly) vary. There doesn’t seem to be a magic bullet, but that’s true for everything in life.

    • Elin says:

      There is actually a lot known about what works (once you decide what works means), how long people pay attention to lectures before drifting off, what kinds of active learning are helpful. I mean you can for sure criticize the designs of almost all of the studies, but I’ve still learned a lot by reading and thinking about what’s out there and what other people are doing. Lectures seem to work best for the kinds of people likely to become academics, that love the topics, can handle complexity, have good attention spans for things they are interested in, actively engage with the material. I mean, I love lectures, I go to talks all the time if I can. As long as your lectures are like going to a talk, not just transferring information that they should get by doing the reading, they are probably interesting and engaging for good students. Everyone else … it’s an open question that you could certainly collect some data about what works from your own students since it’s all so contingent on the instructor and students anyway.

  10. hjk says:

    These seem like decent tips.

    I never really liked pair/peer based activities as I preferred to think/work through things myself, but I can’t say I didn’t gain anything from these. As long as it’s not overdone it could be okay. Students generally seem to rely a little too much on each other though and maybe this could hinder independence? Similarly with the idea of not exposing them to things beyond what’s assessed – it seems a little restrictive and counter to the spirit of a university. One of my main goals when teaching is to make the students feel a least a little (mentally!) uncomfortable.

    A clear and careful set of notes and the chance to see how an experienced person really thinks/solves problems (whiteboard/chalkboard/computer/doc cam etc) seem like the most important things a lecturer can provide. Access to libraries/books/similar resources, contact with peers and an environment of expectations seem like the most important things a university offers generally. The rest is largely up to the students.

    • Rahul says:

      Another real risk is that of not assessing students on things beyond what’s explicitly taught in class. When the teacher also doubles as the sole evaluator this is a very real risk I feel.

      I wish there was a way I could tell a fellow Professor, “I’m teaching my class linear regression this month” and then have him compose a test that I administer the students end-of-month without me ever looking at what he put on the test while I teach the class.

      • hjk says:

        Agree, yet again. Wonder if Andrew could give more details on his views of the ‘don’t teach/assess beyond what’s taught’ issue? I know students get annoyed when you do go beyond the material when testing (the ‘not fair!’ emails…), but that’s probably a good sign.

        • Rahul says:

          And the fact that students do get annoyed reflects a bad precedent other courses have set. We’ve too often cultivated this unhelpful notion that tests must be restricted to what’s taught in class.

          • Ok, but to take the devils advocate position, it would be insane to test students on things the students didn’t even know they were supposed to learn. If you want to test on a wider variety of materials than what is detailed out in lectures, you need to at least restrict it to some materials that the students had the opportunity to know they should study. For a statistics related example (let’s all drop the issue of whether you should teach tests for the sake of example), if you teach the t-test in detail, and you then mention that there are 3 or 4 other common tests, like say the chi-squared for goodness of fit, and some kind of binomial test for proportions or something… then if one of those other tests shows up on the final, the students were at least warned. If on the other hand, you pull an esoteric test for goodness of fit of some specialized distribution which you never even mentioned in class nor the textbook or handouts… students would be right to be irate.

            Going back to non-devils-advocate though, I frequently tried to focus my teaching on core concepts rather than on working individual example problems. This was my teaching-review downfall, but I think it’s far more important that students learn the basic concepts that organize material than that they learn to solve particular types of problems. If you teach a student conservation of linear momentum and conservation of energy as concepts, and then give them a problem where they need to recognize and use those two concepts together… even though it looks nothing like any of the homework problems, the students should be able to solve it, given enough time (test-taking time is always an issue !)

            To me, that’s the more important way in which testing can and maybe should vary from the “homework” and “example” problems.

          • Martha says:

            I see interrelated semantic and communication problems that need to be taken into account in discussing whether or not “tests must be restricted to what’s taught in class.”

            The semantic problem is the interpretation of “what is taught in class” — is the interpretation “just the specific techniques/types of problems ‘covered’ in class?” Or is it also the thinking skills that the student is expected to develop from class work and homework? The communication problem is that the instructor needs to communicate clearly just what her/his intention is. I typically aim to take the latter semantic interpretation, which then means I have the obligation to communicate that effectively (i.e., both clearly usually repeatedly) to the students. In my experience, the communication problem is a non-trivial issue (as is also the teaching of thinking skills)

  11. Sean says:

    For the quality of the post, I have to say that the commentary has felt quite negative and depressing.

  12. Eric Rasmusen says:

    I complain a lot about blog comments not being moderated, since often the top 5% of comments is better than the post itself, but it’s not worth finding them. I just had an idea. The problem is that moderation takes time, and the blogger is too busy maybe even to read the comments. How about if a commenter does it for him? One of us could pick the comments he liked, put them into a web page, and then put the address into a new comment. Since I ought to getting back to other things, though, I won’t do that here and now; I invite someone else to. Whoever does it is the one who gets to decide what is a good enough comment to include.

    For a post like this one, Professor Gelman, you might want to write up the essence of the most useful comments yourself, as a new post.

  13. […] later, half an hour later, at the end of the day and a week later if possible. I also worry that too much fun means the message is lost. Give them a snap question to tell you what a p-value is, with some multiple choices on a […]

Leave a Reply