My favorite statistics demonstration is the one with the bag of candies. I’ve elaborated upon it since including it in the Teaching Statistics book and I thought these tips might be useful to some of you.

**Preparation**

Buy 100 candies of different sizes and shapes and put them in a bag (the plastic bag from the store is fine). Get something like 20 large full-sized candy bars, 20 or 30 little things like mini Snickers bars and mini Peppermint Patties. And then 50 or 60 really little things like tiny Tootsie Rolls, lollipops, and individually-wrapped Life Savers. Count and make sure it’s exactly 100.

You also need a digital kitchen scale that reads out in grams.

Also bring a sealed envelope inside of which is a note (details below). When you get into the room, unobtrusively put the note somewhere, for example between two books on a shelf or behind a window shade.

**Setup**

Hold up the back of candy and the scale and write the following on the board:

Each pair of students should:

1. Pull 5 candies out of the bag

2. Weigh the candies

3. Write down the weight

4. Put the candies back in the bag!!

5. Pass the scale and bag to your neighbors

6. Silently multiply the weight of the 5 candies by 20.

(And, as Frank Morgan told me once, remember to read aloud everything you write on the board. Don’t write silently.)

The students should work in pairs. Explain that their goal is to estimate the total weight of all the candies in the bag. They can choose their 5 candies using any method–systematic sampling, random sampling, whatever. Whichever pair guesses closest to the true weight. they get the whole bag!

Demonstrate how to zero the scale, give the scale and the bag of candies to a pair of students in the front row, and let them go.

**Action**

The demo will proceed silently while the rest of the class proceeds. So do whatever you were going to do in class. Take a look to make sure the scale and bag are moving slowly through the room. After about 30 or 40 minutes, it will reach the back and the students will be done.

At this point, ask the pairs, one at a time, to call out their estimates. Write them on the board. They will be numbers like 3080, 2400, 4340, and so forth. Once all the numbers are written, make a crude histogram (for example, bins from 2000-3000 grams, 3000-4000, 4000-5000, etc.). This represents the sampling distribution of the estimates.

Now call up two students from the class (but not from the same pair) to look at all the estimates. Ask them what their best guess is, having seen this information. As the class if they agree with these two students. Now give the bag to the two students in the front of the room and have them weigh it.

**Punch line**

The weight of all 100 candies will be something like 1658. It’s always, always, always lower than *all* of the individual guesses on the board. Write this true weight as a vertical bar on the histogram that you’ve drawn. This is a great way to illustrate the concepts of bias and standard error of an estimator.

Now call out to the students who are sitting near where you hid the envelope: “Um, uh, what’s that over there . . . is it an envelope??? Really? What’s inside? Could you open it up?” A student opens it and reads out what’s written on the sheet inside: “Your guesses are all too high!”

**Aftermath**

Now’s the time to talk about sampling. Large candies are easy to see and to grab, while small candies fall through the gaps between the large ones and end up at the bottom of the bag. You can draw analogies to doing a random sample by going to the mall or by sending out an email survey and seeing who responds. Ask, How could you do a random sample. It won’t be obvious to the students that the way to do a random sample is to number each of the candies from 1 to 100 and pick numbers at random. Also, as noted above, this is an example you can use later in the semester to illustrate bias and standard error.

P.S. My feeling about describing these demos is the same as what Penn and Teller say about why they show audiences how they do their tricks: it’s even cooler when you know how it works.

P.P.S. Remember—it’s crucial that the candies in the bag be of varying sizes, with a few big ones and lots of little ones!

You shouldn't hide the envelope somewhere, you should have it in plain sight. Otherwise, smart students will think that you hid multiple envelopes, and just directed them to the suitable one. (This would be like the trick where you say "think of any card from the deck and tell me what it is", and you say "I knew you'd pick that" and pull the card out of your pocket. Of course you have the spades in your right front pants pocket, diamonds in your left front, clubs in your right jacket pocket, etc., and you just reach in and count off the right number, and pull out that card).

All through your description I was expecting the punchline to be that some of the students would surreptitiously eat some of the candies!

Phil,

No, I want them to not be completely sure. That's part of the fun of the trick. Anyway, I explain to them in the debriefing that every guess has always been too high, whenever I've done the demo.

Derek,

I do keep an eye on them to make sure that doesn't happen!

Does this mean that you tend to advocate stratified samples rather than "random" ones with a much-less-than-100% response rate?

Lemmus,

It's the usual tradeoff: a simple random sample can in many cases be easier to do and easier to analyze, a stratified sample can be more efficient. In this case, either option would work better than the "grab a bunch of candies" approach.

Frank Yates's classic book on sampling has an example where many stones were laid out on a table and people had to sample to estimate the mean mass of the stones. The overall bias was about 20% too large, although some samples underestimated the known true mean. This was presumably originally a demonstration like Andrew's but in geomorphology and sedimentology sampling stones at a site (say a bar in a gravel-bed stream) is a real scientific problem. (This example is I think in all editions: the last was 1981 from Griffin, London.)

Andrew,

I thought the classroom example you used was quite akin to a random sample when some of the selected participants are quite hard to reach or unwilling to participate (and systematically so).

What is the exact point of the example, if I may ask?

A long time ago I was a shipping clerk and I had to weigh and mark many outgoing packages. I got quite good at estimating the weight of a package.

Wouldn't the best way to estimate the weight of all the candies be to hold the bag and say—hmm–about 3 pounds 4 oz? Don't you ever have a situation where students use the subjective weight of the big bag to estimate the measured weight?

I'd like to take a stab at Lemmus^2 and Chuck's questions, unless Andrew has already beaten me to it.

L^2, I think the point is that a convenience sample can have quite a large, systematic bias.

Chuck, I don't think Andrew cares about the weight of the bag of candies — he can weigh them himself a lot more easily than going to all this trouble! I think the point is to teach the students that how you get your sample can be really important.

But Andrew, Chuck raises a good issue, perhaps before doing any of this you should have each student take a guess at the weight of the bag, and compile these into some kind of prior distribution. (You could even have each student assign a precision, so students who think they're good judges will be, um, weighted more heavily). But perhaps this would just confuse things. I guess it depends on how sophisticated the students are.

Lemmus: What Phil said.

Chuck and Phil: The "weigh by hand" thing is another good idea for a demo (which, in fact, I think I'll steal). It fits in with some of our other successful demos such as the one where groups of students guess the ages of people in photographs.

My instinct is to do the "weigh by hand" demo completely separately, on a different day and not using bags of candies. Then once it's over I can point out that it's another example of bias and standard error in measurement.

P.S. I think the whole thing of measuring in grams distracts students from thinking about the true weight of the candies in the bag. By the time they've taken the sample, weighed it, and multiplied by 20, they've sort of forgotten what the numbers mean.

The purpose of this could be something as specific as "watch out for sampling bias" but, to me, you can't emphasize enough how inaccurate most people's statistical intuition is and how people always, always overestimate how accurate their intuition is.

It would be nice to have another demonstration in which the class's sample comes out too low. If the distribution what I suspect is is in that bag, then if they did the exact same experiment with truely random draws they would come out too low; due to the low chance of getting one of the few big peices.