Skip to content
 

The hare, the pineapple, and Ed Wegman

Commenters here are occasionally bothered that I spend so much time attacking frauds and plagiarists. See, for example, here and here. Why go on and on about these losers, given that there are more important problems in the world such as war, pestilence, hunger, and graphs where the y-axis doesn’t go all the way down to zero?

Part of the story is that I do research for a living so I resent people who devalue research through misattribution or fraud, in the same way that rich people don’t like counterfeiters.

What really bugs me, though, is when cheaters get caught and still don’t admit it. People like Hauser, Wegman, Fischer, and Weick get under my skin because they have the chutzpah to just deny deny deny. The grainy time-stamped videotape with their hand in the cookie jar is right there, and they’ll still talk around the problem. Makes me want to scream.

This happens all the time. All. Over. The. Place.

Everybody makes mistakes, and just about everybody does things they shouldn’t, every now and then. But to not apologize when you’re caught, that to me just seems evil, showing a disrespect not just for the people involved but for the very concept of truth. (As you can see, I wouldn’t make a good criminal defense lawyer.)

Anyway, here’s the latest story. There’s always an outrage-of-the-week, and I don’t mean to make a big deal about this particular scandal. It’s just another example of what I consider the disgraceful pattern of people refusing to admit error.

Part 1: The mistake

The following question was on a New York State eighth grade reading exam:

The Hare and the Pineapple

by Daniel Pinkwater

In olden times, the animals of the forest could speak English just like you and me. One day, a pineapple challenged a hare to a race.

(I forgot to mention, fruits and vegetables were able to speak too.)

A hare is like a rabbit, only skinnier and faster. This particular hare was known to be the fastest animal in the forest.

“You, a pineapple have the nerve to challenge me, a hare, to a race,” the hare asked the pineapple. “This must be some sort of joke.”

“No,” said the pineapple. “I want to race you. Twenty-six miles, and may the best animal win.”

“You aren’t even an animal!” the hare said. “You’re a tropical fruit!”

“Well, you know what I mean,” the pineapple said.

The animals of the forest thought it was very strange that tropical fruit should want to race a very fast animal.

“The pineapple has some trick up its sleeve,” a moose said.

Pineapples don’t have sleeves, an owl said

“Well, you know what I mean,” the moose said. “If a pineapple challenges a hare to a race, it must be that the pineapple knows some secret trick that will allow it to win.”

“The pineapple probably expects us to root for the hare and then look like fools when it loses,” said a crow. “Then the pineapple will win the race because the hare is overconfident and takes a nap, or gets lost, or something.”

The animals agreed that this made sense. There was no reason a pineapple should challenge a hare unless it had a clever plan of some sort. So the animals, wanting to back a winner, all cheered for the pineapple.

When the race began, the hare sprinted forward and was out of sight in less than a minute. The pineapple just sat there, never moving an inch.

The animals crowded around watching to see how the pineapple was going to cleverly beat the hare. Two hours later when the hare cross the finish line, the pineapple was still sitting still and hadn’t moved an inch.

The animals ate the pineapple.

MORAL: Pineapples don’t have sleeves

Several questions follow, including:

The animals ate the pineapple most likely because they were

A Hungry
B Excited
C Annoyed
D Amused

Which animal spoke the wisest words?

A The hare
B The moose
C The crow
D The owl

Nope, I don’t know how to answer these either.

Part 2: Exposure

The story hits the Daily News and other media outlets. To its credit, the state education commission takes the question off the test. More on the story here.

Part 3: No admission of error

From Time magazine, a memo from Jon S. Twing of Pearson, the company that made the test with the silly questions:

Pearson is confident that the NYS Grades 3-8 English Language Arts (ELA) and Mathematics assessments have been developed to support valid and reliable interpretations of scores for their intended uses. The “Hare and the Pineapple” passage and associated items were placed on the Grade 8 ELA test after the NYSfield test data associated with the multiple choice items and the feedback from the “final eyes” committee determined that this was an appropriate passage and set of items to include on the test. . . .

“The Hare and the Pineapple” passage is intended to measure NYS Standard “interpretation of character traits, motivations, and behavior” and “eliciting supporting detail”. . . .

Blah blah blah . . . But now the good part. Jon S. Twing writes:

There have been two items of the set of six that have been challenged by NY teachers and students as the test was under way April 17-19, 2012 -Item 7 and Item 8. The correct answers and rationales to Item 7 and Item 8 are as follows:

• Item 7: The correct answer is C. The question regarding the animals’ possible motivation for eating the pineapple requires a reader to infer the correct answer from clues conveyed in the text. While all of the options are plausible motivations, the most likely answer is that the animals were annoyed. Paragraph 13 indicates that the animals support the pineapple to win the race because they assume the pineapple has a clever plan. However, the pineapple never moves during the race. From these clues and events, a reader can infer that the animals are annoyed. The text does not support the inference that the animals are motivated by hunger, excitement, or amusement.

• Item 8: The correct answer is D. The question regarding the wisest animal requires the reader to apply close analytic reading skills to determine which of the choices represents the wisest animal based on clues given in the text. The moose and the crow are the two animals that present the incorrect idea that the pineapple has a clever plan to win the race. This idea is proven false when the hare wins the race. The hare is presented as incredulous that a pineapple would challenge him to a race, but overconfidently agrees to race a pineapple.

Huh? The hare “overconfidently agrees to race a pineapple”??? What’s overconfident about that? A pineapple can’t run at all!

Twing continues:

Finally, the owl declares that “Pineapples don’t have sleeves,” which is a factually accurate statement. This statement is also presented as the moral of the story, allowing a careful reader to infer that the owl is the wisest animal.

Sorry, Jon S. Twing, but that doesn’t make sense at all. The moose makes it perfectly clear that “a trick up the sleeve” is just a figure of speech. The owl is wise for taking a figure of speech literally? In that case, why not kill the joke entirely with “hares can’t talk”? Grrrrrr.

You gotta remember that the people taking this exam are 8th graders who are trying to not get tricked themselves.

And, by the way, I have a feeling “the moral of the story” is a joke, in the style of James Thurber’s Fables for Our Time.

Twing wraps things up with a bunch of words and numbers suggesting that these questions have good psychometric properties, i.e. that students who got other questions right tended to get these right also. That’s interesting: even an impossible-to-answer question has some patterns which some students are able to match. But that doesn’t make these good exam questions.

The psychometrics are relevant but I just think it’s embarrassing that this poor guy is in the position of claiming, first, that the animals are annoyed, when there is no evidence at all of that in the quoted passage, and, second, all that business of pineapples and sleeves.

Why not just say: Hey, it’s not easy to write a test! We made a mistake! Sorry!

But nooooooo, he can’t do that, he’s gotta defend defend defend defend defend. OK, I’m not calling him a Wegman—there’s no reason to suspect that any botched wikipedia copying went down—but that makes the whole thing even worse, in a way. These people didn’t even do anything intentionally wrong. They just made a mistake. But they still won’t admit it. “The correct answer is C,” indeed.

Here I am, railing against universal human nature again. That’s what blogging is for, I suppose.

89 Comments

  1. revo11 says:

    The story is almost a metaphor for the test question itself. Can we eat Mr. Twing now?

  2. jrkrideau says:

    Mistakes were made (But not by me): Why we justify foolish beliefs, bad decisions, and hurtful acts, Carl Tavris and Elliot Aronson. (2007)

  3. Anonymous says:

    For some reason, people have an easier time finding fault with the good guys who kick up a fuss than with the bad guys who cause the problem. This applies whether it’s academics railing against plagiarism, or volunteer watchmen looking out for their neighbours. A large chunk of the world’s problems with “war, pestilence [and] hunger” surely come from a failure of decent folk to deal with the psychopaths like Wegman. Keep up the good work.

  4. fraac says:

    Animals only ever eat anything because they’re hungry or they want to make themselves sick. They don’t comfort eat or spite eat. The story isn’t allegorical, it’s a joke, a parody of an allegory BUT if the moral is that pineapples don’t have sleeves then the moral is that animals eat when hungry. Twing is an idiot.

  5. K? O'Rourke says:

    > railing against universal human nature again

    Yes, but what would be much better in science, would be a culture of admitting to being wrong, or at least capable of becoming less wrong.

    On the other hand, academic culture does seem to be very much the opposite – never admit or allow an inadequacy to come to light – no matter what.

  6. adam says:

    I like the eating question because it really makes evident a lot of the assumptions that go into testing. I think the answer to why they ate him his pretty obviously that they were annoyed, though this is not clear from anything else in the text. Rather, it is obvious because that is how these fairy tales tend to go. Answering the question isn’t a matter of analyzing the passage at hand, but rather having a textual background.

    And of course if that’s the case then the people who score highly will tend to get this correct. Who tend to score highly? Wealthy kids with involved parents. Who tends to hear a lot of fairy tales? Wealthy kids who get these stories read to them by their literate and involved parents. Okay, strictly speaking we can probably strike the wealthy bit, but I think everyone understands my point here…?

    PS. This was how I always felt about the SAT and the GRE. There were often what I would consider – and still do consider! – two correct answers to the problem, but one “felt” right in the sense that I could tell that’s what the test was looking for; if this were an essay I could have strenuously argued for the other case, but in a fill-in-the-blank it’s all about having the cultural knowledge to know what they’re looking for.

    • Manoel Galdino says:

      +1
      Exactly what I’d say. I’m glad that you said it, though, cause I don’t need to write it myself. And English isn’t my first language, so you probably said it better than I could myself.

    • fraac says:

      Interesting. I said ‘hungry’ *because of* my awareness of story forms. This appeared to be a joke where the punchline is reversion to reality: pineapples don’t win races or have motives, animals eat what they want. If I went too meta it was because the animals were genre-savvy, referring to the hare and the tortoise.

  7. Mike G says:

    I don’t necessarily think the first question is entirely bad. For one, the choices other than “Annoyed” are not sensible answers, whereas “Annoyed” makes sense from a judging someone’s “Social Intelligence” standpoint. A student could infer that the animals ate the pineapple because they were annoyed based on the social context clues of the animal’s unmet expectations and the assumption of a causal link between the animals eating the fruit and the fact that the fruit failed to meet their expectations. Any of the other answers requires one to assume that the answer is completely independent of the rest of the story (The animals were not happy or excited after the pineapple did not move and being hungry would have been independent of any given information).

    The second question is a complete failure on the test writers part, because more than one answer could be legitimately defended as being correct. The biggest problem is that “wisest” in this case is a poorly defined quality, especially when one is in a magic intelligent, talking animal/fruit world.
    As you pointed out overconfidence here seems entirely subjective and otherwise basically everything the hare said was as wise as what the owl said.
    Additionally, the moose seems to be somewhat wise in identifying the fact that it seems unlikely a fruit would challenge a hare to a race if it did not have a means to compete. The moose is basically basing his assumption on the pineapple not being entirely irrational and challenging the hare for no reason. Just because the moose was wrong does not mean he was necessarily unwise.

    Would the moose have been the wise one if the pineapple did have a trick up it’s sleeve?

    Maybe it’s just me, but I think you have to judge the wiseness of the animals based on their judgement with the information they had at hand, not based on the results.

    • Scott says:

      “Being hungry would have been independent of any given information” is not quite true – the animals were waiting around for about two hours. But even if it were true, why does that make ‘A’ a bad answer ? I’d say that the hypothesis that the animals ate the pineapple because they were hungry has the highest prior probability. It’s the commonest reason for humans, animals, & fictional talking animals to eat. If the animals were annoyed, even fructicidally so, wouldn’t they have vented their annoyance in some other way unless they were also hungry ? And who’s to say they were annoyed anyway ? There was nothing at stake in the race; it was just a friendly competition: and there’s nothing to show that they’re untypically peevish. There’s just not enough evidence in the text to shift the prior odds much.

      Pedantry aside, I read the story like Fraac – as a ‘reversion to reality’. The pineapple’s lustre faded, they put it to its customary use & forget all this nonsense about racing. A little dark perhaps, considering that it’s a talking pineapple; but funny, & in keeping with the tone of the story. They eat it as revenge for their being misled in a trivial matter !? That’s horrible. Why jump to that conclusion ?

  8. MAYO says:

    I don’t think it makes sense to call this kind of absurd test question an “error”, as if they were even close to asking something that measures analytic ability fairly.
    When I read this I think: the pineapple showed how silly the hare was in being tricked into running a race (without even a prize for winning), and how easy it was to manipulate him. The pineapple did have something up her sleeve, and I say “her” deliberately. (I should think the other animals would have appreciated her nerve.)
    One other thing the testmaker fails to consider: my anger, were I taking the test, at having to read and actually take seriously something so inane almost, and possibly would, make me “spite-answer” the question!
    It is precisely why I always detest those parts of standardized tests for “college-readiness”.

    • Shoe says:

      Mayo, You are right. The moose was right. The pineapple did have something “up her sleeve” in that she got the hare to run a 26 mile race, without the pineapple having to do anything at all. And of course a fruit is a “her” if one thinks about the basic biology.

      Thank you for posting this!

      • revo11 says:

        As someone who enjoys running and who has many runner friends, I find the assumption that running for no good reason is a negative thing to be quite presumptuous. People are not only are willing to run for no good reason, they’re willing to pay money to run in races.

        Not to say that one should have to assume running is a positive thing – just that one shouldn’t have to bring such a subjective outside bias one way or another in order to answer the question.

  9. J says:

    The questions are a bit silly, but I would have answered them correctly. If the item statistics are good, what’s the problem? You cannot judge how items function just by looking at them.

    • Andrew says:

      J:

      See Adam’s comment above. Also, Twing did not merely refer to the psychometrics. He also explicitly said that these questions had “correct answers.”

      • Mark Palko says:

        Someone else may have made this point already (there are a lot of comments here) but I suspect Twing pretty much has to toe the “correct answer” line. He’s a spokesman for a huge corporation in a situation where a non-evasive reply would expose his employer to countless lawsuits. I’ve been out of teaching for a awhile (and out of the public schools for even longer), but I’m relatively sure that these tests factor into a large number of decisions affecting districts, administrators, teachers and students. If Pearson started conceding flaws, the company’s lawyers would undoubtedly worry about floodgates opening with every question coming under scrutiny.

        An honest answer — “These questions correlate well with the internal and external metrics we use to validate our tests, but they’re still pretty stupid, so we removed them and we apologize to any students who might have scored a couple of points lower than they deserved.” — would have been nice, but I could see it costing Twing his job. It’s a sad state but Pearson’s handling of this is pretty standard.

  10. Roger says:

    Twing was not a cheater who got caught. He was asked about a question, and he explained it. What did you want him to do — withdraw the question without comment? Considering that the question had been used for several years, he owed an explanation. You don’t know that it is his question, or that he even agrees with it.

    I think that the questions are worse than what you say. The story says “The animals agreed” that the pineapple would win. So the owl’s opinion was that the pineapple was going to win somehow. Other questions are problematic also. “What would have happened if the animals had decided to cheer for the hare?”, “When the moose said that the pineapple has some trick up its sleeve, he means that the pineapple”

    These questions have no sensible answers from the story.

    • Andrew says:

      Roger:

      I agree that nobody cheated here. What happened is that a bad question got onto the exam. That happens all the time. What bothers me about Twing is that he did not acknowledge the problem. Instead of saying that these questions have no sensible answer and moving on, he defended the question. He was unwilling to admit error.

  11. Fernando says:

    “Why go on and on about these losers, given that there are more important problems in the world such as war, pestilence, hunger, and graphs where the y-axis doesn’t go all the way down to zero?#

    You made me laugh!

  12. MAYO says:

    Well I still say that the pineapple showed superiority in demonstrating how silly the hare was in being tricked into running a 2-hour race (without a prize for winning or penalty for losing), and how easy it was to manipulate him. Like, I’ll race you to the top of the Empire state building, and you go running, and I stay put. Of course, the moral of the story would have to be the opposite, and instead of eating the pineapple, the answer would have included their electing her to office.

  13. konrad says:

    I like the story – it’s clearly tongue-in-cheek, but its theme of questioning conventional assumptions is serious. The hare, moose and crow all fall for the incorrect but conventional assumption that the pineapple would only issue the challenge if it intends and expects to win (to be fair, the owl also gets talked into this, but at least it resisted initially) – but the point of the story is that apart from convention there is no reason to assume this: there are no consequences for losing (except that the animals got annoyed on having their foolishness shown up – which the pineapple clearly did not foresee), and there is satisfaction to be gained from making the arrogant and foolish hare run for two hours. I’m not saying that the questions cannot be nitpicked, but anyone who has understood the key point of the story (defy convention! question assumptions!) should get the 2nd question right – it’s not perfect, but it’s a sensible quesiton.

    The 1st question is rather easier, testing basic understanding of emotions – a sensible question for children, and I’d be worried about any adult who got it wrong.

    Sure, Twing could have done better at justifying the questions, but that’s not the point. Backing off without attempting to justify them would just have been despicably spineless.

    • Andrew says:

      Konrad:

      I hardly think that admitting error is “despicably spineless.” I am in fact disturbed by this attitude; it represents the sort of don’t-back-down escalation that I associate with all sorts of conflict. Why can’t Twing just say that the fable is fun but those two questions don’t have correct answers? Everyone makes mistakes—I know I do!

      • konrad says:

        Because he genuinely disagrees with you. The questions do have correct answers, and they do test understanding.

        I’m all for admitting error, but only when you agree that you were actually _in_ error.

        • Andrew says:

          Konrad:

          It’s not a matter of disagreeing with me. These questions were a laughingstock before they ever came to me. At the very least, the fact that many people (not just me) who have thought this through, disagree with the test developers, suggests that it’s silly to describe these questions as having “correct answers.”

          If Twing wanted to, he could’ve taken an intermediate position, something along the lines of: There seems to be genuine disagreement on the answers to these questions, indicating there is “correct” option in the way this is usually understood.

          • konrad says:

            Sure, he could have, and I take your point that in this situation it would have been an advisable and diplomatic rhetorical ploy to state the obvious (“there is disagreement” – duh!). But the bottom line is people disagreed with him, so it was incumbent on him to either justify or abandon his position. He did the former, which I think is the right thing to do when confronted with criticism one doesn’t agree with.

        • Andrew says:

          Konrad:

          But the very fact of all the disagreement is informative in itself. In what sense can these questions be said to have “correct answers” given this level of disagreement? It shouldn’t even matter whether this guy agrees with the criticism; the more general point is that it does not make sense to think of these questions as having correct answers at all.

          As I noted above, it’s hard to write a test, and it’s not uncommon for bad questions to end up even on carefully-vetted tests. There’s no shame in admitting that a question which seemed reasonable at the time, actually was ambiguous.

          • konrad says:

            You seem to be arguing that the existence of disagreement means that a question cannot have a correct answer. Somehow this reminds me of the “debate” in the USA about evolution – does it follow from current public opinion that evolutionary biology cannot constitute knowledge?

            There’s no shame in admitting that a question was ambiguous if and only if you agree that it was, in fact, ambiguous.

          • Eric says:

            I have to agree with Andrew on this one (not that it’s hard to agree with Andrew!). It’s a middle school English test. If university graduates who spend much of their professional time writing don’t agree on the answer, then it’s a bad question. This is not analogous to the cultural disagreement over evolution.

          • Shoe says:

            Andrew,

            I am actually very glad that the rationale for these questions was explained. Doing so satisfies curiosity, and shows that there was a rationale for the questions and answers in the first place. So I have no problem with that aspect.

            The problem is in the omission. I agree with you: the problem seems to be in the lack of recognizing that there could be anything wrong with the items.

            However, the text of Twing’s memo was never intended to be made public. Given the complete lack of introduction, it seems to be a response to a question (to which the public reader of the memo is not privy). Every sentence in the memo seems to be based on truth, and seems to be answering a question that might have been something along the lines of, “please tell me about this passage and question, because we are getting a lot of questions about it.”

            So, this memo might not have been the proper place for an apology, at all. We don’t know.

            Personally, I agree that the choice of passage and questions is, well, questionable. See below, for my particular problem with the “which animal spoke the wisest words” question. And Pearson ought to make a public statement agreeing that the questions, statistically relevant as they are (see my comment on that, as well), are not of the quality Pearson aspires to, given their substantial power in the lives of children, and that they will do all they can to fix this, and make sure they keep higher standards in the future.

            But Andrew, you write a column about research and statistics; the researchers in this case, did not lie about their statistics. I think you could pick a better muse for your soap box. I fully AGREE with your soap box stance, that researchers who lie are bad. I’m just not convinced that is what happened here.

            Fascinating read, however! Either way, I enjoyed your column, for the intellectual exercise. :)

        • Shoe says:

          konrad,

          The pineapple did trick everyone. The hare was tricked into running 26 miles for nothing. The rest of the “society” was tricked into rooting for the wrong one. The pineapple made fools of them all.

          The moose was right when he said the pineapple probably had a trick up its sleeve, but got the “trick” wrong.

          “The owl” NEVER SPOKE. AN owl said something about pineapples not having sleeves, but there are NO QUOTATION MARKS. Why does this matter? Because the test is supposed to look at the very detailed reading abilities of children, and a very detailed reader would see that the answers are not satisfactory, and be relegated to “guessing the best one.”

          The fact that better students are better at guessing the best one, when none of them is actually correct, is what leads to the positive correlation between correct answers on these questions, and other correct answers on the tests.

          Nonetheless, you might be able to support the “annoyed” answer, but you cannot support the “the owl” answer, because there simply was no specific owl and no quoted words from an owl.

    • Roger says:

      You justify a question because it is “testing basic understanding of emotions” — meaning testing how the child understands the emotional feelings that a moose might have for a talking p;neapple. How is this even related to reading comprehension, or to anything that is the business of the schools?

      • konrad says:

        We are given that the characters in this story converse like real people; and as the story unfolds it becomes clear that they act like real people, down to details like playing the fool and getting fooled. Given this context, it would be very strange to assume that their emotions in response to getting fooled run counter to those of real people. I wonder if the question would have generated less controversy if the characters were replaced by humans with names like Alice and Bob, and eating was replaced with (say) insulting or punching. It sure would have taken all the fun out of the question – reminds me of when the South African library service deemed the work of Richard Scarry unfit for children because it featured animals acting like humans and wearing clothes.

        • Andrew says:

          Konrad:

          I don’t think anyone’s objecting to the story or the appearance of talking animals and plants. It’s just those two questions that have problem. In my youth I took lots of tests such as SAT, GRE, with passages and reading comprehension questions. These just happen to be bad questions (according to the Daily News, me, and lots of other people). Bad questions happen. You can have bad questions in a story with all human characters too.

          • Anonymous says:

            Surprised to see that Andrew offers such a social constructivist position (majority rules).

            I thought the questions were interesting and probably annoy quite a few people (like, e.g., statisticians) because they aren’t boring and strictly analytical. I’d never choose them in a test, but I wouldn’t blame anyone that did offer such questions – in addition to more boring ones.
            The answers were, given the fairy tale context, fairly obvious – to some.

          • Andrew says:

            Anon:

            I didn’t say that majority rules. I said that, at the very least, if many or even most people think the answer is different, that should call into question Twing’s confident assertion about what is the “right answer.” Also, I personally did not think those particular answers made sense. So what you have is an unclear answer where some people (Twing and maybe you) think the answers are C and D, while many many others (including me and the people who resulted in the story getting into the Daily News) thinking there were no good answers.

            Again, I’ve written tests myself and I know that bad questions can happen, even in exams that are vetted. Sometimes you have to go beyond the colleagues of the item writer to see what’s wrong with a question.

          • Scott says:

            Anonymous:

            The point’s not how many people find the ‘correct’ answers obvious. A good question shouldn’t be too obvious. The point’s that lack of consensus, among educated adults who’ve had time to think about it, that the argument to the ‘correct’ answer is the only reasonable one is already a bad sign. If someone comes along & refutes all the arguments to ‘incorrect’ answers I’m sure we’ll stand corrected, or make counter-arguments – no-one’s suggesting a vote.

            And yes, there is more slack in deciding what’s a reasonable argument outside a strictly analytical context, but that’s not the issue here. It seems from the fuss that it wasn’t statisticians, but the children who had to answer this silly question who were the most annoyed.

          • Anonymous says:

            Yes, you do say majority rules – you clearly refer to it as “unclear answer”, and furthermore refer to how “many” people view it. I am not saying your argument doesn’t make sense, it’s just the warrant for your position. Well, not important.

            I showed it to my 13 year old niece who only has had english for 3-4 years. She was puzzled why I gave her the test, because it was so easy – for her. But she has read tons of this kind of literature recently.

            This question is different and because it is different it has to be used in the right way. I have no idea whether that is possible in the suggested setting – but it does capture something, I think.

  14. Jonathan (a different one) says:

    I had an extended discussion with an educational maven of my acquaintance last week. I seem to be alone (OK… joined by Mr. Twing) but I like the story and I like the questions and I like the answers. Admittedly, the overconfidence of the hare 9which doesn’t actually figure into any of the questions) is an error by Mr. Twing. But this is what i thought every one of these reading comprehension stories was supposed to be: (1) Try to draw inferences from the story and your background knowledge about how such stroies are constructed; (2) Put in (incorrect) answers that could well be right in some other story but are not as well supported in the story you have; and (3) Use the psychometric results to justify the questions.

    You seem to feel, Andrew, that the questions are intended to have deductive rather than inferential answers. Why? The GREs have (or had) a test for logical deduction. That’s just not what’s being tested here. And once you grant (if you do) that the answers to these questions are inferential, then we enter a world where lots of kids can miss a question just because they aren’t particularly good at this sort of free-floating inference.

    adam 9above) is on the right track. Doesn’t the fact that the kids who get all teh other questions right tend to get this one right indicate that they have some set of skills which will seve them well in school? adam disagrees whether this set of skills is fairly distributed, but that depends on definitions of fairness that are sort of beside the point here.

    • Andrew says:

      Jonathan:

      I think there’s a sliding scale, with meaningless questions at one end, clear questions on the other, and various possibilities in between. The fact that many people (not just me) find these questions problematic suggests that this question falls on the bad end of the scale. In this case, I find the owl’s remark pointless rather than wise, and I see no reason to think the animals were annoyed. A much more reasonable hypothesis to me is that they were hungry.

      • konrad says:

        Yes, the owl’s remark was pointless – the question asks you to decide which is “wiser”: a pointless remark which had no consequences, or a foolish remark which had negative consequences. Seems clear-cut to me.

        “A much more reasonable hypothesis to me is that they were hungry.” – Oh dear, please tell me you’re joking.

        • Alex says:

          “Hey guys, that hare’s probably finished by now, and the pineapple’s still doing nothing. I’m hungry—shall we eat it?”

        • mpledger says:

          The owl’s question could be pointless or it could be meaningful. It depends if you take the owl’s meaning literally (i.e. pineaples don’t have sleeves – well, duh!) or figuratively (i.e. the pineapple can’t be playing a trick because it can’t trick). And I don’t really see any way of telling the difference except for the “moral” at the end.

          I read it literally myself and thought the owl was being (socially) silly – it’s the joke about literal people getting confused over the obscure nature of sayings (I think I’ve been watching to much Big Bang Theory).

          However, I am not surprised that kids who scored well overall got this question right. If you’ve read lots of kid’s literature and had to guess a brainy animal then you most likely to plump for an owl.

          Eating the pineapple was a bit confusing (although I got the question right). If they had of stamped on the pineapple that would have been more easily understood but I guess it would clash with the violence codes about question making (and eating didn’t?).

      • Jonathan (a different one) says:

        I’m not sure I know what “finding problematic” means in this context. It can’t mean that the psychometrics are messed up, because no one finding it problematic has looked at the psychometrics. It shouldn’t mean that the questions are unusual because, well, all of the question should, in my opinion, require something unique, not just the rote Stanley Kaplan tricks — that’s a good thing. It might mean that the someone doesn’t know the answer themselves, but that assumes that your introspection trumps the psychometrics. On what basis does anybody believe that? FWIW (to me) there is exactly zero evidence that the animals were hungry in the story. there is certainly some evidence that they are annoyed, since their anticipation of discovering the pineapple’s trick was folied since there was no trick. I’d certainly find that annoying. The likelihood ratio in favor of the annoyance hypothesis over the hunger hypothesis must favor the annoyance hypothesis. And I think it’s quite clear that a story this fanciful can’t sustain a strong prior either way — QED.

        • Scott says:

          Eating the pineapple is evidence of being hungry. There’s some justification for guessing that the animals might be annoyed; but people & ordinary animals don’t eat because they’re annoyed, at least not in my neck of the woods, & nothing in the text indicates that these talking animals are different.

          So after the race they thought “Oh well, a pineapple’s just a pineapple after all” (see Fraac’s comment), & put it to its natural use & ate it. The alternative hypothesis is quite nasty really, & if I knew someone who’d chosen ‘C’ I’d be worried about his psychometrics.

      • Scott says:

        Our prior knowledge concerning owls’ wisdom gives greater weight to the interpretation of this one’s comment as a gnomic utterance meaning “You’re over-complicating things by speculating about the pineapple’s motivations – whatever they are, we shouldn’t lose sight of the obvious fact that a pineapple has no means of beating a hare in a race”. If a duck or a peacock had said the same thing we could safely assume it was just taking the metaphor literally.* All the same, it hardly seems fair to penalize those who lack such prior knowledge.

        *Though perhaps the _words_ would still be wise, with the benefit of hindsight – “Out of the mouths of babes & sucklings [...]“.

  15. Howard wainer says:

    Let’s try another question:
    What is the right answer?
    a. 128 quarts
    b. 128 pints
    c. 1,280,000 quarts
    d. 536 quarts

    When this was asked to a lunch table full of test developers at ETS they all agreed what was the correct answer, as did 90% of the students in a statistics class at the University of Pennsylvania.

    After some folks offer their choice of the right answer, with the logic supporting their choice I will continue — to make the connection with hares and pineapples.

    • Andrew says:

      Howard:

      I have no idea. The 1,280,000 answer looks wrong somehow—anything that big wouldn’t be measured in quarts—but beyond that I have no idea.

      • Phil says:

        I’m going with 128 quarts. I say that on impulse and without even thinking about why that is my choice. I’m pretty sure though.

        On a possibly related note, David Owen’s old book “None of the Above” gives the questions and answers for a reading comprehension portion of the SAT, but not the passage that the questions are supposedly about. Owen asserts that anyone who is reading his book will not need to read the passage in order to get all of the answers right, and at least in my case that is correct. Pretty stupid “reading comprehension” test if you don’t have to read anything in order to get the right answer. (The questions were things like “The author would probably agree with which of the following statements:
        (a) Even though hard work isn’t always rewarded, it’s your best chance of getting ahead.
        (b) Since hard work isn’t always rewarded, there’s sometimes no point in putting a lot of effort into something.
        (c) There’s not much relationship between how hard you work and how well you do in life.
        (d) People shouldn’t worry too much about whether they get ahead in life.)

        Anyway I had no problem getting the pineapple questions “right” but not because they’re actually the right answers.

    • konrad says:

      (a), as explained by David. This is a common flaw in multiple choice questions and I agree that the 2nd question under discussion suffers from something similar – one could sniff out the intended answer by finding the odd one out. But it’s hard to see how to modify the question to avoid this – the aim of the question is basically to ask “did you get the joke?”, and finding a way to test that is never easy.

      • konrad says:

        Another way of explaining why (a) is the right answer: it’s the only answer that has all three of the properties (“128-ness”, “quart-ness” and order-of-maginitude=100) which the question designer evidently considered important for making it to the status of candidate answer. Each of the other answers has 2 out of the three properties.

        Of course the argument only works when the question designer is not actively trying to counter question-gaming – but they rarely do.

    • K? O'Rourke says:

      Howard:
      As my old anthropology prof (Tom McFeat) would argue, small-group cultures can quickly arise in a lunch table full of co-workers (Small-Group Cultures, Permagon Press, 1974) and within that culture, I am not surprised that there is a definite right answer.

      But I think you raise a very important point – except for math and inescapable hard facts of history – it is essentially impossible to know – without error – that someone is wrong.

  16. David J. Harris says:

    Howard: If I were to try to game that question, this would be my thought process:

    There are two “128” answers, and 128 pints is obviously “wrong” because everything else is in quarts. So 128 pints is there to distract from the real answer, therefore the real answer must be 128 quarts.

    Of course, the test maker could be operating under a different scheme, but this is what’s most consistent with my experience on multiple choice tests.

  17. Francis says:

    Actually, the true moral of the story is that murder is justified if someone is really annoying you. This does not bode well for future generations if this is part of the curriculum they’re trying to teach 8th graders.

  18. Stephen Senn says:

    It reminds me of the famous 1970s joke (well, famous to me):
    Q “What’s the difference between a duck?”
    A “One leg’s both the same”

    • Andrew says:

      Stephen:

      Of the same vintage:

      Q: What do you get when you cross an elephant with peanut butter?

      A: Peanut butter that never forgets, or an elephant that sticks to the roof of your mouth.

    • Corey says:

      Q: How many surrealists does it take to screw in a light bulb?
      A: Fish.

  19. Henry says:

    This is an awesome story. I would have put “annoyed” but I too suspected hungry for the reason as fraac. The story is sufficiently absurd that it would make a great anticlimax. Oh well.

  20. Manuel says:

    Well, I am with those that do not see what’s wrong with this. I guess there is no way a question about an allegorical (or parody-of-allegorical, if you wish) text is “correct”. But I really like the story, and its moral. Maybe because I am an economist, and what this story tells to me is “distrust statements saying that there can be no $100 notes on a sidewalk”.

  21. (This was posted only yesterday, and already I’m late to the party.) Thanks for posting this. I love Pinkwater — if you can find Wempires, read it — and regret losing touch with him when my kid got too old for me to read aloud to him, since he had left home for college. I hadn’t read “The Hare and the Pineapple.” But I laughed more at the test questions and especially the answers than at the story.

    The trouble with this is that the test is supposed to find out what kids understand about the story. But the questions themselves miss the whole point of the story, which is that it’s a spoof on fables. It’s like telling a joke and then asking,”What was the Jew’s motive for walking into the bar?” or “Who made the wisest choice of beverage — the Irishman, the Jew, or the orangutang?”

    • Andrew says:

      Jay:

      The answer is B: He came into the bar because they were offering free food.

      P.S. See, for example, here, for background.

      • Shoe says:

        Andrew,

        No joke: did you just make an anti-Semitic joke about Jews? I think you did. I think you need to apologize.

        With this thread, you could think I am kidding. And I am very, very serious. I am personally offended. Your “punch line” plays on the prejudice that accuses Jews of being “tight-wads.”

        Given your position on admitting a mistake, I hope you will adhere to your own soap box, and apologize. However, I also want you to remove this offensive post. Perhaps you could replace it (and my post) with a comment that you have removed offensive comment, and apologize for not realizing how offensive it was before you posted it.

        • Nick Cox says:

          So, jokes, real or implied, about Irishmen are OK for you? As some of my relatives are Irish I want to protest about this unevenness. Actually, biology tells us that orangutangs are our relatives too, and I am not tremendously in favour at mocking species without the scope to answer back on the internet.

          You have the right to be offended, but it’s Andrew’s blog. I really hope he doesn’t apologise (much).

      • Shoe says:

        Seriously? You edit your post to add explanation for your comment, without apologizing? Hypocrisy!

      • Shoe says:

        Speaking as a Jew with a fully functional workroom in the house, including table saw, I say that your “background” is just as offensive as your punch line.

    • Martyn says:

      Pinkwater’s story was adapted for the test. In the original story, it was an eggplant not a pineapple, and the moral was “Never bet on an eggplant”. Pinkwater is not happy with the modified version according to an article in The New Yorker: http://www.newyorker.com/talk/2012/05/07/120507ta_talk_mcgrath

  22. Scott says:

    By the way, this question wouldn’t be stupid at all if you had to write an answer & justify it, & if the marking allowed for differences of opinion. That’s how it was in my day (& in my country). You don’t have to be Derrida to wonder at childrens’ being taught that there’s only ever one correct interpretation of a text. The limitation of having to choose between a small selection of given answers with no opportunity to explain your reasoning seems appropriate for pub quizzes or opinion polls, but surely not for exam questions, or indeed those that arise in real life.

  23. Rick says:

    There is nothing wrong with defending the question, because the answers are fairly obvious (to many of us). Plus, judging from the comments and informal survey of my colleagues, the various people who got them right are getting there by the same line of reasoning, so clearly it is not nonsense. Quit conflating things you find challenging with nonsense. You should apologize for failing to see the logic in the question, and then covering up your failure by accusing its author of peddling nonsense.

    • Phil says:

      Rick, I was able to correctly guess the answers they wanted, but not because they’re the right answers; they’re just the answers that conform best to the cultural norms (owls are wise, right?).

      In fact, the right answers are unambiguous:
      (1) the animals ate the pineapple because they were hungry: as fraac points out, the only reason animals eat is because they’re hungry. If they were just annoyed they could have simply smashed the pineapple or torn it apart.
      (2) The hare is the only one whose words were wise. He did not _act_ wisely — he got tricked into running for 2 hours for nothing — but the question asks about his words, and his words were the only ones that were true and to the point. (The other animals’ words were either obviously wrong the moment they were spoken, or, in the case of the owl, either pedantic or based on misunderstanding a figure of speech to be interpreted literally).

      So the “right” answers according to the test are in fact not the right answers. And yet, I knew which answers they were looking for.

      This is a very bad test question. I do like the story, though.

  24. Clark Andersen says:

    I find it amusing that a question implicitly designed to have a subjective error distribution, rather than an unambiguously correct response, stirs so much angst amongst the readers of a statistics blog. As statisticians, shouldn’t we embrace error as we find it, and work with it as it is? As I see it, there’s nothing wrong with this question, so long as it is not the sole question on the exam; if there’s enough other questions (subjective or otherwise), then the odd subjective question will have little effect. We’re interested in the expected value of the overall score. Some exams, like IQ tests, are not designed to allow a perfect score to be achieved.

    That being said, I thought the story was amusing and the correct answers obvious. Of course, people who didn’t grow up reading this sort of story would be less likely to reach those conclusions, but isn’t one of the points of a test to distinguish between people? Wouldn’t a math problem be discriminatory towards people who didn’t spend as much time doing math as opposed to those who did?

    • Scott says:

      I’m puzzled by that. I don’t know the background to this question, but I understood it to be from an exam, not a psychological test. Why then is it unreasonable to expect the ‘correct’ answer from a forced choice to be unambiguously correct ? And I’ll grant you that one bad question out of many isn’t a big deal – but why shouldn’t the response be “Well spotted ! We’ll scratch that one directly” ?

      You found the ‘correct’ answers obvious; perhaps most people do. Yes, the point of tests is to discriminate; but do you think, given the justifications made in several posts on this blog for ‘incorrect’ interpretations, including some by – call me a credentials snob if you like – eminent academics, that this question is making the right discrimination ? Really, honestly ?

      • Clark Andersen says:

        I would say that if a (predefined) raw score from the exam determines a grade or pass/fail, then I would agree that all questions should have unambiguously correct versus incorrect solutions. However, if the intent is to quantify the level of knowledge, then ambiguous problems have a place as they are capable of measuring a higher order of understanding.

        I quite appreciate the legitimate “correct” alternative explanations presented above, and if this were not a multiple choice exam there would be room for these to be accounted for.

  25. Corey says:

    “Item 8: The correct answer is D… The moose and the crow… present the incorrect idea that the pineapple has a clever plan to win the race. This idea is proven false when the hare wins the race. The hare… overconfidently agrees to race a pineapple.”

    Accepting the rules of the story world for the moment, this explanation criticizes the moose and the crow for making the assumption that the pineapple knows something they don’t — and also criticizes the hare for failing to make that same assumption!

    • Scott says:

      Well I’m glad we agree about exams, though I don’t follow the bit about predefined raw scores. To dock marks from a child who gives one answer to an ambiguous question because children who generally do well in these sort of tests tend to give a different answer is unfair, just as to dock his marks because children who generally do well in these sort of tests tend to have richer parents would be unfair.

      My knee-jerk reaction to the rest is to say that if you want to measure a higher order of understanding you should set essays. But, sticking to forced-choice tests, I’m afraid I only have a hazy idea of what you’re getting at. I can’t believe it’s beyond the wit of educationalists to set unambiguous yet taxing questions. Do you think these two ‘Hare & Pineapple’ questions measure something that unambiguous ones couldn’t ? If you’ve got examples I’d be grateful to see them.

      I would have thought even if you’re trying only to quantify the level of knowledge (of a child, a school, a country, or whatever), requiring that the questions have unambiguous correct answers would be a useful way to protect against various types of bias and give you a better chance of measuring what you really intend.

  26. Eli Rabett says:

    Everyone with a room temperature IQ knows that to take these tests you leave imagination and thought at the door, they are not interested.

  27. [...] science department and covered a mix of statistical and political topics. Followers of our recent discussion on test questions won’t be surprised to learn that some of the questions are ambiguous. This [...]

  28. Igor Carron says:

    Since when do pineapples talk ?

    If by chance a pineapple could talk, couldn’t we just assume that the same synthetic biology experiment has given him some means (i.e. the genes) to walk or run ?

  29. JSB says:

    I don’t see any mistake with these questions and answers. This is an reading comprehension question which requires the test taker to guess what the author was intending to communicate. This can be a fuzzy art, but not in this case. In question 1, the author never mentions anything about hunger in regards to eating, but directly paints a picture of annoyance. Where is any indication from the question that the animals are hungry? Eating does not automatically equate to hunger, especially in the context of an allegory.
    For question 2, the clue to D being the only correct answer comes from the very last line where the moral of the story is that pineapple does not have sleeves, an echo of the Owl’s statement. Being wise in the context of this question is lies in the morale of this allegory. Don’t expect to find depth in the shallows.

    • Scott says:

      It would be a better question if the author did directly paint a picture of annoyance. But he doesn’t – not even an indirect picture. None of the characters does or says anything to even hint they’re annoyed.

      No, there’s no mention in the text of hunger, nor is it the only possible motive for eating. Anyway, I’m off to get dinner ready. (Can you guess why ?)

      • JSB says:

        “You aren’t even an animal!” the hare said. “You’re a tropical fruit!”

        “Well, you know what I mean,” the pineapple said.
        I’d wager the hare was more annoyed than hungry based on the above.

        • Scott says:

          Sorry, I don’t accept the wager – if I win you might get annoyed & eat me.

          But seriously, that’s not a direct picture of annoyance; it’s your guess. The hare sounds incredulous to me, but not annoyed. Hunger’s the default explanation, which it requires better evidence than a guess to topple. In my view anyway – but if the ‘correct’ answer depends so much on prior assumptions about what annoys people it’s not a good test of reading comprehension.

    • Anonymous says:

      Excellent summary that clearly shows that the test makes sense.

      No we all just have to wait for AG to acknowledge that “many people” think this and that he made a mistake – unless he has double standards of course, god forbid.

      • Andrew says:

        Anon:

        Of course I recognize that some people think the official answers are correct for these problems. My point was that it doesn’t make much sense to describe these as “the” correct answers, given the ambiguity and clear lack of consensus here.

  30. Zach says:

    There’s gotta be a special place in hell for people who talk at the theater and make graphs with misleading axes.