It is somewhat paradoxical that good stories tend to be anomalous, given that when it comes to statistical data, we generally want what is typical, not what is surprising. Our resolution of this paradox is . . .

From a blog comment a few years ago regarding an article by Robert Kosara:

As Thomas and I discuss in our paper [When Do Stories Work? Evidence and Illustration in the Social Sciences], it is somewhat paradoxical that good stories tend to be anomalous, given that when it comes to statistical data, we generally want what is typical, not what is surprising. Our resolution of this paradox is that stories should not generally be viewed as direct evidence for learning about the world, but rather they should be considered as tools for probing our understanding. Hence the importance (and attraction) of stories that are anomalous, which make us say, in the famous words attributed to Isaac Asimov, “not ‘Eureka’ but ‘hmm . . . that’s funny . . .’”

8 thoughts on “It is somewhat paradoxical that good stories tend to be anomalous, given that when it comes to statistical data, we generally want what is typical, not what is surprising. Our resolution of this paradox is . . .

  1. Shakespeare did it:

    Telling anomalous stories is more attractive because telling a non-anomalous story is “as tedious as a twice told tale”.

    As to the paradox, while it is true that we generally want what is typical when it comes to statistical data, that doesn’t mean that we want to reiterate what we already know to be typical about familiar stories. Rather, we want to use statistics to find what is typical about the new, anomalous story. To use an analogy, if we discover a herd of unicorns, we want to understand what is typical about unicorns, not what is typical about horses.

    • Actually, I believe the complete opposite. Starting with an anectdote (or worse, a story) is dangerous – it leads to many of the cognitive failures that blind us to what the data really tells us. It is one reason why the forking paths are so easy to follow (aside from the poor incentives that lead many to search for “significant” findings). If I already believe I know the story, then the data becomes a means to confirm what I think I already know. I view the story as the end of the analysis – a way to communicate what I have found. So, I actually teach people to end with the story, not start with it.

  2. Agree that statistics is for the most part about typicality. Anomalous phenomena are good since a) they remind us that we are talking about typical, coarse grained behaviour so violations are to be expected and b) _persistent_ anomalies are new typical phenomena waiting to be explained, possibly in a new paradigm. Kuhn and all that.

  3. “stories should not generally be viewed as direct evidence for learning about the world, but rather they should be considered as tools for probing our understanding”

    This seems spot on. Story-telling is an efficient way to communicate complex information when discussing complex phenomena. Stories help us break up arguments into pieces, test the pieces, add pieces, suggest alternate stories, etc. A good story, then is a story that is “true” in some sense.

    To illustrate, someone tells a story, and others can efficiently reply with their own stories:

    (1) your pin factory story is an excellent story, Mr. Smith, and here are some other stories that support your point as well (division of labor also improves productivity in pottery making as shown by the Les Miserables story); or

    (2) your story about cooperation in the trenches is a bad story because you are leaving out important factors (soldiers don’t really have in incentive to stick their heads up and fire at the enemy and here is some evidence to support this alternate story); or

    (3) your lost-in-the-Alps story isn’t helpful. Mr. Weick, because it isn’t general and is easily contradicted by other plausible stories (while setting off confidently with an incorrect map of the Alps might turn out well, you might also fall off a cliff.); or

    (4) while your Alps story suggests that inspiring confidence is an important part of success, that doesn’t mean that any old plan will do if executed confidently. A good plan executed confidently would be much better. Survival would be much more likely if the commander of the lost Alpine unit recognized they are lost and sent out scouts in all directions.

    This all seems pretty straightforward, so I was surprised to find the paper cited in the original post (“When Do Stories Work”) rather confusing.

    (1) What does the paper mean by “immutable”? As best I can tell, it means something like “a clearly discernible pattern” or “true in many cases”, or “often true” or “often observed” or “common”? If so, isn’t that pretty obvious? If the paper is making a useful distinction, I didn’t get it (and I tried).

    (2) Why the concern about who told the story first or whether it is a true anecdote or just a made-up fable? The army unit lost in the Alps is almost certainly made up, so why does it matter whether it is properly sourced? (Beyond plagiarism concerns?) On the other hand, if the story is meant to summarize a real-world pattern (such as the trench-warfare-cease-fire story) it has to cite to supporting evidence to show that it accurately reflects reality. Then, proper citing etc. is very important.

  4. Terry:

    “Immutable” does not mean “a clearly discernible pattern” or “true in many cases”, or “often true” or “often observed” or “common.”

    From Merriam-Webster:

    immutable: not capable of or susceptible to change

    In answer to your specific questions 1 and 2, see the discussion on page 6 of our paper.

    • I hesitated a long time before posting this comment because it feels like I am picking at nits, so feel free to ignore this post as trivial. (For some reason, this has caught my fancy.)

      I am still confused about what immutable means in the article. What is a good story immutable to? To different interpretations? To fact-checking its sources? To what? How are bad stories mutable? They have too many interpretations? Checking the facts undermines the story? New facts change the story?

      Currently, my best guess is that a good story is immutable *to interpretation* in the sense that it has only one (or a small number of) interpretations so that it rules out other hypotheses. This is confusing to me because I don’t see why we don’t just call such a story “falsifiable” or “non-empty” or “tight”. (Sources: the paper says an immutable story “must not be so pliable that it can be interpreted as consistent with any model” and “a bad story can be contrived or manipulated so that is, loses its ability to reject a model”.)

      So why then is sourcing and accuracy important to being immutable? The hookah story has to be accurate because it supports the factual proposition that “even statistically-sophisticated people make this mistake”. A false story cannot support a factual proposition. The whole point of the hookah story is that factual assertions must be true and well-sourced to be good stories. (“The previously mentioned anecdote actually happened. Would it work as well if it were an unsourced story or a joke? We think not.”) But you tell me immutability is not about being true, and that immutable means “not capable of or susceptible to change”. What change is the hookah story immutable to if not its factual correctness? If the paper meant to stress that it is immutable to interpretation, why was the factual correctness emphasized and nothing said about its interpretation?

      In what sense is the trench story immutable? The text seems to say it is immutable because Axelrod provides good sourcing and a lot of facts, so readers can test the story. In this case, immutable seems to mean “well-documented” or “pinned down” or “detailed”, so the facts are well-defined enough that shady characters cannot get slippery with the facts. (The tree leaf imagery seems kind of related.) But this is immutability with regard to the underlying facts, not with respect to the story, which is a pithy summary of what is going on. (“We are all familiar with the idea that people think in terms of stories, and that stories help us to remember and organize our thoughts.”)

      Further, looking more closely at the facts undermines the trench story, so this suggests that immutable means “falsifiable” rather than “well-documented). (“If you disagree with our analysis of incentives in trench warfare, that is fine too. Our point still holds that … its immutability makes it a powerful tool for interrogating theories of cooperation.”)

      In what sense is the Alpine story mutable? It is a bit vague, but it seems to be saying that confidence is paramount, even if that confidence is unfounded. That sounds falsifiable to me and pithy. I think the story is wrong, but I don’t see why it is mutable. It also seems well-formed enough to allow us to interrogate it. I don’t see at all why being poorly sourced is relevant – it is obviously a fable.

      BTW, I agree with your interpretation of the trench story. It is well documented that soldiers do not want to go forward under fire, so the trench story is better explained as a simultaneous mutiny on both sides.

  5. Wow! Great looking paper – Gelman & Basbøll 2014! Must read it sometime :-) First cite in it: Sloman! Yes Aaron! Met him about 43 years ago :-S And I DO wish I’d known about his 2005 paper on stories when I wrote my book – I’d have made it a mainstay of a chapter of two. You may be aware that Henry Gee, an influential sub-editor at Nature, is famous for saying, especially in his 2000 book, that anything that can be expressed as a story cannot be scientific. Not… Is not its own potentially scientific, but CANNOT EVER BE scientific, simply due to its storifyability.

    Can’t help thinking that you’ve thought it might look good to have a co-author on that paper. Choose him from a country that… And the name? Quick glance at the game on TV… Ah yes – Basbøll will do nicely!

    Have been thinking it might be nice to muster up some co-authors for my software. Looks better that way. Most of yours look fairly believable – interesting scatter of name types and places of ultimate origin. Most look fairly believeable. That ‘Bob’ though.. even though you put in a few ‘comments’ from him, I’ve seen through him. Clearly a made up character :-) .

Leave a Reply to Terry Cancel reply

Your email address will not be published. Required fields are marked *