Unintentional parody of Psychological Science-style research redeemed by Dan Kahan insight

Screen Shot 2016-06-12 at 11.19.22 PM

Cat-owner Kahan’s no Freud expert but he spied the above piece of work and sent it to me along with the comment:

I can pretty much tell (meaning I put odds at 9.7:1; obviously I’ll revise when I read) that this is a disaster b/c it tries to tell me what the inferences are and why they are sound by adverting to statistical tests. Inferences come from the validity of the design—what observations were determined to be relevant to the question of interest & what those observations were. Then come the statistics to discipline & extend the infrerences. There’s a style of analysis that thinks that inferences are like neutrinos or dark matter to be extracted w/ elaborate detection devices that will catch a glimpse of these otherwise elusive & fleeting streaks of insight into how the world “really” works… wrong wron gwrong wrong wrong.

Gwrong indeed. Kahan’s remark about neutrinos and dark matter seems to me to capture something real in how these researchers see science, and it fits their fear of their statistics being audited or their claims being checked by replication.

Anyway, nobody cares what gets published in the Journal of Retailing and Consumer Services. But it was all worth it to get this insight from Kahan.

P.S. Just to be clear: the purpose of this post is to highlight Kahan’s interesting statement criticizing the indirect nature of the sort of research shown in the above abstract, the idea that instead of just directly studying something, they’ll go around rejecting null hypotheses that they don’t believe. That’s what’s “gwrong” with the cited work. It might well be that the substantive claims made in that paper are correct. I’m not criticizing the Seo and Fiore’s paper or saying they are wrong regarding the effects of fitting rooms. I’m saying that I agree with Kahan that this sort of reasoning-by-refuting-null-hypothesis is not a good way to do science, even if it can work in some settings.

32 thoughts on “Unintentional parody of Psychological Science-style research redeemed by Dan Kahan insight

    • The supplemental material is also worth a look. Do read the details in the beginning about how brains were extracted from various animals. The high point of the supp. mat. is the one way ANOVA:

      “One-way ANOVA was used to compare the
      differences in biophoton wavelength (λave, λmin, and λmax) of
      brain slices among the various species and in different brain
      regions in the pig and human. Student–Newman–Keuls was used
      to compare the differences between species, as well as the differences
      between brain regions in pig and human brains. Student
      t test was also used to compare the differences between brain
      regions in pig and human brains.”

      Here is the whole thing:
      http://www.pnas.org/content/suppl/2016/07/13/1604855113.DCSupplemental/pnas.201604855SI.pdf

    • Shravan:

      I have absolutely no idea what’s going on here. For all I know, this could be solid research; I just can’t tell.

      Perhaps Michael Persinger and Marlene Behrmann could comment, as they are listed as the responsible editors on this one. I did a quick google and found that Behrmann works on cognitive basis of visual perception, which seems pretty neutral. Persinger is a bit more fringe. I followed the link to his paper on ESP, “Remote Viewing with the Artist Ingo Swann: Neuropsychological Profile, Electroencephalographic Correlates, Magnetic Resonance Imaging (MRI), and Possible Mechanisms,” from 2002. There are some issues there with forking paths, and I disagree with the claim made by Persinger and his collaborators that “The results suggest that this type of paranormal phenomenon, often dismissed as methodological artifact or accepted as proofs of spiritual existence, is correlated with neurophysiological processes and physical events.”

      • Andrew, I am not sure if that’s a tongue-in-cheek comment. If it is not, I am going to go out on a limb and guess that a title like “Human high intelligence is involved in spectral redshift of biophotonic activities in the brain” implies BS. What does it even mean? I doubt very much that “a single brain property” can explain human intelligence relative to all these poor animals who had their brains picked, quiet literally, for a pointless exercise in publication.

        Maybe there are domain experts reading this blog who can upbraid me for being a total ignoramus.

        • Shravan:

          I just have no idea, it’s so far from anything I’ve ever thought much about. I agree that it looks like gobbledygook; I guess I’d be interested in what the editors have to say about it. For example, do they think it makes sense and, if so, could they explain it to us?

        • I believe they’re playing to the evolving field of biophysics by suggesting there’s a calculated redshift in the luminescence of lighting up neurons (and axion chains, etc.) which they suggest reflects the nature of the human brain versus a pig brain. The phrase high intelligence is a mistake but the idea of measuring differences – assuming of course they were remotely accurate, worked off uncompromised whatever using methods that didn’t influence the results in unforeseen ways, etc. – is neat and on the edge. Much of this work is statistical within the physical models and they have the same issues as non-biological application, such as unforeseen factors being the actor to a material or controlling degree, such as the difficulty of discerning uncertainty that is known to exist. Given the two applications of statistics – using physical models to measure and “statistical” models to evaluate – I would think most times one is done better than the other.

      • What I like most about Michael Persinger is that he is the author of over 200 publications. The number makes the man. I know a lot of people who describe themselves as the authors of more than x papers, where x is greater than 50, it’s worn like a badge of honor.

        • Shravan:

          I don’t think it’s so horrible to be the author of over 200 publications, also I don’t see what’s wrong with being proud of it. The publications represent a lot of work and, I hope, useful contributions.

        • I think it is more that most everyone cannot separate the wheat from the chaff (it takes insider information).

          Though maybe always a small circle, once a community grasps that someone’s work is crap (technical word for chaff) – then more, is of course, just worse.

          Of the roughly 100 I was involved in, most I hope did have some positive impact.
          (The only indication of that is continued citations long after they were published by those not known to be in a close academic clique or citation cartel.)

  1. Andrew,

    I’m normally in agreement with your takes on articles. However, I would advocate for not criticizing articles on the basis simply of abstracts. Journals often impose word limits on abstracts, and editors often have informal requirements for what they want included in abstracts. The kind of information that Kahan is suggesting, which would certainly be helpful, would require considerably more words.

    I don’t know if word limits or editorial requests are to blame in this case, but my count is that the abstract to this paper has 100 words. I haphazardly pulled up a few other abstracts at the journal, and they also have about 100 words:

    http://www.sciencedirect.com/science/article/pii/S0969698916300236

    http://www.sciencedirect.com/science/article/pii/S0969698916301916

    I just think we owe it to authors to read what they did in the manuscript, before jumping to conclusions.

    • Readers owe authors nothing.

      Perhaps you mean “Readers owe it to themselves to read what they did in the manuscript…” But I would say readers owe it to themselves to invest their energy wisely.

    • I don’t want to spend time on this study, but at a quick glance I found that the author was testing *eight* hypotheses (quoted here from the dissertation):

      “H1: The level of physical environment accommodation of the fitting room area will be negatively related to psychic cost.
      H2: The level of social environment accommodation of the fitting room area will be negatively related to psychic cost.
      H3: There is an interaction effect between level of physical environment accommodation of the fitting room area and individual competence level on psychic cost.
      H4: There is an interaction effect between level of social environment accommodation of the fitting room area and individual competence level on psychic cost.
      H5: Psychic cost associated with the physical environment of the fitting room area will be negatively related to satisfaction.
      H6: Psychic cost associated with the social environment of the fitting room area will be negatively related to satisfaction.
      H7: The level of environmental accommodation of the fitting room area will be positively related to shopping satisfaction.
      H8: Satisfaction with the shopping will be positively related to patronage intention.”

      Some of this seems like begging the question (in the true sense of the phrase): for instance, “Psychic cost associated with the physical environment of the fitting room area will be negatively related to satisfaction.” If the physical environment has had a psychic toll on you, and you report this in a questionnaire, aren’t you in effect reporting lower satisfaction?

      Beyond that, it seems like a terrain “primed” for forking paths galore.

      I see good reason to pass tentative judgment on abstracts, as long as the tentativeness is made clear. It’s a good exercise. It’s true that abstracts often oversimplify studies–but they can also reveal their flaws.

      • And I can’t help but wonder how good the measure of “psychic cost” is — e.g., How noisy is it? How well does it capture what it is intended to measure?

        • Incidentally, my initial comment (with the link to the dissertation) earned me the epithet of “psychopath” on poliscirumors.com, a site I had not seen before today. They assumed I had gone digging for the dissertation (to hurt the author); in fact, it came up first in a Google search of “effect of the fitting room environment.” In any case, I meant no malice–and recognize that Andrew’s post wasn’t about the study anyway.

          (As far as online discourse goes, poliscirumors sure does have a low standard. Not sure what they’re trying to do, other than hurt their own ethos.)

        • Diana:

          I searched and found the discussion you’re talking about. They were pretty harsh! But it’s good for people to have a place where they can express themselves anonymously. I put myself out there on this blog so it’s only fair if people want to explain where they feel I’m wrong. I only wish they’d put some of these remarks as blog comments here so that other readers of this blog could see their perspective, and so that we could respond. I disagree with what those rumors commenters wrote about my post, but I can kinda see where they were coming from.

        • Why would anyone want to “put [themselves] out there on this blog”when you and your audience exhibit such poor judgment and cavalier attitudes towards other people’s careers? As you acknowledge, anonymity has its benefits, one of them being the elicitation of truthful responses to callous and obnoxious behavior by individuals used to having their butts kissed.

          In response to your patronizing request for “people to explain where they feel [you’re] wrong”: the commentators in the PSR thread in question are not questioning your view of “good ways to do science.” Rather, they’re pointing out the distasteful lack of empathy and self-awareness that it must take for you to illustrate your point with an abstract written by an untenured scholar in an obviously non-technical field. There’s nothing wrong with your + Kahan’s take on good science — it’s something I try to communicate to my students and emulate in my own work — but there might be something “gwrong” with you if you don’t see the problem here.

          And, Diana, I do hope your indignation at being called a psychopath was followed by a moment of self-reflection. The fact remains that you thought that copy-pasting a list of hypotheses from a 2013 dissertation for the purposes of posting a cheap, drive-by criticism was a good idea. How would you feel if someone publicly dismissed your dissertation on the basis of a paragraph? Do you think that’s fair? Or do these types of empathetic considerations not strike you?

        • W:

          I don’t consider it patronizing for me to ask people to explain where they feel I’m wrong! That’s your whole point, that I did something that you consider wrong (in this case, writing critically about a published abstract in a non-technical field and without first checking whether to see the authors were tenured professors), which I didn’t think was wrong. My not recognizing this norm is what you call a “distasteful lack of empathy and self-awareness.” I don’t share your perspective—I think that statistical methods used in published work by tenured or untenured scholars in non-technical fields are still open to critical discussion—but I’m glad that you’re able to share your perspective with us anonymously.

        • So serious criticism of important researchers is “methodological terrorism”, and serious criticism of unimportant researchers is “distasteful lack of empathy”? It’s almost as though researchers are vastly more concerned about keeping the welfare checks (grants) rolling in then they are about Truth.

          Time to face the fact that academia has devolved into welfare for useless “researchers” who will do anything to keep the scam going.

        • W:

          I regret the distraction that my comments caused. The whole thread on PSR apparently started in reaction to my posting the link to the dissertation.

          As for commenting on things one hasn’t read in full, I think it’s legitimate IF the commenter states this limitation (and any others) up front. What’s wrong is to pretend to have read it in full when one has not.

          I should have made clear (as I often do) that I am not a statistician. I take full responsibility for my words but do not claim any authority that I lack.

          I meant absolutely no harm; I was curious about the study, looked it up, found the cluster of eight hypotheses problematic, and said so, making clear that I had only glanced at the work. That’s it. The limitations of my comments were clear (except maybe for the authority part).

          All this said, I don’t think the ad hominem (and vulgar) comments on that PSR thread are productive or deserved. They certainly don’t help the commenters’ arguments.

      • So the summary of the research is that, all other things being equal, people are more likely to purchase things from nicer stores. Who knew?

        I think I’m going to put “See the work of Jan Smedslund on pseudo-empirical theories in psychology” on a hot key. It would save me quite a bit of typing.

  2. Justin, Jack:

    It’s no big deal, there are thousands of these sorts of papers published every year. The point of this post was not this particular article but rather Dan Kahan’s interesting statement criticizing the indirect nature of that sort of research, the idea that instead of just directly studying something, they’ll go around rejecting null hypotheses that they don’t believe. That’s what’s “gwrong.” It might well be that the substantive claims made in this paper are correct, but I’m with Kahan in disliking the reveal-the-sculpture-by-chipping-away-at-the-null-hypothesis reasoning that the researchers use to support their claims.

    I added a P.S. to the post to clarify.

  3. Why would you go out of your way to publicly attack a junior female scholar of color from a department of Apparel, Events and Hospitality Management at Iowa State by attacking a 100 word abstract of a paper you didn’t even bother to read? Are you really that much of a jerk? Open up any issue of PSQ, and I’m sure you’ll find plenty of grist for the mill authored by established scholars in your own field. Seriously: what’s wrong with you people?

    • Hp:

      Thanks for your feedback. In retrospect I think it would’ve been better to have just not posted the abstract at all, or to have just quoted one sentence of it to make the point, as some people seem to have been distracted by details which I had not even noticed, such as the sex and ethnicity of the names of the authors of this article.

      Regarding established scholars in my own field: I posted something last week on the notorious claim by Chris Achen and Larry Bartels that shark attacks determined elections. I think Achen and Bartels were wrong on that one.

      From a statistical standpoint, I think Achen and Bartels’s errors have some commonalities with attitudes shown in the abstract posted above, and I do think there is value in seeing how these null-hypothesis-significance-testing issues show up in different settings.

      Perhaps this will clarify: My goal in the above post was not to “attack” anyone but rather to share a general concern expressed by Kahan, and with which I agree, about this sort of research. When I say that I think an abstract represents a wrong statistical approach, I do not consider that an “attack”; I consider it a scientific disagreement. And I think, more generally, it is perfectly proper to disagree with any published abstract. Abstracts are published so that people can read them, and it’s the nature of science that once you read an abstract, you might find reasons to disagree with it. Again, though, I guess I could’ve made the point more clearly by clipping out the authors’ names as that seems to be a distraction here.

  4. All that you need to know: the last time psychologists (around 50) assembled to craft a working definition of intelligence (@ 1997) the paper they authored came up bewildered and empty.

    So the idea that a spectral shift (measurable and descriptive) can have an influence on an undefined entity (or a psychological rather than natural kind) is a clear signal that one is entering the murky waters of psychological conceptual offerings.

Leave a Reply to Shravan Cancel reply

Your email address will not be published. Required fields are marked *