Dan Goldstein made this amusing graph:

in discussing this paper.

Posted by Andrew on 13 December 2006, 12:00 am

Dan Goldstein made this amusing graph:

in discussing this paper.

## Recent Comments

- Andrew on Creativity is the ability to see relationships where none exist
- Andrew on Creativity is the ability to see relationships where none exist
- Kyle C on Creativity is the ability to see relationships where none exist
- Rahul on Creativity is the ability to see relationships where none exist
- Oncodoc on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?
- Rahul on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?
- MatthewKS on Creativity is the ability to see relationships where none exist
- Bruce McCullough on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?
- Keith O'Rourke on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?
- Rahul on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?
- Type-S and Type-M errors | The Etz-Files on This is what “power = .06” looks like. Get used to it.
- Kaiser on What’s the most important thing in statistics that’s not in the textbooks?
- Clyde Schechter on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?
- John Mashey on Weggy update: it just gets sadder and sadder
- Andrew on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?
- Rahul on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?
- Keith O'Rourke on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?
- Anonymous on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?
- Andrew on Can talk therapy halve the rate of cancer recurrence? How to think about the statistical significance of this finding? Is it just another example of the garden of forking paths?

## Categories

Taking the difference between the significance of effects as evidence of a difference of effects is a very common mistake of clinical research trainees – that hopefully is corrected early in their careers.

The comment in the paper that "we are pretty sure the article would not have been published had the results not been statistically significant" identifies an arguably larger inference problem with little hope of practical resolution. (Some have gone as far as to suggest that if such is the case, such studies should neither be funded nor published)

Keith

Bob Rosenthal and colleague made exactly this point empirically about 40 years ago, showing that social scientists (and now lawyers, judges, legislatures, etc.) simply don't credit a finding unless there's a ".05" lying around.

-Robert Rosenthal & J. Gaito, The Interpretation of Levels of Significance by Psychologists, 55 J. PSYCHOL. 33 (1963); Robert Rosenthal & J. Gaito, Further Evidence for the Cliff Effect in the Interpretation of Levels of Significance, 15 PSYCHOL. REP. 570 (1964)

Depressing.

Thanks for the reference Jeremy.

I forgot to attach this one with my post – Sterling TD (1959). Publication decisions and their possible effects on inferences drawn from tests of significance — or vice versa. Journal of the American Statistical Association 54:30-34.

(Topic was suggest to Sterling by RA Fisher)

Keith

In grad school I thought I'd heard there was a Jacob Cohen quote to the effect of "Surely god loves p=.06 as much as god loves p=.05" – I've done some searching and found nothing…

Not that I believed the person telling me, really, well…maybe…

The quote is from Rosnow & Rosenthal (1989, p1277):

"We are not interested in the logic itself,

nor will we argue for replacing the .05 alpha with another level of alpha, but at this point in our discussion we only wish to emphasize that dichotomous significance testing has no ontological basis. That is, we want to underscore

that, surely, God loves the .06 nearly as much as the .05. Can there be any doubt that God views the strength of evidence for or against the null as a fairly continuous function of the magnitude of p?"

Rosnow, R.L. & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44,

1276-1284.

Thom