Skip to content

The limits of open-mindedness in evaluating scientific research

Seth is skeptical of skepticism in evaluating scientific research. He starts by pointing out that it can be foolish to ignore data, just because they don’t come from a randomized experiment. The “gold standard” of double-blind experimentation has become an official currency, and Seth is arguing for some bimetallism. To continue with this ridiculous analogy, a little bit of inflation is a good thing: some liquidity in scientific research is needed in order to keep the entire enterprise moving smoothly.

As Gresham has taught us, if observational studies are outlawed, then only outlaws will do observational studies.

I think Seth goes too far, though, and that brings up an interesting question.

In the discussion on his blog, Seth appears to hold the position that all published research has value. (At least, I brought up a notorious example of error-ridden research, and Seth responded that “I don’t agree that this means its info is useless.”) But if all published research, even that with crippling errors, is useful, then presumably this is true of some large fraction of unpublished research, right?

At this point, even setting aside monkeys-at-a-typewriter arguments, there’s the question of what we’re supposed to do with the mountain of research: millions of published articles each year, plus who knows how many undergraduate term papers, high school science fair projects, etc. I think there are some process-type solutions out there, things like Wikipedia and Slashdot or whatever (which have their own biases, but let a zillion flowers bloom, etc.). But that seems like a cop-out to me, since ultimately someone has to read the papers and judge whether it’s worth trying to replicate studies, and so forth. Somewhere it’s gotta be relevant that a paper has mistakes, right?


  1. greg says:

    I think Seth's point is right except that:

    (1) It matters very much if the mistakes in papers are correlated. For example, if a discipline as a whole condones sloppy research, then it may continue to publish papers that are more mistake than reality.

    (2) Mistake-ridden research can get picked up by the press (especially if it's sensational) and can cause people to believe wrong things.

    So yeah you can recover information from mistake-ridden papers but it's incredibly inefficient and potentially harmful to not punish mistakes.

  2. John says:

    "Somewhere it's gotta be relevant that a paper has mistakes, right?"


    (Yarrg! I don't know if the linking works, so here's the reference:… The preview function doesn't do a very good job.)

  3. Phil says:

    There are several different ways that a research paper can fail to meet current standards for peer-reviewed journals. To list a few: it can be judged to be too long; to provide insufficient discussion of methods; to contain no new results; to contain no interesting results; to fail to cite previous research in the field; to include too much speculation and not enough data; to present results in a way that is not detailed enough or that lacks important information; to contain data or analysis that is simply wrong; and more.

    Some of these are much worse than others. For example, "too long" is unfortunate but shouldn't disqualify a paper, at least from online publication. But "data or analysis that is simply wrong" is (and should be) grounds for rejecting a paper!

    I do think professional standards have been set too high in some of the areas that I mention above. For instance, I wonder how many null results haven't been published, just because researchers feel (probably correctly) that reviewers will say "there's nothing interesting in this paper." And it's even harder to find a place for work that duplicates someone else's results — it's hard to even get this funded, much less published.

    So if Seth's argument is that reviewers and editors are too hard on some flaws and some kinds of research, I agree with him. But on the much broader claim (if he is indeed making it) that no research is so bad as to be useless, I completely disagree with that.

  4. Ted Dunning says:

    There has to be a counter-argument based on the amount of effort required to get gold out of dreck.

    Sure, most observations are telling us something about the real world, but the effort required to understand what the limitations and biases of the data, and to estimate the likelihood of outright falsification often (usually) far outweigh the benefit that you might gain from the data. At the extreme, you are left with such doubts about the results that you have no effect on your posterior (except for a distinct pain in the rear).

    Looking for well done, replicated results is an excellent way to quickly estimate how much value the data will have to you as well as how much effort it will take to clean it up to a usable state. Reputation, in turn, is a pretty good predictor.

    In sum, I think that Seth is correct that pretty much all data is interesting at some level, but wrong in that the *net* benefit of the data may be strongly negative in terms of the marginal cost of using the data. After all, I could spend my time looking at other data that is likely to benefit me more in the same time as it would take to some understand poorly collected dataset.

  5. current grad student says:

    A couple more interesting quotes from Seth Roberts' blog:

    "There’s no control group, no randomization (apparently), yet the results are very convincing"

    WOW! How could someone care so little about experimental design.

    He further goes on to say in the comments,
    "If your goal is maximizing patient benefit, you do Test X; if it is “not fooling yourself” you do Test Y."

    The problem is in fooling yourself into believing that it is your treatment that is maximizing patient benefit.

    There are some very sound rational principles behind experimental design, and I think it is important to remember how often experiments contradict observational studies. However, this doesn't mean all observational studies are useless. Instead, we just have to be a LOT more careful about conclusions drawn from observational studies.

  6. Daniel Lakeland says:

    I really like things like the which let people put electronic versions of pretty much anything online. let the interesting and supportable stuff percolate to the top…

  7. Prison Rodeo says:

    My own personal favorite along these lines has long been this:

  8. Kevembuangga says:

    It seems to me that this whole discussion is predicated on the idea that there is "one right way" to do things (which is usually called "scientific" or "rational") and that what's to be debated is the appropriate means to reach this goal of blissfull true knowledge (mystical overtone intended!).I beg to differ.The great payoff of rationality and most especially of mathematics is, that once you have cornered your concepts and data into well defined "rigorous" definitions the succeeding computations, deductions and whatever horrendously complex logical tinkerings you have to go thru to reach your results will not introduce spurious distortions in the meanings and reliability of the outcome.So far, so good, except, except…That in most real life cases you have to first turn your concepts and data into Spherical Cows!And this is where the glorious "Scientific Method" and her many rationality oriented cousins breaks down in laughters and unfortunately, in some cases in disasters.
    Before you can rely on "observations expressly collected for the purpose of a study" you have to make sure that you are looking for the right criteria from the right samples with respect to your problem.
    Alas, assessement of this adequacy of the concepts and samples to be used is not entirely within the realm of pure rationality, by definition, because that would imply an endless regression of rationality judgments.
    One cannot prove the consistency of any theory from within the theory itself, a point the "rationality junkies" seem to conveniently forget…
    I cannot speak on behalf of Seth but maybe his point is close to the evidence that in a huge number of practical cases we just cannot afford to be rational because we have neither the concepts (the model appropriate to the question at hand) nor the data samples to make up a (statistically) reliable estimate of the best course of action.
    Yet we "solve" hundreds (literally!) of ill defined problems everyday, are we doing better than chance (with respect to our goals and interests), I think so.
    Looking more seriously into intuition would probably yield a better return than obssessive fights for rationality, which just smell like retarded 19th century anticlerical propaganda.
    (Ah! Well… I know! I am European, maybe the USA need an exception with this)

    Ted Dunning: Sure, most observations are telling us something about the real world, but the effort required to understand what the limitations and biases of the data, and to estimate the likelihood of outright falsification often (usually) far outweigh the benefit that you might gain from the data.

    Every observation is telling us something about the real world, the real question is how far do we go with the processing of this observation?
    If we restrict ourselves to the only cases where we have a consistent model onto which to match the observation and a perfect knowledge of the context biases for this observation we are loosing a lot, Spherical Cows are only good up to a point.

    I also want to emphasize that EVERYBODYs, EVERYBODYs personal experience is nothing but a bunch of anecdotes.
    Aren't you ashamed?
    Time to reassess ALL your ideas with "serious" placebo-controlled double-blind studies!

  9. current grad student says:


    Even if we are measuring the "wrong" quantities, as you seem to suggest, wouldn't you rather know that you are at least measuring this "wrong" quantity, rather than also measuring the effects of confounders and biases, both of which are common in observational studies. In such cases, it is hard to tell what you are really measuring, and the knowledge gained can be minimal. Every observation tells us "something" about the real world, the only problem is that sometimes we don't really know what that something is.

    In regards to Ted Dunning's quote, are you seriously advocating not attempting to understand the data collected?

  10. Kevembuangga says:

    the only problem is that sometimes we don't really know what that something isWe do know something about anything as soon as we can refer to that "anything" in discourse no matter how confused our ideas are (including people with schizophrenia, etc…).What I am saying is that we actually make some use of this fuzzy information no matter how garbled it is, since out of this we make associations which are not pure chance.The output may be noisy but the noise is "tainted" by whatever we confusely know about the inputs.I know of no formalisation of this feature of human speech and I even deem improbable that it could be fully formalized, YET I assert that this allows us to make decisions which are more beneficial than random choices (or more detrimental!) and that this is what is usually called intuition when it appears beneficial.are you seriously advocating not attempting to understand the data collected? Not at all!On the contrary the more structured the model and data can be the better.My true critiscism is twofold:- We sometimes drop too much valuable information (Spherical Cows) in our efforts to come up with a clean and consistent model.- Assessement of the adequacy of this model building phase cannot be made entirely in the realm of "well received" rationality, and this is a formal impossibility.

  11. I'm not sure the quality now is any worse than it ever was, there's just more quantity. It's a good thing we have search engines.

  12. Luciano says:

    Every written paper has its value.
    The problem is that sometimes you need more effort to get that value out than the value of the paper itself. And papers are a renewal commodity.