Richard Morey writes:
Rink Hoekstra and I are undertaking some research to explore how people use classical statistical results to evaluate the weight of evidence. Bayesians often critique classical techniques for being difficult to interpret in terms of what scientists want to know, but there is clearly information in the statistics themselves. We wonder how people extract that information. Below is our official announcement; it would be great if you could let people on your blog know about the survey, as we want to get a wide variety of statistical users to take the survey.
Announcement follows:
Empirical science is grounded on the belief that data can be used as evidence. The convincingness of data — the “weight” of the evidence they provide — is crucial to deciding between rival scientific positions. In situations with no uncertainty, reasoning about evidence is often straightforward; in practice, however, most conclusions from data involve uncertainty. In these situations, we obviously prefer strong evidence to weak evidence, but beyond this, strikingly little is known about how scientists actually evaluate the strength of evidence and to what extent scientists differ in their evaluations.
We are looking for researchers with experience using statistics to complete a short survey about the weight of evidence provided by statistics. Participants are asked to assess the weight of evidence in several research scenarios. The survey takes about 15 minutes to complete; if you would like to participate, click or copy/paste the link.
I found the survey quite hard to answer. I’d be glad if someone posted here which are the “correct” answers…
+1
That one seemed like a Mensa test! Whew. Was that meant for career statisticians only?
Hi – I did the first question, but haven’t completed the rest yet. Here’s the reason why:
it says something like this: all females have skulls of size blah and all males have skulls of size bleh, and *there are no exceptions*. That’s problematic. There are always exceptions. There could be a genetic mutation. So should we treat this as a real world problem, or a hypothetical one where whatever was stated in the problem can be assumed to be 100% true, always?
Depending on the answer, my answer would change.
This is certainly interesting but by suggesting implicitly that there is a correct way of doing this you may do more harm that good.
Hi everyone, thanks for helping out with the survey. I’d love to discuss the survey, but we’re still collecting data, so I don’t want to say too much. I would like to point out. though, that this is not meant as a test of quantitative skills – we’re trying to gauge peoples’ impressions, as they might be formed, say, when they’re reading a paper, and what might affect them. There may be no consistency at all, and we’d find that to be interesting as well. If you’re interested in similar research exploring evidential reasoning, you might check out Katya Tentori’s work (for instance, here: http://www.princeton.edu/~osherson/papers/conf33.pdf – which uses a “ball-urn” setup, but with as similar response setup to our survey).
Hillary, I was confused as well about the meaning of “no exception” and exactly. Later on the same question, they say something like: “the exact length is…” What I am supposed to thing about the word ‘exact’? There are no measurement errors? No uncertainty on this measurement?
In any case, I suppose it’s almost always possible to derive a result with the properly set of assumptions. So, in what sense can I say that there is no possible way to asses the evidence? That’s a rather radical statement. I guess most of time I went with the option I’m not sure blablabla