Inti Pedroso writes:
Today during the group meeting at my new job we were revising a paper whose main conclusions were sustained by an ANOVA.
One of the first observations is that the experiment had a small sample size. Interestingly (may not so), some of the reported effects (most of them interactions) were quite large. One of the experience group members said that “there is a common wisdom that one should not believe effects from small sample sizes but [he thinks] if they [the effects] are large enough to be picked on a small study they must be real large effects”. I argued that if the sample size is small one could incur on a M-type error in which the magnitude of the effect is being over-estimated and that if larger samples are evaluated the magnitude may become smaller and also the confidence intervals. The concept of M-type error is completely new to all other members of the group (on which I am in my second week) and I was given the job of finding a suitable ref to explain it. They acknowledge that the CI would decrease with large sample but not necessarily that the mean effect itself could be wrongly estimated. The group is formed by biologist some of which have good stat knowledge but there are several undegrads which are just starting. I was wondering if you know of an article describing M-type errors and how large they can be on small sample sizes?
I think there’s room for more research on the topic.