Charlie Saunders writes:
A study has recently been published in the New England Journal of Medicine (NEJM) which uses survival analysis to examine long-acting reversible contraception (e.g. intrauterine devices [IUDs]) vs. short-term commonly prescribed methods of contraception (e.g. oral contraceptive pills) on unintended pregnancies.
The authors use a convenience sample of over 7,000 women. I am not well versed-enough in sampling theory to determine the appropriateness of this but it would seem that the use of a non-probability sampling would be a significant drawback. If you could give me your opinion on this, I would appreciate it.
The NEJM is one of the top medical journals in the country. Could this type of sampling method coupled with this method of analysis be published in a journal like JASA?
My reply: There are two concerns, first that it is a convenience sample and thus not representative of the population, and second that the treatments are chosen rather than randomly assigned, hence there will be pre-treatment differences between the groups. That said, perhaps this descriptive information is valuable. From a statistical perspective, the strengths of the study are realism of the conditions and accuracy of the measurements.
You would have thought they would have at least used propensity scores to deal with some of these issues.
I completely agree with your stated concerns. Additionally, I don’t see how this ever made it into the NEJM since we knew all of this before. I don’t think this provides any new information at all.
I did not read the paper and hope I understood yout question correctly.
If the objective is to measure the prevalence of contraceptive use I agree with your concerns about non-random sampling. However, if the objective is to measure pregnancy outcome given the contraceptive used I do not see a problem with a convenience sample. The problem that the contraceptive use is not randomly assigned is valid. However, an argument has been made that it is unethical to randomised prescribe a contraceptive method to a woman, as would happen when someone is randomised to contraceptive use.
Anne,
I don’t see the problem with randomizing. I worked at a non-profit organization that specialized in contraceptive research for 8 years, and we randomized women to various contraceptive methods all the time. Moreover, they readily agreed to be randomized.
If the conclusions about the efficacy of contraceptive use are meant to apply to future American women, it’s hard to see how they could ever get a random sample of that population given all the immigration/births/deaths in the years ahead. So just like Physicists (who’ve never even come close to taking a random sample of the universe’s electrons) they’re stuck with convenience samples.
Pretty much no medical trials are random samples of the general population since that’s pretty much impossible/unethical. That’s the problem with all clinical trials – they’re all on idiosyncratic selections of people willing to be experimented on, raising the question of external validity. So the question is which is it – getting formal inference right within a idiosyncratic and possibly irrelevant population or getting data on more realistic population but not having a proper sampling frame?
Even if these women were a random sample of some population (i.e. all women in the US now), that just kicks the argument back a step to whether the sampling frame used is really an appropriate population from a scientific or policy perspective (should we really mean future women, or people likely to use the contraception?). Except in very specific circumstance, like political polling, it usually isn’t.
As for the effect of pre-treatment differences, in this case I think we can fairly sure they’ll be swamped by the effects given the type of intervention. It is unlikely there’s something in the biology of treatment vs non treatment women that will dramatically influence the effectiveness of IUDs. The scale of the mechanical intervention is going to dominate the effect of any covariates.
I think this is a case where statistical reasoning is so formal that insisting on it is inappropriate and a distraction. Statisticians like sampling uncertainty, because they know how to deal with it, but in lots of cases is isn’t the most important type of uncertainty researchers have to worry about. The impact of practical questions like accurate measurement, defining effects, and getting data on a wide population are also important. It’s useful to have data aimed at at dealing with these other issues as well, even if it’s not within a sampling framework. Sure, sampling is important, but it’s not the only game in town.
“It is unlikely there’s something in the biology of treatment vs non treatment women that will dramatically influence the effectiveness of IUDs” – perhaps, but the point of the study was to compare IUDs with the pill &c., & they say contraceptive failure is “largely owing to inconsistent or incorrect use”. The pre-treatment differences – propensities to choose among different contraception methods – might surely have a large effect on compliance to the contraceptive regime.