Saw a video link talk at a local hospital based research institute last Friday
Usual stuff about a randomized trail not being properly designed nor analyzed – as if we have not heard about that before
But this time is was tens of millions of dollars and a health concern that likely directly affects over 10% of the readers of this blog – the males over 40 or 50 and those that might care about them
Its was a very large PSA screening study and and
the design and analysis apparently failed to consider the _usual_ and expected lag in a screening effect here (perhaps worth counting the number of statisticians in the supplementary material given)
for an concrete example from colon cancer see here
And apparently a proper reanalysis was initially hampered by the well known – “we would like to give you the data but you know” …. but eventually a reanalysis was able to recover enough of the data from the from published documents
but even with the proper analysis – the public health issue – does PSA screening do more good than harm ( half of US currently males get PSA screening at some time? ) will likely remain largely uncertain or at least more uncertain than it needed to be
and it will happen again and again (seriously wasteful and harmful design and analysis)
and there will be a lot more needless deaths from either “screening being adopted” if it truly shouldn’t have been or “screening was not more fully adopted, earlier” when it truly should have been (there can be very nasty downsides from ineffective screening programs, including increased mortality)
OK I remember being involved in this PSA screening stuff many years ago in Toronto and think we argued that given the size of the study required – scarce research dollars would likely have a much better return studying some other health concerns (most of us were males but young)
But the PSA screening studies were funded – but apparently defectively designed and analyzed.
Now I had been involved in the design of a liver screening trial around 1990 (not funded because it was percieved as being too expensive) and the lag of an effect did not actually occur to me
until I started to write up the power simulation studies (discouraged by my advisor who told me professional statisticians should not have to stoop to simulations to calculate power)
and then I had to think up a treatment effect.
The immediate effect would likely not appear right away – the early treatable tumours would not have a mortality outcome for a while – after all they were early.
But maybe as important – I did some literature searching (breaking Ripley’s rule that statisticians don’t read the literature) and there were papers discussing the lag effect in treatment effects in screening trials and they suggested ways to design and analyze given these.
Then to hear of a huge disaster happening much later – in the 2000,s – why does it happen?
Statisticians have to think through the biological details of studies – somehow
Simulating planned trials is very important – even if you can get away with (i.e. fool reviewers) highly non-robust over simplified closed form professional looking power formulas
Statisticians have to confer widely – especially when designing a large expensive trial
Anyone can be “blind sided”
Do literature searches specifically on the study design and clinical topic
Read some of that literature
Try to contact other statisticians who have worked with such designs and that clinical topic
Try to have some of them look at your design
Simulate the details – that’s were the devils are
And if you notice someone else has blown it and you could fix it if you just could get their data…
Well you should be able to get their data – but there are good and not so good reasons why that won’t be feasible – but sometimes you can get more from the published data that you might think
First and technically challenging – there is always the marginal likelihood – the probablity of the published (rather than actual|) observations gives the appropriate likelihood (some math details here “justimportance.pdf” )
But sometimes you can get everything:
under Normal assumptions just the means and variances are sufficient (that just means the marginal likelihood exactly equals the full data likelihood)
in correspondence analysis there is something called the Burt matrix which is a summary from which you can (with some algebra) redo the full correspondence analysis as if you had the actual data
and for survival data the Kaplan-Meier curve – with enough resolution – will allow you to read off the raw data (event and censoring times). Modern pdf’s can provide full resolution?
Perhaps most importantly to avoid statistical bridges falling down:
We should try to worry more about public health rather our public (professional) images or even our publications!