Sanjay Kaul writes:
I am sure you must be aware of the recent controversy ignited by the 2013 American College of Cardiology/American Heart Association Cholesterol Treatment Guidelines that were released last month. They have been the subject of several newspaper articles and blogs, most of them missing the thrust of the guidelines. There is much to admire about these guidelines as they are more faithfully aligned with high-quality ‘actionable’ evidence than the 3 previous iterations. However, the controversy is focused on the performance of the risk calculator introduced for initiating treatment in individuals without established atherosclerotic disease or diabetes (so-called primary prevention cohort). The guidelines recommend statins for primary prevention in individuals who have a 10-year risk estimated to be 7.5%. The risk calculator was derived from population cohorts studied in the 1990s. The discrimination for predicting the risk of atherosclerotic cardiovascular events defined as coronary heart disease deaths, myocardial infarctions and strokes is fair as measured by the area under the ROC curve (c index of 0.7 to 0.8 across the ethnic spectrum). [As an aside, very few risk models developed to predict risk of clinical events have c index >0.80]. However, when applied to recent population and randomized controlled trial cohorts, the risk calculator is limited by ‘miscalibration’ (overestimates risk by 75% to 150%). This is of course understandable as the prediction cohort is quite different from the development cohort in terms of baseline risk (many subjects in the former were on statin treatment that modifies risk). The single-event probability estimate (which is essentially what calibration characterizes) is referenced to the ‘state’ of the development cohort. If the state is different (as in the prediction cohort), should it surprise anyone that the risk model miscalibrates risk? So, the question is does calibration trump discrimination in risk prediction?
I guess, this harks back to the age-old conundrum about weather forecasting, what is important – that it will rain tomorrow (patient will develop an event as a measure of discrimination) or there is a 30% probability of rain tomorrow (there is a 30% probability of the individual patient developing an event in the future as a function of calibration)?
I have attached the Ridker and Cook paper published in Lancet and the response from the chairs of the guidelines (Stone and Lloyd-Jones) published in Lancet. The original risk tool publication is also attached (JACC paper).
Would appreciate posting this on your blog to get useful insights from you and your well informed readers.
This looks interesting but now I’m feeling too overwhelmed to look at it in detail. (It’s too bad that I read all sorts of crappy papers but then feel too busy to read the interesting stuff. . . .) Maybe some of you will have useful thoughts?