Comment of the year

In our discussion of research on the possible health benefits of a low-oxygen environment, Raghu wrote:

This whole idea (low oxygen -> lower cancer risk) seems like a very straightforward thing to test in animals, which one can move to high and low oxygen environments . . .

And then Llewelyn came in for the kill:

Why do the animals always get first pick at new treatments? Seems unfair.

12 thoughts on “Comment of the year

  1. I have a recurring discomfort with the notion that it is better to experiment on animals rather than ourselves, particularly when s much of this is poorly designed, underpowered, and usually badly analyzed and interpreted.

    A few years ago I volunteered for a clinical trial in order to get a better sense of what patients experience, and how this translates to the data I see. Underpowered, null result, but it provided real insight into the nature and quality of the data I’ve been asked to analyze, and the questions I need to ask. Investigators often don’t tell us the full story about the data they bring us.

    • Kudos to you for your personal (underground?)investigation into clinical trials. Please tell us more (e.g., the questions that need to be asked) if you are willing.

      • Martha, sorry about the late response, I’ve been on vacation.

        The trial involved a study of metabolism, and I’d previously collaborated as statistical analyst with several of the investigators. The study involved two overnight stays several months apart, each spending a day cold but not to the point of shivering while they took many blood & breath samples, measured metabolism from several approaches, as well as a couple of biopsies, DEXA, PET & CT scans. Some issues…

        Needles & pain: We’re all familiar with the occasional mild sting or burn associated with a blood draw or injection. On my first morning, they needed to start 3 IV lines, but the nurse had trouble finding the veins. Altogether it took about a dozen sticks to establish these 3 lines. I found that, perhaps compounded with being cold, each needle stick hurt more than the last, so this became progressively more unpleasant. I’ve learned that patients in our routine clinical trials likewise become very needle shy, and this probably drives a lot of failure to follow-up — and the missing data undoubtedly skews the results, in spite of my efforts to compensate. Patients who do return for follow-up tend to be those with more serious conditions, and probably follow-up for the additional care rather than in service of the clinical trial. Incidentally, nurses have the option of using a numbing gel prior to needle sticks, but it is almost never used; I’ve brought this issue up at several research meetings, and the investigators agree it needs to be addressed, but nothing seems to change.

        Biopsies & pain: There was little pain control with deep biopsies at my first visit — quite uncomfortable. I complained — a lot — and communicated frankly with the MD doing the biopsy on the next visit, and he did a much better job of numbing the biopsy site the next time. As with needles, I’ve learned that patients in our routines have similar issues, and this contributes to lack of follow-up or withdrawal from trials or specific procedures. Patients often don’t realize that they can communicate about this sort of discomfort, and that something can be done about it, though there is also a problem where physicians ignore them (I think that physicians often are more focused on fixing a problem than making the patient feel better).

        Data quality: Some of the measurements taken were clearly very subjective. Oxygen metabolism was measured by draping a tent over me while a lay in a bed, and measuring some combination of changes in oxygen and CO2 for a fixed period of time — it seemed to me that the amount of air trapped with me under the tent as well as imperfections in the seal added a lot of noise to the measurement, plus I was supposed to be relaxed but not sleeping or talking, which is a somewhat arbitrary state of relaxed and ignores intrinsic stress responses. Isotope ratios were measured at a minimum number of time points, which means that modeling corresponding rates of change are very subject to errors in measurement or lost samples (both common). Also, patients are quite variable in percentages of muscle/fat mass, which are imperfectly measured from DEXA and other imaging or BMI, and these measures contribute to metabolic modeling. Biopsies don’t always sample the specific tissue they’re targeted at. I was subjectively cold and stressed on my first visit, not so much on my second, which undoubtedly skewed the metabolism.

        Overall, there is a lot of implicit variability due to patient characteristics and circumstances. The investigators try to control for these, but there’s only so much they can do. I’ve learned to be very skeptical of data quality from clinical studies — often it is noisy and biased. Missing data and lack of follow-up introduce additional biases. Investigators tend to hand us data without conveying a lot of these issues, and we analyze it naievely, with results published accordingly. I’ve found it is quite helpful to look for impossible outliers and to graph raw data (by patient) over time to get a sense of variability, bad measurements, underlying trends, and missingness. Problems with reproducibility are to be expected. We need to learn the what/why/how of the measurements. When an investigator brings us data, it is not data from some ideal simulation which we can simply model automatically. There is usually a story behind the data, patients or data points may have been excluded, some data may be invalid/impossible, and biases need to be ascertained. In this particular trial, the investigators were conscientious about doing the right thing, but be careful of investigators who don’t take blinding seriously, or shift patients from one treatment to another, or exclude data or patients they don’t like. Clinical trials are messy.

  2. Years ago, a physician remarked,

    “Thousands of Americans are alive today because they had the good fortune to be in the control arm of a randomized clinical trial.”

Leave a Reply to Alex Chernavsky Cancel reply

Your email address will not be published. Required fields are marked *