As someone who relies strongly on survey research, it’s good for me to be reminded that some surveys are useful, some are useless, but one thing they almost all have in common is . . . they waste the respondents’ time.
I thought of this after receiving the following email, which I shall reproduce here. My own comments appear after.
Recently, you received an email from a student asking for 10 minutes of your time to discuss your Ph.D. program (the body of the email appears below). We are emailing you today to debrief you on the actual purpose of that email, as it was part of a research study. We sincerely hope our study did not cause you any disruption and we apologize if you were at all inconvenienced. Our hope is that this letter will provide a sufficient explanation of the purpose and design of our study to alleviate any concerns you may have about your involvement. We want to thank you for your time and for reading further if you are interested in understanding why you received this message. We hope you will see the value of the knowledge we anticipate producing with this large academic study.
We are decision-making researchers interested in how choices differ when they are made for “now” versus for “later”. Previous research has shown that people tend to favor doing things they viscerally want to do over what they believe they should do when making decisions for now, while they are more likely to do what they believe they should when making decisions for later (for a review, see Milkman, Rogers and Bazerman, 2008). The email you received from a student asked for a meeting with you either today (if you were randomly assigned to the “now” condition) or in a week (if you were randomly assigned to the “later” condition). This email was actually from a fictional student. It was designed for a study of the responsiveness of University faculty to meeting requests from prospective students of various backgrounds made on short notice versus well in advance. Faculty members at the top 260 U.S. Universities (as ranked by U.S. News and World Report) and affiliated with Ph! .D. programs were identified as potential participants in this study, and a random sample (6,300 faculty in total – one per Ph.D. program) were selected to receive emails. In addition to examining the responsiveness of faculty to meeting requests for “now” versus “later”, we are also interested in how the identity of the applicant affects, or does not affect, response rates, and as such, the name of the student sending a meeting request was varied (by race and by gender). We expected that students from underrepresented groups would receive fewer meeting acceptances than other students, though we have competing hypotheses about whether this would effect would be stronger in the “now” or the “later” condition.
The email you received as a part of this study contained the following message:
“I am writing you because I am a prospective Ph.D. student with considerable interest in your research. My plan is to apply to Ph.D. programs this coming fall, and I am eager to learn as much as I can about research opportunities in the meantime.
I will be on campus today/(next Monday), and although I know it is short notice, I was wondering if you might have 10 minutes when you would be willing to meet with me to briefly talk about your work and any possible opportunities for me to get involved in your research. Any time that would be convenient for you would be fine with me, as meeting with you is my first priority during this campus visit.
Thank you in advance for your consideration.”
As soon as the results of our research are available, we will post them on our websites. Please rest assured that no identifiable data will ever be reported from this study, and our between subject design ensures that we will only be able to identify email responsiveness patterns in aggregate – not at the individual level. No individual or university will be identifiable in any of the research or data we publish. Of course, any one individual email response is not meaningful as there are multiple reasons why an individual faculty member might accept or decline a meeting request. All data has already been de-identified and the identifiable email responses have already been deleted from our databases and related server. In addition, during the time when the data was identifiable, it was protected with strong and secure passwords. And as is always the case when academics conduct research involving human subjects, our research protocols were approved by our universities’ ! Institutional Review Boards (the Columbia University Morningside IRB and the University of Pennsylvania IRB).
If you have any questions about your rights as a research subject, you may contact the Columbia University Morningside Institutional Review Board at 212-851-7040 or by email at firstname.lastname@example.org and/or the University of Pennsylvania Institutional Review Board at 215-898-2614.
Thank you again for your time and understanding of the work we are doing.
Katherine L. Milkman
Assistant Professor of Operations and Information Management
The Wharton School, University of Pennsylvania
Assistant Professor of Management
Columbia Business School
Dear Drs. Milkman and Akinola:
I believe that it would be appropriate and ethical for you to compensate all of the participants in your study. My understanding is that standard ethical guidelines require that research subjects (which, in this case, include me) be compensated for their time. I am surprised that your study passed the IRB without this–especially in a situation such as this in which we were not asked ahead of time whether we wanted to participate.
According to your email, 6300 of us have participated in your study. Perhaps you could pay $10 to each of us for our involuntary participation. This would come to $63,000 (plus the costs of checks, letters, and stamps)–surely a small price to pay for this valuable increase in scientific knowledge?
P.S. Some might say that it is mean of me to send such a sarcastic email to two evidently serious researchers. If I had been asked to participate in the study, I would respond more discreetly, but the unsolicited nature of the project seemed to demand an equivalent response. I am indeed sensitive to the ethical difficulties of survey research, but this does not stop me from feeling that my helpful impulses toward inquiring students are being abused by this sort of study, which I think belongs in the trash heap of ill-advised research projects along with Frank Flynn’s notorious survey from a few years ago when he tried to get free meals out of NYC restaurants by falsely claiming food poisoning. What is it with Columbia Business School researchers?