When we suggest a new method, we are duty-bound to not just demonstrate that it works better than existing approaches (or is superior in some other way such as simplicity or cost). We also need to explain why, if this new method is so great, people aren’t already using it.
Various answers are possible, for example:
- The new idea is technically advanced, requiring a level of mathematical or engineering complexity such that it could not easily have been discovered by accident. Hence its novelty can be explained as a product of some particular historical process.
- The new idea is clever and unexpected, as with the mechanical device underlying Rubik’s Cube.
- The new idea could only exist given recent technological developments (perhaps hardware developments such as a new composite material or an ultralight battery, or software developments such as a new MCMC algorithm).
- The new idea usually isn’t so impressive but it shows its virtues in some previously hidden domain (for example, you wouldn’t have much need for relativity theory if you’re modeling sub-relativistic velocities).
- The new idea violates some principle or taboo (for example, until recently there wasn’t much formal research on weakly informative prior distributions because the literature on Bayesian statistics was divided between quests for noninformativity and assertions of complete subjective informativity).
- The new idea was actually out there all along but people didn’t use it because until recently they had no need for its benefits (for example, a method that offers a 10% improvement in speed but at the cost of requiring an elaborate computational implementation, this might not be worth it for a desktop regression but can come in handy if you’re analyzing billions of data points).
You can probably come up with some more. My point is that when we promote a new idea, we must—explicitly or implicitly—explain why our brilliant predecessors did not already discover and use it.
I thought about the above after reading this from John Cook:
According a recent biography of Henri Poincaré,
Poincaré … worked regularly from 10 to 12 in the morning and from 5 till 7 in the late afternoon. He found that working longer seldom achieved anything …
Poincaré made tremendous contributions to math and physics. His two-hour work sessions must have been sprints, working with an intensity that could not be sustained much longer.
I [Cook] expect most of us would accomplish more if we worked harder when we worked, rested more, and cut out half-work.
I agree, but . . . I’ve thought this for a long time, as I’m sure has almost anybody who works at a flexible job. It’s long been my goal to work intensely for whatever number of hours a week is possible and then relax the rest of the time, rather than spending hours and hours each week rearranging my files, responding to email, etc.
Yet this doesn’t always happen (hence it’s long been my goal etc.). I occasionally make some progress (for example, my strategy of reviewing journal articles immediately, just reading the article and writing the report in 15 minutes, or my strategy of not reading email before 4pm), but I still spend lots of time doing essentially nothing yet still hanging out at work. (Not to mention blogging, but that at least serves some socially useful purposes.)
Which brings us to the question: if this is such good advice, and such obviously good advice, why aren’t we doing it already? I think one reason is that it’s hard to work intensely, and once you’re at work it’s easier to spend your time quasi-goofing-off. I was once at a workshop where the person next to me was checking email on the laptop, literally more than once per minute. Seems pretty boring, but it beats working!
I’m reminded of the advice in the classic book, How to Talk So Your Kids Will Listen and Listen So Your Kids Will Talk. It’s all such clearly good advice, but somehow so difficult to carry out in real time.
P.S. One of the comments on Cook’s post led me to this website by computer scientist Cal Newport which is full of advice for students, along the lines of: “‘follow your passion’ is bad advice if your goal is to end up loving what you do.” I like what Newport has to say. Unlike the usual in-your-face internet self-help gurus, he doesn’t seem to feel the need to be obnoxious or to supply having-it-all parables. He does do a little of that B.S.—for example, a post entitled, “How to Get Into Stanford with B’s on Your Transcript”—but, even there, the substance of the post is interesting. Gladwellian, one might say, and I mean that in the best possible sense of the word.