Emanuel Derman and Paul Wilmott wonder how to get their fellow modelers to give up their fantasy of perfection. In a Business Week article they proposed, not entirely in jest, a model makers’ Hippocratic Oath:
- I will remember that I didn’t make the world and that it doesn’t satisfy my equations.
- Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.
- I will never sacrifice reality for elegance without explaining why I have done so. Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.
- I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.
Found via Abductive Intelligence.
Agree – but it would put a damper on most academic careers?
Like Peirce's comment of "good thing we die as otherwise we would come to realize everything we ever thought we understood we were wrong about".
And in the short run we do have to ourselves and others seriously.
Thanks for the post Aleks.
K?
I would have liked to see something like:
Our model only describes the data we used to build it; if you go outside of that range, you do so at your own risk.
Neat idea, but not nearly pithy enough to be adopted.
K?, would love to find that original Peirce quotation. Haven't read him in some time.
I think that the following version lies closer to truth:
"Our model is not falsified by the data we used to test it but it will certainly be falsified by future data."
Point 3 should really be subdivided into 2:
a) I will make sure I understand fully all the assumptions made explicitly or implicitly by the model.
b) I will never give false comfort to people of the model's accuracy. I will always explicitly measure the model's accuracy.
On a), note the first person. It is often the case that the modeler has not pondered fully the implications of various assumptions, which often includes hidden assumptions.
On b), you can only report what you measure.
Pithier reminders that the meal is not the menu and the map is not the terrain often serve well enough.
Dean: Sorry it was from memory
But in Philosophical writings of Peirce By Charles Sanders Peirce, Justus Buchler at the bottom of page 3 in italics is what Pierce took as the greatest compliment he ever recieved "…that I did not seem to be absolutely sure of my own conclusions"
and a bit before "if this practice [affixing a probable error] is not followed … it is because … the probable errors are too vast to be estimated."
I do miss being at a university with searchable writings on Peirce and others.
Also Peirce is now getting more exposure to statisticians – getting as many lines as Fisher here -
http://en.wikipedia.org/wiki/Founders_of_statisti…
K?
Epanechnikov, probabilistic models aren't really "falsified" in contrast to logical ones; it's just that one probabilistic model ends up being beaten by a better probabilistic model.
Nick, I know and love the "map is not the territory", but didn't know of the "meal is not the menu" ("you can't eat a menu?"). Although one could push the limits with edible menus.
Nick: I do remember my finace professor in MBA school realy not getting this – after my comment "that's how the model suggests people evaluate price" I got a "What model" thats just "what happens".
And Peirce guestimated "ninety-nine out of every hundred good heads are reduced to impotence by that malady [blight of cocksureness]" though thats likely a bit too high.
K?
Aleks: for "beaten by a better" perhaps "not lost yet"?
Last summer in a research group on evaluating model fit, I suggested we use terms less wrong, least wrong rather than the better, best, etc. type words. Dont have a count, but I was voted down.
But langauge can be importanat or at lease reveals the tendency to be cocksure.
K?
Keith, interesting point about language – your suggestions had higher cognitive cost than alternatives, if you could find something with the right vibe but cognitive cost lower than better or best, then you'd have a fair shot at infusing more honesty into dialogue.
Aleks Jakulin my view is that a probabilistic model can be falsified. Assume that we have come up with a theory involving probabilities as theoretical terms. These are attached to physically observable well-specified events. So what happens if we realize that the limiting relative frequency of an event does not equal the probability assumed by our model?
"it's just that one probabilistic model ends up being beaten by a better probabilistic model."
Is that not equivalent to saying that from all conceived (but not all possible) probabilistic models the fittest are the ones who survive? This last premise implies that the rest are refuted or falsified by experience.
K? O'Rourke "not lost yet" definitely sounds better to my ears.
Epanechnikov: Your getting close to Peirce's definition of truth – roughly what unending critical inquiry would eventually settle on.
(This truth is real in the sense of it not depending on particular inquirers because any damage they do will eventually be shaken off. And he did point out for it to be unending, it likely would not just involve humans but some other cognizant beings.)
As for sounding better (I might be wrong – actually am sure to be wrong) – the point is to sound less better or more worse. There is a semiological clash likely happening and a semiotic consult might be worth having, but I think its just all tied up with our striving to seem more positive than we should be given the need – in the short run – to take ourselves and others seriously.
K?
Epanechnikov, well, a probabilistic model can be falsified if there an event occurs that had zero probability predicted for it. In other cases, it can always be interpreted as chance. Now, you will never get the limiting relative frequency for a complex system that depends on many uncontrollable factors in a finite amount of time, so it's better to abandon such unrealistic philosophies and embrace the philosophy of vagueness and model comparison.
Who's fantasy? The only fantasy is maintained by quantitative bystanders with zero knowledge of organizational behavior. They have seized the opportunity to present themselves as geniuses, though to date their desperate word searches for 'correlation' etc in their pre-2006 emails have turned up very little.
There is a reason it is called risk management.
Peter
Aleks Jakulin probabilistic statements are not "logically" falsifiable but they are "practically" falsifiable. But of course you would be right to say that we face the "problem of decidability". We need to decide a methodological rule which could help us find the degree of approximation at which the probabilistic hypothesis is held to be refuted. But we certainly know that the rule does not need to be drawn completely arbitrarily. It is possible to draw the dividing line between what is permitted and what is forbidden in such a way so that the outcome of our tests is optimized according to some preset criteria.
Claiming that probabilistic statements cannot be falsified is equivalent to saying that they are metaphysical and without empirical significance (i.e. unscientific). And this is definitely what somebody would call weird …
i would like to suggest that we strike the word "better' from our vernacular, in the absence of any additional clauses. a model may be better with regard to one measure, and will likely be worth with regard to infinitely other measures. unless god tells us what the right measure is for any particular question, better as an objective terms seems somewhat meaningless. while it doesn't roll off the tongue, something like: "our model fits the data better than the other proposed models according to measure X." note that even this claim could include additional caveats, such as, assuming the bugs in our code don't change things too much, etc.