Let’s accept the idea that treatment effects vary—not as something special but just as a matter of course

Tyler Cowen writes:

Does knowing the price lower your enjoyment of goods and services?

I [Cowen] don’t quite agree with this as stated, as the experience of enjoying a bargain can make it more pleasurable, or at least I have seen this for many people. Some in fact enjoy the bargain only, not the actual good or service. Nonetheless here is the abstract [of a recent article by Kelly Haws, Brent McFerran, and Joseph Redden]:

Prices are typically critical to consumption decisions, but can the presence of price impact enjoyment over the course of an experience? We examine the effect of price on consumers’ satisfaction over the course of consumption. We find that, compared to when no pricing information is available, the presence of prices accelerates satiation (i.e., enjoyment declines faster). . . .

I have no special thoughts on pricing and enjoyment, nor am I criticizing the paper by Haws et al. which I have not had the opportunity to read (see P.S. below).

The thing I did want to talk about was Cowen’s implicit assumption in his header that a treatment has a fixed effect. It’s clear that Cowen doesn’t believe this—the very first sentence of this post recognizes variation—so it’s not that he’s making this conceptual error. Rather, my problem here is that the whole discussion, by default, is taking place on the turf of constant effects.

The idea, I think, is that you first establish the effect and then you look for interactions. But if interactions are the entire story—as seems plausible here—that main-effect-first approach will be a disaster. Just as it was with power pose, priming, etc.

Framing questions in terms of “the effect” can be a hard habit to break.

P.S.

I was thinking of just paying the $35.95 but, just from the very fact of knowing the price, my satiation increased and my enjoyment declined and I couldn’t bring myself to do it. In future, perhaps Elsevier can learn from its own research and try some hidden pricing: Click on this article and we’ll remove a random amount of money from your bank account! That sort of thing.

28 thoughts on “Let’s accept the idea that treatment effects vary—not as something special but just as a matter of course

  1. Though researchers know that we’re usually talking about average treatment effects, maybe it would be useful to non-researcher consumers to use the word “average” in presenting such results. E.g., “the presence of prices accelerates satiation *on average*”.

    Another observation is that, when summarizing a quantitative variable, we usually present mean (SD), and part of the reason for presenting SD rather than SE is that we’re not just interested in how precisely we’ve estimated the population mean, but also in how informative the population mean is about the population. A large SD means that the mean is not a very good guess for the value of a randomly selected instance. Similarly when we present average treatment effects, it seems we should not just present the precision with which we’ve estimated it (SE or CI), but also some measure of how good of a guess it is of the treatment effect of a randomly selected instance.

    • “it seems we should not just present the precision with which we’ve estimated it (SE or CI), but also some measure of how good of a guess it is of the treatment effect of a randomly selected instance.”

      I agree this would be great, but I’m not aware of a simple way to do this. Do you know of one?

  2. I’m always mystified by the decision of publishers to charge $35 for an article. I would probably pay around $5, and I suspect the number of people willing to pay 5 versus 35 has to be greater than 7 to 1. Does anyone understand what the rationale is? Greed would imply that make more money that way, but I find it hard to believe people would pay that much for an srticle they could easily get for free via legally dubious means.

    • I believe they are worried about cannibalizing subscriptions to the journal. If articles you are interested in are reasonably cheap, then people might never subscribe to the journal in the first place. I’m not sure they are wrong about this.

        • Journals make essentially all their money off University subscriptions. If the university sees that it’s cheaper to buy individual articles they will end their subscription and just provide a purchase card to their department members that draws on a library pool of funds or something. Imagine in a given issue the average number of articles read at a university is 5 per month. Then it needs to be more expensive to buy 5 articles per month for 12 months than to subscribe to the whole journal for a year… If the journal charges $35 per article, x5x12 = $2100/yr which sounds like the kind of money a typical journal probably does charge a university… so there you have it ;-)

        • Imagine in a given issue the average number of articles read at a university is 5 per month.

          I must read journal articles very differently than most… I find it is rare to actually study an article, instead I will skim through many, many of them looking for some specific information, then move on. There is no way this is worth $5 a read or even $0.05 since most are nothing but teases that are closed soon after being opened. It could easily be hundreds of articles in an evening.

    • @Jonathan: My impression is that the number of people who use quantile regressions is very small indeed.

      @Ram: A histogram is a much better way to summarize a quantitative variable than just giving mean and standard deviation — that way, the reader can see the shape of the distribution. (Who cares about SD if the distribution is really skewed, or strongly bimodal, or …?)

        • Z:”Ram is talking about a causal effect.”

          Ram wrote, “when summarizing a quantitative variable, we usually present mean (SD), and part of the reason for presenting SD rather than SE is that we’re not just interested in how precisely we’ve estimated the population mean, but also in how informative the population mean is about the population. A large SD means that the mean is not a very good guess for the value of a randomly selected instance.” That’s what I was responding to.

    • Quantile regression gets at one type of heterogeneity in treatment effect, but not always an interesting one. I mean, we can learn “where in the outcome distribution” the effects are largest or smallest, but not necessarily anything about “who” the people are that are, other than they are people around the tau’th quantile of the outcome distribution. And once you start adding additional covaraites beyond the “treatment group” indicator, interpreting quantile regression coefficients becomes incredibly difficult.

      Take, for example, an experiment on educational outcomes measured by test scores. And suppose the “treatment” under consideration is a new pedagogical method, and this method (for purely illustrative reasons) strongly improves girls’ test scores but not boys. Any correlation between gender and test score external to the intervention (say, because parents spend more time at home teaching boys) will mean that the quantile estimates are higher in some parts of the outcome distribution than other parts (based on the distribution of girls across test scores). But that isn’t telling us anything about the heterogeneity of the impact itself, which you could estimate only by estimating separate (probably mean) effects for boys and girls.

      Now of course, if you have a heterogeneous treatment effect that affects people differently based only on where they are in the outcome distribution (say, an intervention that targets low-scoring students on a well-measured and similar pre-test), then quantile effects get you exactly the heterogeneity you are looking for. But it isn’t obvious to me that we are usually interested in heterogeneity across Y. More often I think we are interested in heterogeneity in treatment effects across X’s, that is, observable characteristics of the study participants.

      • In working on my current project one intermediate thing I need is to come up with some Bayesian estimate of a local regional poverty measure. I’ve been thinking about how to tease out what is the “practical safe minimum” cost of housing + utils sufficient for a family of a given size. I started thinking about how the difference between what you actually spend on housing and the practical minimum should be bigger if your income is bigger, but not necessarily that much, it depends on whether you like nicer housing more than you like say eating fancy restaurant meals or saving for your kids private school tuition or whatever… Still, on net, in a given city, people who make more should probably spend at least as much or more on housing than people who make somewhat less, at least when they’re spending their own money (not a housing voucher or whatever).

        And so, thinking about the quantity (ActualExpenditure – MinimumSafeExpenditure) you expect this to be right tailed, but of course there will be some people who really scrimp or get a good deal on rent because their uncle owns the apartment complex, or whatever, so you have some people to the left of zero… The end result is going to look like a quantile regression even if it’s not actually formally equivalent (Without having thought too carefully, I believe a quantile regression would be equivalent to two exponential distributions back to back with different averages on each side).

        Anyway, it seems to me that quantile regression is inherently about finding out where in the outcome space some “critical” point lies, and it’s probably best done when you first estimate which quantile you’re interested in by looking at how some outcome changes with respect to your quantile point…. and then you want to find that same quantile point under alternative conditions.

        So for example, suppose you see two interesting populations in some histogram, with a lump around 1 and another lump around 5, and few people in the region say 3… You estimate the quantile point of 3 and it’s say 10%, so now you go and find out what are the 10% quantile points of this outcome across different regions of space or something… this can help you find a way to partition the distribution unto two groups where two different causal factors are involved.

        Such a thing is much less likely to work when there is no principled reason to choose a particular quantile.

        • Daniel,

          Check out the (long) research literature regarding Engel Curve: https://en.wikipedia.org/wiki/Engel_curve

          “The best-known single result from the article is Engel’s law which states that the poorer a family is, the larger the budget share it spends on nourishment.”

          There should also be some similar types of estimates for housing, though it may end up being that you do just as well using food expenditures as a fraction of household budget (hey, dimensionless measure!) to get at poverty or general cutoffs relating to basic needs.

          Re: using quantiles to find cutoffs in outcome space – I think that is very context specific but potentially a good use. My concern is about the nature of the comparison being made. For instance, the tau’th percentile of the income distribution is clearly different across regions (in its level), but things like the federal poverty line are not – but the federal poverty line may be very important for getting access to social programs including rent subsidies. So if you have an actual, dollar-valued thing that could be coming into play, that will hit different quantiles in different regions and quantile regression won’t necessarily help find your cutoff. That said, what you describe sounds plausible, if a bit overly complicated. I think the literature on estimating Engel Curves might help… but maybe I’m just not exactly seeing what you are doing. We should probably discuss over brunch soon.

        • I’m not so much interested in the estimation of the curve of how much people actually spend for various incomes, but rather the estimation of a particular cost to buy a particular quantity of housing, utilities, and food as a function of location and time and size of family. But, I also simultaneously want to estimate what it means to have this “minimum” level of housing (economies of scale in the family). Obviously, everyone *could* live in those little Japanese “kapuseru hoteru” but in practice in the US they aren’t available, and there are cultural norms that limit how small your lodging can be for a given family size (esp. with kids) before people start calling Child Protective Services etc. It’s this “minimum culturally stable quantity” of housing whose price I want to estimate. Fortunately it’s a Bayesian model so I can acknowledge both the difficulty of defining and estimating in terms of probabilities.

          Once I’ve got this dollar quantity I can then build some additional models based on dimensionless ratios of after tax income to this minimum quantity of housing, utils, food.

          It’s a kind of alternative to the 10 years of research into a “supplemental poverty measure” that the Census cooked up several years back. https://www.census.gov/hhes/povmeas/publications/SEHSD2013-28.pdf

          I want a Bayesian measure (one with acknowledged uncertainties) and one that focuses only on costs not on sources of income, as I’m not primarily interested in estimating the frequency of poverty but rather in modeling household security and decision making as a function of several dimensionless numbers, one of which relates after tax income to “cost of necessities”

          Would love to get together. email me

        • @Daniel

          >>>”come up with some Bayesian estimate of a local regional poverty measure.”<<<

          Isn't this a case where you are trying to estimate something which is in itself ill-defined.

          i.e. Bayesian estimate of (say) mortality or income or BMI is a difficult estimation problem but at least what you are trying to estimate is well defined. To that extent it is strictly a statistical problem.

          In your case, isn't this more of a normative problem? You are trying to pass (implicitly) a judgement on what poverty means in the first place since "local regional poverty measure" in in itself can mean so many different things.

          So, I'm not criticizing what you are doing but just trying to differentiate two aspects: (a) Defining the appropriate metric (judgement call / normative) (b) Using the best technique (Bayesian or not) to *estimate* and already well-defined metric.

          I just fear that if you mix (a) and (b) you may run around in circles and trying to do (b) before (a) is in place can be a bit of a fishing exercise. One may tweak around the metric a hundred different ways till the profile (regional, demographic, spatial whatever) of "poverty" matches whatever pre-conceived notion one may have in mind? Especially given the power & flexibility of Bayesian techniques your researcher degrees of freedom are quite large here?

        • Hi Rahul: yes I agree with you that the precise quantity is ill defined, but the statement “there exists a dollar amount, call it X_r, such that your risk for malnutrition, chronic disease, chronic high levels of stress hormones, lead poisoning, alcoholism, crime victimization, arrest…. is a strongly varying function of E/X_r for region r where E is your total consumption expenditures and X_r is a scale bar for poverty which is itself related to the cost of housing, utilities, food, and sufficient extras to be able to hold a job.”

          I think this is probably uncontroversial (the existence of such a number) but estimating its precise quantity is not easy. Therefore, Bayesian statistics is a great way to go here. You can acknowledge that uncertainty and work with it, find lots of sources of information that can constrain your estimate…

          I’m not primarily interested in the quantity “fraction of population below X_r” so it doesn’t bother me if that number is strongly varying. My main interest is in using X_r as a scale bar to help define commonality among regions in patterns of consumption and outcomes.

        • I still think what you are trying to do here is too broad. Too vague. But I may be wrong. I’d love to see your results.

        • I am struggling with these ideas myself, so I’m glad to hear that it’s not obvious what should be done :-)

          I’m not committed to a view that this quantity has to be a definitely identifiable thing. Bayes will happily tell me that the posterior for this number is enormously broad if that is indeed the case. That will be an indication that the fundamental model is not tenable.

          If you’re committed to “getting a result” then of course, this area of research is risky, you might need to tweak things using forking paths until you get your p less than 0.05 so you can publish. Fortunately, that’s not my commitment. What I want is the truth about how people are doing in the US, and if it comes from some other line of research, at least by pursuing this one until I know it isn’t doing a good job, I will have ruled out what seems to be an obvious type of theory (ie. that poverty is a meaningful state of being which relates to health and education and etc and is also related to the cost of buying some essential goods which of course is a spatially varying thing).

          Ruling this out is to me, as valuable as “getting the result” because if this line of research doesn’t lead to some approximate answers, it will be extremely valuable to know why, don’t you think? I mean, what if in the US it doesn’t really matter how much housing and food cost, whether you’re obese, sick, uneducated, and constantly stressed is essentially independent of income all the way up to the top 1% of incomes… umm… wow! that would be a huge discovery itself!

  3. So price is like cocaine: quick high, rapid satiation (even condensed satiation) and addiction to the experience. Kind of like an addiction to shoes. But the counter argument is also true: pay more for a hooker and you can tell that story for a long time. (If that’s too offensive, try a Mercedes versus a Nissan Versa.) And in some cases, like Floyd Mayweather, an excess of cash means no one has any idea what the guy actually enjoys because he has so much so what exactly would his satiation profile be? But in contrast, Jay Leno has mega doses of cars and takes long-lasting satisfaction from the experience of his collection. Or maybe satiation effects only occur when you think of enjoyment as if it were a drug with an absorption profile or some other relatively inappropriate metaphor. There is a point to this rambling, which is that nearly every psychologically measurable or interpretable effect is not close to being unique in cause or expression. So I agree totally. And now I wonder why I bothered typing this.

    • The hooker comment is indeed offensive and massively so, given the trafficking of women is a horrible thing and also because euch statements minimize women in general. As a general rule if you need to type, ” if that is offensive substitute”, you might want to Just backspace and substitute in your writing.

      I like this blog but do not want to have to read statements about the selling of women for sex.

Leave a Reply

Your email address will not be published. Required fields are marked *