What Do We Learn from Narrow Randomized Studies?

Under the headline, “A Raise Won’t Make You Work Harder,” Ray Fisman writes:

To understand why it might be a bad idea to cut wages in recessions, it’s useful to know how workers respond to changes in pay–both positive and negative changes. Discussion on the topic goes back at least as far as Henry Ford’s “5 dollars a day,” which he paid to assembly line workers in 1914. The policy was revolutionary at the time, as the wages were more than double what his competitors were paying. This wasn’t charity. Higher-paid workers were efficient workers–Ford attracted the best mechanics to his plant, and the high pay ensured that employees worked hard throughout their eight-hour shifts, knowing that if their pace slackened, they’d be out of a job. Raising salaries to boost productivity became known as “efficiency wages.”

So far, so good. Fisman then moves from history and theory to recent research:

How much gift exchange really matters to American bosses and workers remained largely a matter of speculation. But in recent years, researchers have taken these theories into workplaces to measure their effect on employee behavior.

In one of the first gift-exchange experiments involving “real” workers, students were employed in a six-hour library data-entry job, entering title, author, and other information from new books into a database. The pay was advertised as $12 an hour for six hours. Half the students were actually paid this amount. The other half, having shown up expecting $12 an hour, were informed that they’d be paid $20 instead. All participants were told that this was a one-time job–otherwise, the higher-paid group might work harder in hopes of securing another overpaying library gig.

The experimenters checked in every 90 minutes to tabulate how many books had been logged. At the first check-in, the $20-per-hour employees had completed more than 50 books apiece, while the $12-an-hour employees barely managed 40 each. In the second 90-minute stretch, the no-gift group maintained their 40-book pace, while the gift group fell from more than 50 to 45. For the last half of the experiment, the “gifted” employees performed no better–40 books per 90-minute period–than the “ungifted” ones.

The punchline, according to Fisman:

The goodwill of high wages took less than three hours to evaporate completely–hardly a prescription for boosting long-term productivity.

What I’m wondering is: How seriously should we use an experiment on one-shot student library jobs (or another study, in which short-term employees were rewarded “with a surprise gift of thermoses”), to make general conclusions such as “Raises don’t make employees work harder.”

What I’m worried about here isn’t causal identification–I’m assuming these are clean experiments–but the generalizability to the outside world of serious employment.

Fisman writes:

All participants were told that this was a one-time job–otherwise, the higher-paid group might work harder in hopes of securing another overpaying library gig.

This seems like a direct conflict between the goals of internal and external validity, especially given that one of the key reasons to pay someone more is to motivate them to work harder to secure continuation of the job, and to give them less incentive to spend their time looking for something new.

I’m not saying that the study Fisman cited is useless, just that I’m surprised that he’s so careful to consider internal validity issues yet seems to have no problem extending the result to the whole labor force.

These are just my worries. Ray Fisman is an excellent researcher here at the business school at Columbia–actually, I know him and we’ve talked about statistics a couple times–and I’m sure he’s thought about these issues more than I have. So I’m not trying to debunk what he’s saying, just to add a different perspective.

Perhaps Fisman’s b-school background explains why his studies all seem to be coming from the perspective of the employer: it’s the employer who decides what to do with wages (perhaps “presenting the cut as a temporary measure and by creating at least the illusion of a lower workload”) and the employees who are the experimental subjects.

Fisman’s conclusion:

If we can find other ways of overcoming the simmering resentment that naturally accompanies wage cuts, workers themselves will be better for it in the long run.

The “we” at the beginning of the sentence does not seem to be the same as the “workers” at the end of the sentence. I wonder if there is a problem with designing policies in this unidirectional fashion.

11 thoughts on “What Do We Learn from Narrow Randomized Studies?

  1. Tggp:

    I followed your link and the explanation given there ("the real value to Ford of his high wages was . . . that he was paying higher wages than his competitors. . . . Paying hugely higher wages than everyone else in the nascent manufacturing industry of the time meant that he lowered turnover, thus recruitment and training costs."), and this seems completely consistent with Ray's first paragraph above ("Ford attracted the best mechanics to his plant, and the high pay ensured that employees worked hard throughout their eight-hour shifts").

  2. Of course the "we" and "workers" aren't the same. Didn't you get the memo? We're heading towards aristocracy and if you want to live on the scraps (read, through study) you've gotta join the party.

  3. Are these conclusions symmetric? Would paying you less make you work less harder?

    Is there any discussion about how a wage-cut experiment might work?

  4. Of course you are correct but bear in mind the costs and benefits of experiments. The "gift exchange" theory is pretty radical to begin with so I see this as a small experiment that is asking whether the theory is worth investigating in greater depth. Since the gift exchange effect seems to be small and temporary then the answer is no. Useful in the grand scheme of things.

    By the way, note that the gift exchange theory is different than the selection theory (pay higher wages to get a larger pool from you select better workers) and different from the efficiency wage (fear of firing) theory and this experiment is testing gift exchange only.

  5. "The goodwill of high wages took less than three hours to evaporate completely–hardly a prescription for boosting long-term productivity."

    I am worried that this makes the leap from the causal contrast and into conjecture as to the mechanism behind this contrast. It could also be that the "gifted" employees became fatigued at the higher pace of work and so 40 books an hour represented a higher level of effort overall (tired people working harder might shelve as many books as less tired people who were not motivated to work hard). In other words, I am not sure that we can claim that effort in hour six is independent of the time course of effort. The crude causal contrast shows overall higher productivity.

    So not only do we have the issue of external validity (especially due to the "one time" nature of the experiment) but we also don't have a clean test of mechanism. What if the $20/hour students had the same pattern of book shelving for a second shift? What would that tell us?

    In particular, I worry about strong claims like "The goodwill of high wages took less than three hours to evaporate completely". Not only is the term "goodwill" vague, but the experiment was measuring productivity and not the emotional state of the workers. Maybe goodwill is being expressed in other ways (like a higher propensity to apply for future shelving jobs)?

  6. Suppose that each job has a rate of work that is sustainable over the period of a working day. If you are paid more than you may try and work harder than that but you can't keep going at a rate that is unsustainable so you fall back to the sustainable rate. That's not a lack of goodwill, that's a reality that people have physical limits.

    The interesting thing would be to give them all another days work at the same pay rates and see if the gifted employees followed the same work pattern.

    And then more days work to see if the gifted employees can build up their endurance to work at a higher rate for longer.

    So I guess I agree – the reported randomised trial doesn't really tell us anything.

  7. From your description it sounds like this is a case of simply asking the wrong question.

    The purpose of paying the workers more has little to do with incentivizing them to work harder, and a lot to do with preventing them from leaving as they gain more expertise because it is costly to constantly hire and retrain new people (of course this applies only to jobs that require some skill).

  8. I think there is a problem of internal validity if you take the title seriously. The experiment doesn't show that high wages don't matter, but that unexpected and unexplained wage increases don't matter (or at least not much). And the article notes that experiments suggest that unexpected and unexplained wage cuts reduce effort. That point seems to support Akerlof's model: the employer's "gift" is taking care of workers during a recession by maintaining wages rather than cutting them to the level the market would permit and the workers' "gift" is working harder than they need to.

  9. Re: perspective of the employer – or the manager of the endeavour.

    First, my favourite experience as a student was by orders of magnitude in MBA school.

    To me, it was about bringing science to the management of any endeavour (and why it was great training for going into research where managing the research endeavour makes it replicable).

    In statistics, how often do we write from the perspective of the randomized subject or survey respondent (OK Andrew occassionally)?

    And there was no marks deducted for spelling mistakes in my MBA school.

    More seriously the money comes from the employers. When I was consulting to a university on their plans to set up an executive MBA program, I raised the concern about employers _sticking_ their problems employees there until they can be made strategically dispensible (this happens). It was turned into a reason for _needing_ to charge such large tuition fees!

    Cuurently, my former policy professor has told me that new b-school faculty obtain their tenure by doing less than ideal step-wise regression on any available data set, and yarn on for pages interpreting any statistically significant regression coefficients …

    So maybe a field statisticians could take over (or at least join).

    K?

  10. "Since the gift exchange effect seems to be small and temporary then the answer is no. Useful in the grand scheme of things."

    The theme of the original post (as I see it) was that this experiment should not have found much of an effect, that it was set up to minimize such an effect (not deliberately).

Comments are closed.