Skip to content

Evaluating the impacts of welfare reform?

John Pugliese writes:

I was recently in a conversation with some colleagues regarding the evaluation of recent welfare reform in California. The discussion centered around what types of design might allow us to understand the impact the changes. Experimental designs were out, as random assignment is not feasible. Our data is pre/post, and some of my colleagues believed that the best we can do under these circumstance was a descriptive study; i.e. no causal inference. All of us were concerned with changes in economic and population changes over the pre-to-post period; i.e. over-estimating the effects in an improving economy.

I was thought a quasi-experimental design was possible using MLM. Briefly, my suggestion was the following:

Match our post-participants to a set of pre-participants on relevant person level factors, and treat the pre/post differences as a random effect at the county level. Next, we would adjust the pre/post differences by changes in economic and population factors at level 2 (county level) in order to produce an estimate of the change.

I was wondering if you might share some thoughts on this approach?

In reply to your colleagues who believe all that can be done is description, recall Jennifer’s dictum that the goal of inference is always causal. A good description is fine—I spend much of my time as an applied researcher doing descriptive inference—but, implicitly or explicitly, it will usually be used for causal purposes, so it’s worth making that link and understanding the assumptions required for any given causal interpretation.

Regarding your idea, it seems to me that you’re considering variation of the treatment at the county level. From the perspective of data analysis, multilevel modeling is definitely the way to go. But I don’t know enough about the substantive context to say more than this. Maybe my commenters will have more to offer.


  1. John Pugliese says:

    Thanks for responding to my question.

    In terms of context, most welfare programs seek to aid participants in finding employment or increasing earnings. The ultimate goal is obviously for them to exit aid and in a sustainable fashion, remain off-aid. That outcome is achieved through various program activities, such as job-preparation, volunteering, or education. “On the ground” these activities are implemented differently across counties. How much so would be a study in itself. The key point is that the legislative changes apply to ALL counties and were effective at the same time. These high-level changes may or may not change how the program looks on the ground. The question is did these broad legislative changes impact participant outcomes?

    It would have been great had the changes been staggered, but they were not. We only have pre/post data at this time. As stated above, my colleagues were concerned with the improving economy. I was hoping to take advantage of the variation in changing economic conditions at the county level to possibly tease out these factors so a reasonable pre-change group of participants could be compared to new participants.

  2. BP says:

    Why not construct a synthetic control group like in Hainmueller and Abadie and coauthor’s study of the cigarette tax change in California? (Published in JASA)

    If there is some reason to think county-level effects are different, you could match California counties to counties outside the state, or do synthetic control groups for each county even. That is how I would be thinking about the problem.

    • Fr. says:

      I’m not an expert on either welfare reform or Hainmueller and Abadie’s method, but I also thought of that one when I read the question.

      Also, I would suggest submitting the question to CrossValidated to see what the StackExchange hive mind comes up with on that topic, which I find very interesting.

  3. Deborah says:

    I suggest someone look into special needs trusts, possibly putting a limit on them.
    I know a gentleman who has $750,000 in a trust and continues to live on SSDI, Medical, Section 8 and food stamps.
    Might be a small savings for the state but something is better than nothing.