Multilevel regression

Mike Hughes writes:

I have been looking a your blog entries from about 8 years ago in which you comment on the number of groups that is appropriate in multilevel regression. I have a research problem in which I have 6 groups and would like to use multilevel regression.

Here is the situation. I have racial attitudes data from samples of University of Alabama students drawn in 1963, 1966, 1969, 1972, 1982, and 1988. (I also have data from 2013, but I am not including them in this analysis).

I have run a number of analyses predicting social distance attitudes in which the level-2 group variable is year. The main predictor is endorsement of racial stereotypes. I am interested in whether the association between endorsement of racial stereotypes and social distance declines over time (it does), so I specify a random slope for racial stereotypes, and I look at the cross level interaction between the racial stereotypes coefficient and year, included in the fixed part of the equation as number of years since 1963 (1963=0, 1966=3, etc.). I use xtmixed in Stata14.

The models run, and the likelihood ratio test vs the linear model is significant. AIC and BIC indicate that the xtmixed model is better than the OLS model that I run in Stata. Also, the OLS model predicts values of the dependent variable that are beyond its range (1 to 5), but the xtmixed model does not.

So my question is, is it ok to present my multi-level model, with 6 groups, rather than the OLS model? If so, is there something I can cite to provide some backing for using the model with 6 groups?

My reply: I agree that it makes sense to include a linear predictor for time and also allow intercepts and slopes to vary by discrete survey (so that you have 6 values for each coefficient). And once you do this, there’s not problem using the model to extrapolate. Whether that’s a good idea, is another story.

5 thoughts on “Multilevel regression

  1. hi dr gelman,

    given that this is not my area of expertise, and you are clearly much more seasoned than i am, i am refraining from impulsively hitting the big red button many times when i read the term ‘social distance’.

    the analysis seems fine, although i concede it is not my area of expertise (i am fortunate to not have to worry about ‘level-1’ and ‘level-2’ in the analysis i’ve performed so far).

    that being said: don’t you think the ‘root’ data, ‘social distance attitudes’, is, at best, questionable? i am assuming this measurement is loosely based upon https://en.wikipedia.org/wiki/Bogardus_social_distance_scale ?

    don’t you, as a prominent statistician in the community, feel that performing correct, rigorous statistical methodology on input data (social distance measurements) is misleading?

    i am not faulting you or any other statistician for your role in such studies, as you are often put in unenviable situations (my would-have-been mentor didn’t make it to 40). i am moreso raising a question about the usefulness of studies whose statistical methodology is correct, but the original input (social distance) is grossly oversimplified out of numerical convenience.

    i could be wrong, but it really does seem that robust methodology is not as fruitful as we’d expect when the input data is… i don’t know what it is… words fail me when people start arbitrarily conjuring measurements like social distance.

    • Just because you measure something and put a name on it, doesn’t mean what you measure and what the name conjures up in other people’s heads are in any way related. (statistical significance vs practical significance anyone?)

      That being said, you are measuring SOMETHING. Sometimes it’s almost purely noise. Other times it is a signal. Figuring out what the signal means for your theory is different from figuring out what your measurements mean for your signal.

    • I hardly ever see Guttman scales now days, yet years ago they were considered essential kinds of measures and we could ask the question “does it scale?” and mean does it follow the Guttman pattern of deterministic agreement (if you agree with a harder item you will agree with a softer item). For 1925 it was pretty forward looking. That said, I think I’d at least want to think about looking at the items separately as a within-subjects measures and use a logistic model. That might give me a better handle on how the meaning of the different items might have changed over the years and a more effective way to think about the ordinality issue. Back in 1925 or even 1975 they were just not able to do that easily.

      Specifically, for example, when the social distance scale was created interracial marriage was illegal in many places in the US, so saying you would be comfortable with it for a close relative probably meant something different. Certainly over the timeframe of the study that would have changed too, at least if the GSS results are reflected. I’d really like to know if the assumption of equidistance was ever reasonable or if it changed in its reasonableness.

      I would like to hear the idea behind the hypothesis more. We know that the GSS general approval of interracial marriage has changed a lot (though less in the south) but that approval of marriage with a close relative has change dramatically less. is that the idea? That there is a social acceptability change in willingness to express racial stereotypes such that change in acceptance at a personal level can no longer be predicted by such attitudes? It seems to me that the hypothesis might actually be exactly about the issue of whether the underlying assumptions of the scale hold up.

      Ha! I knew something related came up before, that’s why I read about Borgadus scales. http://statmodeling.stat.columbia.edu/2014/11/27/quantitative-literacy-tough-idea-1958-96-americans-disapproved-interracial-marriage/

  2. I can’t comment sensibly on the more fundamental measurement issues raised by the other commenters, but: based on my reading of the linked Wikipedia article, it sounds like the social distance attitude score is essentially an ordinal variable (i.e. ordered, but with possibly unequal distances between categories). If the grouping factor in your mixed model is the social distance attitude score, wouldn’t it make more sense to treat it as an ordinal predictor? Furthermore, I’m not sure that your sociological hypothesis (“the association between endorsement of racial stereotypes and social distance declines over time”) matches up with your conclusion (which I take to be that there is significant variation among slopes, i.e. that different social distance categories change differently over time) — I think that the right configuration of intercepts and slopes among social distance categories could make them either more *or* less correlated over time …

Leave a Reply

Your email address will not be published. Required fields are marked *