Someone sent me a question about whether it makes sense to use multilevel modeling in a study of polls from many countries. I’ll give the question and my response. The topic has been on my mind because I just wrote a discussion on this issue for the forthcoming issue of Political Analysis.

The question:

This question concerns a set of surveys with 35,000 individual respondents in a total of 20 countries. The model of opposition to immigration has a six-question scale as the dependent variable, and includes a a large number of individual-level predictors (~30) as well as three country-level variables (percent foreign-born in country, the unemployment rate, and the interaction of the two — extant literature suggests the interaction is often meaningful). Other model specifications will include cross-level interactions but that’s a secondary issue for the moment.

One question is why do HLM instead of OLS with clustered standard errors. One reviewer seems to prefer the latter and asks us why HLM is necessary. (He cites a paper by Rob Franzese at the U of Michigan that suggests that it makes little difference whether you use HLM or clustered OLS.) I have been unable to find a discussion that explicitly weighs HLM against OLS with clustered standard errors.

The other question is whether to do a more simple HLM with random intercepts but with “fixed” level-1 coefficients, or do a more complex model that includes random effects for each level-1 coefficients. We originally wrote the paper with only random intercepts, but I am now wondering if that was the right model. I am mindful here of your post of 1/25/05 (“Why I Don’t Use the Term ‘Random and Fixed Effects'”), so I want to think about this in a nuanced way. A handful of things:

1) Because the paper is about “European” attitudes towards immigration, and we don’t have the “population” (all European countries) in the ESS data, we could conceive of our level-2 units as a sample from this population. In which case, if I understand things I’ve read correctly, random effects can be appropriate.

2) In a simplistic statistical sense, the difference in fit between a random-intercepts/fixed-effects model and a random-intercepts/random coefficients model is statistically significant. Adding the random effects improves the fit, even though it adds many more parameters. In my mind, this is by itself no reason to include random coefficients.

3) This is what puzzles me the most. Adding random effects to the level-1 coefficients does not change the coefficients, though it does increase the standard errors because the d.f. is now the number of countries minus 1 instead of the full 35,000. More importantly, it does affect whether the *country-level* variables are significant. These level-2 effects are only significant when the level-1 slopes include random effects. I have no explanation for this and one other, more methodologically adept person with whom I consulted was similarly puzzled. I can send you output that shows the results if you’re interested. This is why I care even more about getting the specification “right”: it does change the substantive conclusions one derives from the model, and the role of contextual effects is an ongoing debate in the literature. (The models are estimated using HLM 6.0, by the way.)

My response:

Before doing multilevel modeling, I would do a so-called “two-stage regression”–that is, fitting the regression model separately in each country, then estimating level-2 effects by running regressions on the country-level regression. By fitting the models separately, you’re automatically allowing slopes as well as intercepts to vary by country. (If you then want to interpret the intercepts, you have to make sure that your level-1 predictors are coded so that their zero-levels are interpretable.)

It sounds like there are two sorts of things you’re interested in: the level-1 coefficients and how they vary by country, and the level-2 coefficients that describe variation between countries. To understand the level-1 coefs, I’d make a series of plots showing the ests and se’s for all the countries, for each of the level-1 predictors. The level-2 coefs should be easy enough to interpret. With the two-stage regression, the se’s on the level-2 coefs automatically account for the variation between countries.

OK, so what about multilevel modeling? MLM takes more effort; the payoff, compared to two-stage regression, comes when the level-1 coefficients in the individual countries cannot be estimated accurately–when their se’s are large compared to their unexplained variation (an issue we discuss a bit in this paper). If that’s an issue then, yes, I’d recommend multilevel modeling as a way of better estimating the level-1 and level-2 regression coefficients.

It’s irrelevant whether your study includes all the countries of Europe, or just a subset. Multilevel modeling is fine in either case.

P.S. The Political Analysis issue is Autumn, 2005; see here.