What predicts whether a school district will participate in a large-scale evaluation?

Liz Stuart writes:

I am writing to solicit ideas for how we might measure a particular type of political environment, relevant to school districts’ likelihood of participating in federal evaluations (funded by the US Department of Education) of education programs. This is part of a larger project investigating external validity and the generalizability of results from randomized trials to target populations that I am doing with Rob Olsen, Steve Bell, and Larry Orr. In this piece of the work we are trying to model the factors that may impact whether a school district is likely to participate in one of these large-scale evaluations. One factor that occurred to us was the political environment in the state, and how likely a school district might be to want to “go along with” and participate in a federal evaluation. (And that, for example, this might depend on the party alignment between the state politics and the federal government.). We were wondering if you or your readers might have any suggestions of a parsimonious way to measure this, again as a reflection of their potential likelihood of cooperation with a large-scale federal evaluation.

Any thoughts? This would’ve been a good one to post on the old Monkey Cage blog.

8 thoughts on “What predicts whether a school district will participate in a large-scale evaluation?

  1. Some thoughts:

    – Size & type of central office staff; large scale federal evaluations require a lot of different bits and pieces of information and districts that already have the capacity to produce these are at an advantage. Similarly look for divisions that deal with special projects or somewhat amorphous things like youth engagement/development or evaluation; they tend to house the program management staff that works with these evaluations; you also need people to write/apply for grants. If a district is funding this kind of staff they’re likely to have the political climate that would make them interested in evaluations.

    – Need for funding; All school districts need funding but some need it enough to jump through federal hoops. The district as a whole or one particular segment of it needs funding theyre more likely to bite. This happens a lot in urban districts where the demographics of a city are different than the demographics of the state but the city is big enough to be a player at the state level.

    – Previous engagement with large philanthropic organizations or big grants. These can be similar (reporting-wise) to fed grants. Its a good proxy for capacity to meet evaluation requirements and willingness to get in the game. This is less about politics and more about finding some districts that might be good “go along” candidates and looking at them in more detail.

    – Not sure which way this will cut but I think looking at how centralized control of the district brass is would help. Does a particular group have enough representation on the school board to push through what it wants? Is there mayoral control?

    – Not sure if this will yield anything but the frequency w/ which district organization and funding allocation (as opposed to anything property tax related) questions turn up on local ballots might speak to something in the local political climate

    – In the district: presence of the “school reform” movement. A district that’s pushing hard for charters is often one that’s internalized the rhetoric of evidence based practice, evaluation & metrics. At the state level a good proxy is whether the state applied for (and how far they got) in Race to the Top funding.

    -party alignment between the state and the feds doesn’t take into account the politics of the district. In the example of the large urban district in the state where most districts are not large & urban you might see the district and state being less in alignment than the district and the feds.

    (background: This is based on observation & anecdote from time spent working with a few large & midsized districts on federal evaluations and other large scale data/evaluation initiatives)

  2. Any consideration of the district’s chief administrator (Superintendent) receptiveness to external evaluation in view of the increased security by state and/or federal standards initiatives and the resultant impact upon the interpretation of relative viability/ performance of the district?

  3. The pattern I’m noticing is likely pre-hoc collusion between liberal localities and the Obama Administration. For example, the Obama Administration has been investigating and suing school districts for various kinds of discrimination. It started with suing LAUSD for discriminating against Hispanic English learners, even though the LAUSD school superintendent at the time was Hispanic.

    But my phrase “even though” in the preceding sentence is naive. The feds and the locals are not typically adversaries, they are colleagues in using agreements to settle lawsuits to shake down local and federal taxpayers for favored constituencies. It probably wouldn’t be hard to come up with some metrics for modeling this pattern of the Obama Administration suing its friends.

  4. Why not just ask them? As the OP pointed, those factors are unknown, so an inductive approach might be warranted here. Sacrificial interview question for district leaders: “How would you decide on whether to participate in a federal evaluation program?” Obviously the question may need to be a bit more nuanced, but you get the idea. Then use an open coding scheme (gasp) to sort the responses into categorical buckets.

  5. FWIW366 suggestion resonate with my own experience working with school boards in order to get permission to collect data, and seem worth considering. Here are a couple of quick thoughts:

    + Education of administrator in charge of approving research (do they have a doctorate, a research degree etc?)

    + Number of research requests per year in the district (some urban districts are deluged by requests)

    + Size of the district, also urban vs. rural (getting permission from larger districts is a more formal process than from small in my experience)

    + Demographics of the district (besides the point of funding, the wealth of the constituents, of the students, the racial composition of the district, and the education level).

    + Speaking of the above, a measure of segregation in the district– would be curious if that had an effect.

    Also, regarding politics, what about a state/region level variable, to see if there are patterns amongst states? Because each state may have crazy politics/rules regarding research in school plus indigenous feuds and testing that need to be navigated.

    JB’s idea would be good in theory, but non-response would surely be a problem LOL! It sounds like the study is secondary-analysis. Interesting to see if there are patterns and looking forward to seeing the results of this study.

  6. I’d also try to include some measure of the competitiveness of school board elections, as well as how fractious the board is. That might be the most proximate political measure at the district-level.

  7. How much is the district interested in having something to publicize? This might be measured by the number of press releases from the district (possibly normalized to the size of the district). Might want to exclude press releases that seem necessary (e.g. snow day closings).

Comments are closed.