An incredibly useful method is to fit a statistical model repeatedly on several different datasets and then display all these estimates together. For example, running a regression on data on each of 50 states (see here as discussed here), or running a regression on data for several years and plotting the estimated coefficients over time.
Here’s another example:
The idea is to fit a separate model for each year, or whatever, and then to look at all these estimates together to see trends. This can be considered as an approximation to multilevel modeling, with the partial pooling done by eye on the graphs rather than using a full statistical model.
One reason the secret weapon is so great can be seen in various analyses of repeated cross-sectional data, with estimates every two or four years (for example, in studying Congressional or Presidential elections). The horrible alternative often involves people pooling data over decades in order to get stable estimates, but as a result it is then difficult to see time trends, and models get oversimplified.
We call it this technique the “secret weapon” because it seems to be done much less often than it could be. I suspect the technique is not used more because people are fixated on point estimates and don’t realize that a graph can tell a clearer story. Another failure of classical statistical estimation!
(For some examples of the secret weapon with repeated cross-sectional data, see Figures 2, 4, 9, and 10 of this paper
Well, I guess it’s not a secret anymore…