Brendan Nyhan and Jacob Montgomery talk sense here. I am perhaps too influenced by Steven Rosenstone’s 1983 book, Forecasting Presidential Elections, which is the first thing I read on the topic. In any case, I agree with Nyhan and Montgomery that the difference in vote, comparing a centrist candidate to an extreme candidate, is probably on the order of 1-2%, not the 4% that has been posited by some.
Among other things, ideological differences between candidates of the same party might seem big in the primaries, but then when the general election comes along, party ID becomes more important.
I also disagree with the model in which presidential elections are like votes for high school prom king.