Eric Loken writes:
If you’re used to Google upending conventional wisdom, then yesterday’s interview with Laszlo Bock in the New York Times did not disappoint. Google has determined that test scores and transcripts are useless because they don’t predict performance among its employees. . . .
I [Loken] am going to assume they’re well aware of the limits of their claim, and instead I’m going say that as readers of the interview we should not lose sight of a fundamental fact -
Across a wide variety of employment settings, one of the most robust findings in organizational psychology is that tests of cognitive ability are strong predictors of job performance. If Google has found otherwise, what they have found is that grades and test scores are not predictive of performance at Google. In general, in the workplace tests are still highly predictive of success.
If all the research says that test scores and grades predict performance, why would the people at Google want to ignore this information?
There are at least two factors in play here (and again I’m assuming the folks at Google are well aware of both of these points). First, when a company has built its brand on attracting only the brightest prospective employees, through self-selection and through sheer volume of applicants the pool will be extremely competitive. Google likely doesn’t have much variability among those hired with respect to test scores and grades. And when there is no variability, there is no correlation with anything. It’s a similar argument to MIT saying that the SAT is useless for their admissions. Their applicant pool is so vast and highly qualified that the incoming class is largely homogeneous on those measures.
The second point is a bit more subtle. We of course need to wonder how all those people with lower test scores and grades would have fared at Google had they been hired. But furthermore, when an organization has used a certain instrument to select their employees, the correlation with job performance goes down. It’s not just that the range has been reduced (through selection). It’s also that the information has already been acted upon. If someone is hired despite their lower test scores, it usually means some compelling compensating characteristics made that person look like a good bet. That is why the correlation between a valid selection instrument and job performance can be dramatically depressed when only looking at the hired sample.
It’s common to misinterpret that low correlation as a sign of poor prediction. Again – thousands of research studies have confirmed the predictive validity of tests of cognitive ability for job performance. Google may well say, “Not here.” – but they cannot (and did not) say, “Not anywhere.”