My major pet peeve are rankings on various web sites. This picture of restaurant rankings should clarify the problem:
Shouldn’t the 4-review 5-star ranking be above the 1-review 5-star ranking? Four excellent reviews should increase our confidence in the quality more than a single review.
The mistake lies in using the frequentist mean instead of a Bayesian posterior mean for aggregating the reviews, which disregards the number of reviews. The Bayesian prior would inform us that all rankings tend to be centered at, say, 3. The reviews would then be able to push the estimate away from 3.
How to implement this? We can assume that the prior for rankings of restaurants is N(3,1) – a Normal distribution with the mean 3 and the variance of 1. Let us also assume that the rankings for a particular restaurant have a standard deviation of 1: under this assumption our prior is conjugate. Second, we have the average y of the n rankings. The posterior mean (see page 49 of the second edition of the BDA book) will then take the following form:
μn = [ 3 + ny ] / [ 1 + n ]
If we take the averages and the counts from the table above, compute μn and sort by it, we get the following table:
Now, the 5-star rankings with a single review appear even below the 4.5-star rankings with multiple reviews. In a practical application, the μn column could be converted into a graphical representation.
Some websites do first sort by average rating and then by the number of reviews, but such a solution is problematic: a restaurant rated only with 4 and 5 will appear above a restaurant rated 99 times as 5 and 100 times as 4.
A less questionable conjugate prior would be the Normal-Gamma, but I won’t go into this here.