I did say that it’s a bit like decathlon however while conceptually similar to a decathlon-style ranking, and unlike decathlon, we have a varying set of participants across different “events” (the different leaderboards/Maps metrics). In essence, we need a way to compute a composite score that fairly represents a Local Guide’s achievements in the year across multiple metrics while accounting for the varying number of metrics they qualify in and this is where the comparison to decathlon breaks down as in the decathlon all athletes compete in all events.
So how then do we calculate a fair score?
Weighted Rank Sum: Yes one way is as you have indicated, to assign inverse rank points in each metric and sum them up:
- The top-ranked Local Guide on a leaderboard gets 100 points (since there are 100 in the ranking).
- The second-ranked gets 99 points, third gets 98, … last gets 1 point.
- The overall score for a Local Guide is the sum of their points across all leaderboards (metrics) they appear on.
It’s a simple and intuitive approach but it rewards higher rankings more heavily and does not allow for the overall strength of the achievements of Local Guides on the different leaderboards. There are statistical ways to address this. For example, we could compute the
Weighted Average Rank (Relative Performance): and instead of simply summing up raw inverse rank points, we normalize it by the number of leaderboards qualified for and by doing so, Local Guides that have qualified on many leaderboards are not unfairly advantaged over those excelling in fewer metrics. While undervaluing the advantage of excelling in more metrics, what I like here is that we ensure fair comparison across specialists and generalists. A more complex decathlon-style,
Scientific ranking approach would be a Z-score normalization approach where the mean and standard deviation of ranks is computed in each event (leaderboard in our case) and the athlete’s event score (Local Guide’s leaderboard score) would be the Local Guide’s rank minus the mean rank divided by the standard deviation. This approach accounts for the variation in ranking difficulty across different metrics and makes rankings comparable across different distributions but of course it is computationally more complex.
ps: Thanks for liking the new avatar 