$30.00
Analyzing the BGG ranking system: the Good, the Bad, and the Graphickly
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
Recommend
374 
 Thumb up
219
 tip
 Hide
BGG is the most comprehensive resource for board games, which provides a wealth of information about all you need to know on all games. With a massive group of users, we now have an impressive dataset of opinions, rating and comments for more than ranked 120.000 games.

Because of its fame and power to influence gamers on what games to buy (or not), the BGG ranking has always been contentious, because some look at it as a “holy ranking” speaking the truth on the quality of games, while other will see it as a “sheeple maker”, where the opinion of the masses will dictate how a game will sale or not.

The reality is, as often, probably in between, and while this analysis has probably been done many time, I wanted to look at it from a different perspective and therefore give a grain of salt to the discussion.

The fact is: BGG ranking, is a ranking system, and therefore inherently has its good and bad aspects.

Let’s crunch the number so we can simply understand how it works, and how to read it with some relativity.

Note: data is valid as of June 10th 2016. It is probably different now, as you read this…

Feel free to comment the data. My opinion is as subjective as any other user
Your Tags: Add tags
Popular Tags: Geek_List_Useful [+] ratings_episode_999,999 [+] [View All]
  • [+] Dice rolls
1. Board Game: Rate it! [Average Rating:6.12 Unranked]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
The BGG ranking is primarily using two variables:

1-The value of the ratings given by users
2-The number of ratings given by users.


To be eligible for the ranking system, a game has to fulfill two conditions:

1-Be a standalone game (expansions are not included in the ranking)
2-Have received at least 30 ratings.

This explains why, out of 84k+ entries, only 12k+ games are included within the ranking system.

This means that some obscure little gems are not even in the BGG rank. For example, my favorite word game, Wyps, only has 25 ratings, and therefore not even ranked.
10 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
2. Board Game: Ye Mean and Mighty [Average Rating:0.00 Unranked] [Average Rating:0.00 Unranked]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
Another important aspect to understand is the two different rating averages available.

1-The “user average”, doing a simple arithmetic mean of all ratings for a given game, regardless of the number of users who rated it.

2-The “BGG average”, which is a secret formula, which includes a number of dummy ratings giving a value of 5.5. This average is used for the BGG ranking.

There is a perfectly good rationale for using the BGG average, because games with very few votes, but well appreciated by its microniche gamers, would otherwise be placed on the top of ranking system.

See here (https://boardgamegeek.com/wiki/page/ratings)

This BGG average comes with interesting perks on how it influences the ranking of games. Let’s check it out.
11 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
3. Board Game: Rank [Average Rating:7.00 Unranked]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
First, let’s look on how the game ranking is distributed according to their BGG average.



Let’s look at some familiar landmarks so we know where we are…



You will notice that there is a rapid decrease in the BGG ratings within the first 1000 ranked games, and then the BGG ratings reach a plateau…



…which is at 5.5, of course. This is due to the weight of all the 5.5 dummy votes for games with few votes and such votes are not that great (5s or 6s…). At 5.5, the rating is going through a point of inflection, which drops sharply after 110k, for games that received A LOT of negative ratings (1s, 2s etc..), and for which the dummy votes don’t have any effect anymore.
31 
 Thumb up
0.30
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
4. Board Game: Levée en Masse [Average Rating:6.92 Overall Rank:3892]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
Let’s now compare the BGG average to the user average.



We can see a clear departure of the User average, above the BGG average for games rated above 5.5, and below the BGG average for games rated below 5.5.

The way to read this graph is to consider that all the blue points very close to the red line, have received so many votes, that the two averages are essentially the same.

However, there are many outliers and there are two possible explanations for these:



1-Some games are relatively new, for example, Gloomhaven, is “hot”, but has so few ratings that it is currently rated around 6000. Within a few months, it should climb easily in the top 500, but as more ratings are provided, different opinions may also bring it not so high. Same is true for Unpublished Prototype

2-Some games are a “niche” game. Primer: The Gamer's Source for Battles from the Age of Reason, may be rather obscure, but all the ones who rated, loved it. It did not get much hype in 2013, missed its window of opportunity to receive massive ratings from users, or is just too far “out there” for the masses...

Using the same rationale, games with few ratings, but terrible ones, are creating clear outliers, for reason we can imagine.

However, a few oddballs are showing up. For example 90degres has a user average of 7.75, but has a BGG average of 5.456, which is below 5.5 !. It’s a clear outlier to the maths behind the formula and I can only see a bug somewhere to explain such value.
41 
 Thumb up
1.00
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
5. Board Game: Massive Darkness [Average Rating:8.25 Overall Rank:5345] [Average Rating:8.25 Unranked]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
Unfortunately, there is vast majority of games that may never receive additional ratings, which will (or already has) send them to the limbos of the BGG ranking, while being perfectly good games… It is particularly true nowadays with the never ending flow of new games, which chase to oblivion the ones published just yesterday…

Let’s see where are the games with more than 2000 ratings



Which make perfect sense…, most of them are in the top 500 games on BGG.

Some are historical games that nobody really like anymore (on BGG), for various reason (Risk, monopoly, Clue etc..) are also visible down the line…

And now let’s compare with games with less than 500 ratings



Wow. We can see that all games within the top 600 have received more than 500 ratings So, even if you have the best game in the world (to your own subjective opinion), If you cannot have more than 500 users to rate it with 8, 9 and 10s, then it will never make it to the visible forefront of BGG ranking. This explains how important it is for a new game to be “hot” even for a few days, so that it can receive this sooooo needed critical mass of ratings.

Note the inevitable positive feedback: popular games will get more visibility, therefore more ratings… etc.
34 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
6. Board Game: Fluxx [Average Rating:5.70 Overall Rank:5829] [Average Rating:5.70 Unranked]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
The ranking for many, many, many games is always in flux. The reason is, as a game receives more ratings, it will bring it closer to its actual “stable” position.
Let’s look at our previous example Gloomhaven. As it will receive more ratings, it will get its stable position in the ranking, with a range of possibility, depending if the game is as good as the original hype said, or if it will flop with more users.



You can see that each game’s destiny (on each side of the good or bad) is in flux toward stability within the BGG ranking, but how fast it will get stable only relies on the number of ratings.

Let’s look at the same idea but with a different kind of visualization: map out the difference between the User ratings and the BGG rating, according to their number of ratings. The departure from the two averages scatters in an interesting way (on a log scale here)



Elegant. To the point. What else to say?
20 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
7. Board Game: The Sky's The Limit [Average Rating:7.50 Unranked]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
One of the problem with the current BGG rating, is that it brings most games to an “acute” normal distribution, where 70% of all games are within 5.5 and 6.5, to the point where the rating is irrelevant for the actual ranking. See why:



While the User averages are actually following a perfectly good looking normal distribution



And we can see it back to the original graph



And this is the major problem with this system. After the rank 2000, your position in the ranking is irrelevant. You can be ranked 2,000 with a BGG rating of 5.9, and ranked 10,000 with a BGG rating of 5.5. The vast majority of the games are in this zone, which means, that some perfectly good games are lost in the noise, but because they did not received enough vote, they will never receive the love they deserve due to the lack of exposure…

Another way to look at it is to compare two games with a similar ranking, but with two completely different potentials.



Here, we show that a game for which the User rating and the BGG rating are similar (for example: Aliens vs. Predator), probably won’t move much more from their current position, because most ratings legitimized the position.

However, some other games (for example: By Fire and Sword) for which the user rating is very far from the BGG rating (therefore beloved by a small group of users, and not rated by others) may be stuck in this position forever, despite the original enthusiasm about the game.

You could do the same comparison for thousands of games: Same ranking, but two different quality games… In the end, there is a lot of noise that blurs the signal and many gems are buried in the mud…

24 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
8. Board Game: Winner's Circle [Average Rating:6.96 Overall Rank:619]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
But each ranking system has its winner, and this is all the value of the current BGG ranking: popular games are, well, popular. The system put the games that have received the most positive ratings on top. Therefore, it is a popular vote, and in the end, it’s what matter the most: the games at the top, are the most appreciated by most of the users on this website.

Let’s look at the breakdown of the number of votes according to the BGG ranking:



This is quite clear and we can recognize some outliers (good or bad), like munchkin, Risk, Monopoly and others for historical reasons that we don’t need to discuss here.

But let’s look at it with a log scale so we actually can see things.



We will appreciate the lack of games with less than 30 ratings. And we clearly see that games at both end of the spectrum received the most ratings, (you love it, or you hate it)…

Let’s transform the data with a moving average…



This is pretty much the “finale” of my demonstration. The number of ratings (within a log-margin of error) will determine the game popularity, and therefore its place in the top BGG ranking.

Edit: this is the same data, but just focusing on the top 500 (as discussed in comments)


here is the distribution of ratings for the top 500 only




but at this level, the trend is only visible on the log scale, which is much more appropriate here to buffer outliers.

We are looking for trends here, not exceptions.



I think that overal, it is still a popular vote, even for the top 500 games. In general, the more vote, the more popular a game is (not looking at outliers, again)

On the top 200, about 10% of the games are outliers, which is a lot, as 1000rpm mentioned. However, 90 of the games create a nice trend, so I think the popular vote is still important on the top 200.


23 
 Thumb up
6.00
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
9. Board Game: Helix [Average Rating:5.00 Unranked]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
Let’s finish with a correlation between BGG Rating vs User Rating



We can see that the distribution is restricted by two axes:
1-the perfect 1:1 ratio for BGG=User
2-the 5.5 dummy vote effect.



And this graph summarizes the position of most games and how we feel about it:

36 
 Thumb up
6.25
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
10. Board Game: Where shall we go? [Average Rating:4.00 Unranked]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
Well, with all of these great games buried in the BGG ranking, there is a need to show them some love: more ratings are needed and it is up to you to raise awareness on some of these gems, instead of just focusing on the hotness. Revisiting some (not so) oldies, bringing them a new life, and place them in a stable position that actually reflect their quality.

In my humble opinion, the ranking would be more relevant if it would not place 80% of all ranked games in a bug mushy blur (games with a BGG rating between 6.5 and 5.5) , so that the distribution would look like this:



However, I have no clue on how to tweak the formula to get such result…


Another thought: in order to shine some light on some buried gems, we could keep the exact actual ranking system, but on a different page, create a second league only for games with fewer than 500 ratings (or any other arbitrary cutoff).



As soon as a game receives more than 500 votes, it would leave the second league, so that other games can received some exposure, more votes…etc.

It would allow many games to bring them to a legitimate position within the BGG ranking system as it would reach this critical mass of 500 ratings (as explained above)
64 
 Thumb up
0.60
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
11. Board Game: What's Missing? [Average Rating:4.94 Unranked]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
If there is an analysis that I could have done, and that I can do, please feel free to suggest. I will try to look at the data and see if I can do it.
5 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
12. Board Game: Shear Panic [Average Rating:6.40 Overall Rank:1439]
tom-le-termite
United States
ft lauderdale
Florida
flag msg tools
badge
I think we're going to need a bigger tub.
Avatar
mbmbmbmbmb
After thoughts...

I was mentioning earlier about the Sheeple maker effect of the BGG ranking...

well... I own 18 out of the 20 most rated games.
blush

How about you? (go to the game ranking page and filter by "Num voters"
8 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
Loading... | Locked Hide Show Unlock Lock Comment     View Previous {{limitCount(numprevitems_calculated,commentParams.showcount)}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
{{error.message}}
{{comment.error.message}}
    View More Comments {{limitCount(numnextitems_calculated,commentParams.showcount)}} / {{numnextitems_calculated}} 1 « Pg. {{commentParams.pageid}} » {{data.config.endpage}}
Front Page | Welcome | Contact | Privacy Policy | Terms of Service | Advertise | Support BGG | Feeds RSS
Geekdo, BoardGameGeek, the Geekdo logo, and the BoardGameGeek logo are trademarks of BoardGameGeek, LLC.