Confirmation Bias and the Stealth Nerf Delusion

From a thread I posted in the forums.

Confirmation Bias. 

It is the tendency of humans to to only consider evidence that supports their own theories, while ignoring or pushing aside evidence that goes against their theories. It is most prominent in the “Drop rates were stealth nerfed” threads.

Consider a person who has an unshakable belief that a fair die is rigged to roll a 1 less often than it should. Now suppose the person conducts 6 tests of 6 rolls.

4 2 5 6 1 3
3 2 1 6 2 5

1 1 6 2 3 4
2 4 1 5 1 3

5 2 4 5 2 6
6 4 2 5 3 2

Statistically, this is far too small of a sample size to indicate anything wrong with the die. In fact, in total, the die rolled a 1 exactly 6/36 times, as expected. A statistician will look at this and conclude that the experiments show no evidence of the die being rigged. However, humans are not fully rational, and most humans are not statisticians, and thus do not think of the roll in a rational way.

Instead, if a person already had the pre-conceived notion that the die was rigged to roll 1’s less often, he would only consider experiments 5 and 6, in which 1’s rolled less often, to be relevant experiments. When you ask him, “What about the experiments 3 and 4?”, he will come up with some explanation such as, “It is just luck” or “That’s just a few cases and it does not indicate that the die is not rigged.”

This applies a double standard because his own good cases, experiments 5 and 6, were likewise based on luck and do not indicate anything about the die.

Stealth Nerf Delusions

Now, replace rolling a 1 with rolling an ilvl 63 rare, with proper adjustment for the actual drop rate percentages.

What goes through the mind of someone who is already assuming that the drop rate was nerfed? Basically, when this person has a “normal” run, such as experiments 1 and 2, he do not notice it. When he gets a “good” run, such as experiments 3 and 4, he dismisses the evidence because it is just randomness and luck. But when he gets a bad run, such as experiments 5 and 6, it suddenly becomes solid evidence of a drop rate nerf.

Just based on those two runs, a biased sub-sample of the data, they believe that the drop rate has been stealth nerfed.

This bias is a self-propagating. The more posts there are claiming drop rate nerfs, the more likely it becomes for someone who was previously skeptical to join in despite the lack of evidence. They start only noticing when they get unlucky streaks and start dismissing lucky or normal runs. When enough people believe in such a claim, it becomes a mass delusion.

Of course, under a delusion, people often become overly defensive about their position even though they have no rational reason to agree with it. When questioned, they start making even more ridiculous claims to support their original claim, akin to making up more lies to cover up the original lie. Only, they aren’t aware that it is a lie and they genuinely believe what they are saying to be the truth.

Randomness and Statistics

Besides confirmation bias, many people just don’t seem to understand what randomness is. People mistakenly think the following statements are true:

  • If the drop rate says 5% ilvl 63, then most of the time in a set of 100 random items, I should find 5 ilvl 63 items.
  • If I don’t get an ilvl 63 drop for a long time, then I am due to get one soon.

The first point is statistically not true. In fact, most of the time (82%), you will find less than or more than 5 ilvl 63 items in 100 runs. The Act 1 drop rates for ilvl 63 were analyzed in this post.

The result is that out of 100 random items from Act 1 Inferno, with a drop rate of 5% (rounded up from 4.8%), the chances to get exactly N ilvl 63 drops is the following:

0 : 0.592%
1 : 3.116%
2 : 8.118%
3 : 13.958%
4 : 17.814%
5 : 18.002% (Expected Value)
6 : 15.001%
7 : 10.603%
8 : 6.487%
9 : 3.490%
10: 1.672%
11: 0.720%

When you scale this up to 100,000 players, there should be hundreds of players (0.592%) who should find zero ilvl 63 drops in a random set of 100 drops in Act 1. Here the selection bias kicks in hard. These people who don’t find any ilvl 63 drops become much more likely to complain on the forums, thus becoming a vocal minority and vastly over-representing their actual distribution (0.592%) and try to make it seem as if they a much larger proportion (50%+) of players.

As far as the second point goes, that an event should be “due” when it hasn’t happened for a while, it is simply the Gambler’s fallacy. When an independently random event does not happen for a while, people erroneously assume that it is bound to happen soon. For example, if a coin has landed 10 Heads in a row, people will erroneously bet significant amounts of money on Tails because the chance to get 11 Heads in a row is tiny. This misses the real issue, as the chance for the next flip to be Tails is still 50-50. The chance to flip Tails does not increase when there is a Heads streak.

Similarly, when you don’t find an ilvl 63 item in a while, that does not mean the next item you find has more than a 4.8% chance to be ilvl 63. It still has the same chance as before.

Final Notes

The truth is often painful. But as life works, ignoring the truth does not make it false, and believing in a lie really hard does not make it true. A bunch of people believing in a lie also does not make it true. Someone might have all the faith in the world that the Earth is flat, and cite all the other people who believe this to be the case, but unfortunately, that does not make his theory correct.


  • Due to confirmation bias, players tend to ignore lucky streaks and normal runs, and only include unlucky streaks as evidence. This is wrong. All evidence should be included.
  • The above confirmation bias builds on itself, leading more and more people to “notice” the same thing, creating a mass delusion.
  • By the binomial distribution, the chance to actually obtain the expected number of ilvl 63 drops (for instance) is very low in any given run.
  • Not getting the expected number does not imply any stealth nerfs.
  • Those who get very unlucky streaks are much more likely to post, and thus over-represent themselves on the forums.
  • Good drops are never “guaranteed” to occur after an unlucky streak. Expecting that a certain item must drop soon is the Gambler’s fallacy.
This entry was posted in Forums and tagged , , , . Bookmark the permalink.

2 Responses to Confirmation Bias and the Stealth Nerf Delusion

  1. SomeNoob says:

    Hi, are you nargaque on the forums?

    I am of the opinion that the economy is everything that is wrong with D3. I suspect the most common complaint about D3 is that “the loot sucks”, but what does “suck” or “good” really mean here? The idea that something “sucks” or “good” is probably relative; an item with +1000 STR would “suck” if every other item gave +10000 STR, but that same item would be “awesome” if every other item gave about +100 STR. So a possible heuristic of determining the “average” item could be to look at the easily purchasable items in the auction house.

    In my opinion, the problem is that the “average” gear in the auction house (which has been shown to be not so average after all) is too cheap with respect to how rarely it drops. If the “average” gear in the auction house is the top 1% of all gear ever found, then each item drop has only a 1% chance of being rewarding when found!

    • Sean Li says:

      Yep, I go by nargaque on most game-related things.

      And your analysis focusing on the relative power of items is spot on. The AH pretty much puts the best, top 1% of items out there for everyone to gaze at, but of course, one only has a 1% chance of finding such an item. For the next tier of items, in the top 0.1%, it would be a 0.1% chance, and so on.

      In real life, this would be like there being 50 Lexus’s and Ferrari’s just lying on the street for you to look at every time you leave your home. So of course you would feel that your own car “sucks” and that you deserve a Lexus/Ferrari, even if the car you have is already pretty great.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s