An experiment on confirmation bias

This post summarizes the article “Confirmation bias with motivated beliefs”, by Charness and Dave, published in Games and Economic Behavior in 2017¹.

Confirmation bias (CB) can be defined as an agent’s tendency to seek, interpret and use evidence in a manner biased toward confirming her existing beliefs or hypotheses. This constitutes a misjudgment that limits the individual’s ability to learn (Rabin and Schrag, 1999 ²), induces changing in beliefs to justify past actions (Yariv, 2005 ³), and may result in an increasing polarization of beliefs among a population (Wilson, 2014 ⁴). The implications for real life situations are plentiful: excess volatility and momentum trading in the stock market, perpetuation of stereotypes, and inaccuracy of diagnosis, among many others.

One important question is, then, what kind of environments make individuals integrate the new information without falling for a confirmation bias. To this end, Charness and Dave conduct an experiment in which rational learning should follow Bayesian updating, measure the confirmation bias as the distance from this Bayesian update, and compare different measurements in different scenarios to draw conclusions.

The experiment is as follows:

• There are two urns, the “more black” (MB) urn contains 7 black and 3 white balls, while the “more white” (MW) urns contains 3 black and 7 white balls.
• One urn is chosen at random. If it happens to be the MB urn, experimental players play game MB game. If the MW urn is chosen, they play the MW game. Both are simple games. If players play correctly, the Odd player will get 20 tokens, while the Even player will win 25 in the MB game and 30 in the MW game. The role of the player (Odd or Even) is determined in advance, and is known to the experimental subject.
• Previous to the disclosure of the information about the urn, a ball will be extracted, shown to the players, and replaced back in the urn. This is done six times before the contents of the urn are shown. Players are asked to estimate the type of urn after one of the urns is randomly chosen, and also the probability after each extraction of a ball. Better estimates are rewarded with more tokens.
• Players play this game with an anonymous opponent for 10 times, each time with a different opponent.

In this experiment, there are 50-50 prior probabilities that the urn is of either type, and there is a correct way to update these probabilities after each ball extraction, according to Bayes formula. Deviations from these probabilities provide a measure of a possible confirmation bias. Finally, the authors present the hypothesis that, relative to the Even players, Odd players should show a higher confirmation bias in the sense that they would not use the information of the color of the extracted balls efficiently to update their estimate of the urn to be MW or MB, and would estimate updated probabilities too close to their prior predictions. According to the hypothesis, Odd players should not care about the type of urn, as their equilibrium payoffs do not depend on the type of urn, whereas the payoff of the Even player does, and thus have a greater interest in paying attention to the probability updates.

The findings of the experiment are explained by the authors:

One can imagine at least three possible outcomes ex ante. First, there is the null hypothesis that the background environment has no effect on updating behavior across experimental conditions. Second, the strategic element in the Strategic condition could serve to exacerbate updating errors for the interested party, particularly when the observed draws favor her preferred state of the world having been realized. Third, this additional consideration might serve to ameliorate updating errors for the Even role in the Strategic condition, perhaps because this causes her to pay more attention to the task at hand or perhaps because the CB helps to counteract other decision-making biases.

In fact, we find that players in the Even role are less susceptible to confirmation bias. We find a considerable degree of conservatism (under-updating) for players in both roles and in both conditions; people rarely update sufficiently on the basis of observed draws. People do guess close to 50 percent in the easy case when the same numbers of black and white balls have been drawn. However, we find that the average belief is closer to 50 percent than the Bayesian prediction for all other combination of draws; furthermore, our regression results indicate that the tendency to under-update is greater for Odd players than for Even players.

Since an individual suffering from CB would be expected to give a confirmatory signal relatively more weight than would a Bayesian, one might see this improvement as a simple consequence. However, the effect on updating occurs both when the preferred state of the world (which for Even players is that more white balls have been drawn) is likely and when it is unlikely. Our reading of the situation is that a person who has a vested interest in the underlying state would be emotionally prone to (if anything) give less weight to an unhappy signal that confirms the unfavorable state; in this case, one would expect to see a disconfirmation bias, or at least less of a confirmation bias. This suggests that the effect may be driven by factors such as increased attention to the updating problem or the complexity of the background environment.

One aspect of the experiment is the games individuals play after the updates. An alternative specification would directly give a payoff to the players. The reason to choose a game was to force them to pay more attention to the states of nature (the type of the urn). The games are very simple and indeed subjects play the equilibrium strategies most of the times (about 85 percent for odd players, and 90 percent for even players). To check this feature of the experimental design, the authors conduct an experiment with the alternative specification, and find that Even players seem less conservative than when end states are associated with games. However, these Even players are more affected by the confirmation-bias in the non-strategic condition. This suggests that beliefs that are motivated by end states being games translates into less of a CB due to heightened attention being paid by people, while the level of conservatism may not be that different.

Finally, another experiment was conducted with different final games to make sure that there was nothing special in those games that affected the behavior of Even and Odd players. No substantial differences were found with the new games.

The authors conclude:

It is perhaps surprising that such a strong effect on behavior resulted from our rather modest manipulation for motivating beliefs. The only difference in our design is that one set of players has slightly higher equilibrium payoffs in one state of the world than the other, while the other set of players do not. And yet this difference led to a substantial and significant difference in updating. It does in fact appear that people paid more attention when they cared about the underlying state of the world.