In an intriguing new paper, David Mandel suggests that the most famous demonstration of framing effects – Tversky & Kahneman's (1981) “Asian Disease Problem” – is caused by a linguistic artifact. His paper suggests that eliminating this artifact eliminates, or at least strongly reduces, the framing effect. Does it?
This is the perfect sort of paper for a replication: The original finding is foundational and the criticism is both novel and fundamental. We read Mandel’s paper because we care about the topic, and we replicated it because we cared about the outcome.
The Asian Disease Problem
Imagine that an Asian disease is expected to kill 600 people and you have to decide between two policies designed to combat the disease. The policies can be framed in terms of gains (people being saved) or in terms of losses (people dying).
Tversky and Kahneman found that whereas 72% of people given the gain frame chose Program A’s 200 of 600 certain lives saved, only 22% of people given the loss frame chose Program C’s 400 of 600 certain deaths. This result supports prospect theory: people will take risks to avoid losses but will avoid risks to protect gains.
Just a Linguistic Artifact?
David Mandel argues that when people read “200 people will be saved” or “400 people will die” they interpret it as “at least 200 people will be saved” or "at least 400 people will die”. That small difference in interpretation would switch the finding from irrational to sensible, as the certain option would potentially save more people in the gain frame (>200 of 600 will be saved) than in the loss frame (>400 of 600 will die). Mandel resolves the ambiguity by adding the word “exactly” (“exactly 200 people will be saved”). Because the word “exactly” makes it clear that no more than 200 will be saved, he predicts that including that word will eliminate the framing effect.
In Study 2 (the study most faithful to the original), Mandel used Tversky and Kahneman’s wording and replicated their result (58% chose to save 200 people for certain; 26% chose to kill 400 people for certain). When he added “exactly,” that difference was reduced (59% vs. 43%). [1]
We replicated Mandel’s procedure. We showed mTurk workers the same scenario and asked the same questions. We collected ~2.5 times Mandel’s sample size; Mandel had ~38 per cell and we had ~98 per cell. (Following Mandel, we also included conditions with “at least” as a modifier; here are those results).
Unlike Mandel, we found a strong framing effect even with the use of the word “exactly” (p<.001) (materials; data):
For completeness, we should report that Mandel emphasized a different dependent variable – a continuous measure of preference. We measured that too and it also failed to replicate his result.
In sum, our replication suggests that Tversky and Kahneman’s (1981) framing effect is not caused by this linguistic artifact.
We Could Have Just Asked Uri
When we told Uri about all this, he told us that he conducts this experiment in his class each year and that he uses the word “exactly” in his materials. The experiment has replicated every single year. For example, in the past two years combined (N=250), he observed that 63% chose to save 200 lives for sure whereas only 25% chose to let 400 die for sure.
David Mandel Responds
As is our policy, we sent a draft of this post to David Mandel to offer him the chance to respond. Please check out David's response below. We are very thankful he took the time to do this:
I welcome this replication experiment by Joseph Simmons and Leif Nelson. I think we all agree proper replications are important regardless of how the results turn out. They have kindly offered me 150 words to reply, but that would hardly get me started. There are many points to cover, both about the replication and some of the broader issues it sparks. Joe contacted me on the Friday before the post went live with the “unwelcome news” and it’s been a weekend of changed plans, but I wanted to have a reply ready when the post goes live. Here it is on my website [and here it as a pdf]. I hope you read both. Feel free to email me if you have comments. Lastly, Joe invited me to comment on their post, but I haven’t since I cover what I might have otherwise recommended they alter in my reply. Readers can make up their own minds. 151, 152…
- However, compared to the condition with the original wording, this reduction was nonsignificant (p=.293). [↩]