In the ninth installment of Data Replicada, we report our attempt to replicate a recently published Journal of Marketing Research (JMR) article entitled, “Advertising a Desired Change: When Process Simulation Fosters (vs. Hinders) Credibility and Persuasion” (.htm).
Some products, such as weight loss programs, exist to help consumers attain a desired change. In this paper, the authors investigate how these products should be advertised. To understand their hypothesis, consider the following advertisements for a weight loss program called MRMDiets:
Both ads show a person’s weight before using the product and after using the product. The one on the left shows a traditional “Before/After” image, but the one on the right shows a fuller progression of weight loss. The authors of this paper hypothesize, and find, that consumers judge Progression ads (like the one on the right) to be more credible than simple Before/After ads.
This JMR paper contains eight studies, and six of those used MTurk participants. We chose to replicate Study 2 (N = 213) because (1) it, along with Study 3, represented the simplest test of the authors’ hypothesis, and (2) it contained stronger statistical evidence (ps < .01) than Study 3 (p = .02). In this study, participants were randomly assigned to see one of three ads for the tMRMDiets weight loss product. Some saw the ad on the left, others saw the ad on the right, and a third group saw three different Before/After exemplars (shown below). The authors found that those who saw the Progression ad judged it to be more credible than those who saw either version of the Before/After ads.
We contacted the authors to request the materials needed to conduct a replication. They were extremely forthcoming, thorough, and polite. They immediately shared the original Qualtrics file that they used to conduct that study, and we used it to conduct our replication. They also promptly answered a few follow-up questions. We are very grateful to them for their help and professionalism.
We ran two identical replications. The first was run on MTurk using the same criteria the original authors specified. The second was run on MTurk using a new feature that screens for only high-quality “CloudResearch Approved Participants” .
In the pre-registered replications, we used the same survey as in the original study, and therefore the same instructions, procedures, images, and questions. These studies did not deviate from the original study in any discernible way, except that our consent form was necessarily different, our exclusion rules were slightly different , and we added an attention check to the end of the survey to help us measure participant quality . After exclusions, we wound up with 761 participants in Replication 1 and 769 participants in Replication 2 (~3.5 times the original sample size). You can access all of our pre-registrations, surveys, materials, data, and code on ResearchBox (.htm).
In each study, participants were asked to “imagine that you have decided to lose some weight.” They then saw one of three ads for a weight loss program called MRMDiets. Specifically, they either saw a Before/After ad, a Before/After (Information Rich) ad, or a Progression ad:
Notice that the Before/After (Information Rich) condition serves as an extra control: it depicts the same number of images as in the Progression condition, without showing weight loss progression.
After viewing the advertisement, participants answered a number of questions designed to assess how credible they found it to be:
The six scale items were averaged to yield a measure of credibility .
Participants then went on to answer a few additional questions, which we will discuss in Footnote 5 below.
Here are the original vs. replication results:
As you can see, we successfully replicated the authors’ key findings.
First, we find highly significant evidence that participants judged the Progression ad to be more credible than the simple Before/After ad. Our effect sizes are also similar. Whereas the original authors found d = .51, we found d = .32 and d = .49 in our first and second replications, respectively.
Second, we also find highly significant evidence that participants judged the Progression ad to be more credible that the version of the Before/After ad that had six images. In this case, our effects were a bit smaller (ds = .26 and .27) than in the original (d = .69), but the effect is demonstrably larger than zero .
In sum, this was an unambiguous success. We of course can’t speak to all of the studies in this paper, but we can confidently conclude that people judge this progression weight loss ad to be more credible than these Before/After ads.
We appreciate the importance of these replication efforts. We are happy to read that both replications were unambiguous successes.
- Specifically, in the first replication, we used MTurkers with at least a 99% approval rating and at least 1,000 HITs completed. In the second replication, we used only “CloudResearch Approved Participants” with at least a 99% approval rating and at least 1,000 HITs completed. We have been doing this since Data Replicada #7 [↩]
- For quality control, we pre-registered to exclude all observations associated with duplicate MTurk IDs or IP Addresses, and to exclude those whose actual MTurk IDs were different than their reported IDs. [↩]
- For our attention check question, we asked participants: "How many siblings do you have. Please use the dropdown menu to answer this question. Actually, to signal that you are paying attention to this survey, please ignore that question and submit 10 as your answer." It turns out that this question was insufficiently sensitive, as only 2% failed it in Replication #1 and 1.8% failed it in Replication #2. It is thus not surprising that none of our results meaningfully change if you exclude those who failed the attention check. [↩]
- The results shown below remain significant if you only analyze the "Not credible/Credible" measure rather than this composite measure. [↩]
- In the original study, the authors also collected measures of “process simulation”, which is the average of three measures that assess whether the ad evokes (1) “The process of losing weight”, (2) “The steps involved in losing weight”, and (3) “The means by which one loses weight”. You can see from this graph that we replicated those results too: