An impressive team of researchers is engaging in an impressive task: Replicate 21 social science experiments published in Nature and Science in 2010-2015 (.htm). The task requires making many difficult decisions, including what sample sizes to use. The authors' current plan is a simple rule: Set n for the replication so that it would have 90%…
Category: Statistical Power
[26] What If Games Were Shorter?
The smaller your sample, the less likely your evidence is to reveal the truth. You might already know this, but most people don’t (.html), or at least they don’t appropriately apply it (.html). (See, for example, nearly every inference ever made by anyone). My experience trying to teach this concept suggests that it’s best understood…
[18] MTurk vs. The Lab: Either Way We Need Big Samples
Back in May 2012, we were interested in the question of how many participants a typical between-subjects psychology study needs to have an 80% chance to detect a true effect. To answer this, you need to know the effect size for a typical study, which you can’t know from examining the published literature because it…
[6] Samples Can't Be Too Large
Reviewers, and even associate editors, sometimes criticize studies for being “overpowered” – that is, for having sample sizes that are too large. (Recently, the between-subjects sample sizes under attack were about 50-60 per cell, just a little larger than you need to have an 80% chance to detect that men weigh more than women). This…