A number of authors have recently proposed that (i) psychological research is highly unpredictable, with identical studies obtaining surprisingly different results, (ii) the presence of heterogeneity decreases the replicability of psychological findings. In this post we provide evidence that contradicts both propositions. Consider these quotes: "heterogeneity persists, and to a reasonable degree, even in […]…
Category: Meta Analysis
[63] "Many Labs" Overestimated The Importance of Hidden Moderators
Are hidden moderators a thing? Do experiments intended to be identical lead to inexplicably different results? Back in 2014, the "Many Labs" project (.pdf) reported an ambitious attempt to answer these questions. More than 30 different labs ran the same set of studies and the paper presented the results side-by-side. They did not find any…
[58] The Funnel Plot is Invalid Because of This Crazy Assumption: r(n,d)=0
The funnel plot is a beloved meta-analysis tool. It is typically used to answer the question of whether a set of studies exhibits publication bias. That's a bad question because we always know the answer: it is "obviously yes." Some researchers publish some null findings, but nobody publishes them all. It is also a bad…
[33] "The" Effect Size Does Not Exist
Consider the robust phenomenon of anchoring, where people's numerical estimates are biased towards arbitrary starting points. What does it mean to say "the" effect size of anchoring? It surely depends on moderators like domain of the estimate, expertise, and perceived informativeness of the anchor. Alright, how about "the average" effect-size of anchoring? That's simple enough….