Methods people often say – in textbooks, task forces, papers, editorials, over coffee, in their sleep – that we should focus more on estimating effect sizes rather than testing for significance. I am kind of a methods person, and I am kind of going to say the opposite. Only kind of the opposite because it…
[19] Fake Data: Mendel vs. Stapel
Diederik Stapel, Dirk Smeesters, and Lawrence Sanna published psychology papers with fake data. They each faked in their own idiosyncratic way, nevertheless, their data do share something in common. Real data are noisy. Theirs aren't. Gregor Mendel's data also lack noise (yes, famous peas-experimenter Mendel). Moreover, in a mathematical sense, his data are just as…
[18] MTurk vs. The Lab: Either Way We Need Big Samples
Back in May 2012, we were interested in the question of how many participants a typical between-subjects psychology study needs to have an 80% chance to detect a true effect. To answer this, you need to know the effect size for a typical study, which you can’t know from examining the published literature because it…
[17] No-way Interactions
This post shares a shocking and counterintuitive fact about studies looking at interactions where effects are predicted to get smaller (attenuated interactions). I needed a working example and went with Fritz Strack et al.’s (1988, .html) famous paper [933 Google cites], in which participants rated cartoons as funnier if they saw them while holding a…
[15] Citing Prospect Theory
Kahneman and Tversky's (1979) Prospect Theory (.html), with its 9,206 citations, is the most cited article in Econometrica, the prestigious journal in which it appeared. In fact, it is more cited than any article published in any economics journal. [1] Let's break it down by year. To be clear, this figure shows that just in 2013, Prospect Theory got about…
[13] Posterior-Hacking
Many believe that while p-hacking invalidates p-values, it does not invalidate Bayesian inference. Many are wrong. This blog post presents two examples from my new “Posterior-Hacking” (SSRN) paper showing selective reporting invalidates Bayesian inference as much as it invalidates p-values. Example 1. Chronological Rejuvenation experiment In “False-Positive Psychology" (SSRN), Joe, Leif and I run experiments to demonstrate how easy…
[12] Preregistration: Not just for the Empiro-zealots
I recently joined a large group of academics in co-authoring a paper looking at how political science, economics, and psychology are working to increase transparency in scientific publications. Psychology is leading, by the way. Working on that paper (and the figure below) actually changed my mind about something. A couple of years ago, when Joe,…
[11] “Exactly”: The Most Famous Framing Effect Is Robust To Precise Wording
In an intriguing new paper, David Mandel suggests that the most famous demonstration of framing effects – Tversky & Kahneman's (1981) “Asian Disease Problem” – is caused by a linguistic artifact. His paper suggests that eliminating this artifact eliminates, or at least strongly reduces, the framing effect. Does it? This is the perfect sort of paper…
[10] Reviewers are asking for it
Recent past and present The leading empirical psychology journal, Psychological Science, will begin requiring authors to disclose flexibility in data collection and analysis starting on January of 2014 (see editorial). The leading business school journal, Management Science, implemented a similar policy a few months ago. Both policies closely mirror the recommendations we made in our…
[9] Titleogy: Some facts about titles
Naming things is fun. Not sure why, but it is. I have collaborated in the naming of people, cats, papers, a blog, its posts, and in coining the term "p-hacking." All were fun to do. So I thought I would write a Colada on titles. To add color I collected some data. At the end…