I recently joined a large group of academics in co-authoring a paper looking at how political science, economics, and psychology are working to increase transparency in scientific publications. Psychology is leading, by the way. Working on that paper (and the figure below) actually changed my mind about something. A couple of years ago, when Joe,…
[11] “Exactly”: The Most Famous Framing Effect Is Robust To Precise Wording
In an intriguing new paper, David Mandel suggests that the most famous demonstration of framing effects – Tversky & Kahneman's (1981) “Asian Disease Problem” – is caused by a linguistic artifact. His paper suggests that eliminating this artifact eliminates, or at least strongly reduces, the framing effect. Does it? This is the perfect sort of paper…
[10] Reviewers are asking for it
Recent past and present The leading empirical psychology journal, Psychological Science, will begin requiring authors to disclose flexibility in data collection and analysis starting on January of 2014 (see editorial). The leading business school journal, Management Science, implemented a similar policy a few months ago. Both policies closely mirror the recommendations we made in our…
[9] Titleogy: Some facts about titles
Naming things is fun. Not sure why, but it is. I have collaborated in the naming of people, cats, papers, a blog, its posts, and in coining the term "p-hacking." All were fun to do. So I thought I would write a Colada on titles. To add color I collected some data. At the end…
[8] Adventures in the Assessment of Animal Speed and Morality
In surveys, most people answer most questions. That is true regardless of whether or not questions are coherently constructed and reasonably articulated. That means that absurd questions still receive answers, and in part because humans are similar to one another, those answers can even look peculiarly consistent. I asked an absurd question and was rewarded…
[7] Forthcoming in the American Economic Review: A Misdiagnosed Failure-to-Replicate
In the paper “One Swallow Doesn't Make A Summer: New Evidence on Anchoring Effects”, forthcoming in the AER, Maniadis, Tufano and List attempted to replicate a classic study in economics. The results were entirely consistent with the original and yet they interpreted them as a “failure to replicate.” What went wrong? This post answers that…
[6] Samples Can't Be Too Large
Reviewers, and even associate editors, sometimes criticize studies for being “overpowered” – that is, for having sample sizes that are too large. (Recently, the between-subjects sample sizes under attack were about 50-60 per cell, just a little larger than you need to have an 80% chance to detect that men weigh more than women). This…
[5] The Consistency of Random Numbers
What’s your favorite number between 1 and 100? Now, think of a random number between 1 and 100. My goal for this post is to compare those two responses. Number preferences feel random. They aren’t. “Random” numbers also feel random. Those aren’t random either. I collected some data, found a pair of austere academic papers,…
[4] The Folly of Powering Replications Based on Observed Effect Size
It is common for researchers running replications to set their sample size assuming the effect size the original researchers got is correct. So if the original study found an effect-size of d=.73, the replicator assumes the true effect is d=.73, and sets sample size so as to have 90% chance, say, of getting a significant…
[3] A New Way To Increase Charitable Donations: Does It Replicate?
A new paper finds that people will donate more money to help 20 people if you first ask them how much they would donate to help 1 person. This Unit Asking Effect (Hsee, Zhang, Lu, & Xu, 2013, Psychological Science) emerges because donors are naturally insensitive to the number of individuals needing help. For example,…