There is a recent QJE paper reporting a LinkedIn audit study comparing responses to requests by Black vs White young males. I loved the paper. At every turn you come across a clever, effortful, and effective solution to a challenge posed by studying discrimination in a field experiment. But, no paper is perfect, and this…
Category: Discuss Paper by Others
[127] Meaningless Means #4: Correcting Scientific Misinformation
Before we got distracted by things like being sued, we had been working on a series called Meaningless Means, which exposed the fact that meta-analytic averaging is (really) bad. When a meta-analysis says something like, “The average effect of mindsets on academic performance is d = .32”, you should not take it at face value….
[125] "Complexity" 2: Don't be mean to the median
In Colada[124] I summarized a co-authored critique (with Banki, Walatka and Wu) of a recent AER paper that proposed risk preferences reflect 'complexity' rather than preferences a-la Prospect Theory. Ryan Oprea, the AER author, has written a rejoinder (.pdf). Its first main point (pages 5-12), is that our results with medians are 'knife edge' (p.8),…
[124] "Complexity": 75% of participants missed comprehension questions in AER paper critiquing Prospect Theory
Kahneman and Tversky’s (1979) “Prospect Theory” article is the most cited paper in the history of economics, and it won Kahneman the Nobel Prize in 2002. Among other things, it predicts that people are risk seeking for unlikely gains (e.g., they pay more than $1 for a 1% chance of $100) but risk averse for…
[122] Arresting Flexibility: A QJE field experiment on police behavior with about 40 outcome variables
A forthcoming paper in the Quarterly Journal of Economics (QJE), "A Cognitive View of Policing" (htm), reports results from a field experiment showing that teaching police officers to "consider different ways of interpreting situations they encounter" led to "reductions in use of force, [and] discretionary arrests" (abstract). In this post I explain why, having spent…
[121] Dear Political Scientists: Don't Bin, GAM Instead
There is a 2019 paper, in the journal Political Analysis (htm), with over 1000 Google cites, titled "How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice". The paper is not just widely cited, but is also actually influential. Most political science papers estimating interactions now-a-days, seem to…
[119] A Hidden Confound in a Psych Methods Pre‑registrations Critique
A forthcoming paper in Psych Methods (.pdf) had a set of coders evaluate 300 pre-registrations in terms of how informative they were about several study attributes (e.g., hypotheses, analysis, DVs). The authors analyzed the subjective codings and concluded that many pre-registrations in psychology, especially those relying on the AsPredicted template, provide insufficient information., Central to…
[101] Transparency Makes Research Evaluable: Evaluating a Field Experiment on Crime Published in Nature
A recently published Nature paper (.htm) examined an interesting psychological hypothesis and applied it to a policy relevant question. The authors ran an ambitious field experiment and posted all their data, code, and materials. They also were transparent in showing the results of many different analyses, including some that yielded non-significant results. This is in…
[99] Hyping Fisher: The Most Cited 2019 QJE Paper Relied on an Outdated Stata Default to Conclude Regression p-values Are Inadequate
The paper titled "Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results" (.htm) is currently the most cited 2019 article in the Quarterly Journal of Economics (372 Google cites). It delivers bad news to economists running experiments: their p-values are wrong. To get correct p-values, the article explains, they need to…
[98] Evidence of Fraud in an Influential Field Experiment About Dishonesty
This post is co-authored with a team of researchers who have chosen to remain anonymous. They uncovered most of the evidence reported in this post. These researchers are not connected in any way to the papers described herein. *** In 2012, Shu, Mazar, Gino, Ariely, and Bazerman published a three-study paper in PNAS (.htm) reporting…