A forthcoming paper in the Quarterly Journal of Economics (QJE), "A Cognitive View of Policing" (htm), reports results from a field experiment showing that teaching police officers to "consider different ways of interpreting situations they encounter" led to "reductions in use of force, [and] discretionary arrests" (abstract). In this post I explain why, having spent…
Category: Preregistration
[119] A Hidden Confound in a Psych Methods Pre‑registrations Critique
A forthcoming paper in Psych Methods (.pdf) had a set of coders evaluate 300 pre-registrations in terms of how informative they were about several study attributes (e.g., hypotheses, analysis, DVs). The authors analyzed the subjective codings and concluded that many pre-registrations in psychology, especially those relying on the AsPredicted template, provide insufficient information., Central to…
[115] Preregistration Prevalence
Pre-registration is the best and possibly only solution to p-hacking. Ten years ago, pre-registrations were virtually unheard of in psychology, but they have become increasingly common since then. I was curious just how common they have become, and so I collected some data. This post shares the results. The data From the Web of Science…
[101] Transparency Makes Research Evaluable: Evaluating a Field Experiment on Crime Published in Nature
A recently published Nature paper (.htm) examined an interesting psychological hypothesis and applied it to a policy relevant question. The authors ran an ambitious field experiment and posted all their data, code, and materials. They also were transparent in showing the results of many different analyses, including some that yielded non-significant results. This is in…
[64] How To Properly Preregister A Study
P-hacking, the selective reporting of statistically significant analyses, continues to threaten the integrity of our discipline. P-hacking is inevitable whenever (1) a researcher hopes to find evidence for a particular result, (2) there is ambiguity about how exactly to analyze the data, and (3) the researcher does not perfectly plan out his/her analysis in advance….
[55] The file-drawer problem is unfixable, and that’s OK
The “file-drawer problem” consists of researchers not publishing their p>.05 studies (Rosenthal 1979 .htm). P-hacking consist of researchers not reporting their p>.05 analyses for a given study. P-hacking is easy to stop. File-drawering nearly impossible. Fortunately, while p-hacking is a real problem, file-drawering is not. Consequences of p-hacking vs file-drawering With p-hacking it’s easy to…
[44] AsPredicted: Pre-registration Made Easy
Pre-registering a study consists of leaving a written record of how it will be conducted and analyzed. Very few researchers currently pre-register their studies. Maybe it’s because pre-registering is annoying. Maybe it's because researchers don't want to tie their own hands. Or maybe it's because researchers see no benefit to pre-registering. This post addresses these…
[12] Preregistration: Not just for the Empiro-zealots
I recently joined a large group of academics in co-authoring a paper looking at how political science, economics, and psychology are working to increase transparency in scientific publications. Psychology is leading, by the way. Working on that paper (and the figure below) actually changed my mind about something. A couple of years ago, when Joe,…