You can’t un-ring a bell. Once people receive information, even if it is taken back, they cannot disregard it. Teachers cannot imagine what novice students don’t know, juries cannot follow instructions to disregard evidence, negotiators cannot take the perspective of their counterpart who does not know what they know, etc. People exhibit “Outcome bias”, “hindsight…
Category: Discuss own paper
[73] Don't Trust Internal Meta-Analysis
Researchers have increasingly been using internal meta-analysis to summarize the evidence from multiple studies within the same paper. Much of the time, this involves computing the average effect size across the studies, and assessing whether that effect size is significantly different from zero. At first glance, internal meta-analysis seems like a wonderful idea. It increases…
[62] Two-lines: The First Valid Test of U-Shaped Relationships
Can you have too many options in the menu, too many talented soccer players in a national team, or too many examples in an opening sentence? Social scientists often hypothesize u-shaped relationships like these, where the effect of x on y starts positive and becomes negative, or starts negative and becomes positive. Researchers rely almost…
[49] P-Curve Won’t Do Your Laundry, But Will Identify Replicable Findings
In a recent critique, Bruns and Ioannidis (PlosONE 2016 .htm) proposed that p-curve makes mistakes when analyzing studies that have collected field/observational data. They write that in such cases: p-curves based on true effects and p‑curves based on null-effects with p-hacking cannot be reliably distinguished” (abstract). In this post we show, with examples involving sex,…
[45] Ambitious P-Hacking and P-Curve 4.0
In this post, we first consider how plausible it is for researchers to engage in more ambitious p-hacking (i.e., past the nominal significance level of p<.05). Then, we describe how we have modified p-curve (see app 4.0) to deal with this possibility. Ambitious p-hacking is hard. In “False-Positive Psychology” (SSRN), we simulated the consequences of four…
[43] Rain & Happiness: Why Didn’t Schwarz & Clore (1983) ‘Replicate’ ?
In my “Small Telescopes” paper, I introduced a new approach to evaluate replication results (SSRN). Among other examples, I described two studies as having failed to replicate the famous Schwarz and Clore (1983) finding that people report being happier with their lives when asked on sunny days. Figure and text from Small Telescopes paper (SSRN) I…
[30] Trim-and-Fill is Full of It (bias)
Statistically significant findings are much more likely to be published than non-significant ones (no citation necessary). Because overestimated effects are more likely to be statistically significant than are underestimated effects, this means that most published effects are overestimates. Effects are smaller – often much smaller – than the published record suggests. For meta-analysts the gold…