Pre-registration is the best and possibly only solution to p-hacking. Ten years ago, pre-registrations were virtually unheard of in psychology,1 The word "preregistration" first appeared in Psych Science in the January 2014 issue (htm). “Preregistration” does appear in an editorial (of sorts) published in 1990, in the very first issue of Psych Science, but it refers to people preregistering to attend the first APS conference, which was adorably “characterized by one observer as ‘a kind of psychology Woodstock’” (.htm). but they have become increasingly common since then. I was curious just how common they have become, and so I collected some data. This post shares the results.
The data
From the Web of Science I obtained a list of all papers published in 2022 in four top journals that publish psychology experiments: Psych Science, JEP:G, JPSP and JCR.2(1) JEP:G = Journal of Experimental Psychology General, (2) JCR = Journal of Consumer Research, and (3) JPSP = Journal of Personality and Social Psychology Data and code available from ResearchBox.org/2116.
I opened each paper and looked for a preregistration, keeping tabs of whether any study had been pre-registered.3I did the first part of data collection myself, and then was helped by a great undergraduate RA here at ESADE, Aleksandra Lyubarskaja I also kept track of where they were pre-registered. The figure below shows the results. Overall about 43% of papers in the sample had at least one pre-registered study.
That 43% was a bit lower than I expected, but it's a lower bound of sorts as the sample includes papers that could not have included pre-registrations (e.g., review articles & observational studies). Among papers with pre-registerable studies, pre-registration rates are higher than 43%. I expected Psych Science to have more pre-registered articles, and JCR to have fewer than they did.
I did not read any of the pre-registrations in my sample, let alone evaluate or compare them to the published analyses. So I can't comment on their quality. But, in the process of collecting the data, I couldn't help but notice a problem that deserves attention: lousy links.
Lousy Links
About half of the papers linking to an AsPredicted pre-registration link to the anonymous version that is meant only for peer-review. This is not too big a deal, as it mostly means that readers do not learn which of the authors were involved in the pre-registration process. In terms of OSF links, about 17% point to the anonymous version of the project. In addition, papers often link to the OSF project page rather than the pre-registration per se. And finding the pre-registration within that project page is sometimes easy but often difficult (and in a handful of cases I simply could not find it at all).
Suggestions For a Better Future
1. Preregister or justify. It's time for journals to require authors to justify when an experiment is submitted without a pre-registration. Papers without compelling justifications could be desk rejected.
2. Journal checks. Journals should dedicate the 10 minutes per-paper that it takes to ensure that links to pre-registrations actually lead to (the public version of) a pre-registration.
3. The 20 second rule. Authors (and journals) should ask: "How long would it take a reader visiting the paper's repository to find the pre-registration for Study 2 and the data for Study 3"? If the answer is "more than 20 seconds", the paper is not ready for publication.
4. Single links. Toward that aim, I recommend that authors include in their papers a single link to a well organized repository with all files (code, data, materials, and pre-registrations). This is done by default at ResearchBox (an open research platform that is fully integrated with AsPredicted). See screenshots below. Authors posting to the OSF should spend the 20 minutes that it may take to properly organize and label their files.
Since not everyone is familiar with ResearchBox yet, below I include three screenshots of ResearchBox pages for three different published papers. They all look the same, which is why it takes very few seconds to know what's available and where.
Just scroll down to see them.
This is ResearchBox.org/45
This is ResearchBox.org/1237
This is ResearchBox.org/15
