A science preregistration revolution? Not yet.

Within a short span of only 24 hours or so, I received two opposing types of news concerning preregistered science. Its basic idea is that with good preregistration of researchers' intents, science would be more objective by reducing researcher degrees of freedom (e.g., HARKing: Hypothesising After Results are Known; p-hacking: using different analyses, data, variables until the hypothesized effect is found).

• Yesterday, May 19 2014, the journal Social Psychology published an issue stacked with preregistered replication studies. Not all of these replications were (fully) successful, including our own attempt to replicate the idea that a single exposure to music and an object would transfer your attitude about the music to that object (i.e., Gorn’s 1982 concept of single exposure musical conditioning). If you want to read more about this issue and the preregistration idea in (social) psychology, I would suggest Chris Chambers’ blog in the Guardian. For a more critical voice, John Bohannon expressed the idea (or fear) that preregistration relates to academic bullying.

Though I participated in the preregistration replication effort for Social Psychology I am still in doubt about the exact status of preregistration. I believe it is a good and even necessary tool for honest replication attempts. I am not sure whether we should use it for all of our research, as long as we are sure to refrain from HARKing our exploratory analyses. In any case, I think we should move on in honest reporting. Preregistration will only foster scientific integrity if we link it with specific reporting guidelines. Sure, a researcher can also do exploratory analyses that were not preregistered. But we should not conceal them with persuasive wording suggesting that all was planned (cf. HARKing, p-hacking)

"Preregistration will only foster scientific integrity if we link it with specific reporting guidelines"

Sadly, this is not an opinion everybody shares…

• The Lancet finally gave me an editorial decision for the Letter I submitted, addressing precisely the lack of good preregistration and the persuasive framing used by authors to cover up their HARKing. See an earlier blogpost for the (astonishing) details of this. Lancet decided not to publish the submitted commentary. Of course I can understand that all kinds of editorial choices have to be made, but I did not receive a true explanation for the rejection. I sure hope they do acknowledge the seriousness of the issue I tried to address.  

Above, I already talked about HARKing and p-hacking as a serious threat to science integrity. For the non-informed: compare them to a sports game where the rules change. Claiming a soccer victory not because you scored more goals but because you had higher ball possession percentages increases your degrees of freedom to a non-acceptable extent. Your degrees of freedom are limited meaning that when one team wins the game, we understand that this could be due to different facts but at least their is something common to all wins, namely scoring more goals. In (empirical, quantitative) science, an implicit set of (bad) rules has long been dominating "the game". Problem is that science does not pretend to be a mere game. So we should have good rules on interpretation of findings. The problem with ill-executed preregistration of research is that researchers can start pretending to follow the rule book while, in fact, they don't.

That is a serious threat to the preregistration idea – as demonstrated in the Morrison article that I discussed in that commentary that was rejected by Lancet. Such ill-suited practice conceals the researcher degrees of freedom and concealing is worse than not caring about it, I believe.

Submitted, but rejected commentary

References for the commentary:

1 Morrison AP, Turkington D, Pyle M, Spencer H, Brabban A, Dunn G, Christodoulides T, Dudley R, Chapman N, Callcott P, Grace T, Lumley V, Drage L, Tully S, Irving K, Cummings A, Byrne R, Davies LM, Hutton P. Cognitive therapy for people with schizofrenia spectrum disorders not taking antipsychotic drugs: a single-blind randomised controlled trial. Lancet 2014 

2 http://www.controlled-trials.com/ISRCTN29607432/morrison

3 Morrison AP, Wardle M, Hutton P, Davies L, Dunn G, Brabban A, Byrne R, Drage L, Spencer H, Turkington D. Assessing Cognitive Therapy Instead Of Neuroleptics: Rationale, study design and sample characteristics of the ACTION trial. Psychosis 2013; 5(1): 82-92.

4 Schulz KF, Altman DG, Moher D. Protocols, probity, and publication. Lancet 2009; 373: 1524.

5 Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, Michie S, Moher D, Wager E. reducing waste from incomplete or unusable reports of biomedical research. Lancet 2014; 383: 267-76.