The pitfalls of pre-registration: The Morrison et al CBT paper

There has been quite a fuss about the recently published (online prepublication) Lancet paper on unmedicated cognitive therapy for schizophrenia patients (see below for a reading list of posts about the study). This paper reports on a longitudinal (9 to 18 months) follow-up study of a treatment group and a control group and its basic claim is that the cognitive therapy treatment group fares better than the control group. Though I contributed to a letter to the editor on a study in the same domain that had tremendous reporting flaws, this research is well outside of my discipline. Still, the criticism on the Lancet paper sparkled my interest in it and even for an outsider, there are strange things going on with said manuscript. In this post I will focus on the peculiar timeline of things that shows one of the greatest pitfalls of scientific pre-registration. Though I am greatly in favor of preregistration for a lot of studies, I do think that the mass application of preregistration might lead to new problems. One evident problem is that merely refering to a preregistered study protocol does not imply that the protocol was sensible or that the researchers actually followed it.

The summary
It took quite some investigative academic journalism, I think, but the summary of the study's timeline is presented below. Plain grey timeline moments are based on assumptions following the different pieces of evidence that are available; those with a colored border are based on one of the four public sources.

Timeline - click to enlarge

The facts
There are a few different public sources about this particular manuscript.
First, there is the Lancet article itself. It mentions the study being registered "as an International Standard Randomised Controlled Trial, number 29607432". This implicitly references to a second source, the controlled-trials.com website. The Lancet article also states that the authors "report on all outcomes that were specified in our published protocol and analysis plan". Here, the article references to a third source, namely an article published in Psychosis that contains an extensive rationale for the study, an extensive sampling plan, data about the included cases, and a smaller part on an analysis plan. Note that the quote from the Lancet article does not say they also report on all the planned analyses. This is important because, in fact, they do not follow the analysis plan from said Psychosis paper, though those planned analyses are far superior to the analyses reported in Lancet.
So, the second source was a registered protocol at controlled-trials.com. This is a very short summary of the protocol, includes an approximate start and end date, inclusion criteria and no reference to planned analyses. The first registration of this protocol was eight months after the inclusion of patients already started. Also remarkably, the last changes to this protocol (unclear what was changed, added or deleted) were done even after the online publication of the Lancet article.
The third piece of evidence comes from the article published in Psychosis. This is the first and only piece that refers to planned analyses. This article was first submitted to the journal four months after the inclusion of patients already ended. It seems likely that many included patients already reached the 9-months follow-up at that point in time, or even the final 18-month follow up.
A final piece of evidence comes from the ukcrn.org website which seems to claim that the study inclusion and follow-up already ended in June 2012, but we'll just discard this because that would be too strange, even for these standards.

So what?
Preregistration is a joke if it does not include an analysis plan. That analysis plan marks the distinction between your confirmatory analyses and your exploratory analyses (read the magnificent Nature piece by @ReginaNuzzo about that and other aspects of sensible use of "significant findings").
Refering to preregistration is a joke as well if it (willingly or not) blinds the reader with an illusion of rigorous and planned research while in fact it just employed the same researcher degrees of freedom as non-registered studies do.

So, either we should use a proper registration tool with a timestamp for registration documents (such as, for instance, the open science framework) together with sensible and fool proof procedures that make sure authors transparently report on what they did different compared to that preregistration. Or, we should apply a good extra deal of scrutiny in reviewing and publishing papers that do not refer to preregistration. Whatever we do, I think we should get rid of artefactual preregistrations that are merely ornamental marketing claims alluding to scientific scrutiny.


Further reading
More in depth insights into this line of research have been given by people more knowledgeable about the topic itself. A reading list:
http://keithsneuroblog.blogspot.co.uk/2014/02/my-bloody-valentine-cbt-for-unmedicated.html
http://jcoynester.wordpress.com/2014/03/07/whos-to-blame-for-inaccurate-media-coverage-of-study-of-therapy-for-persons-with-schizophrenia/
http://jcoynester.wordpress.com/2014/02/12/my-response-to-the-hacking-of-this-blog-site/
http://jcoynester.wordpress.com/2014/02/08/offering-us500-wager-to-authors-of-lancet-article-help-me-pick-a-suitable-charity/
http://blogs.plos.org/mindthebrain/2014/02/25/much-ado-little-lancet-study-cognitive-therapy-persons-unmedicated-schizophrenia/
http://blogs.plos.org/mindthebrain/2014/03/11/much-ado-modest-misrepresented-trial-cbt-schizophrenia-part-2/
http://www.thementalelf.net/treatment-and-prevention/medicines/antipsychotics/pilot-study-suggests-that-cbt-may-be-a-viable-alternative-to-antipsychotics-for-people-with-schizophrenia-or-does-it/