tag:blogger.com,1999:blog-5125261180654017789.post7270068867241426954..comments2024-03-15T08:27:44.775+01:00Comments on Persuasive mark: A science preregistration revolution? Not yet.Anonymoushttp://www.blogger.com/profile/06408559623511378013noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-5125261180654017789.post-24671885992331524832014-11-11T03:38:49.303+01:002014-11-11T03:38:49.303+01:00Also, published protocols can just be ignored or a...Also, published protocols can just be ignored or abandoned for little to no reason.<br /><br />I wrote this comment elsewhere about another trials that was published in the Lancet, and it seems relevant to this blog post so I thought I'd re-post it here if that’s okay. The full comment and references can be found at: http://www.bmj.com/content/347/bmj.f5963/rr/674255<br /><br />The PACE trial's published protocol defined 'recovery' as requiring an SF-36 Physical Functioning (SF36-PF) questionnaire score of at least 85 out of 100, while the trial's entry criteria required a score of 65 or under, which was taken to indicate that patients' fatigue was disabling[2]. The post-hoc criteria for recovery allowed patients with an SF36-PF score of 60 to be classed as recovered. This change was justified by the claim that a threshold of 85 would mean “approximately half the general working age population would fall outside the normal range.”[3] In fact, the data cited showed that the median score for the working age population was 100, less than 18% of the general working age population had a score under 85, and 15% had declared a long-term health problem[4,5].<br /><br />An SF36-PF score of 60 was claimed in the Lancet PACE paper to be the mean -1sd of the working age population, and thus a suitable threshold for ‘normal’ disability[6]. They had in fact used data which included all those aged over 65, reducing the mean physical function score and increasing the SD[4]. For the working age population the mean -1sd was over 70, requiring patients to score at least 75 to fall within this ‘normal range’[5]. Also, the trial's protocol makes it clear that the thresholds for recovery (including ≥85 for SF-36 PF) were intended to be more demanding than those for the mean -1sd, reporting that: “A score of 70 is about one standard deviation below the mean... for the UK adult population”[2].<br /><br />The post-hoc criteria for recovery so clearly overlapped with the trial's own criteria for severe and disabling fatigue that an additional element came into play, mandating that ‘recovered’ patients not also fulfil every aspect of the trial's criteria for CFS[3]. Even so, patients could still have been classed as recovered when reporting no change, or even a decline, in either one of the trial’s primary outcomes.<br /><br />Even using the loose post-hoc criteria for recovery, only 22% of patients were classed as recovered following treatment with specialist medical care and additional CBT or GET[3]. Regardless, the BMJ had reported that PACE showed CBT and GET “cured” 30% and 28% of patients respectively[7], a Lancet commentary claimed that about 30% recovered using a “strict criterion” for recovery[8], and a paper aimed at NHS commissioners stated PACE indicated a recovery rate of 30-40% for CBT and GET[9,10]. It is wrong for such misstatements of fact to be allowed to go on affecting how doctors treat their patients, how funding decisions are made, and the information that patients are provided with before deciding whether to consent to particular interventions.<br /><br />The changes to the outcome measures used in the PACE trial may not be “representative of a hidden agenda”[1], but they were misguided, justified by inaccurate claims, and have been misleading to others. The refusal to allow patients access to data on the outcome measures laid out in the trial’s protocol reflects a sad dismissal of their right to be informed about the medical treatments they are being encouraged to pursue[11,12,13].<br /><br />Dan Clarkenoreply@blogger.comtag:blogger.com,1999:blog-5125261180654017789.post-59060373158746120742014-05-21T20:28:55.369+02:002014-05-21T20:28:55.369+02:00Hi Chris, thanks for your comment. I love the arti...Hi Chris, thanks for your comment. I love the article you linked (and its graph that visualizes p-hacking and harking in the science process)!<br />Still, I am a bit pessimistic about the odds of canceling out these practices in registered reports. Imagine a world in which registered reports have scaled to a major publishing format. The editorial task of checking both the registration and the final manuscript will be very time consuming. Thus, an author could quite possibly still sell something as a conceptually preregistered analysis. Careful wording could draw a fine line between exploratory versus confirmatory analyses. One way to deal with this is the demand a particular disclaimer or phrase.<br />We should do this, I believe, because practices in Medicine proved us that authors will keep using the degrees of freedom they have at their disposal. This was already documented in 2004 (http://jama.jamanetwork.com/article.aspx?articleid=198809) and my recent experience with the Lancet paper proves that a decade later it is still true. Anonymoushttps://www.blogger.com/profile/06408559623511378013noreply@blogger.comtag:blogger.com,1999:blog-5125261180654017789.post-10368995452594143842014-05-21T08:35:06.067+02:002014-05-21T08:35:06.067+02:00Hi Tim, fair comment. I completely agree that extr...Hi Tim, fair comment. I completely agree that extremely vague (or post-dated) pre-registration is useless - and potentially worse than useless because it creates an illusion of credibility without actually earning it. But that's not a concern for journal-based pre-registration (i.e. Registered Reports) because the protocol is peer reviewed and there is editorial continuity between the protocol and final submission. This mechanism can work just as well for novel research as for replications. I'm sure you're already aware of this, but for readers of your blog, we wrote a detailed piece on it here: http://orca.cf.ac.uk/59475/1/AN2.pdfChris Chambershttps://www.blogger.com/profile/10437328364681252945noreply@blogger.com