Openness initiative for reviewers - an editorial discussion

One of my former mentors, Gert Storms, a KU Leuven professor in psycholinguistics, sent me this information. Caring about scientific scrutiny, he committed to the Peer Reviewers' Openness Initiative. Following up on that commitment, he recently asked for the data pertaining to a manuscript he was asked to review for the APA's Journal of Experimental Psychology: Learning, Memory & Cognition. This resulted in a discussion with the journal's editor, Robert Greene. Greene raised issues against data openness prior to the manuscript being accepted and I understand we need to be careful in this process. However, this is covered within the openness initiative. Please find below the editor's response and Gert Storms' reaction to this.

The bottomline is that the editor asks Gert to resign from the editorial board because of the fundamental disagreement. I find it astonishing that a journal is so rigidly opposed to a measure that should in the end increase/restore the self-correcting nature of science and the quality of its own publication. Comments are open for your perspective ...

Science first, communication second. A prequel to the Food and Brand Lab episode

The Food and Brand Lab at Cornell University, led by Brian Wansink (@BrianWansink) caused quite some academic turmoil these recent weeks. This contrasts sharply with the media turmoil usually generated by this lab's research. A series of academic social media posts and a critical article target the lab's inferior methodology and old school approach to rendering null-effects into (a set of) publishable papers. In this post, I want to give my account of a previous similar situation that I had with the same lab in 2012. I think the take-home-messages of this post are the following. First, I want to claim that the recent controversy fits within a longer tradition that pertains to this lab (but maybe also a large part of their discipline) and this just shows that a science revolution for reproducible findings is not a matter of a few years. Second, I do think that given previous criticism on their work, it is astonishing that it has not improved to some extent. Researchers should be more responsive to how the field reacts to their work. Third, the case seems to demonstrate a troublesome trend where labs and universities tend to invest more in PR than in research methodology and ethics. Although I am an enthusiast of science communication, we should not neglect the order in these words: science-communication. "Science first, communication second"

How credibility spoiled this mini-lecture on statistics

In a recent reply to one of our commentaries (Smits et al., 2015), Domenic Cicchetti gave an interesting mini-lecture on effect sizes, power, and sample sizes. You'll find it here. Though interesting in itself, it is a pity that the full scope of his contribution will largely remain unnoticed given a series of credibility issues with the text itself and his position as the "statistics editor" of Journal of Nervous and Mental Disease. Given that some of my recent research deals with the credibility of online word-of-mouth communication, and because I feel a bit misunderstood as the author of two earlier commentaries to this journal, I will elaborate these credibility issues.

The pitfalls of pre-registration: The Morrison et al CBT paper

There has been quite a fuss about the recently published (online prepublication) Lancet paper on unmedicated cognitive therapy for schizophrenia patients (see below for a reading list of posts about the study). This paper reports on a longitudinal (9 to 18 months) follow-up study of a treatment group and a control group and its basic claim is that the cognitive therapy treatment group fares better than the control group. Though I contributed to a letter to the editor on a study in the same domain that had tremendous reporting flaws, this research is well outside of my discipline. Still, the criticism on the Lancet paper sparkled my interest in it and even for an outsider, there are strange things going on with said manuscript. In this post I will focus on the peculiar timeline of things that shows one of the greatest pitfalls of scientific pre-registration. Though I am greatly in favor of preregistration for a lot of studies, I do think that the mass application of preregistration might lead to new problems. One evident problem is that merely refering to a preregistered study protocol does not imply that the protocol was sensible or that the researchers actually followed it.

The summary
It took quite some investigative academic journalism, I think, but the summary of the study's timeline is presented below. Plain grey timeline moments are based on assumptions following the different pieces of evidence that are available; those with a colored border are based on one of the four public sources.

Timeline - click to enlarge

Don't get all psychotic on this paper: Had I (or we) Been A Reviewer (HIBAR)

Academic publishing sometimes displays a strange parallel with advertising. There's the celebrity endorsement by reknown authors, a credibility inference evoked by the academic journal and the assumed thorough peer review, and the sales talk in the paper's abstract.

Original article with some marketing claims added

As a researcher, we should of course know to read beyond that abstract, but a recent Twitter discussion pointed me towards a paper that has a very blatant discrepancy between its sales talk and the article's actual core. The paper appeared to be hard to understand due to a substandard use of statistics and equally bad reporting of the study's methods. This resulted in a "letter-to-the editor" written by Daniel Lakens (@lakens), Stuart Ritchie (@StuartJRitchie) and Keith Laws (@Keith_Laws). The story behind that Letter-to-the-Editor is interesting in its own sake? Be sure to read about it on Daniel's blog.

Below, you'll find a summary of our most important issues with the original paper and a list of other issues that were not included in the letter to the editor. I guess that for many students (irrespective of their field of interest) it would be a good excercise to go through the paper and find at least some of these errors themselves. Indeed, as I already said in the Twitter discussion, the paper would not pass as a Master's thesis (at least, I hope so). This, of course is troublesome with regard to the paper itself, but also concerning the review process that should be able to reject such a write-up.

Shame on the authors, reviewers and editor for sloppy reporting of data. http://t.co/9n2JXJLrV4 HT @Keith_Laws Wouldn't pass as a MA thesis
— Tim Smits (@TimSmitsTim) January 14, 2014