Openness initiative for reviewers - an editorial discussion

One of my former mentors, Gert Storms, a KU Leuven professor in psycholinguistics, sent me this information. Caring about scientific scrutiny, he committed to the Peer Reviewers' Openness Initiative. Following up on that commitment, he recently asked for the data pertaining to a manuscript he was asked to review for the APA's Journal of Experimental Psychology: Learning, Memory & Cognition. This resulted in a discussion with the journal's editor, Robert Greene. Greene raised issues against data openness prior to the manuscript being accepted and I understand we need to be careful in this process. However, this is covered within the openness initiative. Please find below the editor's response and Gert Storms' reaction to this.

The bottomline is that the editor asks Gert to resign from the editorial board because of the fundamental disagreement. I find it astonishing that a journal is so rigidly opposed to a measure that should in the end increase/restore the self-correcting nature of science and the quality of its own publication. Comments are open for your perspective ...

Robert Greene's statement:

I understand your viewpoint.  However, given that your policy conflicts with that of the journal, I think that it's best that you step down from the editorial board. This is not a matter of whether or not I agree or disagree with your view on this.  You mention in your email that I did not agree with your view expressed in our earlier correspondence.  However, to the best of my knowledge, I have not expressed an opinion on this issue, either in my correspondence with you or in correspondence with anybody else.  My concern is with the fairness of the process.  This is an APA journal, owned and published by APA, and the policies of the journal are set by APA.  APA has given me the honor of editing the journal for a limited period of time, but this has always been with the understanding that the journal is to follow the policies established by APA.  These policies range from the mundane (e.g., formatting of references) to the very consequential (the subject matter covered by each journal).  I do not have the power to pick and choose which APA policies to follow. APA policy traditionally has been clear about this.  Once a paper has been accepted for publication and for five years after that, authors must make their data available to other investigators.  Over the course of my editorship, I have intervened several times to make sure that this policy was being followed.  However, APA has not required authors to make data available to others prior to acceptance or to physically place the data in any particular repository. I know that there is an active and vigorous discussion happening as to the best way to ensure integrity of scientific data, and APA has been involved in both internal conversations among its various constituencies as well as external conversations with other scientific organizations, publishers, and funding agencies.  I am not privy to the details of such discussions, but I know that this is a complex process that is ongoing.  It is certainly possible that APA, perhaps in coordination with other organizations, will arrive at a different policy in the near future.  However, the journal as of now must be guided by APA's current policy. It would be unfair to our authors for one of our 96 consulting editors to unilaterally establish a policy that must be followed for consideration for publication.  I think that this establishes a terrible precedent.  I also do not want to impose on our 16 associate editors the task of determining whether a particular paper meets with the personal submission policies of each of our consulting editors before knowing whether or not to request a review.  This would reduce the whole process to chaos. I also believe in transparency.  The policies of APA in general and the journal in particular have been published and are available to authors prior to submission.  All changes in policies result from an open policy of consultation and deliberation, and they publicized before they go into effect.  I think that it is unfair to authors to add another requirement (making data available prior to submission) that authors would have no way of knowing ahead of time. I am grateful for your contributions to the journal and the field.  However, your proposal here simply is not consistent with the requirements of the journal.  If you feel that you must follow this principle, I think that it is best that you step down from the editorial board.

 And Gert Storms' reaction

Dear Dr. Greene,
In this mail I want to answer your mail from February 12, in which you ask me to step down from the editorial board of the Journal of Experimental Psychology: Learning, Memory & Cognition (attached below this letter).
Since I want to distribute this mail to all associate editors and all consulting editors of the journal, I first want to clearly express my motives for the issue that we disagree on and I want to mention a previous e-mail conversation on the topic from a couple of years ago.
First of all, I personally have nothing to gain, nor anything to lose by being (or no longer being) a consulting editor for the journal.  At my age, I don’t need my name on the cover of a journal and with my duties in my own university, I can only gain time if I get less papers to review.  I always considered reviewing an service to the field, an obligation, if you will, for the peer review process to be possible.  If I take my job as a scientist seriously, than it is important that the review process is done as thoroughly as possible.  After all, peer review is, together with replication, an essential ‘gatekeeper’ in guarding the quality of scientific contribution.
As you know well, the field of psychology has been confronted with some heavy challenges in the past five years.  There was the prominent fraud case of Diederik Stapel in the Netherlands, the discovery of which made a lot of researchers (and outsiders) wonder how it is possible that his large-scale misconduct remained unnoticed for so long.  Around the same time, however, several publications, most of which were based on empirical facts, argued that questionable research practices are far more prevalent in psychology than generally assumed.  (To give just a few examples, see the publications of Fanelli of 2009 in PLoS One, indicating that 2% of anonymously questioned researchers admit to having fabricated data; Simmons et al., 2011, and John et al., 2012, in Psychological Science; the special section on replicability of Perspectives on Psychological Science in 2012.)
Reading all this, I, as many of my colleagues, was amazed.  A few other findings made me raise my eyebrows even more.  In their 2006 American Psychologist publication, Wicherts, Borsboom, Kats and Molenaar reported that only 38 of 141 (27%) corresponding authors of four APA journals (including JEP:LMC!!) sent the raw data for re-analysis upon request.  Furthermore, in a later publication, Wicherts and colleagues found that a considerable number of the authors who were not willing to share their data claimed significant findings while they reported test statistics that corresponded to non-significant values.
The fact that so many authors of papers in APA journals do not send their data upon request from competent professionals is astonishing, especially because they all signed to adhere to the ethical principles of the APA, which include sharing research data for verification.  The above mentioned Wicherts study was based on papers of 2004, that is, before the beginning of the so called replicability crisis in psychology.  Therefore, we were wondering whether the situation improved after 2012.  Thus, Wolf Vanpaemel and I, in collaboration with two undergraduates, set up the largest study to date to investigate data sharing.  We found that still only 38% of nearly 400 requests, all sent to authors of APA papers, resulted in data sharing responses (http://www.collabra.org/articles/10.1525/collabra.13/) .That is why, in a review for JEP:LMC of February 2012, I asked for the data of two of the reported experiments.  I explicitly added: “Let me be very clear: I have no reason whatsoever to think that there is something suspicious about the data or the analyses in this particular paper.  I would just like to formulate such a request and redo some analyses once and a while when reviewing papers”, and I motivated this request by referring to the above mentioned worrying findings in general.
In the mail from a couple of days ago in which you ask me to step down as consulting editor, you mention that, and I quote, “to the best of my knowledge, I have not expressed an opinion on this issue, either in my correspondence with you or in correspondence with anybody else.”    This is simply not true.  In your mail from February 27, 2012, in reply to my above mentioned review, you wrote, and I quote again:  “I think that inserting a request like this into the review process is a mistake, one that I strongly disagree with.”  (In another e-mail from a couple of days later, you wrote: “I suppose that we will have to agree to disagree here on the costs and benefits.”  That seems like a clear opinion to me.)
But anyway, let me come back to what I consider my responsibility as a reviewer.  APA guidelines ask me to evaluate whether the methodology (design and execution) permits one to draw the conclusions the author wishes to make.  (Note the explicit mentioning of ‘and execution’!)  It is a mystery to me how can I do that properly if I’m not allowed to see the data.  I also really don’t see the point of requiring authors to share their data only after the paper is published.  That seems to argue that it’s better to retract all of Stapels contaminated papers after they are printed in the journals than to prevent the damage from happening if we can.  I really cannot see the logic of this argument.
For all these reasons, I decided to support, as several hundred researchers in the field of psychology did, the call from the ‘openness initiative’ and I signed to follow the guidelines formulated at https://opennessinitiative.org/guidelines-for-action-editors-and-reviews/ (and unlike most authors from APA who signed the APA guidelines and don’ts share their data, I plan to do as I promised).  I do want to stress that the openness initiative leaves the authors the options to simply give a reason why they don’t want to share their data, which is enough to proceed in the review process.  However, this seems for you enough to ask me to step down as consulting editor.
To conclude this long letter, I want to say that I don’t want to do what you’re asking.  That is, I will not take the initiative, as you seem to want me to do.  Of course, as editor in chief, it is up to you to cross me off the list.  But then it is clear that that is your decision, not mine.  I will continue to do my part of the service for the scientific community for those journals that are in line with what I think is necessary in our attempts to prevent sloppy science.