Sherman and Fugh-Berman Respond

On January 14th I published a critique of an editorial by Thomas Sherman and Adriane Fugh-Berman on the Hasting Center for Bioethics blog. A few days later they were moved to respond.

The issue at hand was the retraction of the notorious Séralini rat study.

Sherman and Fugh-Berman held that the retraction was the result of industry pressure and that the retraction didn’t cite reasons that fell within accepted guidelines for retraction [pdf]. I quote them at length to avoid misrepresenting them.

According to the Committee on Publication Ethics, a group that advises medical editors and publishers on ethical issues, particularly, how to handle cases of research and publication misconduct:

Journal editors should consider retracting a publication if:

• they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error)
• the findings have previously been published elsewhere without proper crossreferencing, permission or justification (i.e. cases of redundant publication)
• it constitutes plagiarism
• it reports unethical research

There are hundreds of studies that should be permanently removed from the scientific literature, but the Séralini study is not one of them. The FCT retraction announcement very clearly states: “Unequivocally, the Editor-in-Chief found no evidence of fraud or intentional misrepresentation of the data” – and then goes on to say, incredibly, that the study is being withdrawn because the journal’s own review of the primary data show that the results are inconclusive.

Inconclusive? Until a hypothesis is proven, all results are inconclusive.

It would have been perfectly appropriate for the journal to have written an editorial expressing its concerns. Instead, it seems the editors may have succumbed to industry pressure to do the wrong thing.

. . . The retraction of the Séralini study is a black mark on medical publishing, a blow to science, and a win for corporate bullies.

My response was three fold. I agreed that the retraction had a political element, but that it did not seem to be in response to industry pressure.

Second, Sherman and Fugh-Berman had ignored Séralini’s own ethical lapses. The two that I pointed out were his unusual and manipulative press embargo on the study and his decision to allow the rats to die from massive tumors rather than euthanize them. I did not bring up the conflict of interest that the funding of the study represented. This was a conflict of interest not stated in the paper. Séralini wrote in his book that he funneled industry money through CERES to obscure the funding sources for this study. He failed to disclose any conflicts of interest in the paper. That seems like a major no-no to me.

Third, Sherman and Fugh-Berman had thrown around a lot of innuendo about conflicts of interest. While conflicts of interest raise red flags and call for heightened scrutiny, they do not justify jumping to conclusions. Instead, they should be seen as presenting a hypothesis which should be tested. Sherman and Fugh-Berman say, “The quality of Séralini’s work aside, the process by which his paper was retracted reeks of industry pressure.” But how can you judge whether the retraction can be confidently attributed to industry pressure if you put the quality of Séralini’s work aside?

Before moving on to Sherman and Fugh-Berman’s response, let’s take a quick look at the crux what I wrote:

I agree with critics that the retraction was political. I disagree that it was meant to harm Séralini’s reputation. He had managed that quite well all by himself. The politics behind the retraction almost certainly had to do with salvaging the journal’s reputation. Absent Séralini withdrawing the paper, it’s hard to blame them.

The quality of Séralini’s work aside, the process by which his paper was retracted reeks of industry pressure.

This is absolutely wrong. Questions about the motivation of Séralini’s critics are only valid if their criticisms are invalid.

This is what is infuriating about Fugh-Berman and Sherman’s piece. They state that the quality of the Séralini’s work is beside the point. This is wrong. They seem to think that the Séralini Affair is a he said/she said affair; as if it were impossible for bystanders to assess whose position is stronger. It isn’t. Anyone with an 8th grade science education can understand the issues with the paper. Unless they are trying not to. The insinuation that the motivations of those who slammed the study could be explained by conflicts of interest is beside the point. It is beside the point because Séralini’s work was clearly substandard.

It’s valid to be aware of conflicts of interest. It is a reason for heightened scrutiny. However, those potential conflicts only become salient when presented with questions which can’t be explained otherwise. We ask first order questions. Is the evidence and analysis consistent with basic principles of how we understand the world? Is the analysis solid? Do results seem consistent with common experience?

If those first order questions haven’t raised any flags, there is no point in asking a second order question about conflicts of interest. If the criticisms of the Séralini paper were unsound, then you should ask, What is driving this? There are cases that call for following the money. This wasn’t one of them. You don’t need to follow the money to understand criticism of poor quality science.

When you start hollering ‘Conflict of Interest’ before evaluating the evidence and analysis, it becomes a ‘Get of Jail Free Card’. It becomes an excuse for discounting inconvenient evidence. Asking about conflicts of interest should be safeguard against getting snookered. Instead, it becomes a way to justify motivated reasoning. Awareness of conflict of interest should be a tool for explaining weak evidence and poor analysis. Instead it becomes an excuse for dismissing strong evidence and sound analysis. It leaves you lost in a hall of mirrors, surrounded by industry funded research, revolving door regulators, and defending bad research that confirms your biases. It leaves you lost in a fever swamp of paranoia without firm footing.

Examining the soundness of the evidence and the strength of the analysis must come first. Then you can decide whether questions of funding and loyalties are relevant. This is how you maintain a firm footing and hew to solid ground. This is how you can use awareness of conflicts of interest to avoid motivated reasoning. Otherwise you are only fueling the fire of your own biases. Fugh-Berman and Sherman level charges of conflict of interest while dismissing the questions about the quality of Séralini’s work. This is upside down and backwards.

I think it is clear that I acknowledge that conflicts of interest can be trouble and produce unethical outcomes, but that one must take steps to test whether they are actually affecting events.

Here is Sherman and Fugh-Berman’s response in full (emphasis added):

By all accounts in this case, Gilles-Eric Séralini and colleagues followed the normal route when seeking publication of their study. The journal Food and Chemical Toxicology expressed interest in considering this study for publication, when other journals did not, and the study passed peer review with only minor revisions required. Most importantly, the editors then accepted the findings of their reviewers and published the paper. One can disagree with the reviewers, or one can disagree with the editors, but this is the process we as scientists have accepted.

The scientific peer review process is imperfect and is subject to manipulation. Reviews, especially anonymous reviews, can be colored by conflicts of interest, intellectual biases, personal allegiances, or petty jealousies. As a result, and as we stated in our Bioethics Forum piece, the literature is filled with imperfect, inconclusive and simply bad science. Perhaps the reviewers or editors even recognized the shortcomings of he Séralini study, but decided that elements of the study made important contributions to a GMO literature that is similarly filled with imperfect, inconclusive and simply bad science conducted by industry: who knows? In any case, we deliberately chose not to comment on scientific aspects of the Séralini controversy, focusing instead on critiquing a journal’s unusual retraction of a study that had passed the accepted sequence of peer review and that had not subsequently been shown to be fraudulent, plagiarized, or mistaken.

Mr. Brazeau argues that the conflicts of interests of those in the orchestrated letters to the editor are beside the point because their criticisms are valid. He seems to view conflicts of interest as only a petty annoyance, or as a red herring. In fact, industry-paid researchers will always have more resources and incentives to drown out the voices of non-industry-paid researchers, and that fact interferes with the self-correcting nature of science.

Mr. Brazeau’s claim is a classic “the ends justify the means” argument that is only attractive to those who concur with the result. A strong, well-subscribed publication standard, such as that of the Committee on Publication Ethics, provides justification not only for actions that some approve of, but for what is right and just. Ethical standards provide the bulwark that helps protect science from fraud, error, and commercial interests. In this case, the editors’ actions violated the international standard for the peer review and publication process.

Thomas G. Sherman, PhD
Adriane Fugh-Berman, MD

Before moving on to the meat of our disagreement, I have to flag this statement:

In fact, industry-paid researchers will always have more resources and incentives to drown out the voices of non-industry-paid researchers, and that fact interferes with the self-correcting nature of science.

If you’ve ever done an internet search for information on anything relating to GMOs, then you know that this is a ridiculous assertion.
It is not the voices of industry that are doing the drowning out. If industry was actively drowning out hostile voices, they would have paid somebody to debunk Earth Open Source’s “GMO Myth’s and Truths” by now.

As far as our disagreement goes, let me start by stating as clearly as I can what I believe about the retraction.

1. The retraction was justified by a number of Séralini’s actions. Those included his failure to disclose funding conflicts of interest, his decision not to euthanize tumor riddled rats, the clear and confident conclusion that were drawn in the discussion section of the paper were unsupported by the inconclusive data making the paper unreliable. Clouding the waters further were his press embargo of the paper, the decision to include misleading pictures of the rats,  and timing the release of the paper with the release of his book.

2. The FCT’s written text did not provide sufficient justification of the retraction. That could be seen as an ethical lapse. It certainly was unhelpful in it’s lack of clarity.  It was likely a legal strategy to minimize risk in litigation.

3. The retraction was possibly politically motivated. The likely motivation was rescuing the journal’s reputation from the harm that Séralini had caused.

4. The assertion that the retraction was motivated by industry pressure is lazy, irresponsible and unsupported by the evidence for reasons that I stated originally and will explain further.

Reading their response to my criticism, it is not clear Sherman and Fugh-Berman understood the points I was trying to make. Perhaps I did not express them with sufficient clarity. I certainly didn’t say that conflicts of interest are a petty annoyance or that the ends justify the means.

What I said was that when we see them, we should proceed with caution. We need to be aware of them, in the case that they are affecting outcomes. BUT. If those outcomes are not anomalous, then charges of conflict of interest have no explanatory power. As I said previously, “You don’t need to follow the money to understand criticism of poor science.”

Before getting back to Sherman and Fugh-Berman’s original editorial, we need to flag two items in their response to my criticism. The first is this:

Mr. Brazeau argues that the conflicts of interests of those in the orchestrated letters to the editor are beside the point because their criticisms are valid.

They are now asserting without qualification that the letters to the editor were orchestrated. They don’t refer to them as likely or possibly ‘orchestrated’. They seem to know that they were orchestrated. If they ‘know’ this, they should share how they ‘know’ it.

Then:

Mr. Brazeau’s claim is a classic “the ends justify the means” argument that is only attractive to those who concur with the result.

I am not making a “the ends justify the means” argument. What I’m saying is that if the ends are not unethical, then they need no justification. Maybe the letters to the editor were orchestrated, if the reasoning and evidence contained in the letters is sound, then what does it matter. A corporate campaign of misinformation and confusing the issue is unethical. A corporate campaign of education and clarity is not. That is why the quality of Séralini’s work is relavent to judging the ethic of the players in this drama.

The argument in Sherman and Fugh-Berman’s original editorial has two parts. The first is that the retraction was unethical because the reasons cited in the retraction are insufficient. They appeal to the norms and guidelines that govern the conduct of academic journals to illustrate this. The second is that the retraction was motivated by industry pressure and bullying. Neither of these holds up to scrutiny, especially the latter.

Let’s look to see if Séralini really skates past the accepted guidelines for retraction.

The clear and confident conclusions that he draws in the discussion section of the paper are unsupported by the data he presents.
Unreliable findings. Check.

He allowed the rats tumors to develop far beyond the point that the rats should have been euthanized.
Unethical research. Check.

His did not disclose conflicts of interest in the funding of his research and in fact admitted to intentionally obscuring the sources of funding.
Failure to disclose conflicts of interest. Check.

So while the stated reasons for the retraction were mealy mouthed and did not fulfill the normal guidelines for retraction, sufficient reasons existed whatever the stated reasons.

Wallace Hayes writes:

A second concern that has been raised is whether this retraction follows the COPE guidelines. The COPE guidelines were consulted when making this decision. According to the COPE guidelines, “Journal editors should consider retracting a publication if… they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error).”(COPE, 2009). The retraction statement could have been clearer, and should have referred to the relevant COPE guidelines. The data are inconclusive, therefore the claim (ie, conclusion) that Roundup Ready maize NK603 and/or the Roundup herbicide have a link to cancer is unreliable. Dr. Séralini deserves the benefit of the doubt that this unreliable conclusion was reached in honest error. The review of the data made it clear that there was no misconduct. However, to be very clear, it is the entire paper, with the claim that there is a definitive link between GMO and cancer that is being retracted. Dr. Séralini has been very vocal that he believes his conclusions are correct. In our analysis, his conclusions cannot be claimed from the data presented in this article.

Sadly, the guidelines are silent about is the unethical conduct by Séralini that brought shame upon the journal that had magnanimously published his work. He designed the study to be a bombshell, this can be seen in the unusual decision to included photos of three tumor riddled rats but to exclude a photo of a rat from the control group. He timed this bombshell to coincide with the release of his book. He embargoed the paper for reporters so that they could not seek informed opinion about it prior to writing about it. The dean of American science writing, Carl Zimmer called this a rancid, corrupt way to report about science. Throughout the book tour, Séralini did not portray the study as inconclusive. None of this reflected well on a journal that obviously should never have published this paper in the first place. Inconclusive studies are published all the time, that’s as it should be. Bad papers slip through peer review. Mistakes happen. The guidelines on retraction are silent on the question of what to do when a scientist refuses to withdraw bad work but instead makes a spectacle of it and turns the journal’s error into an international media event.

I think it is clear that there were reasonable grounds for retraction, but that FCT should have been clearer about their reasons. As I said, that may be considered an ethical lapse. This is not what Sherman and Fugh-Berman are saying. They are saying that it was unethical for FCT to try to rectify their error when Séralini refused.

That they take that position is understandable and forgivable. What I cannot understand, is writing about ethics, while so casually tossing around charges of corruption and bullying.

Yes, some of the people criticizing the paper had industry ties. That’s a certainly a red flag. But, the observation of a potential conflict of interest sets up a hypothesis, not a conclusion. The lazy observer just uses that observation to make a leap of logic and assume the conflict of interest explains the criticism. The rigorous response is to figure out ways to test that hypothesis. Does it have any explanatory power?

Sherman and Fugh-Berman state, “11 of the authors of letters to the editor slamming Séralini’s study had undisclosed financial relationships with Monsanto.” This sets up the proposition that only relationships with Monsanto can explain the criticism of Séralini’s work. If that was the case, you would expect criticism of Séralini’s work to be confined to people with industry ties. But criticism of Séralini’s work was not confined to people with industry ties. It was near universal. The paper thoroughly failed post-publication peer review.

After carefully reviewing the study, six French national academies (Agriculture, Medicine, Pharmacy, Science, Technology and Veterinarians) issued an extraordinary joint statement condemning it and the journal that published it. The paper was reviewed and refuted by the most prominent independent international science organizations and every food standards agency of note, including French HCB and the National Agency for Food Safety, the Vlaams Instituut voor Biotechnologie, Technical University of Denmark, Food Standards Australia New Zealand, Brazilian National Technical Commission on Biosafety and the European Food Safety Authority

So much for that idea.

What about the case of Richard Goodman an incoming associate editor brought in to clean up the journals peer review? Goodman had previously worked for Monsanto. He was not part of the decision to retract the Séralini paper, but (cue the theremin music) months after he was brought on board Elsevier, FCT’s publisher, announced the retraction. It’s always amazing the power people will attribute to former employers when it suits there purpose. Think back to who you were working for ten years ago. Are they influencing your decisions today? Hmmm? I didn’t think so. It’s all the more striking when that power is imagined to work through people who are not even in the room making the decision.

Jon Entine reports:

Richard Goodman, who runs the AllergenOnline database at the University of Nebraska. Goodman is an internationally respected expert on allergies and the health effects of GM foods—but also a former Monsanto scientist. He was brought in by the FTC earlier this year to clean up the journal’s peer review process. As Robinson acknowledges, “there is no proof that Goodman was responsible for the retraction of Prof Séralini’s study.”

Goodman declined to comment directly, but numerous people have confirmed to me that he was not involved in the evaluation process and was not even aware of what if any action the editor had been contemplating.

Sherman and Fugh-Berman write:

Meanwhile, back at Food and Chemical Toxicology, a new position for an associate editor was filled by Richard E. Goodman, a University of Nebraska professor who previously worked for Monsanto, and who has a longstanding association with the industry-funded International Life Sciences Institute (ILSI). Months later, Elsevier, FCT’s publisher, announced the retraction.

THAT … is a smear, pure and simple. A smear made by people who are purporting to write about ethics.

Wallace Hayes, editor in chief of FCT writes:

The membership of the editorial board is composed of academic, government, and industrial scientists. Contrary to what has been suggested by some, the appointment of Professor Richard Goodman, University of Nebraska, as an Associate Editor was not influenced by Monsanto or any other party. Members of the editorial board are chosen based on their expertise as scientists. It is the goal of this journal to have a variety of different viewpoints. In this case, as in other cases, I as Editor-in-Chief listened to as wide and diverse a set of expertise as possible. To wit, Professor Goodman, along with all other members of the editorial board was involved in initial discussions of the Séralini paper and the request to view raw data.  When the request was made to Dr. Séralini to review the raw data, the journal suggested to Dr. Séralini that all parties involved sign a confidentiality agreement. This confidentiality agreement was designed to protect Dr. Séralini and his data so that it was (A) not viewed by anyone he did not want to view his data and (B) that it would not go beyond the people he agreed would review the raw data.  Not initially, but during the process, Dr. Séralini made a direct request that Professor Goodman be excluded, and we at FCT readily and quickly agreed. It is understandable that Dr. Goodman’s involvement, however small, might be cause for concern for some. However, the decision to retract the paper was mine alone, made by me exclusively and not by a vote of the editorial board. Further, when Dr. Séralini asked for Dr. Goodman’s involvement to stop, I agreed, fully and promptly.

The retraction is done and over, the world moves on. Séralini’s work had been thoroughly discredited already. What matters to me is the object lesson in critical thinking. All too often people will point to conflicts of interest, previous employers and potential regulatory capture and jump to conclusions. In doing so, they may feel like they are wised up realists, real critical thinkers. They aren’t. They are lazy and paranoid. They are leaving the job half finished.

I’m often asked, “That study was funded by the industry, don’t you think we should question that?”  And I say, “Yes. We should question that. So, let’s think of some questions. Because what you are doing is not questioning it. You are simply dismissing it and using industry funding as a convenient excuse.” That’s not critical thinking. It’s uncritical thinking. You’ve set up a hypothesis and drawn a conclusion, but you’ve skipped the part where you test the hypothesis.

That’s what was particularly galling about Sherman and Fugh-Berman’s editorial. They were lecturing us all about how ‘science works’ while proposing hypotheses and then jumping to conclusions without testing. They were lecturing us all about ethics while blithely smearing others. To crib from Carl Zimmer, that’s a rancid and corrupt way to write about the ethics of science.

Sources:

Rounding Up Scientific Journals
Adriane Fugh-Berman and Thomas G. Sherman | Bioethics Forum

The Ethics of the Séralini Retraction and Charges of Conflict of Interest
Marc Brazeau | REALFOOD.ORG | Daily Kos | 14 January 2014

Bioethics Forum author reply
Thomas G. Sherman, PhD and Adriane Fugh-Berman, MD | 18 January 2014

Food and Chemical Toxicology Editor-in-Chief, A. Wallace Hayes, Publishes Response to Letters to the Editors
A. Wallace Hayes | Elsevier | 10 December 2013

The Carrefour-CRIIGEN Connection
Before It’s News | 22 September 2012

Auchan and Carrefour financed CRIIGEN studies on GMOs.
David Tribe | GMO Pundit | 23 September 2012

Séralini et al. study conclusions not supported by data, says EU risk assessment community
EFSA | Press Release | 28 November 2012

Séralini Threatens Lawsuit In Wake Of Retraction Of Infamous GMO Cancer Rat Study
Jon Entine | Forbes | 29 November 2013

Stenographers, anyone? GMO rat study authors engineered embargo to prevent scrutiny
Ivan Oransky | Embargo Watch | 21 November 2012