Eric has appeared in comments at Eli’s. Evidently, Eli interprets the recent update at John Nielsen Gammon as supporting Eric. I clicked back to John’s blog, and read the update. I wasn’t so sure what was written exactly supports Eric.
It seems Eric’s interpretation may be similar to mine. I took a screen shot of Eric’s comment at Eli’s post
Folks, while I appreciate that much of the speculation being done here is of a higher quality than that being done, umm.. elsewhere.. it is still speculation.
John Nielsen Gammon, while you seem to be supporting me, which I appreicate it, it isn’t very useful support.
I did not, repeat not, repeat not, “not remember that this amounted to his own insistence on iridge as a reviewer.” I did NOT recommend iridge. I did NOT bring it up. I SIMPLY DID NOT THINK I COULD ARGUE WITH THE EDITOR OR THE KNUCKLE-HEADED REVIEWERS that is should not be used.
This is not complicated folks. O’Donnell and gang, not liking my criticisms of the way they used TTLS, and in particular the fact that the truncation parameter they wanted to use, suddently started using IRIDGE. This has the advantage of having a build in verification function, which means you can’t see what the verification statistics are, which means that it is much easier to NOT SHOW THE BAD VERFICIATION STATISTICS I was criticizing them for. Maybe that is not why they used iridge. I don’t know WHY they used IRIDGE but I did not suggest it to them nor endorse it.
GET IT?
P.S. Yes I am shouting. That’s what the CAPS mean.
P.P.S. Thank you ‘chris’ above for your clear, rational, non-speculation.
As it seems eric is constantly being misunderstood, I am going to resort to the dreaded posing of questions instead of telling you what I think eric is saying.
- Is eric calling the other reviewers knuckleheads?
- Is eric suggesting that a reviewer might modify what he says in a review based on whether or not he thinks he can argue with editors? As in: A reviewer can’t just write what he thinks, or tell the editor he’s done and uninterested in continuing to act as a reviewer?
Eli has been hopping mad that I haven’t reported on this other aspect of JNG’s post. JNG says it both a violation of AMS policy and unethical for RyanO to post both the reviews and the reviewers name. You can read JNG’s comments fully in context at JNG’s.
Ryan posted his own view in comments here.
Ryan O (Comment#69290) February 12th, 2011 at 3:10 pm Edit This
Bugs:
If you can make this sound so reasonable now, why the over the top snark and backstabbing on CA and other blogs?
First, I disagree that there was backstabbing involved. Second, the literal answer to your question (minus backstabbing) is that I am calmer now.
The longer story is that, throughout this whole process, I have bent over backwards to be professional and courteous to Eric and RC. If you do not wish to take my word on this, you can go back through my previous posts and examine it for yourself. You can also go to RC and see my comments there. When people had questioned Eric’s motives or integrity in my threads, I chastised them and asked them to refrain from doing so in the future. I made frequent reference to the professionalism and courtesy that Eric had shown me in our email communications (which was, indeed, an accurate characterization of the emails). When I noticed people making statements concerning either S09 or Eric that I knew to be false, I corrected them (as I have continued to do, even after the eruption of Mt. O’Donnell).
When I made statements questioning the S09 method, I backed them up with calculations and code that could be replicated. I was very careful to do this. I was also very specific in my criticisms, and made note where a specific criticism could not be generalized as some folks seemed wont to do.
Throughout this, I tolerated the claims on RC that were not backed up by calculations and the claims that our criticisms were inaccurate when the evidence presented did not relate to our criticisms. I tolerated this because I believed that the folks had the honest belief that their arguments held water. I was annoyed that they had not checked if this were true prior to making public statements, but this is blogland and those things happen. So I let it go.
I also ignored the snark and condescension in some of the inline responses – not to me, but to other posters in which a snarky or condescending reference back to me or my coauthors was made. I did this because I recognize that the personalities involved had a history that the mere existence of my presence could not put to bed.
I think I my efforts to keep this civil were more than the situation necessarily called for. But I felt that it was the right thing to do, so I did it.
But for everything, there is a straw that breaks the camel’s back. In this case, I think it is quite clear what the straw was. And while you may question my decision to do what I did (as is your right), I stand by it. I apologized for the point of fact on which I was mistaken (that Eric had seen our third response) and I provided edits for the various blogs to remove the statements of dishonesty and duplicity. These are exactly what I agreed to do in the email exchanges between myself, others, Eric, and Revkin.
That does not mean that I regret having done what I did. A less volcanic approach probably would have better conveyed the message, but the message would have been the same. The practice of recommending something in a review, and then requesting justification in a subsequent review, and then making a public criticism against that very same recommendation without having ascertained what the response was cannot be condoned. The only way to properly address this is to let it be known that the same person who is currently criticizing the choice is the same person who previously recommended it. That requires doing what I did (though not necessarily with the accompanying invective).
I am not an academic. Perhaps I would not fit well in that world. I understand that many academics may see my actions as wrong or unethical, but I disagree, and I will continue to disagree regardless of how many might try to convince me otherwise. My actions were certainly different than what they might have chosen, and they were certainly outside the norm for expected behavior between academics . . . but that is not the same thing as unethical. Besides, I am not an academic anyway, so it should not be much of a surprise that my behavior would not conform entirely to what is expected in a relatively insular environment of which I am not a part.
I find Ryan’s position persuasive– it accords with my view on the balance of the ethics of revealing the reviewer. I realize others including John Nielsen Gammon (JNG) may disagree with me and they have a perfect right to do so.
Since people keep asking me this in comments, I will also restate my position that it is up to the AMS to make and enforce whatever policies they have and to tune those to their needs as publishers of journals and to their view of the ethics on this. John Nielsen Gammon is more familiar with those than I. As this incident does involve the AMS, I imagine that body will eventually weigh in formally in some way and we’ll all learn precisely what the AMS as a whole decides to do.
====
Update 5:06 pm Chicago time. There seem to be some additions at rabett. Eric prefaces a 2nd version of the comment with
Eli, thanks for elevating my comment to a real post, on Rabbet Run no less!
How about now replacing it with this, without the typos?
Inserts edited comment. And adds
P.P.P.S. Michael Tobis thank you. Much better. I still wonder why so many people don't sum it up in just one sentence, such as your excellent closing one:
This constitutes an excellent test of whether O'Donnell is interested in science or in McIntyrism. The results of this test are unambiguous to say the least.
P.P.P.P.S. We probably don't need a new term for McCarthyism. The original one still works pretty well. Any similarity to the name Simple J. Malarky is of course entirely coincidental.
P.P.P.P.P.S Here is another relevant cartoon
update: 9:28 pm. Eric clearly loves his own comment so much, he reposted it at JNG’s blog. I can’t help wondering who the three “KNUCKLE-HEADED REVIEWERS” were and when they will learn Eric’s opinion of them.
yep… calling all of them knuckleheaded… and he seems to imply the same of the editor
How quickly things turn in blogland — when you are full of it.
jeff Id (Comment#69300) February 12th, 2011 at 4:20 pm
You’re not one to talk. You really need to drop the conspiracy theories.
lucia (Comment#69296),
On what blog is Eric getting a little testy?
.
Notice what he is getting testy about… yes that’s right, people take a plain reading of the review to mean that Eric suggested (insisted upon?) the change to iridge. Seems most everybody must have reading comprehension problems; this causes the more perceptive folks like Eric to become frustrated and start yelling.
Between the Climategate emails and S(09)/OMCL(10), I’m kind of glad I DIDN’T go into academia. I have problems enough with the politics when they’re open and acknowledged as part of the processes I have to deal with.
It’s up to AMS whether they choose to pursue any allegations here but I would love to ask JNG what the author is supposed to do when a reviewer is publicly denying something s/he did in the review process, e.g., critique use of a technique or process that the reviewer him/herself suggested.
There is an obvious discrepancy between Dr. Steig’s comments in the snapshot you posted and the suggestions from Reviewer A that have been posted previously. No need for blogland to work itself into a tizzy over it because it’s glaringly obvious; watching Dr. Steig comment further is like watching slow motion of Joe Theisman back up into Lawrence Taylor in Theisman’s last Super Bowl.
JNG at stoat scienceblogs.com/stoat/2011/02/are_the_wackos_wacko_enough.php#comment-3270757
I’m puzzled about Steig’s wording which suggests that he had tried to argue with the other reviewers. I wonder if this was a case where the process permitted the reviewers to see each other’s reports? We’ll likely never know. But it’s an interesting thought.
Steig wants to criticize the authors for iRidge, but he could not do that during review? Please.
In fairness to Eric, he very well may not have been suggesting iRidge. In fairness to the authors, that might not have been clear based on the second review:
.
“Perhaps, as the authors suggest, kgnd should not be used at all, but the results from the ‘iridge’ infilling should be used instead. The authors state that this “yields similar patterns of change as shown in Fig. 3, with less intense cooling on Ross, comparable verification statistics and a statistically significant average West Antarctic trend of 0.11 +/- 0.08 C/decade.†If that is the case, why not show it? I recognize that these results are relatively new – since they evidently result from suggestions made in my previous review – but this is not a compelling reason to leave this ‘future work’.
.
What is clear after re-reading the three rounds of reviews from Eric is that he took every possible opportunity to justify using only three ‘eigenvalues’ and insisting (many times) that the paper be written so as to be a ringing endorsement of the S(09) results. I imagine the editor saw that his reviews were more than a bit too self serving to force further re-writes, and took the opinions of the other three reviewers.
I hope we didn’t break him. Wow, he needs to settle down. He’s wrong and loud — bad combo.
—-
Ouch Bugs….hehe
Shub–
Plain reading of what Eric wrote seems to suggest that he, the reviewer could not make the argument iridge should not be used for the reasons he gave in that comment. Given how literally he interprets O’Donnells interpretation of the idea that they made changes because the reviewer insisted I should think that we could all observe that as a matter of literal truth, Eric could certainly write whatever he wanted in his review.
Or at Eric was at least as free to write what Eric wanted in his review as O’Donnell was free to not make changes to his manuscript!
Jeff Id,
.
Yes, the “knuckleheads” comment is not good. Eric may or may not know who the other reviewers were, but I am sure they and Broccoli know who he is, and also what he thinks of their intellects. IMO, he needs to take a week off. Most of all, he needs to drop the snark level two orders of magnitude (starting when he began to participate at RC!). I expect he may be under a lot of pressure from people (not on blogs) telling him he handled this very badly, and he did.
“I did NOT recommend iridge. I did NOT bring it up. I SIMPLY DID NOT THINK I COULD ARGUE WITH THE EDITOR OR THE KNUCKLE-HEADED REVIEWERS that is should not be used.”
I would think that the proper thing to do here would be to show what Eric argued as Reviewer A that is pertinent to his claim above. If it is in direct opposition, I would take note of that and perhaps size up whether further discussion with or about Eric would be helpful. If indeed Steig has wiggle room in what he originally said I am sure his defenders will produce that view.
There is something about the tone of Steig’s insistence that is making me a bit uneasy.
Kenneth Fritsch,
Steig’s first review does not contain the word iRidge. He does complain about TTLS.
Ken–
Given what Eric wrote here: http://www.realclimate.org/index.php/archives/2011/02/odonnellgate/ I think we might look at this:
And not that it is equally true to observe that “..the decision about what gets written in the review is ultimately up to the
authorsreviewer. “I don’t know where Eric is at right now emotionally but reading his comment concerns me a bit. Hopefully he’s able to regain some perspective.
jeff Id (Comment#69330),
.
I don’t think a personal email message from you right now would be helpful (and I am not joking).
He’s had several email exchanges with me since this began. I don’t think he’s that far gone, but I’m not going to be commenting on blogs about it for some time after reading the above.
Antarctic temperatures are a hobby for us, they are a very big deal for him.
Kenneth Fritsch,
.
In his third review (the one that was ignored), he argues that the paper had to be modified to explicitly justify the use of iRidge, since Mann had published a paper saying that IRidge underestimates trends with sparse data (IIRC). As I noted earlier, he argued (endlessly!) from the beginning of the first review that S(09) did not overestimate the trend in west Antarctica and the Ross Ice shelf, and that O(10) were just underestimating the trend badly. That is, he defended the ‘novel finding’ of S(09), tooth and nail, from the start of the first review to the end of the third.
A learning experiance for me.
I just read in detail the O(10) reviews by Reviewer A and O’Donnell’s replies. My single take away is that Reviewer A was all over the place. No doubt his strategy was to somehow get a result more in line with West Antarctica warming from S(09), but his tactics came across as desperation hit and run.
I am afraid that Jeff Id’s comment, that with the authors of O(10) it is a hobby, but that with Steig it is much more than that, comes through in my reading of the reviews. I was saddened by what I read. My recollection of memories of working with scientists could be affected by my being away from it for a while, but in those days the technical discussions, even the heated ones, were fun and meant to be a learning experience for all involved. Some of the best moments would come when someone argued against your point and then it clicked and you understood how you had been wrong and how much sense the correct point made.
I am with Kenneth on this – a little uneasy. There is more to this story.
Is it possible the “THE KNUCKLE-HEADED REVIEWERS ” he refers to are not B and C, but other A’s?
Recollection of memories. Did I really say that? Mrs. Fritsch please come tuck me in.
But wait a minute guys… is it possible that the other reviewers were in fact knuckle-heads?
Another aspect of the peer review process that has received a lot of attention on the blogosphere is the claim that Steig wrote an 88 page review of an 8 page paper. This claim was made by Id and Watts in separate blog postings. It turns out that examination of the data show the following, as partly discussed elsewhere. The three reviews were 14, 6 and 4 pages long in each successive version. There never was an 88 page review. What’s more, the final pre-publication, non-formatted article as posted by J Clim is 52 pages long. Submitted versions are generally 6-10x longer than the final printed version, because submitted versions are expected to be double spaced in at least 12 pt font etc. Published versions cut down the page numbers drastically, an entirely normal process. The comparison of the inflated “88 pages” of review with the purported “8 pages” of manuscript is of course completely misleading and wrong. A more honest version would be “a 14 page review of a 52 page paper.” I leave it to the readers to judge for themselves if that’s unduly burdensome or not. All I can say is that in my experience as an author, reviewer, and editor in the field, it’s not typical but by no means extreme. I don’t wish to pass judgement on either paper in this post, merely point out the way that “page statistics” were manipulated by third party bloggers to give a decidedly false impression – and it appears that manipulation was not accidental. Those “facts” were then echoed in many postings, but apparently very few checked to see if any of it was actually true. It wasn’t. The speed at which disinformation gets circulated is breathtaking. I wish there was a way to assay what percentage of the people commenting on the blogs have actually read the papers, the reviews or anything other than other postings. My guess is that the percentage is very low. It’s another example of why rapid fire blogging is so often not only wrong, but deeply and disturbingly misleading.
lderry,
I did not report that stat, it is from Ryan. I am certain that he counted 88 pages of interaction for review or 88 pages of reply – just because Ryan is very accurate. The original paper was also quite a bit longer than the final so there was one heck of a lot of writing.
lderry–
Ryan agrees there is no 88 page review— it is a sum over a number of communications of reviews and replies to reviews. If you find quotes by Ryan, you will see he never said that. Since Ryan has posted very clear statements on this, I believe him, and in anycase, all the reviews and replys or on line, I’ve been perfectly aware that no single review is 88 pages for sometime.
I’m a bit puzzled why you are dashing in here to “rebutt” an “inflated” page count, when, as far as I can see, you have no evidence that people commenting here harbor any misconceptions about the page count nor that their views are based on any misconception about the page count. Of course, you can easily prove there is no 88 page review– the difficulty is… well.. whose misconception are you rebutting? (Please look up the term strawman. )
We are aware of the correct page counts based on reading the original quotes which provide the correct counts, as well as clarifications by both Ryan and Jeff when people misstate what that means. I’m not going to hunt for their comments for you. I’ll leave it to you to find them using the google search tool in the upper right side of my blog template.
If you want to lecture some third party blogger that their notions about the review process are incorrect and based on inaccurate knowledge of the page count it might be wise for you to actually find that third part blogger. Then you can lecture them.
I know it might be a bit of work for you to find them– but don’t you think that would be wiser than just picking a third party blogger at random and guessing they must harbor the misconception you are guess is out there at some third party blog? (I think it’s better– but you know, that’s just me.)
Yes. Well. You might want to consider the possibility that you are misleading yourself by making conclusions based on your own guesses of what might be true. I’m guessing you are doing an awful lot of guessing and I’m also guessing that you aren’t a very good guesser. But, that’s just my guess.
Jeff,
I’m glad you wrote in. But, I’m not sure I understand your comment. Here is a snip from your Airvent column of Feb 7
“Anyway, Ryan ODonnell is a little tired of it as well and has just added another 14 pages to the unbelievable 88 page review. Read on and you’ll see what I mean.- Jeff”.
Your post gives the strong impression that you think the review was 88 pages long and excessive. Maybe I misunderstand, but my take is that many other people have misunderstood as well, given the frequency of which this claim seems to be repeated. It seems clear that the honest thing to do is to retract that comment. I don’t think the discussion is remotely well served by the “88 pages on 8” theme, no matter who or what the original source for that misrepresentation was. It just isn’t accurate or reasonable, so do the right thing and use your venue at Airvent to say so. I would hope in the future that you and everyone else would be more careful to check the sources before going public with this kind of claim. I firmly believe that accuracy should be much more important than speed in this business. The tenor of the whole discussion needs major improvement, and eliminating clear errors would be one small step. It’s not about who’s fault it is, just fix it. It may or may not be a minor issue, but it should be fixed just the same.
Lucia,
This isn’t a strawman. As you know, I did not say that Ryan said 88/8. I did say (correctly) that the claim appeared on Airvent and Watts Up. This claim, whatever it’s origin, has been widely repeated. You know that too. You also know, apparently, that it is not true. So, what’s the problem with saying that it’s not true? Or putting in context of the length of the manuscript version? This isn’t (or at least shouldn’t be) controversial.
lderry:
Channel Eric much?
It’s also been discussed on this blog, if not on the top page. It’s not controversial, but neither is the fact that people have misquoted Ryan on this.
lderry–
If you read, the bit you quote says Ryan added those pages. Jeff’s wording is inauspicious, but it’s clear he is discussing material that includes stuff written by Ryan.
You can also see that people in comments understand this
Here is Roman:
Note “responses”.
People’s notion’s about the review process are not based on a misconception that Steig wrote an 88 page review!
Lets check this person’s facts that Jeff Id claimed the Dr. Steig wrote an 88 page review by going to Jeff Id’s post and looking:
http://noconsensus.wordpress.com/2010/12/01/doing-it-ourselves/
Hmm nope Jeff Id didn’t claim that Dr. Steig wrote an 88 page review.
You see commentators of the paper made the mistake of attribution of all 88 pages to the Review itself something that the authors particularly Ryan O attempted to correct in the comments section on the Doing it Ourselves post:
Now lets see if Mr. Watts made the claim of Dr. Steig writing an 88 page review by pulling up his post on the subject (which was nothing but a crosspost of the one from the air vent with an intro):
http://wattsupwiththat.com/2010/12/01/skeptic-paper-accepted-on-antarctica-rebuts-steig-et-al/
You see when Anthony first posted it he made the same mistake other commentators did, but he saw Ryan O’s post that I copied on the Air Vent and corrected it the same day which can be seen here from a copy of a comment about that from WUWT:
So Mr. Watts made that mistake but corrected it on Dec 1, 2010. However you seem to be lacking these basic facts and/or you didn’t take the time or effort to research your “facts” before accusing Jeff Id and Mr. Watts or perpetrating the 88 page meme.
Lucia
What lderry is doing usually called – “erecting a strawman” and then bringing it down.
.
“The tenor of the discussion needs major improvement”
You bet.
.
“-retract that comment”.
Apologies, retractions,…that is what the game is about.
.
Jeff also wrote “Read on and see what I mean”, before signing off with his name which lderry has (miraculously) swallowed.
.
I bet this is a clear example of an instance requiring major quality improvement (and not wasting others’ time).
You should see our dear friends moping around that they did not get the apologies they somehow fancied they were going to get, or the retractions they were led to believe, were going to take place. Some are fantasizing about defamation lawsuits.
Re: lderry, Comment #69340
> My guess is that the percentage [of the people commenting on the blogs who have actually read the papers, the reviews or anything other than other postings] is very low.
My first guess is that your guess is wrong, with respect to the blog at which you dropped your comment.
My second guess is that while there are some KNUCKLEHEADS (sic) who have added dross to recent threads at this blog, those participants who have been contributing to the discussion — e.g. Lucia, Kenneth Fritsch, SteveF, Carrick, Nick Stokes, myself, even Ryan O — have shown themselves to be well-read with respect to the content of their comments.
Re: 88 pages, my third guess is that you haven’t seen Eli’s page-count table. It’s here. He’s no great friend of O’Donnell or Lucia, but he can count.
lderry–
Believe it or not, I do not know what claim appeared at WUWT. The reason is that for this story, I’m reading TAV and CA which I read more often in any case. If you want to show me that a misconception exists, show me — providing the link.
You did eventually get around to citing TAV. I looked at it. Jeff said something inauspicious which could be misinterpreted. But as far as I can tell, that did not result in a widespread belief that Steig wrote an 88 page review. Comments clearly indicate that Roman knew 88 pages involved responses.
Moreover, here in comments, Ryan has discussed the ’88’ pages many times– always clarifying that it includes responses.
If it turns out there is a widespread misconception at WUWT, it would be wise for you to go there to disabuse them of the notion. But with respect to rebutting here to my audience, it is a strawman. It’s not a theory widely circulated around here.
If you want to rebutt an inaccurate notion you think has been circulated at some other unspecified 3rd party site, it would be wise for you to go there.
Lucia:
Look up a few posts. I provided the links to not only the TaV post and copy pasted Jeff Id’s comment, I also show Ryan’s Post on the 88 page meme and then I did the same for WUWT and the correction Anthony made on that same day Dec 1, 2010. Even the commentators at WUWT questioned the 88 page meme when Anthony made his mistake (which he corrected long ago).
We’ve had a lot of hyperventilating over the last few days about “ethics” and peer review.
Where does calling your fellow reviewers “knuckleheads” fall on the hyperventilating ethics scale for our harumphing academics?
boballab–
Thanks. I had been unware that Anthony made that mistake back in Dec. and corrected it quickly when his readers advised him that his number was wrong.
It is quite clear that lderry is completely mistaken that there is any widespread misconception that Steig wrote an 88 page reviews. I’m sure if he tries, he can find the infelicitous wording here and there, but, as far as I can see, there is no such widespread misconception here, at TAV, CA, or WUWT.
boballab,
The “10 times” comment remained on Watts, even as he corrected the “88 pages” to include the replies, without specifying how much was comment and how much was reply. What’s wrong with getting this right? I did not propose that anyone “apologize” – just that the misconception be fixed. Simple enough, and if you don’t think it has had an impact …. A lot of readers, for better or worse, were misled by these statements. Do you disagree?
Well… it’s probably not unethical. But it might make them a little grumpy.
Amac,
I have seen the table, which is of course a useful summary (even though reviews are not valued by weight, or it’s inverse). And you’re right, many of the *participants* at this site have read the primary sources and know better. If I’m telling some of those people what they already know I’m sorry if they find that bothersome. But it’s clear from a read around various blogs that the claim in it’s unsophisticated version has been current. As I think most of you will agree, the readers of these blogs tend to significantly outnumber the contributors (otherwise why have them?). I just don’t see why clarifying a misconception like this should be a problem for anyone.
lderry–
Whatever your complaint about Anthony’s blog, I suggest complain to Anthony. Explain to him how you come up with your ratio and he may explain how he comes up with his. Maybe to get the ration he doesn’t count figures in the orginal paper and it’s the ratio of text only. Maybe his ratio is wrong no matter how you look at it. That ratio that is now bothering you hasn’t been a matter of concern for me because
a) I didn’t read that blog post until just now, and even now only skimmed it and
b) as far as I am aware, few of my readers are basing their evaluation of the review process based on Anthony’s blog post and
But if that number bothers you, take it up with Anthony. He changed the first mistake when a reader corrected him way back in Dec. He may be willing to change the or clarify his ratio of 10 based on your explanation.
But complaining to me is pointless. Because as hard as this may be for you to believe, I do not operate his blog.
Lucia:
He is building a second strawman by focusing on Mr. Watts, failing to realize the 10 times number comes straight from Jeff Id. Also he is complaining that Mr. Watts didn’t give a number of pages that were review comments and what was reply on a post dated Dec 1, 2010. At that date Mr. Watts has no way of knowing the breakdown of the exact number of pages is review and with is response since that wasn’t made available until Feb 7,2011. The closet Mr. Watts can get is that Ryan stated on Dec 1, 2010 on TaV that there was almost 40 pages of Review comments alone.
lderry
If you think that’s clear to you, then go to those blogs and explain their misconceptions to them. I don’t know which blogs you are reading and I’m not going to hunt down which have this misconception merely because you have developed the impression there is a misception.
I am especially unmotivated to do it because:
1) you thought the misconception exists at WUWT– yet, it was clearly corrected in Dec.
2) you thought the miconception exists at TAV. You found 1 infelicitous wording among many very clear ones. On the post with the infelicitous wording, it’s clear people in comments did not develop the misconception that reviewer A wrote n 88 comment review.
3) you chose to come here and wrote a comment that suggested you thought the misconception might be rife around here.
I just don’t see why clarifying a misconception like this should be a problem for anyone.
Neither do I. If you need my permission: feel free to go find the blog where the misconception actually exists and post a clarifying comment in the comments boxes. I have no objection.
Meanwhile, I hope boballab posts his comment clearing up the misinformation you posted at dot earth, which can be easily done if he just posts boballab (Comment#69348)
Re: lucia (Feb 12 21:55),
The meme was starting here – eg here and here. I protested here which seems to have some effect.
Protesting at WUWT is no use at all.
I suppose if this were Steig you’d call it a true statement used to deceive. Certainly all the complaints about the length of reviewing mentioned only the 88 page figure, not the 24 pages he actually wrote.
Re: lucia (Feb 12 21:15),
I did write a response with links, as you suggest – but then straight into moderation.
boballab,
but he should have known it wasn’t an 8 page manuscript. In any case, this discussion seems to be getting out of hand/off track, and it’s looking like time to move on from it. As I said, if I bothered you or any one else by repeating what you already knew, sorry. But I don’t think the facts are in much dispute.
Iderry is making the same mistake that many at RC have made: There are some very smart, technically capable and diligent folks carefully and intelligently looking at what is being done and not done. It looks like Eric has made the same mistake.
Re: Nick Stokes (Feb 12 22:33),
> but then straight into moderation.
One reason why I read WUWT only rarely. I disagree with you often enough, but I detest blog-comments-as-cheering-section.
In that regard, it’s nice to see Steig venturing away from his so-comfy home for a bit.
Nick:
You are misleading on the WUWT comment.
Now where does it state that Reviewer A wrote a 88 page Review of the paper? Nowhere.
Reviewer A caused an 88 page review process that in end caused the editor to find a 4th reviewer.
Now lets look at the comment from Kenneth:
Notice Kenneth said three reviews AND 88 pages? That is a true statement after 3 reviews were done there was a total of 88 pages generated both of comments and response. For you to be correct, in your interpretation in anyway, Kenneth would have had to stated: “after three reviews OF 88 pages”.
As to the comment from Kim:
Now that one can be misinterpretated by those not familiar by now with the O10 paper as meaning 88 pages of Review comments alone.
Nick,
You link two 3 comments in a post with 483 comments. The first link contains acomment by Kenneth Fritch which includes this.
This isn’t saying 1 review was 88 pages long. The editor has had to handle three reviews which involved 88 pages. The editor clearly reads reviews and responses. So, this is not the misconception– it is the correct value. It’s not implying any one review was 88 pages long.
The next link you supply is kin.
No offence to kim… but he is not always the precise with words. (Nor am I.) Whether kim has a misconception or not, I can’t say.
You then provided finer detail to what Kenneth — who does not hold any misconception. You wrote wrote
Whie it is true that if you Kenneth’s second sentence from the one immediately proceeding it, it can appear he had a misconception and was somehow communicaiting that a review had 88 pages, that’s not how his comment read when he wrote it. It only reads that way when you took it out of it’s original context. Then, taking it out of context, you add additional information– which is fine.
But the fact that you can make it seem kenneth suggested a single reviews was 88 pages by stripping his comment from it’s context is not particularly good evidence that Kenneth thinks so.
I have no doubt that you can find the occasiona person who write a comment that suggests they think one of Eric’s reviews was 88 pages long. I am even more sure that if you take comments out of context– as you did with Kenneth’s– you can create the impression that more people harbor this misconception. But I don’t think there is any wide spread misconception about this.
lucÃa: “lderry” wrote an identical post in the blog of andrew revkin link http://community.nytimes.com/comments/dotearth.blogs.nytimes.com/2011/02/11/on-peer-review-and-climate-progress/?sort=oldest&offset=3
Nick–
Sorry. I think I’d better go increase the threshold for links before things go into spam. I’m going to bed and so I won’t be able to fish them out. I’ll change it to… 4?
Nick/Amac
All comments at WUWT go into moderation not just yours. However depending on time of day the time until being posted can be short (ie during “primetime”) or long (ie 4 am EST).
As to Cheering section at WUWT I have seen some very highly contested debate between people in the comments. Usually between Dr. Svaalgaard (sp?) and some others over Solar mechanics and Willis with a few others or between Bob Tisdale and Frank Lasner.
Nick– The spam threshold was 5 links. You’re comment contained 6. I changed it to 10 links.
jorge c–
Yes. And raypierre read my comment over at dot earth and emailed to tell me “you responded to Lou Derry’s comments on peer review
as if you knew what you are talking about, but your comments there do
not actually make any sense to me” and asked me if I’d ever published. Of course I told him I had, as that is true. 🙂
Maybe raypierre will now comment to explain how authors don’t feel even the teensiest-beensiest need to modify their manuscripts to comply with reviewers recommendations on things that do not affect the main thesis of their paper… blah… blah… blah…
wowww!!!!
i must tell you that i read mr. revkin blog, but skip the comments of sere, susan anderson, dhogaza, melty, raypierre and others!
Since Dr Derry has been kind enough to stop in here after visiting Dot.Earth, I’ll offer a couple of thoughts on this instance of peer-review (gone wrong).
At the tail end of Chip Knappenberger’s recent guest post, I pointed out that if Steig should submit a rebuttal of O’Donnell ’10, it would be inappropriate for an editor to ask O’Donnell to serve as an anonymous peer reviewer. (Comment #69251).
Ryan O remarked (Comment #69280) –
On reflection, I think Ryan O is wrong on this. I think his error is shared, and contributed its part to this week’s blowup.
.
JCS editor Broccoli was wise to want Steig’s insights on O’10, but he showed poor judgement in offering Steig a spot as one of the three anonymous peer reviewers. Steig had clear conflicts: it was in his personal and professional interests to see O’10 denied, weakened, or delayed. From the tone of Steig’s remarks at RC and at Eli’s, he seems to reject the notion that he was conflicted. That doesn’t matter: notwithstanding, the appearance of major CoIs should have been blindingly obvious to all. Including to Broccoli and including to Steig himself.
Broccoli invited Steig to serve in a role where these (apparrent) CoIs were hidden from the O’10 manuscript’s authors.
Steig agreed to serve in a role where his (apparrent) CoIs were hidden from the O’10 manuscript’s authors.
Neither decision was correct or appropriate.
Together, these decisions put the manuscript’s authors in a very awkward position. In general, MS authors are well-advised to heed the criticisms of peer-reviewers, because they are at the same time expert and disinterested. Usually, pushing back on strongly-worded “suggestions” of these experts is a good way to garner a Rejected notice.
Yet unlike Reviewers B and C (and ultimately Reviewer D), Reviewer A was far from disinterested — he was the most interested party imaginable.
I strongly suspect many of those 88 pages would never have needed to have been written, if the O’10 authors had been freed from the theater of acting as though Reviewer A was highly critical and neutral. This sentence would have covered perhaps a third to a half of the Responses to Reviewer A: “We acknowledge that Dr Steig disagrees with our methods and our conclusions. For the reasons already stated, we stand by our analysis.”
.
Dr Derry, if you are still around at this late hour, I would be very interested to hear your thoughts on this aspect of the O’Donnell/Steig controversy.
AMac:
I agree completely with your position. I have asked the same question on numerous sites – why was Steig an ANONYMOUS reviewer? I have yet to see a response. I have no problem with Steig being a reviewer – simply not an anonymous one.
On a different subject — Dr Steig wrote a large number of different suggestions, objections, and observations over the course of his three reviews. Some people on the ‘skeptic’ side of this argument have interpreted seeming inconsistencies in Steig’s narrative as evidence of his bad faith.
In like fashion, O’Donnell et al. wrote a complex manuscript, performed major surgery on it, and wrote lenthy defenses and explanations in response to reviewers’ criticisms. This has provided grist for the mill as far as O’Donnell’s alleged bad faith.
The subject matter is complex to begin with.
It occurs to me that Steig may not always bat 1000 when it comes to remembering just what he said, and just when he said it. Rather than assuming ill intent, perhaps people should reserve judgment as to motive. it’s easy to make mistakes in a pressure-cooker situation.
Nick, note your links are missing the “/” between the page title and the hash-mark (#)).
(I do have a comment supporting your observation… not sure which thread it’s on anymore.)
AMac –
I think you got it just right. Allow Steig to comment on the paper, but without the anonymity which implies disinterestedness. [Is that a word?] It would reduce the pressure implicitly felt by the authors to respond to the criticisms, as Lucia describes on her dot earth post.
Dr. Derry –
Not only is the 88-page figure understood well enough here, I think you’ll find the same measure of comprehension at Bishop Hill’s site, e.g. http://www.bishop-hill.net/blog/2011/2/10/liz-wager-on-conflicted-peer-review.html?currentPage=2#comments
Re: Carrick (Feb 13 00:06),
Thanks, Carrick – but the links still work for me. Does it cause a problem?
As we all know, this isn’t about truth at all, its about plausibly deniable accusations.
This whole thing is crazy.
.
Amac: Nothing went wrong with this peer review. It was OK for the editors to choose Steig as a reviewer, as long as they take his comments with an appropriately-sized grain of salt – which evidently they did, since Steig’s objections were eventually overruled. Similarly, it would be OK for them to choose O’Donnell as a reviewer of any rejoinder by Steig, especially if this rejoinder takes the form of a comment (though in this case it would likely not be anonymous). Of course, considering the bad blood, the editor would take O’Donnell’s comment with an even larger grain of salt. Which is fine!
.
Harold: It would reduce the pressure implicitly felt by the authors to respond to the criticisms,
.
It would do no such thing. As has been already repeated several times, the final arbiter of publication or rejection is the editor. You can dodge any criticism you like as long as you think you can convince the reviewer that the criticism is unwarranted. The reviewer can just get lost. Which, again, is what happened.
.
But then you are not expected to launch a tirade of crazy, career-threatening accusations when said reviewer publicly repeats the very same criticism he made in his review, which you decided to ignore.
Eric (http://rabett.blogspot.com/2011/02/some-useful-comments-from-elsewhere.html#comments)
I still wonder why so many people don’t sum it up in just one sentence
O’Donnell (http://climateaudit.org/2011/02/11/ryan-odonnell-responds/)
that whatever was original in Steig et al 2009 was based on faulty mathematics; and that whatever was correct in Steig et al 2009 was already known
It is interesting to observe the continuing unhinged rantings of Steig.
O’Donnell, based on the information available to him, made a number of personal statements about Steig that he later withdrew due to a lack of firm evidence. Given what he knew, they seemed to be reasonable conclusions.
Steig, however sees no reason to to continue to bad mouth anyone that in any way questions his faulty Nature publication. His latest assault on the reviewers of the O’Donnell paper provide ample evidence that he has completely lost all objectivity in matters where S09 is involved. It was evident enough in his antics during the review. Then the snarky post on RC where he berated O’Donnell et al for “not doing it right”, using all the hand-waving he could muster.
In theory, the other reviewers are anonymous, he doesn’t even know who they are, but according to him they are “knuckleheads”.
And I’m betting there won’t be a single squeak from the echo chamber at RC. Just like when Mann defended his upside-down varves with pure bunkum.
Obviously this is to be interpreted subjectively. For my part, it seems clear that Steig’s opinion was not treated by the editor as that of a hostile reviewer for a very long time and, taking a broad step back, allowed Steig’s objections far too much weight for far, far too many months.
It’s reasonable to ask if O’Donnell et al. were at least in part responsible for this elevation of the importance of Steig’s opinion, though, by being too accommodating of Steig’s obfuscations and, by doing so, allowed Mr Broccoli (assuming he was himself lacking in expertise in this area) to form the impression that Steig’s input had greater merit and/or value than it actually had.
My impression (and I welcome a substantiated correction, if wrong) is that it was only after Ryan had learned that Eric was Reviewer A that he grew some author-gonads and put his foot down, rejected Eric’s hostile-to-the-paper reviewer objections, causing the editor to finally make a change to the reviewer landscape.
Eric was never going to give the nod to this paper until either its substance could be dismissed or could be peddled by him as being an affirmation and ringing endorsement of a certain front cover of Nature. Only when Ryan could directly and certainly (with evidence, if necessary) point to Reviewer A as a hostile reviewer was it possible for this non-academic to confidently break the peer-review log-jam and move to publish regardless. Had he *known* (rather than *suspected*) that Reviewer A was hostile earlier, or if he’d pushed back harder based on his suspicions and been less accommodating, he might well have been published much earlier and saved himself some grief.
Heh, the leading edge of ‘the shock of an 88 page review’ was sharp indeed, precisely so.
================
Look at this list of CA posts on S09 over the years.
http://www.google.com/search?q=steig+09+%22climateaudit+org+%22+site%3Aclimateaudit.org
From the moment S09 was first published, the “professionalism and courtesy” was the first thing to go missing, as is the SOP for CA, it’s just one long stream of snark. Yet Steig is expected to never be caught up in this mud pit.
Re: toto (Feb 13 05:16),
> Nothing went wrong with this peer review…
To me, the key error is that the process as executed presented the manuscript authors with the form of disinterestedness, without its substance (with respect to Reviewer A).
I am not sure where you disagree with this claim:
* That an actually disinterested review process isn’t important?
* That the appearance of disinterestedness isn’t important?
* That editor Broccoli did create a process that avoided real and apparent conflicts of interest?
* That avoidance of real and apparent CoIs might be somewhat desirable in the abstract, but that peer-reviewed journals have no policies in place on the subject?
.
> The reviewer can just get lost.
This would not be advice that I would give to any aspiring author. I’ve reviewed a couple-dozen manuscripts in a different physical science over the years, so I base this on some experience — not nearly as much as Dr Derry or other seasoned academics. And since I’m pseudonymous, you can believe this or not.
I’m curious: do you base your (seemingly-cavalier) attitude on first-hand experience as an author or reviewer?
From deltoid
There are still plenty of unenlightened folk in the blogosphere peddling this:
” But it appears that he did hide the fact that he was Reviewer A and dumped 88 pages of critique onto O’Donnell’s 8-page paper ”
from http://community.nytimes.com/comments/dotearth.blogs.nytimes.com/2011/02/11/on-peer-review-and-climate-progress/?permid=3#comment3
Louise–
Yes. There are a few people who have information confused. Unfortunately, commenters at dotEarth sometimes don’t read what Ryan, Jeff or SteveM actually wrote and they aren’t responsible for that forum. All three have given the correct statistics– and multiple times.
Corrections– The specific information presented at WUWT, TAV and CA are now discussed at dotEarth. See http://community.nytimes.com/comments/dotearth.blogs.nytimes.com/2011/02/11/on-peer-review-and-climate-progress/?sort=oldest&offset=3
(It’s page 3 of comments, see lderry followed by Boballab and lucia.)
what does Steig mean when he says “…is should not be used”?
Shug–
Who knows? But it seems whatever argument he wanted to advance, he felt he couldn’t argue it in his capacity as a reviewer. Certainly, the review he wrote didn’t say it should never be used. Nor even that it should not be used in this instance. What he wrote read like he was saying that it appeared to be the best method in some context.
“If I’m telling some of those people what they already know I’m sorry if they find that bothersome.”
Nothing personal Iderry, but I do, because instead of discussing the reviews and the tone of the reviews, we end up discussing what someone said about the reviews and than interpreting what they meant when they said it.
Nick, I believe there were 88 pages of correspondence which were written for the review process. I still haven’t bothered to count them myself and apologize for any inaccurate wording on my own part.
” I SIMPLY DID NOT THINK I COULD ARGUE WITH THE EDITOR OR THE KNUCKLE-HEADED REVIEWERS that is should not be used”
.
This sentence is ‘knuckle-headed’. Read it in the context of the post…it just doesn’t make any sense. I am saying this because ES thought it is so good that he posted it a few times.
Amac,
I wasn’t still up, but it’s morning and the coffee is here.
Reviewers all have biases, explicit or not. In many cases, for someone to be truly disinterested they probably don’t know the subject mater that well. I don’t really see how, on a practical basis, you could come up with a system in which reviewers with a somehow defined conflict of interest had to be identified but others not. The shades of gray there would certainly be beyond my ability to navigate. Further, as an editor, one receives submissions in fields you don’t know well, i.e. you don’t know the players and who doesn’t like who etc. And then there are other issues. For example, there is some truth to the notion that young scientists tend to be more critical, but of different things, than more experienced ones. It’s often just impossible to know who might take an initially negative view and who might be inclined to be positive. Puffball reviews are in many ways less helpful than dissections. Hopefully, if everybody is doing their job, the content of the ms will be the main issue, but I don’t believe that we can make this system “bias freeâ€. So you make your best attempt at getting technically qualified reviewers, expect professionalism, and hope for the best. It can sometimes be hard to get reviewers, because good people are very busy. If authors and reviewers don’t agree, it’s up to the editor to navigate disputes. In the end, his/her judgment matters. But it’s a judgment, and so others might do it differently. Furthermore, if a reviewer is especially nasty or unfair, it can backfire in the long run. Credibility and reputation matter in this business. People know this and typically moderate their behavior accordingly. Professionalism matters. It is also not uncommon for an author to rebut a criticism rather than change the ms. If, as editor, I’m reasonably satisfied that the author’s point is sound I don’t ask for changes on that point. In most cases, the changes requested by reviewers result in improved manuscripts, often very much improved. But peer review is certainly imperfect – no argument there. What’s important is that, in the long run, boring papers will be ignored, interesting papers will spur further work, and over time, a better understanding almost always emerges. But the process behind the review of any given paper is a series of compromises and judgments – sometimes easy and sometimes hard. I honestly don’t think that new rules on reviewer disclosure would substitute for judgment. We expect reviewers to behave professionally, and almost all do. Open review is often touted as the solution, but in my experience it doesn’t always work very well. Our journal actually tried an open review system, in which anyone could comment on a paper, but we dropped it for lack of success. Others have had the same experience. I sort of look at the current peer review system the way Churchill looked at democracy – it has flaws, but I haven’t seen a better system.
In this particular case (O’Donnell), it is my opinion that the system worked about as well as one could hope, and I don’t see anything to criticize in either Reviewer A’s comments or the job the editor did. It all seems pretty normal to me, well within the range we see for other papers. The paper as published is measured in its language. Of course most papers don’t have legions of bloggers eager to pass judgment on them (or, having passed judgment a priori, say something anyway). It seems that the amplification of the remaining (and entirely normal) differences between the authors on the web has resulted in an enormous flap over not much. If the authors were left alone, I suspect this would have never escalated the way it did. Much of the commentary trying to establish detailed time lines of who knew or said what when seems just pointless to me, and many of the third party accusations are over the top. I think everybody needs to go back to their day jobs. The relentless attacks on motives, personalities, and process details are a sad and destructive distraction. What matters is what is happening with Antarctic climate, not all the rest of it.
I’m immediately hearing echoes of the Parliamentary Enquiry when, on being challenged about his failure to provide data and methodologies for the purpose of replication, Phil Jones simply responded that it was not standard practice to do so.
.
I accept your assertion that the way O10 has been through the peer review process is commonplace or even standard practice in peer review. The problem, at least for me as an onlooker looking in, is that whatever mistaken impression I had about the integrity of the process was entirely misplaced, because there are simply not sufficient mechanisms as safeguards to defend the integrity of the process.
.
So while you can, I’m sure, state absolutely that O’Donnell et al.’s experiences are not exceptional, are not uncommon and are not outside the normal machinations of peer review, I can most certainly infer from your statement that the whole junk process is not only a heap of crap but is the best thing we have.
.
Consequently, having learned from you what you have graciously taught, and accepting that what Jones stated at the Parliamentary Enquiry is true, you have made me even more sceptical of the integrity and value of climate science than I ever was before. I am horrified to learn from you that this form of junk science gate-keeping is as good as it gets. I am mortified to learn from you that this is common across all of sciences. “Peer review” speaks nothing to the integrity of scientific discipline if what you say is true.
The Nibelungentreue that some people here demonstrate towards Steig is strong and a very german charactertistic. Makes me wonder: Why are there so many german names in the official climatology, names like Schmidt, Mann, Hansen, Steig? (I’m not mentioning Rahmsdorf and Schellen-Huber because these two actually are german).
lderry–
I largely agree with your most recent comment. Possibly more than largely.
I’ve consistently stated I think Broccoli did a good job. Might he have done things a little differently? Sure. Might it be good for the system to somehow capture the fact that venomous comments have flow back and forth pre-submission so as to ensure that editors will be alert to evidence of possible disinterest that goes beyond the usual? Sure. But, it seems Broccoli handled this well.
Absolutely. I once received a manuscript just before it’s lead author was awaiting for galley proofs from J. Fluid Mech. (http://www.che.caltech.edu/faculty/brady_j/index.html). John had overnighted them to me. I read through, and noticed the page containing the equation that represented a main result was missing. He overnighted the page. I then noticed the equation that represented the main result was missing. I phoned him. If I recall correctly, he thought it quite possible that he’s actually submitted the exact set of pages he sent me and the reviewers did not catch the missing page or equation because he’d gotten puffball reviews.
I don’t know if John’s recollection would match mine entirely– but this is my recollection. (The paper was excellent and merited publication. But of course, one would wish to know the main results and reviewers not noticing because they trusted John to write a good paper and only skimmed is something of a disservice. Of course, I’m not sure that’s what happened.)
This is a point I have repeatedly made in various blog comments. It is worth nothing however that I have often made it in a somewhat different context. That is: certain bloggers write things that appear to strongly suggest that they way lay people can identify bad papers are bad papers are rebutted by formal comment to the journal.
Of course, sometimes formal comments are written– but mostly both poor papers and papers that fall outside areas of great interest are ignored for various reasons. Sometimes good papers are also ignored for various reasons. They might appear in obscure journals. Their authors might not be sufficiently well connected to draw attention of busy researchers to them and so on. They might be on a topic that is presently obscure and not of great interesting. But, for the most part bad papers are not rebutted but ignored.
Out of curiosity, what were the difficulties? Chip suggested open review on another thread. (here: http://rankexploits.com/musings/2011/theres-got-to-be-a-better-way-forward/ ) I have mixed feelings about open review and explain a bit in comments. So, that’s being discussed elsewhere. (If you comment on that over there, that might get attention and we could all learn your experiences.)
For better or worse, this isn’t going to happen. Some people may go back to their day jobs, but email, forums, blogs, twitter, and internet discussions are here to stay until something else replaces it. If this is a problem for any field or journals, journals are going to need to adapt to this.
Simon,
Since you’ve concluded that scientific review is a morasse, what do you suggest to fix it? Peer review implies review by people with specific knowledge of the field, and usually the result is a noticeably improved product. Improved quality is a direct result of the back and forth feedback. But, review by “peers†inevitably leads to conflicts of interest. No one disputes this. Integrity, professionalism, judgment and some feedbacks (to reward the above or punish the lack of the above) can manage those conflicts – if you believe that enough scientists have any of those qualities. If you don’t (and I have the impression you don’t), then what? Review by people without knowledge of the field, and therefore presumably fewer conflicts of interest, is mostly useless and rarely adds value but takes up time and money. So, you could have a policy of no review – publish anything you want. The web certainly makes that possible. What are the consequences? A few papers that should be published but would have a hard time getting out because of resistance by the established players would appear more easily. That would be good – but would anybody notice? Because there would be an enormous slew of worthless, error-laden, sloppy or crazy stuff competing for attention, spread out all over the place, and how would anyone sort through it? It’s already very difficult to keep up on developments in any field, and this would make it basically hopeless. I personally don’t think we’d come out ahead – not even close. But of course I’m one of those ethically challenged scientists and what do I know anyway? In all seriousness, what do you suggest? Maybe Google could come up with an algorithm that finds only “good†papers (I can dream, can’t I?), and then we could dispose of journals altogether and just web surf. But we’d miss out on the benefits of peer review, and those are real, just as are some of its failings. I would urge you to really think through the consequences of any system. I agree that “because we’ve always done it this way†is a lame argument. I’m not making that argument.
Further, there is a second filter. Papers get published that shouldn’t – nobody disputes this either. As I said, if they are boring, weak or irrelevant, it’s a loss of time and effort but has little impact on the science because they are ignored. A lot of the scientific literature has few or no citations. Look at ISI Web of Science for papers that have been out a decade or more and have 0-3 citations. Those papers clearly didn’t move the field much one way or the other. Basically, nobody paid attention to them. If a paper is actually interesting, someone will work on the issue, maybe a lot of people will. If the paper has flaws they will be exposed, fixed, reworked. I can’t think of a paper that couldn’t be improved – rarely does anyone have the “last word†on a subject. Over a bit of time, errors get weeded out, assumptions improved, better methods/ideas get developed. That’s what I see happening in the S(09) – O(10) case, and the story is clearly not over. Scientific “progress†is slow, messy, and full of mistakes. Despite all that, it seems to sort of work in the long run. If you can think of ways to make it work better, that would be great. But accusing everyone involved of lack of integrity doesn’t help much.
Alright, back to my day job.
Great discussion. I appreciate Dr Derry’s participation. Just to declare my biases – I think having the lead author of a paper being critiqued as a reviewer should not only be allowed, it makes sense. As to anonymity I am less sure, but I think that the editors usually have the savy to recognize unfair situations when they see it. O10 may well be an example of this.
–
More transparancy in the review process may be useful to support editors who feel they are subjected to undue political pressure
layman lurker–
I don’t think there is any evidence to suggest Broccolli the editor felt he was subjected to undue political pressure. He may not have taken Steig out of the process as soon as some other editor might have done, and maybe he would have done it sooner if he’d been aware of the amount of ill-will on display at blogs. But for all we know he was aware of that and thought Steig would be a good choice both for specific knowledge and to forstall the possibility that potential critics of the O’Donnell paper would complain that potentially harsh critics were intentionally excluded.
At the end of the review process, someone like Steig could present criticisms (and generally without worrying that people might compare what he wrote in his review to his later criticism.) But Steig could not claim his views were not included in the review process or that the editor did not take steps to assure himself that others in the field largely disagreed with Steig and thought the paper worthy of publication not hypothetically, sometime in the remote future because it held the promise of someday being crafted into a valuable contribution if sufficiently guided by a wise and benificent reviewer like Steig who was apparently “willing” to write long review after long review after long reviews.
By the time of publication, the editor knew that the bulk of peers would find O10 a valid contribution to the literature and that this assessment would hold up even if some group of teed-off opponents tried to represent it as otherwise. As arduous as this was for O’Donnell and co-authors, what Broccolli did was helpful to them.
L. Derry,
Thanks for your thoughtful responses.
I think your key point with respect to conflict-of-interest is that it’s not yes-or-no. Thus, any quality-control system — be it based on Editor-centered peer review or something else — is going to have to deal with shades of grey, in how it is structured and implemented.
It seems to me that you implicitly argue that journals have no need for CoI policies — that the potential for apparrent CoIs is everywhere, that any such policy would be unworkable, that such a policy would have unintended consequences, e.g. driving away qualified reviewers (who are scarce in any case).
Am I interpreting you correctly? Do the journals for whom you volunteer as editor and peer-reviewer indeed lack CoI policies? Or, are their policies onerous in practice?
Overall, I would take a different view.
Policies on CoI and apparrent CoI should recognize that shades of gray are nearly the norm, for the reasons you outlined. Their provisions should be triggered by obvious and extreme cases. In my opinion, a good policy would have alerted Broccoli and, ideally, Steig to the problem, regarding Steig’s service as anonymous Reviewer A.
I concur with your response to Simon Hopkinson, supra. As for your challenge to him to envision a better replacement system: basically, I can’t. Using the web to make the process more transparent is, I think, a useful modification, e.g. as is done by Climate of the Past. But you’ve pointed out that such an approach is a tradeoff of some problems for others.
Jeff Id
“I don’t know where Eric is at right now emotionally but reading his comment concerns me a bit. Hopefully he’s able to regain some perspective.”
Maybe dealing with jerks and pompous windbags is merely frustrating and not indicative of mental stress. Just an alternate theory to ponder there, sunshine.
Dr. Derry:
I appreciate your continued involvement in the discussion. However, I strongly agree with AMac on the need for what he terms a CoI policy. I am still at a loss to see the downside of identifying the Reviewer in a cases like S09/O10. The Reviewer with a potential CoI of course should be notified that their review would not be anonymous.
Other cases where the critique is less narrowly focused are probably a fuzzier area.
lderry,
I don’t have any good suggestions but I don’t think I need any at this point of my journey into peer review enlightenment. The first stage of fixing a problem is recognising there is a problem to fix. I don’t pretend to have got further than this, but I have at least got this far. Conversely, I’ve read your descriptions of the way things are, none of which are suffixed with any urgency or impetus to resolve what you surely must also see are serious deficiencies in the process of scientific advancement.
It is clear, though, that this improvement in quality is not assured by the process. It is merely hoped. As much as improved quality may be the result of the review process, it is no more certain that this will happen than it is that advancement may be hampered by the very same process. Absent is a robust mechanism to ensure that the advancement of an improved prevailing theory is made rather than prevented or deferred.
Again, my concern is that it is not enough to hope that a bridge has enough girders to support the traffic that crosses it. Recognising that the bridge is perpetually at risk of failure, do you not perceive the imperative to shore up the bridge, or plan the building of a new bridge?
.
The rest of your post in response descends into strawmen. The alternatives to peer review that you describe are not my suggestions, they are yours, and they are obviously – being your strawmen – silly.
Boris (Comment#69411)
February 13th, 2011 at 11:18 am
Ahhh, the devil made him do it.
http://www.youtube.com/watch?v=0SLifea3NHQ
Lucia,
In response to Eric’s comment, “I SIMPLY DID NOT THINK I COULD ARGUE WITH THE EDITOR OR THE KNUCKLE-HEADED REVIEWERS that is should not be used.”, you ask “Is eric calling the other reviewers knuckleheads?” Later, (in Comment#69356), you respond to a question “Where does calling your fellow reviewers “knuckleheads†fall on the hyperventilating ethics scale for our harumphing academics?” with the following “Well… it’s probably not unethical. But it might make them a little grumpy.”
My take on this is that Steig is NOT referring to the reviewers B, C, D of O10 in the first comment. (First off, I don’t see how he could know who they are.) Instead, he is referring to O’Donnell, Lewis, McIntyre, and Condon; in Steig’s mind, they are not “authors” of O10 but merely “reviewers” of S09, and knucklehead reviewers at that.
chopbox,
Yet another example of why bloggers should not comment in anger.
I think that one could win an award for “I know that’s what I said, but what I really meant requires you to understand what I was thinking”.
Alternatively, maybe it can also be explained by my response to Boris above.
chopbox:
LMAO. That’s good, that’s really good. Who’s the editor he’s referring to? And why did he conflate the “editor” with the “reviewers”?
Steig: “THE EDITOR OR THE KNUCKLE-HEADED REVIEWERS”
Use your brain.
Ascribing ill motives: rarely insightful. Rarely enlightening.
In situations like this, most of people are acting out of good faith, most of the time. We each have different concepts of what “the right thing to do” is. Duh.
Bad idea to assume that we can quickly or easily understand those we disagree with.
chopbox (Comment#69415),
“My take on this is that Steig is NOT referring to the reviewers B, C, D of O10 in the first comment.”
I wonder if you appreciate how badly this reflects on your general reading comprehension skills? It says what it says… the word ‘reviewers’ generally means, well, ‘reviewers’. And ‘knucklehead’, most people understand that word as well, especially when presented with a glaring example.
Chopbox–
Has Eric suggested he meant someone other than the “reviewers” when he used the word “reviewers”?
You seem to think it’s necessary to know who the reviewers before you can describe them as “KNUCKLE-HEADED”, which is an odd but very.. umm.. determined view.
Umm.. nahh. I’m afraid, again, you’d have to be really determined and barrel-visioned to propose this. Maybe it works for you, but I suspect Eric would implore you to hush, too, and not try to help so much.
So, in summary:
Status of the claim Steig demanded a change, then criticized O’Donnell for it: LIE.
Status of the claim Steig had a copy of a paper, and asked for one as though he didn’t: LIE.
O’Donnell’s various allegations of fraud and duplicity: RETRACTED BY ODONNELL, REDACTED BY CA.
On the other hand, O’Donnell has admitted breaking his word, making false allegations, and violating the rules of peer review he agreed to. Lucia’s leash got a hard jerk from Chip Knappenberger, but she continues to double down, despite the fact that all of the allegations she used to accuse Steig of being “the Rod Blagojevich of science” are in tatters.
Remind me why you are celebrating this donnybrook? Oh, right, it’s that defense mechanism, where you retreat from the harsh light of reality into comforting fantasy . . . what was the term? Begins with “D.”
Two syllables, begins “W”. Rhymes with a word meaning “to yearn”.
Robert–
Reading review 1, it appears Steig heavily criticized the choice of kgdn=7, and O’Donnell included changes to address this criticism. In review 2, Steig’s wording certainly reads as if he recognizes that addition of ridge regression is in response to Steig’s criticism. By review 3, Steigs wording can be read different ways ranging from insisting (using the word insist) that results from ridge regression be shown more prominently. As JNG words it in his blog post: “Then, I probably would have taken pleasure in my public rebuttal to Steig’s criticism, already having handy as a response my response to his third review. I might even have included an excerpt from an anonymous review, showing that the reviewers agreed with our choice so strongly that they insisted on it.”
So, notwithstanding Steig’s interpretation that he did not insist on iridge, it appears that at least some people interpret what he wrote as suggesting that at some point in the process, he sure as heck seemed to be insisting on it! (And yes, he later critized its use. In this context his expectation of anonymity does not reflect well on him. )
Who made this claim? I never claimed he had a copy of the paper.
Fraud? O’Donnell retracted a very specific allegation and took back the word duplicity.
I wasn’t aware I was on a leash.
Wording in Steig Review 3 does indicate reviewers were communicating with each other in some way. Steig writes,
This makes interpreting his use of the word “reviewer” in “KNUCKLE-HEADED REVIEWERS” to refer to the other reviewers (B,C & D), and not the authors.
‘Fraud’ Robert?
.
I think you are projecting your loose style of using words onto observers whose comments you must be reading.
.
“Steig: “THE EDITOR OR THE KNUCKLE-HEADED REVIEWERSâ€
Yes, Carrick, can’t you see that it is too painful to even imagine why Steig would damage his own case by calling his fellow reviewers ‘knuckle-headed’. Therefore, it must have been O’Donnell et al that he called ‘knuckle-headed. 😉
When you have a paper that challenges a prior paper, there is something unethical in having a reviewer be both the author of the prior paper, and be anonymous. If a publication chooses to include the author of the prior paper, that reviewer has a vested stake in the challenging papers failure, and therefore should not be anonymous to anyone. How peer review has evolved without this is a mystery to me.
Just so it is clear where I stand, In think it is an excellent idea to have the author of the prior paper be a reviewer, just not anonymously.
lucia (Comment#69427) February 13th, 2011 at 1:02 pm
Thanks, Lucia. I was trying to make sense of Eric’s comment while simultaneously believing that reviewers would not be aware of each others’ comments, and you’ve now laid that to rest. I think my attempt (that Eric was actually referring to the authors of O10 and not the other reviewers) to make sense of that comment can now safely be ignored.
Just for my own information, is it usual for reviewers to read each other comments?
chopbox:
Yes.
Thanks, Carrick.
re: Lucia (Comment#69408)
Agreed. I wasn’t trying to single out O10 as an example of undue political pressure at all. In general it strikes me that editors would appreciate being able to fall back on transparent processes to ward off potential political accusations of bias (from either side of fence).
re: Lucia (Comment#69426)
Steig was ‘insisting’ that the iridge reconstruction should replace TTLS. I don’t see how it can be interpreted any other way. He first ‘insists’, then he clarifies that he is insisting on iridge, not kgnd=5 TTLS. He is making an unequivocal and direct appeal to the editor. What part of ‘insist’ do we all not understand?
Leading you to compare him to a convicted felon who tried to sell a Senate seat — as neat an example of “credibility suicide” as I’ve ever seen.
O’Donnell has seen the writing on the wall and begun the process of trying to rehabilitate his reputation by apologizing and withdrawing his “mistaken facts.” Evidently he would like to be regarded as a serious person again someday. You, however, seem determined to move in the opposite direction, despite having no evidence to back up your shrill allegations of duplicity and fraud.
Maybe O’Donnell will get published again someday, despite his unprofessionalism, but you on the other hand seem to be gunning for Steve Goddard’s old job at WUWT.
Carrick, your short answer should have been preceeded by the words “in my experience”, not all fields are the same. In my own extensive experience of peer review I as a reviewer have never seen the comments of other reviewers except in exceptional circumstances. For example where one of the other reviewers had said something in direct contradiction to one of my own points and the editor wanted us to resolve our differences. In my scientific field the only people who see all the reviews are the originating authors and the editor.
As an additional comment, and as others have suggested, in my own field, in a situation such as the one in debate, the author of the original paper would not be on the review panel of the subsequent paper because of the obvious conflict of interest. However he would be invited to comment on the final outcome of the peer review, and the authors would have been informed that he would be so doing.
In my own view I see nothing wrong in peer reviewers being identified to the author. Indeed in some journals the peer reviewers are identified in the final publication
Layman,
ES’ defense is that he did not ‘insist’,but that he ‘insisted as per O’Donnell’s own wishes’.
Here is a presentation I saw at Roger Pielke Jr. that helps (I think) to characterize what is going on in the ‘climate wars”:
http://www.authorstream.com/Presentation/jhaidt-819710-haidt-postpartisan-social-psychology/
Luica: Sorry, I known it is kind of off topic.
My advise for Prof.Dr.Steig; take 2 Prozacs and call me in the morning.
.
http://static.funnyjunk.com/pictures/knuckle0.jpg
.
BTW Did Dr.Steig made any changes in his paper according O’Donnell et al?
There are a number of larger issues that are raised by this affair.
1. Climate science is too big to be treated as a cottage industry. Because the results of research could have a huge effect on the allocation of government money, there needs to be more openness in the manner that climate research papers are peer-reviewed. Here it is clear that Steig was attempting to protect his own results and not look at the science objectively.
2. I find it hard to believe that Steig was totally out of the loop after he submitted his third review. Since his paper was published in Nature and was important to his reputation, I withhold judgment as to whether his statement is true. It is also possible that his colleagues at Realclimate obtained a copy and informed him of what was transpiring. Steig is a hothead who has not hesitated to accuse others of lying (See Air Vent link, posts #5 & #10 and author’s response to Steig at #33. Link http://noconsensus.wordpress.com/2010/12/05/warnings/) There is no reason to accept his statement by itself that he was out of the loop as true. I admit that it is possible that he was out of the loop, and I am not accusing him of lying. I am simply saying that I would like to see confirmatory evidence (such as typical practices at the Climate Journal) before I make up my own mind.
JD
Arthur, what’s your area?
As far as I can remember, I’ve always seen the other reviews and the authors responses. I don’t know why the editor would want to hide the other reviewers comments or the replies to them from you.
Also, I’ve always been on the review panel (or one of my coauthors) of any paper that was critical of my work. Obviously my comments were treated differently than those of other “independent” reviewers.
I’ve stopped a few papers from getting published too (including writing a few reviews that were longer than the paper), but that was because the paper was so bad that I didn’t think it was in the authors’ or the journal’s interest to have it published.
I think if a person makes obvious errors, and the original author points it out, that shouldn’t be discounted simply because he has a conflict of interest.
“Conflicts of interest are to be managed, not avoided.”
I also don’t see anything wrong with the authors making my reviews of their work public. If you think about it, to the extent that they incorporate my review into their manuscript, that already is “making it public”.
Reviews belong to the authors of the manuscript being reviewed, IMO, unless you specifically don’t give them permission to publish (example: You tell them you have data that contradicts their work, and include a figure demonstrating that their results are falsified by your data).
Shub, If your take on ES is correct then I’m afraid he was quite sloppy (and mistaken) in interpreting O10’s response to the first review. Clearly, O10 are offering support for their counter argument to reviewer A’s critique wrt TTLS (kgnd=5). Only the most casual and careless of readings would interpret it as O10 suggesting that iridge should replace TTLS in the manuscript. Yet the fact that Steig ‘insists’ on this is not in question. Here is the relvant passage from O10’s response to reviewer A’s first review:
I would encourage everyone who has not read to make the effort. Ryan was very meticulous in his responses. There is no mistaking what it is that he is responding to. There is no mistaking here that O10 are supporting their counter argument (that kgnd=5 is not the optimal truncation parameter for TTLS)
–
Another question. If Steig is suggesting that it was O10 who was insisting on replacing TTLS with iridge, then why does he throw in the comment (in the “insit” passage from the second review) stating that O10 should not to hold back with irdige for future work?
Shub–
I’ll preface this with a link to the various communications:
http://www.climateaudit.info/data/odonnell/
It seems more like this to me:
Review 1: Under problem 4, Steig criticizes the authors for choice of kgdn=7. He wants them to use a different value and explains why.
Response: O’Donnell responds to this criticism in their first response defending the choice of kgdn=7. In addition, owing to the discussion of kgdn=7, the O’Donnell include additional methods as cross validation in the revised manuscript. IRidge is one of these.
Review 2: In this review, Steig discusses iridge, and writes
Plain reading for the first sentence at a minimum suggests Steig wanted the authors to show something instead of what they were showing in Figure 3. He wanted this sufficiently strongly to use the verb “insist”.
He says he is not necessarily suggesting kgnd=5, but seems to support showing iRidge results. In fact, the “it” in “If that is the case, why not show it? ” seems to refer back to iRidge results– further suggesting the reading that Steig is insisting (to use the verb at the beginning of the paragraph) that the authors show iRidge results.
Steig’s wording here certainly doesn’t suggest that he
In later parts of the review, Steig continues to argue against kgnd=7.
In their response to review round 2, O’Donnell includes a response to all reviewers, which begins
This indicates that O’Donnell et al. have incorporated the results using iRidge into the main text as a result of a request from one of the reviewers and they are informing all reviewers of this change. The TTLS/TSVD reconstructions — which used to be primary– are now relegated to the SI.
As this is a communication to all reviewers, one might think it possible for Steig to be unaware this is in response to his request, except O’Donnell’s response to Steig/reviewer A reads:
So, presumably, if Steig read this, he should have understood that O’Donnell made these changes based on their understanding of his review #2 and not as a result of recommendations from other reviewers.
At this point, if Steig thought his earlier review had been misinterpreted, and he thought iRidge should not be used, he could presumably have written Broccoli stating that he regrets the misundertsanding, but he did not mean his previous review to suggest that iRidge be used. That far from this, he thinks iRidge should not be used.
Instead, Steig reads Manuscript 3 with iRidge now displaying in the main text as his previous reviews seemed to indicate. His review of that manuscript says this:
And then later writes something that suggests O’Donnell should discuss iRidge in more detail:
So, at this point, it would appear
* Steig read the replies whose plain reading indicates that iRidge has been elevated to the main text based on his previous review and
* Steig said nothing to contradict the idea that his review suggested this be elevated to the text.
* Steig says use of iRidge makes sense to him
* Steig suggests that O’Donnel include a few sentences to justify some issues associated with the use of iRidge and
* Steig suggest O’Donnell include a bit of a tutorial on how the method works.
Those not capable of reading Steig’s mind could rather easily think that Steig agreed he suggested iRidge be elevated, thinks it makes sense, would like O’Donnell to include a bit of text discussing some possible shortcomings, but also encourages him to expand the discussion of iRidge to explain how the method works.
This sequence of events doesn’t suggest ” [Steig] SIMPLY DID NOT THINK {HE] COULD ARGUE WITH THE EDITOR OR THE KNUCKLE-HEADED REVIEWERS that is should not be used.”
I guess I can see how Steig doesn’t think using this method was his idea or that it was elevated because he required or insisted on this. But it also appears that notion had been communicated to Steig during the review process, and he didn’t object to that interpretation of review 2 at the time.
I still haven’t found anything that would explain why Steig thought he couldn’t argue with the editor or the knuckle-headed reviewers. Likely we’d need to see communications between the editor and/or reviewers and Steig. But given the contents of the reviews available to me, I could imagine that by the time he wrote Review 3, he might well have had a difficult time convincing the editors or other reviewers that he really believed iRidge should not be used.
Carrick (Comment#69440) —
In cell/molecular biology, I’ve sometimes seen other reviewers’ reviews, and not at other times. 10-15 years ago, it was most common to not see them. With the migration to web-based platforms over the past 5+ years, it’s become more usual to gain access to them, once the editor’s decision is made. It’s also been more common after the first round, when multiple rounds are needed.
In my experience.
Among other things, it’s obviously easier to upload pdfs to a website than to photocopy.
Lucia,
I think Jeff Id got it spot on earlier on. To parphrase things, to the OD2010 team it was a good sporting intellectual excercise showing up a “seminal” paper. To Steig it is [academic credibility wise] life or death. Right now, it would semm to me, probably more of the latter than the former.
Then again, we all know that it is unwise to show up for a gun fight armed with a knife.
Tetris–
Realistically, being shown wrong would not result in the death of Steig’s academic credibility.
Bugs,
Good list of the posts at CA.
Shame you didn’t read any of them else you would have found that there were almost immediate substantial criticisms of Steig’s methodology and indepth analysis of the paper.
Through a hazy tangled mess of qualifiers, a flicker of honest reappraisal appears. Now she need only finish the thought:
It could feasibly put it in a wheelchair, though.. and while I would never presume to know or understand Steig’s view of his own academic standing, I know how miserable life would be for me without legs.
Boris [Sunshine]
You still fail to understand that the real jerks and windbags are on the Team’s side of the fence. The more counter arguments to their gospel they run into, the more they jerk around and the more they pass wind.
The pains inflicted on all of us by the Team and its hangers-on are not longer counted as philosophical flatulence but rather in terms of intellectual kidney stones.
Problem with that for the rest of us is you don’t have the option of telling them to just piss off.
Sums up CA quite neatly, and the echo chamber that surrounds it.
Robert–
I think Steig can see it that way because it’s
a) possible he didn’t read the reply to all reviewers, or the reply specifically to him.
b) it’s possible he read it but he somehow did not pay enough attention to realize that people understood his review to indicate that iridge should be promoted to the main text
c) it’s possible his style of communication is extremely unclear– possibly as poor as willard’s,
d) it’s possible for people to delude oneself in order to feel better about oneself.
But…. I disagree with Ryan that Steig’s interpretation of the correspondence as a whole is reasonable. I merely think that it I can see how Steig can think what he thinks despite the fact that his interpretation of what things meant in context is rather amazing.
Lucia,
I agree with your conclusion; it matches Ryan’s post at CA.
.
SM has a new post up at CA.
Lucia,
That is correct only to the extent the methods by which he purported to be right in the first place are credible.
If, as it would very much appear, he concocted data as was argued at the time of the 2009 publication by one no one less than Trenberth himself, and then is shown to be active in trying to subvert the “peer review process”, his academic credibility is toast- as it should be. And that is no skin off the OD2010 team’s back.
tetris—
I don’t think Steig concocted data, and I don’t think Trenberth said Steig concocted data. That’s nonsense.
Bugs, oh Bugs,
May I remind you of my suggestion a few days ago that you make an effort and try to produce that liberating sucking sound and get your head out from where the sun don’t shine.
I promise you, you’ll see, it is truly liberating.
Lucia,
That is NOT nonsense. It is a matter of public record. With all due respect, I will remind you that Trenberth – in a direct response/comment to the 2009 Steig paper- is on the record to the effect that one “can not produce data where none exist”. You will no doubt be able to find the interview by googling it.
Where and when I got my academic training, making up data where none exists is “concocting”. No more, no less. Another term that comes to mind is fraud.
tetris–
If you believe what you claim is a matter of public record, please point to the public record that proves your claim.
I do not believe Steig concocted data. I’m willing to believe Trenberth used a flowery metaphor, but I do not believe Steig concocted data.
Simon–
I don’t think the Nature paper being wrong would have put Steig’s academic career in a wheelchair.
tetris, the Trenberth quote is “It is hard to make data where none exist.”
concocting is harsh, what Steig was doing was extrapolation of data, which is done often. Concocting would be applicable if Steig presented his data as observational data but he doesn’t. Using the word “data” for extrapolations is potentially misleading if extrapolated data is confused with observed data but extrapolated data is still data and a read of the paper (S09) makes clear that his data is extrapolated. This is not to say that extrapolated data is “good” data (but then again not all observed data is “good’ data) and the dispute is that Steig’s extrapolated data isn’t very good.
Here is a count of words in the original O’Donnell submission and Steig’s first review:
O10 (sans references and figures): 6275
Steig: 6987
From: wc -w 1A%2020100305%20Review%20A.txt 1%2020100209%20Submission.txt
6987 1A%2020100305%20Review%20A.txt
6275 1%2020100209%20Submission.txt
(After saving the PDFs as text)
So Steig wrote several hundred more words in his review than O’Donnell et al in their submission. That is the fact.
Tetris,
.
Then you should be making exactly the same accusations against me, as we did what Steig did . . . only with some corrections/improvements/refinements.
.
Come on.
.
Besides, Trenberth’s comment had to have been made without the complete context of what Steig had done, as I do not believe Trenberth would have made that comment were he to have fully understood what S09 did. Or maybe I’m wrong. I guess you could ask Trenberth.
Re: lucia (Feb 13 14:53),
No, but a finding that he abused the peer review system might just be fatal.
From his own University’s policies:
Tetris, I need to look up the exact quote and who made it, but the context was the sparseness of the data in the Antarctica and not anybody concocting it. It was early in the publication of S(09) and it may well have been in response to Antarctica data before the commenting person knew the AVHRR measurements were used in S(09) and O(10). I believe RyanO has prefaced O(10) with its intent in evaluating methods and not in establishing the accuracy of the data.
jeez–
Yes. But …universities don’t like to pursue these things unless pushed to do so and even if they are pushed, they like to bend over backwards to interpret things in favor of their faculty. Also, I don’t think anyone is going to push them to pursue any such thing. So, this will be moot.
“I think Jeff Id got it spot on earlier on. To parphrase things, to the OD2010 team it was a good sporting intellectual excercise showing up a “seminal†paper. To Steig it is [academic credibility wise] life or death.”
Tetris, I agreed with Jeff ID’s comment in that Steig takes S(09) and its fate very personally, but that says absolutely nothing about his professional career or Steig’s worry about it or his need to worry about it. I think the whole point being made with these comments was perhaps lost on you.
lucia (Comment#69462)
February 13th, 2011 at 3:43 pm
But publically calling his fellow reviewers “KNUCKLEHEADS” might at least give him a limp for a while.
John M-
I keep marveling on the time stamp on the KNUCKLE-HEAD REVIEWER comment:
12/2/11 1:52 PM
That’s a Friday. Assuming El’s time zone is the US, and Eric was in the US when he wrote that, happy hour had not yet arrived. Yet, Eric wrote it, then revised and reposted, then re-posted at JNG’s.
Still… depending on the reviewers temperament, they may just laugh.
McINtyre;s dishonesty is reaching new depths:
“Again, this is not supported by the record, as discussed above. “Suggest†is defined as “To offer for consideration or action; proposeâ€. Steig’s RC post omitted his Second Review comments in which he clearly suggested iridge as an alternative. Nor was iridge an “innovation of O’Donnell and his coauthorsâ€. It was a RegEM variant in the original Tapio Schneider code.”
What can one say? He redefines words to make it appear that Steig is lying about suggesting iridge to O’Donnell.
It’s clear in the Steig email he is quoting, that Steig is arguing that he did not give the authors the idea to use iridge. They used it on their own, and then when those results were deemed “most likely,” Steig argued they should be promoted to the main text.
If anybody can find a quote where Steig says “Hey, guys, you should use iridge here!” then please do so and show me I am mistaken. Until then, this is just BS from McIntyre and illustrates his preferred MO quite nicely.
Simon Hopkinson, you are off on your facts. The editor had rejected Ryan O’s paper after the second review, calling for a major revision. Ryan learned of Eric Steig’s being Reviewer A after the paper was accepted.
“I don’t think the Nature paper being wrong would have put Steig’s academic career in a wheelchair.”
It shouldn’t — even Einstein had “wrong” papers published. There will always be some wrong turns taken in Science’s search for the truth — what’s critical is that those turns be recognized as wrong when the evidence demonstrates it rather than continue blindly following the “established” path. This is the problem I perceive with climate “science” as conducted by Mann, Jones et al.
Steig COULD have responded to OMCL(10) by accepting it as an improvement on his analysis then using that to solidify his data. Instead, he reacted defensively and criticized the paper and techniques he himself had agreed to. That’s his loss — the record of his demonstrably false statements would make me more inclined to question anything else he said and THAT is something that could damage his career.
Boris,
You are getting really desperate.
The iRidge results were produced as supplementary material to support the TTLS. But as soon as Eric saw the higher trends in WA he insisted that the iRidge results be used as the primary method.
The question of who first introduced the iRidge results is irrelevant. In fact, Eric’s characterization that iRidge was “suggested” by the author’s is false. It was simply an alternate method intended to show that the effect on the results was minimal.
The bottom line: if Eric had not insisted that iRidge be used the paper would have been published with TTLS. Eric is being dishonest when he claims the change was not a result of his demands.
For what it’s worth, my response to Steig’s comment on my blog is here. It addresses the suggestion of iridge and the use of a non-objective reviewer. Sorry if some of these points have been made above…I haven’t had time to read the comments here.
Boris (Comment#69475)
February 13th, 2011 at 4:44 pm
Huh?
http://www.thefreedictionary.com/suggest
Alright, so he “argued” instead of “suggested”.
Good point.
Boris–
No. But then, as far as I can tell, Ryan did not accuse Steig of saying “Hey guys, you should use iRidge here?”
Ryan wrote this:
While Eric’s denial insinuates RyanO accused him of being the first to suggest the use of iRidge, as far as I can tell RyanO did not accuse Eric of that.
I don’t know if Eric is denying something he is not accused of because he did not actually read what RyanO wrote, or if Eric doesn’t understand the difference between what RyanO wrote and what Eric denies or what. But…well.. so much for Eric’s denial.
“If anybody can find a quote where Steig says “Hey, guys, you should use iridge here!†then please do so and show me I am mistaken. Until then, this is just BS from McIntyre and illustrates his preferred MO quite nicely.”
“we did an experiment with iridge, the trend is higher and the results are more likely”
‘you should move the more likely results to the main text’
move iridge to the main text.
‘good to see that irridge is moved to the main text”
Question. Was the suggestion to move iridge to the main text made?
Answer: technically no, those words were never written. Pragmatically yes.
Husband: hey dear, the nike shoes fit better than the Puma shoes.
Wife: I thought so, wear the shoes that fit better.
Husband wears the Nike’s.
Wife: what the hell are you wearing those shoes for, I never told you
to wear the Nike’s.
Husband: slaps forehead.
Re: lucia (Feb 13 17:05),
“But then, as far as I can tell, Ryan did not accuse Steig of saying “Hey guys, you should use iRidge here?—
Lucia, you quoted it yourself:
“So Eric recommends that we replace our TTLS results with the ridge regression ones “
MikeN
Many thanks for the clarification. I know that the Team’s involvement was long suspected, but my understanding of the sequence of events was incorrect.
“The iRidge results were produced as supplementary material to support the TTLS. But as soon as Eric saw the higher trends in WA he insisted that the iRidge results be used as the primary method.
The question of who first introduced the iRidge results is irrelevant. In fact, Eric’s characterization that iRidge was “suggested†by the author’s is false. It was simply an alternate method intended to show that the effect on the results was minimal.
The bottom line: if Eric had not insisted that iRidge be used the paper would have been published with TTLS. Eric is being dishonest when he claims the change was not a result of his demands.”
But the Iridge results were deemed “most likely”, correct? If they are the most likely results, then they should be the main thrust of the paper. That’s all Steig was saying, and it is a reasonable comment.
The kicker is that Ryan his coauthors and apparently the editor agree that the paper was better with iridge.
Yes, Steven. How stupid for someone to suggest moving the most likely results to the body of the paper!
Well, Ryan O et al must have made some good arguments on why that shouldn’t be done.
What? They agreed? We’re seriously arguing over the fact that Ryan O was “forced” to do something that he apparently thought showed better results anyway.
This whole brouhaha is bizarre.
Nick–
I think we are all talking at cross-purposes here. (Not unusual).
I assume Boris is defending what Steig wrote as follows:
The only way I can read this as remotely true is if Eric means he was not the first to suggest the authors use ‘iRidge”. After all, he quite clearly did suggest they ‘use’ it in the sense of replacing the TTLS results with ridge regression ones in at least one instance. He did so in this paragraph:
He sure as shootin’ seems to be recommending they replace the TTLS results with the ridge regression ones.
So, since it appears that Eric did recommend this, assumed when engaging Boris was defending the truth of what Eric wrote, Boris means Eric was not the first to suggest it. If this is not what Boris meant, then I misunderstood, and more over, I was sloppy not to put in the word ‘first’ in replying to Boris– though that’s what I meant.
But if you or Boris agree Eric saying “Hey, use iRidge here”, then Eric did say it. What Eric did not do is be the first to introduce iRidge into the analysis. That is: Eric is pointing out that he did not innovate the idea of using those and then recommending it.
But neither Ryan nor SteveMc ever said Eric innovated the idea.
So, my view:
* If we read Eric’s denial to apply to the accusation he innovated the idea of using ridge regression, then it is true he did not. But neither Ryan nor Steve accused him of that.
* If we read Eric’s denial to apply to the accusation that he never, ever, ever, ever in anyway recommended replacing TTLS results with ridge regression, then… well Eric did recommend that. That he recommended it is evidence by these things:
a) it seems to be the plainest reading of the above paragraph,
b) the O’Donnell replies to all reviewers and Eric in particular said they’d made that change on the basis of one-of-the-reviewers/erics recommending it– quoting from that exact paragraph and
c) during the course of review 3, eric didn’t in any way protest that they had misunderstood what he’d said.
So, it appears that Eric sure as shooting did recommend this in the sense of recommending TTLS be replaced in Figure 3 and this was what he meant at the time.
Boris (Comment#69486)
February 13th, 2011 at 5:30 pm
Boris, if it was such a good idea, as you seem to be arguing, how do you explain Steig’s hatchet job of iridge on Feb 1? He very clearly is arguing that it is an inferior method.
He is shocked, shocked I tell you that O’Donnell et al. favored it. Or as Steig himself might put it, “SHOCKED”.
Ah excellent, moving on to discussing the meta science instead of trying to chew off Eric Steig’s hind leg. You are making progress Lucia.
FWIW, and Eli recognizes that this discussion has moved on, John N-G’s comment(s) pretty much put paid to Lucia’s dissembling about Steig’s ethics.
Boris (Comment#69485)
Sounds reasonable to me except you should have replaced “That’s all Steig was saying” with “That’s why Steig insisted”.
Now, how can Steig insist on something (which the author’s had not) when he has an argument in the can all ready to use against it?
Eli–
Huh? Dissembling?
I’m not under the impression I have concealed my opinion of about anything to do with Steig. I would have thought people might have joked, “So, lucia, tell us what you really think?”
Boris– All Steig was “saying” included the word “insist”.
Boris says:
But the Iridge results were deemed “most likelyâ€, correct? If they are the most likely results, then they should be the main thrust of the paper. That’s all Steig was saying, and it is a reasonable comment.
Talk about moving the goal posts. You started out complaining that SteveMc was lying about Stieg insisting on iridge. You are now arguing that Stieg was right to insist on iridge.
The trouble is Stieg has stated that iridge was the wrong choice and trashed O10 for following his suggestion.
Logically, either he insisted they do X and then criticized them for it, or he didn’t. What they said in response has no bearing on the facts of the matter.
What we have here is an issue of truth or falsehood, which you (having been caught again on the wrong side of the facts) are trying to spin as a matter of literary interpretation. It’s “Gavingate” all over again.
You are in an even weaker position in this instance, however, because you have already accused Steig of deception and fraud, accusations that require “hard” proof. In point of fact, Steig expressed himself perfectly clearly, but had his words been in any way confusing or subject to misinterpretation, your frantic poodle-bark of hyperbolic denouncements would hardly be justified on that account.
Boris (Comment#69486) February 13th, 2011 at 5:30 pm | Reply w/ Link
Yes, Steven. How stupid for someone to suggest moving the most likely results to the body of the paper!
#####
question. does that amount to a recommendation for using iridge?
I never said wear the Nike’s
I never recommended wearing the Nike’s
I did not think you would listen to anything I said.
I never suggested or insisted that you wear the Nike’s
I said ‘wear the comfortable shoes’
Now, that I see they are ugly, I think I’ll say you should wear the better looking shoes. when you come back wearing the Pumas, i’ll suggest that they may not be that comfortable. I can drive you crazy. maybe, you’ll hit me. After all, i want a fight. That’s what I’m really after. A long hard fight and no make up sex.
Whoa. first of all O’Donnell says that they looked at iridge well before Steig’s first review.
Steig’s first review does NOT mention iridge or ridge regression. It does (pg 7) criticize the choice of kgnd:
The first response and the re-written paper are where iridge was introduced
To which Steig in his second review writes
and the authors respond
So, let us stop here for now
a) In the first review Steig never suggested that ridge should be used, but he did have serious problems with using the values of kgnd that Team O’D wanted to favor and he expressed them rather clearly.
b) Steig’s statement was that the authors should justify their choice of kgnd.
c) In response Team O’D brought ridge regression into the paper as support for their using particular values of kgnd (although iridge lead to somewhat different values of warming in W. Antarctica).
d) In his second review Steig said that since the problems with kgnd remained, maybe (as in perhaps, perhaps you don’t understand the English Lucia) it would be better to use iridge WHICH HAD BEEN INTRODUCED BY THE KNUCKLEHEAD AUTHORS who are now confused about what they did.
e) the authors in their second response, agree that the ridge regression results were the most likely and say they will move the TTLS/TSVD results to the supplemental information
f) Lucia decides that this is not ethical.
Somehow the whole thing looks like Calvinball.
I presume you can provide a link and exact quote to where Lucia accused Steig of “fraud”?
Remember, Boris is patrolling the blog to make sure words are used properly.
Would it be possible to ask a question on the scientific import of this dispute between S09 and O10? As I recall, a contentious issue was the apparent cooling of Antarctica which contradicted the accepted climate models. This cast doubt on the utility of these models for predicting possible changes that could be attributed to AGW.
Am I correct in assuming that S09 tended to confirm the model predictions and O10 tended to dis-confirm them?
Is this the basis for the vehemence of the dispute about these results?
What are the next steps that should be undertaken to resolve the issue with the models and Antarctica?
fraud: deceit, trickery, sharp practice, or breach of confidence, perpetrated for profit or to gain some unfair or dishonest advantage
Close enough.
Eli:
JNG wrote another blog post:
http://blogs.chron.com/climateabyss/2011/02/steig_this_is_not_complicated.html
Enjoy.
Eli Rabett (Comment#69499)
February 13th, 2011 at 6:21 pm
hmmm, then I guess those Climategate e-mails show that climate scientists are…well, I don’t like to use the F word myself.
Oh yes, great minds agree. Since this brings the entire mess to an end. . . . Oh well.
Tom Gray, no both results show warming in West Antarctica although somewhat different patterns. In the East neither is very trustworthy because of the lack of data.
Calvinball.
Eli, which shoes are you wearing the Nike or the Puma?
Eli:
I don’t think it was the authors that Steig called “knuckleheaded”.
But perhaps I can’t read?
As usual, a very one-sided presentation on your part, that falls firmly on the side of the professorial old-boys club. Good show, that.
Eli–
Eric’s posted a comment on JNG”s post. My take is Eric is experiencing negative emotions in response to what JNG wrote and is trying to sway JNG to say negative things about SteveMcIntyre.
I read your comment also, and I’m getting the impression you now think things you have said are somehow consistent with this:
I realize can’t read everything you write or say. So, you may well have said that you wrote that you think Eric left the impression that O’Donnell should use iridge and that that you also think what Steig wrote reads as an endorsement of iridge.
Could you provide a link?
“d) In his second review Steig said that since the problems with kgnd remained, maybe (as in perhaps, perhaps you don’t understand the English Lucia) it would be better to use iridge WHICH HAD BEEN INTRODUCED BY THE KNUCKLEHEAD AUTHORS who are now confused about what they did.”
Silly rabbit..
I wonder just what Kgnd would help with since EVERY SINGLE METHOD MATCHED EVERY SINGLE OTHER METHOD!! YESS I’M YELLING — BECAUSE IT’S FUNNY!!
I would agree with Eli Rabett’s observation about chewing ES’s hind leg off if he weren’t so lost in the deep end in bouts of Freudian amblyopia himself.
.
Why is he calling O’Donnell et al names??
.
‘Eric’ has commented at Eli Rabett blog with an analogy about George Bush.
.
One feels like asking the question: you could reserve your judgement about Bush for later, but you have to vote for either Bush or Al Gore. Both are fugly, but only one can be president.
Jeff Id,
If you are still around..
Ryan said some days ago that he planned on posting a sensitivity test with manually added trends using the O(10) method so that the response could be seen side-by-side with those sensitivity tests using S(09). Do you know if he still plans to do this?
Steve,
.
Yes, I do. In a couple of days. 😉
Ryan O (Comment#69514),
.
Thanks, I look forward to seeing them. The best way out of this morass of “he-said-she-said” discussion is to test the two approaches rationally and see which is more reasonable.
Why do we waste so much time on describing what was said by Reviewer A and how various people have interpreted it. The reviews and replies are all available and anyone can read them and come to their own conclusions. Changing minds is probably not going to happen. The second level of chatter is even a bigger waste where the questions are about what others said about what Reviewer A said. And then what others said about what others said …
I personally would rather discuss the approach of Reviewer A in relentlessly attempting to get O(10) to save some of the warming of West Antarctica that appeared in S(09), backing off some of the implications in S(09) , and jumping all over the place attempting to find a chink in O(10) that might save the WA warming. He also appears to be attempting to use his own credentials as a scientist to get the editor to allow him to press the non science O(10) authors to do literature research that S(09) did not do or do adequately. Also it would be of interest to compare Reviewer A’s view of what the literature was saying versus the authors of O(10).
Kenneth,
As always you do keep your eye on the ball
I’m still waiting to see Ryan O’s apology.
I retract my earlier comment about possibly more than one reviewer A, given this statement by Eric at RC ( I had missed this comment before ).
http://www.realclimate.org/index.php/archives/2011/02/odonnellgate/comment-page-3/#comment-200408
I still see no possible rationale for Steig being an ANONYMOUS reviewer.
In addition, I have a hard time understanding why Steig would chose to be a reviewer of this paper once he saw it. Surely he would want to write a rebuttal.
Much of the current mess stems from the JoC’s flawed approach of including anonymous reviewers with patent CoIs and reviewers with strong animus towards the authors from accepting the role of reviewer.
I also note the now concerted effort of the part of Steig and others to denigrate the other reviewers and the editor.
People have asked why O’Donnell et al wished to use the less certain TTLS result than iridge. Steve answered that they wanted to match the method of Steig et al to avoid criticism that using another method would attract, namely it is different so mine’s still good, which Steig has now done.
Shame.
This reminds me why news sites report their most popular stories are always about “stars” misbehaving. People are more fascinated by other people, than anything else.
Steig should re-read his Review comments, and take a chill pill.
bernie (Comment #69531),
You can read some rebuttals to our (shared) view that Steig should have been a non-anonymous reviewer of O’10 at the Rabbett Run post that Lucia links in the body of this post. IMO, the counterpoint with the most traction is that these CoI issues come in shades of gray.
If I saw more insight there, the questions I would have been interested in pursuing are (1) What makes Steig’s position look light gray instead of dark gray?, and (2) Most peer-reviewed journals do have formal CoI policies — if they aren’t to be considered even in a case like this, are they just for show?
ianash (Comment#69527)
> I’m still waiting to see Ryan O’s apology.
Yeah, as apologies go, his wasn’t much of one. He did behave poorly in the aftermath of Steig’s early February RC post, IMO.
A schoolyard fight between members of two popular cliques with a history of mutual antagonism. Each driven by circumstances and likely egged on by their respective alpha leadership. Each drawing the entirely predictable support from the usual quarters (John N-G a notable exception).
No teachers in sight, and no widely-shared lessons drawn, with respect to science or policy.
“question. does that amount to a recommendation for using iridge?”
It’s a stretch, but I could buy it. But note that Eric, in his third review, pointed to problems with iridge. O10 decided to ignore those problems or didn’t agree they were serious. In any case, it is incumbent on the authors to fully research and understand the methods they employ in their papers. And since Eric apparently TRIED to point out shortcomings to them, I don’t see how one could say he is sandbagging them. The assumption you guys have is that Eric is an iridge expert and that he knew all along that iridge overestimated trends and was just waiting to pounce at RC. But that is a conspiracy theory masquerading as an argument (And it makes no sense: why point out the flaws in a method if the whole point is to get them to use a flawed method without realizing it). Given that the other conspiracy theory you guys had goin’ was wrong, you’d think you would be a bit more careful here. But alas.
AMac:
I saw the comments on shades of grey … The litmus test I would use is that if a reviewer could as likely ask to write a rebuttal then that review should not be anonymous. If that means none of the reviewers are anonymous then so what? I understand the benefits of anonymous reviews but in such instances anonymity has no real benefits to the review process as a whole. The OLMC10 authors suspected that Steig or one of his co-authors was Reviewer A according to their communications with the Editor.
I believe however well intentioned JNG’s pronouncement on the ethics of Ryan’s identification of Steig as Reviewer A were, they failed to address the facts of this situation. Steig knew he was the anonymous reviewer before he posted his pre-emptive Feb 1st attack on the paper. The lines of communication with Ryan were apparently open. Why did he not simply discuss his post with Ryan first, especially since he had apparently not read RLMC’s response to his third review: Very, very weird.
Boris,
.
You should, perhaps, read our third response before passing judgment on who understands what.
Boris (Comment#69544)
February 14th, 2011 at 7:26 am
You mean the section of his third review that starts with the sentence “The use of the ‘iridge’ procedure makes sense to me, and I suspect it really does give the best results.”?
I would say they didn’t agree they were serious. They certainly didn’t “ignore” the “problems”.
It sounds like maybe you didn’t bother to read the response to the third review, you know, where the researchers were so sufficiently dilligent that they “understand the methods they employ” that they were able to point out errors in the reviewers citations and the nuances with regard to those erronious citations.
http://climateaudit.org/2011/02/13/steig-and-the-knuckleheaded-reviewers/#more-12949
As far as “sandbagging” and “conspiracies”, I know these have been long threads and I may have missed those accusations, so perhaps you can point them out to me.
Thanks.
A minor sidelight concerns a use of RegEM in the mis-cited Mann (2008) on the … Tiljander data series. These
fourthree series ended in 1985, while the final year of the chosen calibration period was 1995 (IIRC). Thus, values for 1986 through 1995 were absent. Mann et al used RegEM to extrapolate the series. Thus, with respect to Tiljander, about 6% of the data used in the calibration/validation routines was synthetic. Common sense would suggest that calibration/validation performed on extrapolated data would not be as robust as c/v done with primary data. AFAIK, this wasn’t taken into account in their analysis.I don’t understand any of the two methods, but in all that kerfuffle about who suggested what to whom and if TTLS or iridge is the real bees’ knees, I have a question of my own.
If I got that corectly, then Steig didn’t like O’Donnell’s TTLS because of differing opinions on kgdn. O’Donnell then applied iridge. Steig countered by lauding iridge together with the damnation of the same (so to say).
My question is – are there any methods other than TTLS or iridge that could have been used?
I’m not a statistician, but apparently neither is E. Steig…;-)
“You should, perhaps, read our third response before passing judgment on who understands what.”
Hey, maybe Steig is wrong. I have no clue on who is right n this mess and I’m not about to spend the time to find out. My point is that he pointed you to possible problems with iridge, so his piece at RC (while possibly wrong) did not come out of the blue and seems consistent with his review.
I have another problem with these conversations and that is they tend to focus down on details and lose the context and perspective that the entire review and reply process actually involved. A minor point that if interpreted in any number of ways would not change the main point in the reviews or replies.
It would be much more informative if when a blog participant attempts to make a point that we could see an excerpt from the available documents that demonstrates what they attempting to show.
In my view we have been afforded the opportunity to see an example of how the peer review process is applied and all the human elements that come with it. I have published only one peer reviewed paper (years ago as a graduate student) and I am fascinated by how the process (doesn’t always) works and what one can bring away from reading these reviews. In this case Reviewer A was on a mission, and the other 3 reviewers were amongst the types I have seen before that include those who contribute to better form of the final product, those not versed in the important details of the paper but able to offer peripheral suggestions and those versed well in certain technical aspects of the paper. If nothing else I get to feel a little better about myself sometimes in learning that a reviewer has had the same problem that I, as a layperson has had and thought it to be because, well, I am a layperson.
My reading of the reviews (of Reviewer A) and replies were very helpful in getting insight into some of the more vague and nuanced statements in S(09) – and perhaps that observatrion would support having an author of a previous paper now being critiqued doing a review, providing that the reviews are made public. The public reviews also afforded a look at what was originally in the paper and SI, and subsequently taken out that, that are very important and of much interest to the those of us who have taken a special interest in this subject matter.
Re: Kenneth Fritsch (Feb 14 09:51),
It’s interesting to take a fresh look at S’09’s abstract, since that is the authors’ distillation of their views of the important new findings in their work.
“we show that significant warming extends well beyond the Antarctic Peninsula to cover most of West Antarctica…”
“West Antarctic warming exceeds 0.1 °C per decade over the past 50 years…”
According to O’10’s authors, these two claims are not supported if the analysis of temperature trends are done in a mathematically-correct fashion.
According to S’09’s authors, O’10 is simply an alternate approach that modifies but doesn’t invalidate these two findings.
I have pointed to what you show here, AMac, a couple of times in the past and my point has always been that we get this strong message early in S(09) that they have come upon something new with the lesser Peninsula warming and greater West Antarctica warming and attempt to rationalize it the explanation you have excerpted. Late in the paper they note that the variance of the Peninsula may be wrong because of the lower truncation paprameter selected – buit hat that is OK since we know a lot a
Let me try again:
I have pointed to what you show here, AMac, a couple of times in the past and my point has always been that we get this strong message early in S(09) that they have come upon something new with the lesser Peninsula warming and greater West Antarctica warming and attempt to rationalize it the explanation you have excerpted. Late in the paper they note that the variance of the Peninsula may be wrong because of the lower truncation paprameter selected – but that that is OK since we have a lot of instrumental data from there.
What is devasting to S(09) from O(10) is not so much the trend differences in the Peninsula but the showing in O(10) that the Peninsula warmth was transferred from the Peninsula to other areas of Antarctica, including West Antarctica. This impact on S(09) is clearly evidenced by the reviews of Reviewer A.
Let me try again:
I have pointed to what you show here, AMac, a couple of times in the past and my point has always been that we get this strong message early in S(09) that they have come upon something new with the lesser Peninsula warming and greater West Antarctica warming and attempt to rationalize it the explanation you have excerpted. Late in the paper they note that the variance of the Peninsula may be wrong because of the lower truncation paprameter selected – but that that is OK since we have a lot of instrumental data from there.
What is devasting to S(09) from O(10) is not so much the trend differences in the Peninsula but the showing in O(10) that the Peninsula warmth was transferred from the Peninsula to other areas of Antarctica, including West Antarctica. This impact on S(09) is clearly evidenced by the reviews of Reviewer A.
I think that Steig should not have been a reviewer, but that he should have been given space to offer a response. From the AMS Guidelines for reviewers:
According to this guideline, it is very clear that Steig has a huge conflict of interest in this case. And whether or not there is a conflict in Steig evaluating a manuscript critical of his work, there is certainly the appearance of such a conflict.
.
Given that clear policy, it is astounding to me that a) Steig was chosen as an anonymous reviewer, and b) people are defending that choice.
.
Look, Steig definitely deserves a chance to answer his critics, and this is normally done by giving an author space in the same issue to respond to those criticisms.
.
But giving Steig a secret, anonymous vote in whether a paper critical of his work should be published is an assault on both the scientific method and on common sense. Anyone who thinks that practice will not lead to scientific malfeasance and “gatekeeping” to censor opposing views doesn’t understand human nature. Without question, if authors are given a say on whether a paper critical of their work will ever see the light of day, that situation will be abused.
.
And Dr. Steig and the editors of the Journals, you should note that those rules are there for your protection as much as for ours. Steig would have been very smart to follow the “Caesar’s Wife” guideline #5 quoted above, and to recuse himself to write a devastating critique of his critics.
.
Instead, he tried to secretly and anonymously fight against the paper being published, and the editor foolishly let him do it … and now both Steig and the editor are getting their names rubbished all over the web.
.
Given the outcome, I’d say they made the wrong choice, but YMMV …
.
w.
Boris if you had done as Ryan had suggested you wouldn’t have written that. Dr. Steigs “possible problems” came from Steig first looking at the wrong Mann paper which Ryan O put into his response to the review and the Editor sided with Ryan. So lets look at each part since you can’t bother to click two links and read the thing and educate yourself before pontificating:
That is from Reviewer A in his third Review taken from the bullet point responses by Ryan O. Now lets take a look at Ryan O’s reply:
So Dr. Steig gives the wrong paper by Mann as his citation for concern, something the authors knew because of the content of the complaint from Dr. Steig. However Ryan O wasn’t done there is more:
however Dr. Steig isn’t done there with showing his lack of knowledge concerning the two methods as shown in his point 6:
Now here Dr. Steig is claiming that Iridge is not well known and that it wasn’t discussed much in Schneider 2001 and he would like the authors to expound upon it but he says they should not be held to it. However when you read the reply you see how far Dr. Steig was out of his depth here:
Right there Ryan and the other authors are pointing out the iridge is well described in the literature going back to the 1960’s and Dr. Steig has the amount of description of the two methods backwards in the Schneider 2001 paper. That is twice that Dr. Steig has screwed up citations to supposedly support his point. As Dr. Steig himself has admitted, he is no stats guy but you would think that if the editor was satisfied with whatever Ryan wrote in response to that review; he might want to go back and read Schneider 2001 and make sure he got his facts right. Or how about back on Dec 6th when he admitted to Ryan he was Reviewer A in an email, he could have mentioned that he never got to see Ryan’s third response to his third review and asked him for a copy of it to see how he answered those points before popping off on RC about TTLS and iridge.
AMac (Comment#69540) February 14th, 2011 at 6:02 am
In fact, the lessons were learned many years ago, and a process put into place to prevent just this.
Multiple, anonymous, reviewers.
An editor to manage the reviews.
A progression of work, in a formal manner, which over time sees the most likely evidence float to the top.
This has worked exceedingly well for many years, in many disciplines. The evidence of that success is all around you.
The breakdown in this case was the insistence on knowing who reviewer ‘A’ was, and crying like a baby from the start that he was unfair.
http://www.youtube.com/watch?v=-VRBWLpYCPY
The editor did listen to the author’s complaints. He did realize that there was a clash of personalities happening. He managed that situation. The paper did get published.
Despite the history of publicly personalizing the debate, the progress of science was evident. It was O’Donnel and company who broke the rules, and showed exactly why the rules were put in place. There is nothing new to learn here, it was all learned long ago. It’s just that some people chose to ignore that history. What did McI call it “Noble Corruption”?
SteveF (Comment#69333) February 12th, 2011 at 6:29 pm
He did, because it was. This is what he claims is novel.
He is entitled to that claim.
bugs,
.
The very best spin does not make the S(09) reconstruction any more accurate.
.
Yes, it was a very clever idea to combine the data sets. No, it was not very well done; the methods used by S(09) (specifically, using only 3 PC’s) led S(09) to generate regional temperature trends which are misleading… not intentionally misleading, but still misleading. The reason S(09) made its way to the cover of Nature was the very unexpected result of the reconstruction: rather extreme warming in regions where little had been detected before. The selected number of PC’s in S(09) was inadequate to generate the spacial fidelity required to make the claims of significant regional warming that were in S(09). The problems with S(09) were clear enough that the authors of O(10) (along with others) were pointing out these problems on blogs very shortly after S(09) was published.
.
O(10) simply shows that the reported regional trends in S(09) have little to do with the ‘reality on the ground’, and that the use of a larger number of PCs with the S(09) method (plus some other improvements) does generate a more spacially accurate reconstruction…. and most importantly, one that is much more in line with the pre-S(09) estimates of regional warming. The main ‘novelty’ of S(09)…. here-to-fore unknown regional warming… was an artifact of the methods used by S(09).
.
I beg you to look at Ryan’s sensitivity tests, where he adds synthetic trends to station data, using only the S(09) methods, and shows how these added trends consistently show up the S(09) temperature reconstruction in the wrong places! None of this is really all that hard to understand bugs.
Re: bugs (Feb 15 04:49),
Yes. We agree on that. At the end of the day, peer review worked in this case. Not perfectly, but it worked. There’s no Conspiracy.
Most of your other remarks are Talking Points that have already been discussed. I could respond, but it would be boring to write, and boring to read. Certain commenters here say things that are interesting as well as consistent with the record. I try to emulate them in those regards (maybe successfully, maybe not). I am not sure that those are qualities that you strive for, in your writing.
Surely this cannot be an example of peer review “working”. Something looking like the first submission with minor revisions published 9 months ago would have been peer review “working”.
This peer review is going to work, even if I have to redefine the meaning of ‘work’.
============
Re: Steve McIntyre (Feb 15 09:03),
> Surely this cannot be an example of peer review “workingâ€.
That’s an intriguing comment that deserves a full airing.
Two caveats for my point of view. (1) The heart of the paper — the detailed mathematical discussion of the alternative procedures that can be used to produce temperature reconstructions from limited ground-station and satellite data — is beyond me. (2) I’ve scanned the initial and final drafts and read the reviews and responses, but haven’t put in the time that other commenters have.
That said: yes, peer review worked, IMO. The O’10 MS went out for review and got three reviews: One generally-approving and cursory (B, IIRC), one generally-approving and with detailed observations and criticisms (C, IIRC), and one very critical, with detailed criticisms and observations and some compliments, recommending major revisions (A).
RyanO has recounted that the O’10 authors took the reviews seriously, and extensively revised the MS to respond to criticisms (A and C). This was accompanied by detailed Responses to Reviewers. He has opined that the MS was greatly strengthened by this process.
The second submission was sent by Broccoli back to the reviewers. B and C didn’t offer further substantial criticisms (I’m unsure of this point). A offered a second very critical review, recommending further major revisions.
At this point (?), Broccoli sent the MS out to a new reviewer, D.
With some irritation (IIRC), O’10’s authors again made major revisions to the MS, returning the third version to Broccoli along with added detailed Responses to Reviewers.
D weighed in on the version of the MS s/he had seen (2 or 3), offering a generally-approving review with detailed observations and criticisms.
The third submission was sent by Broccoli back to A (and others?). A returned with a third very critical review, recommending further major revisions.
At this point, editor Broccoli instructed O’10’s authors to write a response to A’s third review, but did not require further amendments to the MS. O’10’s authors did so, choosing to make only slight changes.
Broccoli then accepted the MS, asking only for a few proofreading-level alterations. Essentially, the third version of the MS was the one that appeared in print.
Is this chronology mostly correct?
.
If it is, then it seems to me that O’Donnell et al.:
* Submitted a good MS that had problems
* Had their MS sent out for review
* Got back two basically-positive and one basically-negative reviews. One of the positive reviews was fairly cursory, the other two reviews were very careful
* Extensively revised their MS in response to the two careful reviews
* Submitted a much-improved MSv2
* Got a second very critical review from A
* Had their complaints about A’s likely conflict of interest addressed by Broccoli, who sent out MSv2 (or MSv3, later?) to D
* Made some further changes to the MS, without diluting its central points. Submitted this MSv3
* Got a third very critical review from A
* On receipt of D’s positive review, Broccoli decided to accept the MS, requiring only very minor changes to turn MSv3 into the final MSv4. MSv4 did not meet the requirements that A had attempted to impose.
* MSv4 was published.
.
From here, it looks like you guys got a hard time — but not one that’s beyond the pale. I’ve gotten hatchet-job reviews and “unfair” requirements for MS modifications (Rejection decisions, too). Lots of folks are offering war stories like this.
If peer review didn’t work: did it not work for O’10 in a way that is exceptional, or did it not work for O’10 in a way that is — alas — recognizably ordinary for many publishing scientists?
Re: Steve McIntyre February 15th, 2011 at 9:03 am
,
Steve,
In addition to the nine months elapsed time, which you feel is too long, I think you also feel that O10 would have been stronger had it
1. used TTLS results rather than the iRidge results that were eventually used, and
2. included the section on Chladni patterns (that was removed after ES’s request in his first review).
Is this a pretty good summary (of how you feel O10 was actually weakened by peer review)?
Steve McIntyre (Comment#69672) February 15th, 2011 at 9:03 am
Forgive me if I am wrong, but all work by you is correct, and hence there is no need for anymore than a pat on the back by the reviewers, while work by climate scientists that is ‘audited ‘ by you takes many months, finding all kinds of errors that can only be explained by conspiracy and incompetence.
Are you serious?
bugs (Comment#69685) —
SMcI may have opened the door to an interesting exchange…
bugs–
In the ideal perfect peer-review process, if the initial submission is topical to the journal and represents a contribution to the literature, the reviewers would recognize that and the process would play out exactly as McIntyre said. The paper then goes to the next step where the public — including scientists not selected to perform the peer review, get to read the paper, which has passed a certain filter.
The paper is not required to be utterly, totally, correct from every possible hypothetical vantage point. If that were true, no paper would ever emerge from peer reviews.
So, I think one can take this as an example where peer review certainly did not work perfectly, but on the other hand, the paper did get published. But the authors were put through much more rigamorole that the paper warranted, and it appears that Steig’s bias contributed to that. So, depending on the definition of “work” it either did or didn’t.
We could also debate whether and old jalopy that manages to get passengers from Milwaukee to Chicago in 5 hours, backfiring constantly along the drive, and periodically overheating “works”. Yes, it “works” in the sense that the passengers got to their final destination. But it doesn’t “work” in the sense of working as one expects a car to work.
I thought the complaint about peer review was that it is too lenient.
9 months ago would have have been about 3 months after submission. We would all like our papers to be published that quickly with minor revisions, but it is very unlikely to happen (especially if you are saying someone else got something wrong). A time scale of a year or so is the norm, not the exception.
Look at the current (vol 24 Issue 2, Jan 2011) issue of J Clim.
The first few papers were received:
Feb 2010
July 2009
Oct 2009
Oct 2009
Re: lucia (Feb 15 11:43),
I’ll stick to my characterization that peer-review worked in this case.
For a champion of the process, see lderry’s comments upthread (also at dotearth). He isn’t claiming that peer-review produces perfection or even fairly-uniform good results–just that it’s better than the alternatives.
The most interesting thing would be to see the original Steig’09 manuscipt, and follow that paper through the review process at Nature. Because it’s so high-impact, that journal is supposed to have one of the toughest and “best” sets of procedures.
Was it a case of an easy process for the first set of authors, and a difficult process for the second set? Or, was it tough-in-one-way (and uneven) for Steig et al., and tough-in-another-way (and uneven) for O’Donnell et al?
We don’t know.
PDA–
Being too lenient in some instances and too strict in others are not mutually exclusive possibilities.
AMac–
I’m inclined toward work rather than didn’t work. But it does depend on the definition of “work”, what one’s expectations are etc. The paper was a valuable contribution to the literature and did get published in a prestigious journal. The message didn’t get altered much.
But there are aspects of peer review that are jalopy-like. I’m not sure if it’s possible to fix that. But some people are going to see that as “not working”.
Certain comments here say that ‘peer review in the case of O(10) ‘worked’ and other commenters say that it ‘did not work’. These comments of course raise the question of just what are the purposes and objectives of peer review and how it can be said, in any specific case’ to have or have not ‘worked’.
How does peer review differ from the process of vetting engineering designs. Are the criteria for publishing in a paper journal different from the criteria for the construction of a building or machine that will be both costly and liable to cause serious injury or death.
The IPCC uses approval by peer review as a criterion for the assessment of results. Who is this appropriate. The IPCC reports will be sued to create policies that will be both costly and liable to cause serious injury or death if badly formulated. Why is peer review an acceptable criterion fro such a purpose as the IPCC.
My own sense is that peer review does what is does and that is to vet papers for publication in a paper journal. It is not a suitable process fro the vetting of safety critical engineering projects. it is also not a suitable criterion for use in the IPCC reports. No drug would be approved by the opinion of some lead author doing a literature survey. Some procedure more closely tied to engineering practice is needed.
The reason that I made my comment above is that I see some commenters as believing that peer review should be a strict assessment of papers akin to an engineering audit. Other commenters see it only as a first filter to find papers that are worthy of publication, These purposes are quite different and will require a different form of review. So for one commenter, peer review that results in publication of an interesting idea may be seen to work while for another it will be seen to not work since it resulted in a published paper that cannot be relied upon for serious policy decisions.
As Lucia notes, whether peer review “worked” in this case depends on exactly what “working” means. That other people have their own war stories about the process doesn’t really answer the question.
I’ve spent most of life not in academia and I’m trying to understand exactly what is accomplished by peer review in the context of the climate science articles that we discuss at blogs.
In the case at hand, I think that the original submission, based on TTLS, was a worthy contribution to the literature and that something like it should have appeared. The iridge results should have been reserved for a subsequent article as Ryan originally intended.
Now Steig says that he was “totally surprised” that we took his suggestion of iridge at face value and that the Third Version converted from TTLS to iridge. Perhaps we should have taken a stand and refused to make the major revision called for in the editor’s letter on the Second Draft, but this didn’t seem like a practical alternative at the time. But the changes for better or for worse were responding to “peer review”.
And yes, the Third Version, taking a different approach, did present the “future work”. None of the other reviewers weighed in on this approach. And even the editor appears to have been exasperated by Steig. My sense right now is that there are many valid points in the First Draft that remain absent from the academic literature, that should have been in the literature with a peer review process that “worked” more effectively – whatever effectively means.
While the review comments were useful for minor points, the substantive commentary on blogs in 2009 was immeasurably more insightful than the review comments. Chladni patterns are a good example. This is an extremely important point and the reviewers were hopeless on it. Steig called for its removal; one reviewer had no idea what they were. Nor in my opinion did any of the reviewers (including Steig) really have any idea about the mathematical issues involved. They were all passengers. There was zero value added by them, probably negative value.
All of this can hardly be considered a sensible process ‘working”.
Here is Gene Gordon’s (an extremely accomplished inventor who worked for AT & T (See wikipedia) perspective on peer review
http://community.nytimes.com/comments/dotearth.blogs.nytimes.com/2009/11/23/your-dot-concerns-about-climate-files/?permid=134
“141.
Gene G
New Jersey
November 25th, 2009
11:56 am
I am repeating myself going back a year or so but the issue keeps coming up. I comment as a founder of two prestigious, scientific publications in elctrical engineering, and long time associate editor on two. I have also served on the publication committee of IEEE.
Peer review is a process designed to improve the quality of journal articles. It is not a stamp of approval; it doesn’t guarantee that the data is accurate or complete, that the conclusions are correct, and that the author(s) are not playing games. There is an old boys club element. A good and conscientious editor learns to help the peer review process along but editors typically are volunteers, not professionals, they are busy people, and they are not typically experts in all the various subjects being published by the journal. Reviewers, usually only two, are busy also. Sometimes reviews are cursory, sometimes detailed. Sometimes reviewers have fish to fry. There are seldom such controversial issues in EE; (the last big one I can recall in EE was between Tesla and Edison); lots of fish frying in climate science. .
In other words the process is spotty and in no way should be viewed as a stamp of correctness or guarantee of validity, However, it is somewhat better than no peer review.
One clue to look for is the time between submission and publication of various articles. A longer than average review time may indicate some controversy had to be sorted out or the reviewers were too busy.”
JD
http://www.numberwatch.co.uk/2008%20July.htm#refereeing