104 thoughts on “Links to John Cook’s Survey.”

1. Lubos also posted the survey. His site is identified as c=YG6T1272VU5M.

   The survey can also be taken by cropping the URL after the “c=” which removes the identifier. Didn’t test if it will allow you submit your choices though.

   To me, this whole exercise will produce a lot of wasted efffort.

2. All this for another disparaging headline in the Guardian about ‘climate denier nutters’? I think I’ll mow the lawn instead.

3. cui–
   It’s finally warmed up enough that I had to mow this week!!

4. I have better things to do than provide fodder for John Cook to “prove” that AGW skeptics are _______ (fill in the blank).

5. “What else are web surveys for?”
   Umm… for using a cloak of academic research to draw traffic to sites that may financially benefit as a result?

   I’m not saying that is necessarily the case here, but the website-that-cannot-be-named usually only gets a visit from me if I click on a link without realising where it is taking me. In my opinion it is already found far too often by a certain search engine. Do they have the publishers permission to reproduce all of those images?

   Whatever. The devil will ski to work before I afford them my time and attention filling out one of their alleged “surveys”.

   I am curious as to why you are urging your readers to give their time towards such manifestly untrustworthy recipients, especially considering that you are posting a link solely to pre-empt the accusation that you refused to do so?

6. I am curious as to why you are urging your readers to give their time towards such manifestly untrustworthy recipients, especially considering that you are posting a link solely to pre-empt the accusation that you refused to do so?

   The clue of why should come from seeing that I posted multiple links. I’d also like people to fill out the survey with the links that don’t necessarily trace them to a particular site.

7. Hey, now you are providing leading answers to the testers.

8. MikeN–
   Sure. I would think given the fact that he invited 50 bloggers, what I do is part of the study. Presumably, the point of inviting blogger to post links is intended to get the sort of response you get when you ask bloggers to post links.

9. This is interesting. Researchers were sent a similar survey last year, but the answers are worded differently:

   http://pielkeclimatesci.wordpress.com/2012/08/22/skeptical-science-survey-by-john-cook-on-climate-related-research/

   Endorsement: The second drop down indicates the level of endorsement for the proposition that human activity (i.e., anthropogenic greenhouse gases) is causing global warming (e.g., the increase in temperature). Note: we are not asking about your personal opinion but whether each specific paper endorses or rejects (whether explicitly or implicitly) that humans cause global warming:

   Explicit Endorsement with Quantification:paper explicitly states that humans are causing most of global warming.
   Explicit Endorsement without Quantification:paper explicitly states humans are causing global warming or refers to anthropogenic global warming/climate change as a given fact.
   Implicit Endorsement:paper implies humans are causing global warming. E.g., research assumes greenhouse gases cause warming without explicitly stating humans are the cause.
   Neutral:paper doesn’t address or mention issue of what’s causing global warming.
   Implicit Rejection:paper implies humans have had a minimal impact on global warming without saying so explicitly. E.g., proposing a natural mechanism is the main cause of global warming.
   Explicit Rejection without Quantification:paper explicitly minimizes or rejects that humans are causing global warming.
   Explicit Rejection with Quantification: paper explicitly states that humans are causing less than half of global warming

10. As Cook is providing links with identifiers unique to the blogs, he clearly intends to process the results by “skeptic” and “non-skeptic” groups. I imagine the hypothesis he is testing is that “skeptics are so blind they can’t see the evidence even when it is pushed under their noses”.

    I should note that I understand that many posters have figured this out for themselves.

    I would like to ideate the conspiracy theory that this is advocacy and not science. What a crock.

11. I feel stupid. After the party at my house this weekend died out, I decided to take a fresh look at the issue of randomness and this survey. The first thing I did was check the HTTP requests/responses involved. In addition to the normal cookie for the website (with a duration of about a week), there was a cookie labeled PHPSESSID. I immediately felt a sinking sensation.

    Session IDs are basically like any cookie. It’s up to the server to make sure whatever session ID you give is right. Normally you do that by keeping track of who you assign each ID so that nobody uses anyone else’s. You also make sure any session ID being given to the server is one it has assigned.

    With that in mind, consider what happens if you use a session ID as the seed to a random number generator for a survey like John Cook’s. How would you prevent duplication? You would make sure the session IDs you give out are unique. But is that it? A person can modify any cookie on their machine with ease. What if someone decides to change their session ID?

    There’s no way to prevent that. You could try to keep a list of session IDs, make sure people use ones they’re assigned and never use duplicates. That won’t work. It’d require the server check a list of every session ID ever assigned every time someone decided to take the survey. The server load per request would grow with each person, and it would become unbearable.

    So what do you do? If you’re John Cook, you do nothing. You leave your survey unsecured. You make it so people can share session IDs so people can see the same sample sets. You make it so people can change their session IDs to get different sets of questions. Heck, session IDs only last until a browser is closed, so people can just close and re-open their browsers to “refresh” their results.

    It’s stupid. It’s a completely non-standard way of handling the randomness problem that has no benefits. There is no reason it should have been used.

    On a lighter note, it’s possible to send values in the session ID field that couldn’t have actually been generated by the server’s process. That lets you get sample sets not normally available.

12. I spoke too soon. It occurred to me that the last time I drew a conclusion, it seemed things changed after a bit of time. So I tried again. It turns out the same thing happened. The surveys given to me all changed. I checked a dozen different session IDs, and each one gave a different result than two hours ago. However, each one consistently brings up the same surveys each time I use it now.

    I guess time is definitely a factor. Maybe he automatically updates some seed value every so often or something. That’d mitigate the stupidity of using session IDs as a seed by a bit.

    Or maybe John Cook has paid attention to my testing and secretly made changes in response. Conspiracy!

13. Brandon,
    The session ID’s may not be his attempt to solve the randomness problem. It may be an element of his method to prevent the same person from answering more than one time. It does tend to inhibit filling out multiple forms though it doesn’t entirely prevent it.

14. As Cook is providing links with identifiers unique to the blogs, he clearly intends to process the results by “skeptic” and “non-skeptic” groups.

    I don’t know about “clearly”.

    But I suspect he may be doing something like. That’s why I posted multiple links and suggest people cut and paste one chosen more or less at random. Manually cutting and pasting will prevent referrer information from passing and using different identifiers will obscure which blog you came from.

    It’s also why I urge people to take the survey using my method.

15. The rabbet warren has posted a survey link. Here is the ID to add to the above list. c=II7WP4R4VRU7

16. Copy/paste error in the rabbet’s survey url. Extra “c=”

17. lucia, I’d be fine if that was the reason he used session IDs. It isn’t. He could use session IDs like you say without using them as an input for his RNG.

    By using the session ID as an input for his RNG, he gives users the ability to influence the samples they’re presented in a controlled way. It’s not like reloading a page to get a new CAPTCHA. It’s like being able to match individual CAPTCHAs with numerical values then tell a server which numerical values (and thus CAPTCHAs) to use.

    I get that some people might not care because they figure, “Who would cheat like that on this survey?” I care partially because I’m a security person, but mostly, I care because it shows a lack of knowledge/skill. If someone doesn’t know how to properly host a survey, I have to suspect they won’t know how to properly analyze the survey’s results. If I find a problem when I examine a simple and easy step in the computer processes, I have to suspect there will be problems in the more complicated steps.

    Which is something I actually want to write about. But I should e-mail you first.

18. Brandon Shollenberger
    Have you succeeded in submitting session ID’s of your choice?

    I get that some people might not care because they figure, “Who would cheat like that on this survey?”

    Given the survey author’s involvement in “recursive fury”, I should think there would be plenty of people who would want to screw up his survey. Given the extensive discussion of people possibly cheating on Lewandowsky’s survey — which was one of the subject discussed in “recursive fury”, I should think John Cook should be aware that people do cheat on surveys of this type, and probably did cheat on Stephan’s. He’s trying to sample from the same groups- half of which Cook offended pretty badly by the hacker that is “recursive fury”.

19. Lucia, perhaps “clearly” is a bit strong, but there is presumably some reason why Cook issued unique identifiers to each blog. I believe Anthony Watts asked him about this, and he declined to answer.

    I appreciate your caution in suggesting people access the survey via another link, however I suspect that regardless the quality and nature of the obtained data, Cook will write it up as if it supports his hypothesis, whatever it is. After all, that’s what Lew and company did with the “moon hoax” nonsense.

20. lucia, yup. I’ve modified the values at least fifty times now. There’s still some weird issue with time where every so often, it seems the RNG changes. Between those happening though, surveys can be controlled exactly as I described by manipulating session IDs. And anyone can do it. It’s as easy as changing your UA.

    As for cheating, I’m sure some people would like to cheat. I’m just not sure my finding would help them. They could get a new survey by closing their browser and re-opening it. I guess my finding might save them a little time.

21. James–
    He may. But there generally has to be some possible link with the data. As it stands, you can visit rabbet run, sks, deltoid, neven’s blog, this blog, The air vent, scholars and rougues and a few other blogs to enter your survey.

    I’m actually a bit intrigued by the fact that the survey tells us how our answers compare to the paper authors. But
    1) We answered based on the abstract they answered based on their own recollection of the full paper.
    2) The wording of the answers is different. This is especially true of the “reject” type answers. For the abstract evaluation the concept of “minimize” is introduced. That makes quite a bit of difference since “minimizing” is different from “rejecting”.

22. As for cheating, I’m sure some people would like to cheat. I’m just not sure my finding would help them. They could get a new survey by closing their browser and re-opening it. I guess my finding might save them a little time.

    If someone wanted to go to the trouble to write a bot to cheat, it might help. They would know they need to set session id’s. I wouldn’t bother writing a bot, but with these surveys because (a) I don’t have the skills anyway and (b) I don’t care enough. But you never know. And given that Cook and co-conspirators ( 😉 ) like to run surveys, some people might conspire to write bots not only for the current survey but in anticipation that the current script survey would form the basis for future survey security methods.

23. Well, we can only speculate about the purpose of the study, as Cook appears incapable of, or unwilling to, express it in plain English. That sets off my alarm bells right away.

24. Flipping over bacon, conversation of cheating triggered a thought. There is really nothing to prevent a person who wrote a bot from entering odd comments in the “comment” form or attributing the survey response to any email address they feel like entering in the box. Both boxes are optional, but someone could enter john cooks email, stephan lewandowsky’s and so on.

25. lucia:

    If someone wanted to go to the trouble to write a bot to cheat, it might help. They would know they need to set session id’s.

    Alternatively, they could just have their bot erase cookies after each visit. If I was trying to pretend to be many unique visitors, that’s what I’d do.

    given that Cook and co-conspirators ( 😉 ) like to run surveys, some people might conspire to write bots not only for the current survey but in anticipation that the current script survey would form the basis for future survey security methods.

    >.>

26. “As Cook is providing links with identifiers unique to the blogs, he clearly intends to process the results by “skeptic” and “non-skeptic” groups. I imagine the hypothesis he is testing is that “skeptics are so blind they can’t see the evidence even when it is pushed under their noses”

    I would propose another possibility.

    First, it appears that scientists themselves have ranked the abstracts.
    Let’s imagine that an abstract ranks as neutral according to scientists ( perhaps some ranked their own abstracts )

    Now, suppose a reader of SkS thinks the abstract is neutral but a reader of WUWT “sees” an implicit endorsement. There is psychological conclusion one can draw from this.

    Or, suppose a neutral abstract is read by a skeptic as being critical of AGW.. you have confirmation bias.

    Lets put it this way.. IF I had this data I would look for ways in SkS readers and WUWt readers read abstracts differently. ie skeptics cant read.

27. 1. Explicitly endorses and quantifies AGW as 50+%
    2. Explicitly endorses but does not quantify or minimise
    3. Implicitly endorses AGW without minimising it
    4. Neutral
    5. Implicitly minimizes/rejects AGW
    6. Explicitly minimizes/rejects AGW but does not quantify
    7. Explicitly minimizes/rejects AGW as less than 50%
    8. Don’t know

    I am attempting to make sense of why the survey is designed the way it is and for what purposes. Abstracts are seldom sufficiently specific and detailed to allow the reader to rate the underlying paper by the criterion given in the survey ratings. Notice that survey is very simply related to the question of human influence on the modern warming. It does not appear to get beyond the basic physics of GHGs and the warming effects of these gases in the atmosphere. In other words an abstract reporting on historical temperature reconstructions that indicated that the modern period was the warmest would not necessarily say anything about why the modern period was warmer except by implication. The survey ratings are not really framed to consider endorsement of the warming (or its rate) in future times except by inference.

    Looking at the possible ratings and specifically what is being rated, would, I think, put this survey into the realm of something very noncontroversial and hardly relevant for the thoughtful and informed people involved in the AGW debate. The lower boundary for warming in rating 1 above is positioned such that very few informed people would personally disagree with it. The upper boundary on the other hand indicates that the survey is considering only the AGW that has already occurred, as I take the 50% to mean 50% of the warming was cause by AGW as opposed to natural causes. If I carry that limitation for the survey through to the following ratings I would get very different responses than I would if the ratings involved both past and future AGW. Notice that the survey for the AGW ratings following the neutral response have to be related to past AGW or past temperatures in order to make sense.

    If the survey is strictly aimed at past temperatures and AGW then the ratings are based on the instrumental records and what part of the warming in that period is related to AGW. If the survey can be interpreted to be related to future warming there will be a disconnect between the periods of interest below and above the neutral response. It would also remove any upper AGW limit for the survey ratings of 1,2 and 3 and those ratings would have a range of AGW being just above 0 to an unbounded limit.

    How do others here interpret the survey in the terms I have presented?

    Also of note is the fact that almost all climate science and related papers do not deny or claim to show the nonexistent of AGW and within the limits described in all the survey questions. All this leads me to believe that the survey makers are interested in how many survey takers will misinterpret the abstract. That evaluation would require the makers to come up with an independent means of abstract interpretation and for all of the abstracts in the survey. That seems to me like a very difficult method to carry out. Another possibility is that the survey is an indirect method of using the public to evaluate, become aware of and report the “consensus” on AGW. After all the survey ratings have the same ambiguous nature that the consensus does in that the lower limits would include almost everybody and probably be useless for initiating any government mitigation while the upper limits, and thus range of consensus, are undefined.

28. In all likelihood, Cook is not conducting a survey but rather crowdsourcing his work. Cook has presented preliminary data at the AGU from his project (it could be his dissertation), which involves classifying climate science papers. He’s getting people to do work for him.

29. First, it appears that scientists themselves have ranked the abstracts.

    Based on what Pielke Sr. wrote in 2012,
    1) the authors of papers ranked their own full papers based on memory of that paper. They may have been sent a form containing multiple papers. They did not provide ranks of how they would interpret the paper if the only information available to them was the abstract.

    2) The wording of the answers in the survey sent to authors was different than the one used now.

    3) The authors answered a preliminary question which (I think) appeared before the one where they ranked the consensus aspect.)

    Let’s imagine that an abstract ranks as neutral according to scientists ( perhaps some ranked their own abstracts )

    Now, suppose a reader of SkS thinks the abstract is neutral but a reader of WUWT “sees” an implicit endorsement. There is psychological conclusion one can draw from this.

    Or, suppose a neutral abstract is read by a skeptic as being critical of AGW.. you have confirmation bias.

    Lets put it this way.. IF I had this data I would look for ways in SkS readers and WUWt readers read abstracts differently. ie skeptics cant read.

    I would also wish to look to see if we got differences based on who the reader was or where they came from. But if hat is the purpose, it has not been revealed to the participants. It seems to me this violates the ethics requirements at Cooks university. Of course, that university may see things differently, but I think he should reveal his purpose.

    If he does not plan to compare how populations from different blogs interpret things differently, then I can see no good reason to leave breakcrumbs to de-anonymyze users.

30. Steven Mosher:

    First, it appears that scientists themselves have ranked the abstracts.
    Let’s imagine that an abstract ranks as neutral according to scientists ( perhaps some ranked their own abstracts )

    Scientists didn’t rank the abstracts. A number of authors were contacted and asked to rank their own papers.

    Shub:

    In all likelihood, Cook is not conducting a survey but rather crowdsourcing his work.

    Doubtful. They’ve already submitted their paper, and they worked on the rankings for it for quite some time. I can’t see any reason they’d combine the two sets of data. I think the purpose Mosher gave is correct.

31. I took this survey from the link you gave to ScepticalScience. Using best estimate and secondguessing the article autorhors after a quick evaluation (1-2minutes each) my rating was 3.0 compared to the 3.1 reported back by the Cook team.
    My immediate reaction was that the abstracts are to vague (like Rorschach figures in psycology) and i would need access to the articles themselves to make an informed judgement. For most of the abstracts I would be just as confortable giving 1 value up or down, for a few 1 value up and down. I tried to be fair, tending to add to the rating every time i had “rounded down” and vise versa.
    A stricter reading like yours, just considering the actual abstract content, will yield higher values than the Cook team. Probably what they expect and want to show, if I could only understand what it will prove

32. Pingback: The Blackboard » A Random Failure

33. I post the question that I think would be critical to understanding the survey and what the multiple choice replies mean. Is the survey dealing with past AGW/temperatures only and are the replies related to how much of the modern warming can be attributed to AGW?

34. Kenneth–
    It is a features of surveys that ultimately they are “about” whatever the people who answered the questions interpreted the question are “about”. In the case of a pulldown menu survey, the wording of the answers influences that. It some regard even the people who wrote the survey may not really know what the survey is about. The most the authors of the survey know is what they intended it to be about. Sometimes, it’s not clear they even had a clear thing in mind when they wrote the survey.

    Is the survey dealing with past AGW/temperatures only and are the replies related to how much of the modern warming can be attributed to AGW?

    I think the answer to this is no one knows. I think John Cook thinks it is “about” figuring out what ‘where’ the scientific consensus “is” on AGW. Whatever that means.
    I think no one can

35. Brandon,
    True. It is not in relation to the paper that is apparently ready to come in Env Res Lett. But my guess is from the nature of the abstracts – being mostly boring and ambiguous. Cook is likely using crowdsourcing in a tie-breaking exercise. Which is why the questions that appear are not random.

    If he is crowdsourcing, he is not surveying.

36. lucia (Comment #112479)
    May 5th, 2013 at 9:10 pm

    Lucia, I think the answer to my question is critical to the design of the survey and any biases it might provoke in the responses. If the survey is about past warming and what is the partition between AGW and natural, the rating replies in 1, 2 and 3 are bounded . If, on the other hand, the survey includes potential future warming considerations then the replies for 1, 2 and 3 have no upward boundaries and provide a range of warming that would represent what most reasonable people can agree to upward to what the most aggressive warmer would endorse.

    It appears that these abstracts have been rated by the abstract authors and thus it would be informative to determine their understanding of the question I have posed and particularly if Cook plans to make any comparisons. Of course, the authors should be very cognizant of what the main body to which the abstract is attached is endorsing and that could cause a big disconnect between author and layperson interpretations. If a scientist who produced the abstract and attached paper is not interested in clarity of this point I would question their seriousness. I also would be curious as to how many scientists would want to avoid being seen having the contents of their paper “endorsing” a position on AGW – evidential support maybe, but endorsement sounds too strong and too much on the side of advocacy.

    This apparent and intentional vagueness would lead me to believe that the survey is about showing a consensus on AGW for it is that consensus, as used by advocates/scientists, that is always vague, unbounded and pretty much useless in putting forth a position on AGW that might be used to make policy. The consensus statement usually does little more than restate a position to which almost every reasonable person would agree, i.e. GHGs can cause global warming. Unfortunately for those who want and need something to push immediate government mitigation attempts, a clear reading of that consensus does nothing for their advocacy.

37. Kenneth

    Lucia, I think the answer to my question is critical to the design of the survey and any biases it might provoke in the responses.

    I don’t disagree. But whatever the survey design is it has already been designed and implemented by John Cook. The only way for you, me or anyone to guess the answer to your question is to read the survey introduction. That’s all we have.

    As far as I can tell: The survey does not specify that distinction. You can ask over and over and the answer will be “the survey does not specify that distinction”.

    It appears that these abstracts have been rated by the abstract authors and thus it would be informative to determine their understanding of the question I have pose

    Pielke Sr. posted when he was asked to fill out the survey.

    http://pielkeclimatesci.wordpress.com/2012/08/22/skeptical-science-survey-by-john-cook-on-climate-related-research/

    He reproduced the survey for authors

    Survey on Climate Change Consensus in the Peer-Reviewed Literature

    Please select from both drop downs below to rate your paper, specifying category and level of endorsement. You may also add any comments (e.g. – indicate if the paper was erroneously attributed to you). All papers must be rated in one sitting.

    Category: The first drop down indicates what category of research your paper covers. If your paper addresses more than one category, select the category that is the major focus:

    Impacts:effects and impacts of climate change on the environment, ecosystems or humanity
    Methods:focus on measurements and modelling methods, or basic climate science not included in the other categories.
    Mitigation:research into lowering CO2 emissions or atmospheric CO2 levels
    Not Climate Related:This includes social science research about people’s views on climate
    Opinion:Not peer-reviewed
    Paleoclimate: examining climate during pre-industrial times

    Endorsement: The second drop down indicates the level of endorsement for the proposition that human activity (i.e., anthropogenic greenhouse gases) is causing global warming (e.g., the increase in temperature). Note: we are not asking about your personal opinion but whether each specific paper endorses or rejects (whether explicitly or implicitly) that humans cause global warming:

    Explicit Endorsement with Quantification:paper explicitly states that humans are causing most of global warming.
    Explicit Endorsement without Quantification:paper explicitly states humans are causing global warming or refers to anthropogenic global warming/climate change as a given fact.
    Implicit Endorsement:paper implies humans are causing global warming. E.g., research assumes greenhouse gases cause warming without explicitly stating humans are the cause.
    Neutral:paper doesn’t address or mention issue of what’s causing global warming.
    Implicit Rejection:paper implies humans have had a minimal impact on global warming without saying so explicitly. E.g., proposing a natural mechanism is the main cause of global warming.
    Explicit Rejection without Quantification:paper explicitly minimizes or rejects that humans are causing global warming.
    Explicit Rejection with Quantification: paper explicitly states that humans are causing less than half of global warming

    Survey Form (mouseover the paper title to display the abstract)

    Listed below are peer-reviewed articles you have co-authored listed in the ISI Web of Science database matching the exact phrases ‘global warming’ or ‘global climate change’. Due to the specificity of the search, some of your climate related papers may not have appeared in the search.

    would lead me to believe that the survey is about showing a consensus on AGW for it is that consensus, as used by advocates/scientist

    That seems to be precisely what John Cook says it’s about. So if your are concluding it is about that: That’s what John Cook says it’s about.

    Everyone can decide for themselves whether a survey “about” teaches us anything about anything. But that does seem to be what the survey is “about”.

    The consensus statement usually does little more than restate a position to which almost every reasonable person would agree, i.e. GHGs can cause global warming.

    Yes. I think he will “reveal” that people agree that GHG’s cause warming.

38. The survey I took included an paper which said regarding global warming ” there is insufficient evidence to convincingly substantiate” the hypothesis by ” Hydrometerological” means. Where does this fall on his scale. It does mention GW. However, it drew NO Conclusion. It was a true neutral. However, the neutral choice in the survey implies the topic wasn’t mentioned. “Don’t Know” is unexplained as a choice. Does it mean the authors Don’t claim to know the truth about the cause of climate change or does it imply that I am unable to assess the paper. Also, with regard to analyzing the results, I thought taking the mean of a Likart scale was invalid methodology.

39. Brandon,
    I’m not so sure there is something nefarious going on here. The PHPSESSID is a cookie used by the PHP container to keep track of the user’s session when there will be multiple interactions with the web site (as there would be in a survey). This is equivalent to the JSESSIONID cookie which most Java-based servlet containers use to keep track of user sessions. A pseudorandom 128-bit cookie is typically generated by the container and then is passed back and forth between browser and server so that session-based information (which questions you have answered, ability to access the next question in the stack, etc.) can be maintained and used in the user experience. The cookie may be set to expire on the browser at browser close, but it doesn’t expire on the server at that time. The container will have a session timeout setting which tells it that if a session is not accessed for so many minutes, mark it for deletion and get rid of it at the earliest possible opportunity. That may explain being able to reuse the session ID for a short while after you closed and reopened the browser – the session had not yet expired on the server. As to reuse of the PHPSESSID, I guess that people could access the survey and instantly share their session ID with a group in order to try to manipulate the results, but why? If the session ID is a pseudorandom 128-bit number, there is nothing in its material to identify anything about the client. And good look guessing one…
    If I’ve missed a possible security hole, please let me know, but I think there is a perfectly reasonable explanation for the cookie.

40. Jim Bennet–

    If I’ve missed a possible security hole, please let me know, but I think there is a perfectly reasonable explanation for the cookie.

    I don’t think Brandon has suggested there is a ‘security hole’. He is saying the papers drawn from the set do not appear to be drawn randomly from a collection of 12,000. The PHPSESSION cookie seems to be involved, but that doesn’t mean anyone is suggesting a security hole per se.

    It seems John Cook does acknowledge the papers are not randomly drawn from a batch of 12,000. They are drawn from some smaller batch of those that were evaluated by their authors and which also were short.

41. Thanks Lucia for your link to R.A. Pielke Sr.
    The questions Pielke published are far better in terms of evaluating the full range of opinions rather than Cook’s which appears to quantifying or endorse a “consensus” on > 50% AGW. See:
    Brown, F., J. Annan, and R.A. Pielke Sr., 2008: Is there agreement amongst climate scientists on the IPCC AR4 WG1? [http://pielkeclimatesci.files.wordpress.com/2009/09/brown.pdf].

    My view of this survey is that it is much too limiting in the questions they are asking regarding the findings in the peer reviewed literature. It appears they are writing their questions to reinforce a preconceived perspective, rather than complete an actual survey of the diversity of viewpoints in climate system science and the role of humans in its alteration.

    e.g. Cook does not distinguish between the three major beliefs that:
    1) change is natural with little human impact – eg 50%.
    I have seen scientific papers based on available data advocating each of these positions.

42. PS I tried to describe three cases that need to be tested:
    Little AGW 50%

43. Well whatever the result, this post and analysis has ensured that Lew and Cook wont be able to classify respondants as skeptics or warmists so the results will have to be “combined”.

    The question is…will they ignore this and press on with classification by site anyway?

44. Lucia in your ‘mark’ after you did the survey it was indicated:

    “The average rating of the 10 papers by the authors of the papers was 3.4.”

    This reads to me as if the actual authors of each paper were asked where they thought their paper stood between 1 – 7. Not Cook’s own researchers. This would mean Cook would have to have contacted each of the 1,000 (or was it 1,200 authors), and it is not the opinion of his grad, post grad or whatevers. I doubt that was done for such low brow ‘research’.

    All this Social Psycho Climate Science research has a Diederik Stapel feel to it anyway. See: http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academic-fraud.html?pagewanted=2&_r=4&hp&pagewanted=all&

45. James

    This reads to me as if the actual authors of each paper were asked where they thought their paper stood between 1 – 7.

    We now believe that this is what those words mean.

    This would mean Cook would have to have contacted each of the 1,000 (or was it 1,200 authors), and it is not the opinion of his grad, post grad or whatevers. I doubt that was done for such low brow ‘research’.

    We now believe that 10 papers presented are not selected from the 12,000 obtained from the search of the science citation list. Rather, they are selected from a much smaller batch corresponding to those those authors who accepted an Cooks invitation to evaluate their papers. We do not know whether Cook sent out invitations to a subset of of authors or whether an invitation was sent out for each paper. (It does appear that if one author wrote 10 papers, the invitation asked them to fill out the survey for all their papers.)

    Also, only papers with abstracts containing fewer than 1,000 characters seem to have been included.

    We don’t know the number of papers in the smaller set. But my impression is it is no more than 500 and likely less. But I could be totally wrong.

46. http://books.google.com/books?id=UFCNRZ-O7nwC&pg=PA1&source=gbs_toc_r&cad=4#v=onepage&q&f=false

    http://books.google.com/books?id=UFCNRZ-O7nwC&pg=PA155&source=gbs_toc_r&cad=4#v=onepage&q&f=false

47. Something I don’t get. The relationship between whether the RESULTS of the study IMPLY or ENDORSE antrhopogenic global warming, or whether the authors of abstract ASSUMES anthropogenic global warming. In the case of an assumption without a supporting result, is that a NEUTRAL, do you think? The questions seem badly designed, and ambiguous with respect to the question of RESULTS vs ASSUMPTIONS.

    Curiously, in my batch of studies there were a significant number of studies that state that that they intend to measure something relating to anthropogenic global warming, but don’t give indications of their results. (That’s an odd thing in an abstract, I think — not to state results). What’s that? Neutral, I imagine.

48. There’s a link at James Annan’s blog http://julesandjames.blogspot.co.uk/

    http://survey.gci.uq.edu.au/survey.php?c=ITUTR1MY7NSE

49. Thanks for the additional link! They are trickling out.

50. Just went there and left this message at the end of the form (can’t wait to be mentioned as a paranoid in next Lewpaper):

    “Came well intentioned to participate in the survey, but I’m not going to answer the questions because I don’t see the point. Classifying the research paper of an expert, say, in fisheries management, as explicitly endorsing AGW because AGW is mentioned in the abstract and maybe used as a working hypotesis, makes absolutely no sense. Partly because of a well known tendency to mention AGW in abstracts in order to attract readers; partly because the opinion of an expert in fisheries on AGW has no more value than that of an engineer or another informed layperson.
    Moreover, the categorization you propose is clearly skewed: “Explicit Endorsement with Quantification” needs an abstract saying that humans cause more than 50% of AGW; however, when a natural mechanism is proposed as the _main_ cause of warming (that is, more than 50%), the rejection is only implicit (!).”

51. “To best replicate their conditions, I suspect you should familiarize yourself with abstracts because I suspect each evaluator evaluated many, many abstracts. So most were evaluated by people who had already seen many previous abstracts.”

    The author ratings are from the actual authors of the papers, not Cook et al. The results of Cook et al are not part of the survey.

    “In all likelihood, Cook is not conducting a survey but rather crowdsourcing his work. Cook has presented preliminary data at the AGU from his project (it could be his dissertation), which involves classifying climate science papers. He’s getting people to do work for him.”

    The paper has already been submitted. This survey may become a footnote, but it is not a main focus of the paper. Presumably Cook et al will compare their own ratings against the papers that were rated by their authors. That would be a reasonable check to include in the paper at any rate.

52. “Something I don’t get. The relationship between whether the RESULTS of the study IMPLY or ENDORSE antrhopogenic global warming, or whether the authors of abstract ASSUMES anthropogenic global warming. In the case of an assumption without a supporting result, is that a NEUTRAL, do you think?”

    The bar is set at >50% of human contribution to global warming, not just any anthropogenic contribution. I would imagine that would obviate your concern here. The rating should be neutral if all that is discussed in the abstracts is the parameters you’ve set.

    It would be interested if participants looked up any full versions of the papers, and compare their ratings for those with the abstracts.

53. barry

    It would be interested if participants looked up any full versions of the papers, and compare their ratings for those with the abstracts.

    I did. I discussed it comments some where. If ratings are based on reading of a full paper rather than the abstract, the average rating and spread would certainly differ. That test could also be done if someone wanted to do it. Though I don’t think the correct person to do it is John Cook. You need someone who is not deeply committed to predetermined outcomes.

54. barry
    If by ‘the paper”, you mean

    Quantifying the consensus on anthropogenic global warming in the scientific literature

    That is already accepted. Consequently this survey cannot be a footnote of that paper because it’s already accepted for publication. As such, it is frozen. The results of the current survey cannot be shoved in.

    Presumably if the results from this survey are being analyzed, it would possibly become a future paper to be submitted on its own.

    Also, I think you are assuming that if some authors reviewed their own papers, that means an SkS team didn’t also do a review. Both could have been done. Both being done is what comports with what John Cook wrote at SkS:

    In our paper, Quantifying the consensus on anthropogenic global warming in the scientific literature, we analysed over 12,000 papers listed in the ‘Web Of Science’ between 1991 to 2011 matching the topic ‘global warming’ or ‘global climate change’.
    Even though it appears some individual authors analyzed some of their own paper, it seems odd for John to refer to those authors as “we”. But even if he did, it seems unlikely that he got such stupendous compliance that all 12,000 of the papers were analyzed by the authors.

    Beyond that, the current survey reads

    This survey mirrors a paper, Quantifying the consensus on anthropogenic global warming in the scientific literature, to be published soon in Environmental Research Letters, that analysed over 12,000 climate papers published between 1991 to 2011. The purpose of the survey is two-fold. Firstly, to replicate the experience of rating a random assortment of climate papers, gaining appreciation of the diversity of climate research available. Secondly, an invitation to this survey has been sent to a wide range of climate blogs in order to analyse ratings from a diverse range of participants.

    The italicized portion does not describe individual authors rating their own papers. That exercise would not cause those authors to ‘gain[…] an appreciation of the diversity of climate literature available’. The italicized bit describes people rating a random assortment of climate papers.

55. Hi Lucia,

    “Also, I think you are assuming that if some authors reviewed their own papers, that means an SkS team didn’t also do a review. Both could have been done.”

    I assumed both were done. I believe 12,000 papers were rated by John Cook et al, and they asked “thousands” of the authors to rate their own papers as well. That’s a reasonable step to take as a check against their results, I think.

    “If ratings are based on reading of a full paper rather than the abstract, the average rating and spread would certainly differ. That test could also be done if someone wanted to do it.”

    Several participants posted their results compared to the average author rating for the 5 abstracts. In each case the surveyors ranking was closer to neutral than the authors of the papers. I got an averqage rating of 3.5 against the authors’ ratings of 3.3, making my results closer to neutral than the authors.

    Some participants also checked the full papers, where possible, and said that the full versions tended to be less neutral (and generally more towards supporting human contribution at >50%), than the abstracts they had rated. Did you read the comments there?

    I got my understanding of the paper mainly from the lede of the SkS post crowd-sourcing funding for the paper submission.

    “Over the past year volunteers here at Skeptical Science have been quietly engaged in a landmark citizen science project. We have completed the most comprehensive analysis of peer-reviewed climate science papers ever done. Some 21 years worth of climate papers – more than 12,000 in all – have been carefully ranked by their level of endorsement of human-caused global warming. We also invited thousands of the authors of these papers to rate their own papers.”

    http://www.skepticalscience.com/Be-part-of-landmark-citizen-science-paper-on-consensus.html

56. Oh, I see where i may have confused you:

    “The author ratings are from the actual authors of the papers, not Cook et al. The results of Cook et al are not part of the survey.”

    I mean the public survey. The Cook et al ratings are not mentioned there.

57. barry–

    The Cook et al ratings are not mentioned there.

    I agree the magnitude of the Cook ratings are not what is mentioned after you fill out the survey. But it mentions they were done in the introduction and that process is the process it seems to wish us to ‘replicate’.

58. barry-

    Several participants posted their results compared to the average author rating for the 5 abstracts. In each case the surveyors ranking was closer to neutral than the authors of the papers. I got an averqage rating of 3.5 against the authors’ ratings of 3.3, making my results closer to neutral than the authors.

    This is what I expect given that abstracts contain much less information. If you read an abstract, look up the paper, and read the paper, I think you will find that reading the introduction often provides information relevant to answering the survey question. For papers specifically about global warming, the results and conclusions gives some information (quantification and so on.) Often, this information is not in the abstract.

    It’s in the nature of the abstract to focus on contents of the paper in the narrowest possible way. As such, it is not a very useful place to find evaluations for or against the “consensus”!

59. Like the commenters at SkS and yourself, I don’t think rating the abstracts is going to be a great method of testing the consensus, although it has interest for different metrics. It appears to have raised a storm, with people mistakenly thinking it is part of the paper they have submitted, not realizing it has already been accepted.

60. barry

    with people mistakenly thinking it is part of the paper they have submitted, not realizing it has already been accepted

    Other than your comment about the footnote in barry (Comment #112740) , I haven’t read anyone say anything to suggest they think this survey will be included in any paper that has already been submitted nor any that has been accepted.
    Could you clarify what it is people mistakenly think and possibly give an example of a person who thinks it? Because I don’t understand what you mean to communicate.

61. “It’s in the nature of the abstract to focus on contents of the paper in the narrowest possible way. As such, it is not a very useful place to find evaluations for or against the “consensus”!”

    Thinking about it again, While it may not be the ideal place to glean consensus, the abstracts are quick way for punters to contribute. The major part of the work being done, this is a simplified version, which makes it appropriate for a public review. The likely result is more neutral ratings (borne out by the posted results), but there’s no reason to think the results will be misleading if properly caveated. I think there may be something worthwhile in comparing all the results.

62. “Other than your comment about the footnote in barry (Comment #112740) , I haven’t read anyone say anything to suggest they think this survey will be included in any paper that has already been submitted nor any that has been accepted.
    Could you clarify what it is people mistakenly think and possibly give an example of a person who thinks it?”

    An example is Shub’s comment in this thread, above.

    “In all likelihood, Cook is not conducting a survey but rather crowdsourcing his work. Cook has presented preliminary data at the AGU from his project (it could be his dissertation), which involves classifying climate science papers. He’s getting people to do work for him.”

    http://rankexploits.com/musings/2013/links-to-john-cooks-survey/#comment-112451

    And davidmhoffer at WUWT seems to confuse the survey with the paper (see his last comment).

    http://wattsupwiththat.com/2013/05/08/is-john-cook-planning-to-use-systematically-biased-correct-survey-answers-to-make-unbiased-skeptics-look-biased/#comment-1300731

63. barry

    are quick way for punters to contribute

    Contribute what to what? Dung to the dungheap? Coals to newcastle? Food to the foodpantry?

    may not be the ideal place to glean consensus

    Not ideal? It’s so far from ideal it’s difficult to believe crowdsourcing this is anything more than learning what different groups of people see in a rorschach test.

    this is a simplified version

    No. Abstracts aren’t simplied versions of full papers.
    Of course they are publicly viewable. So are youtube videos, photomontages and all sorts of things. Publicly viewable doesn’t turn something that is not useful for diagnosing consensus into something that is useful for that purpose.

    the results will be misleading if properly caveated

    Misleading with regard to what question? The raw results are nothing more than a database of entry results. Whether the story developed from the contents of the database is misleading will depend on what the person spinning the yarn writes. But I don’t think reviewing abstracts that contain no content to permit people to discern where the full content of the paper likes on the 1-7 likert index Cook provides can be useful for learning much of anything about where the consensus of science is on that likert scale.

    I think there may be something worthwhile in comparing all the results.

    Maybe. Or not. It might be more interesting to ask a bunch of people to read the abstract , give a 1-7 rating, the read the paper and re-rate. See how those compare. At least then you are only comparing one change: rating based on abstract vs. rating based on paper.

    We don’t know all the details of what John is trying to do yet. But from the outside it looks like he is comparing apples to rocks. Too many differences.

64. An example is Shub’s comment in this thread, above.

    “In all likelihood, Cook is not conducting a survey but rather crowdsourcing his work. Cook has presented preliminary data at the AGU from his project (it could be his dissertation), which involves classifying climate science papers. He’s getting people to do work for him.”

    Maybe Shub is confused or maybe not. Anyway, quite a few people told him they don’t think John is just substituting their surveys for one he claims he (or his team) are doing. So that’s maybe one person who is confused.

    And davidmhoffer at WUWT seems to confuse the survey with the paper (see his last comment).

    But as it stands, his last comment is

    davidmhoffer says:
    May 8, 2013 at 10:44 pm

    Richard Tol (@RichardTol) says:
    May 8, 2013 at 10:33 pm
    I took the survey three times.
    >>>>>>>>>>>>>>>>>>>

    From the same computer and link? Talk about an opportunity for ballot stuffing!

    I figured you erred by saying his last comment, so I scrolled back to:

    May 8, 2013 at 10:16 pm ?

    I don’t see anything in there that remotely suggest he is confusing the survey with the previous paper. Maybe if you quote the specific text davidmhoffer wrote and show me which words you think suggest he is confused I would see this as evidence. But as it stands, I did a little wild goosehunting and came home with no goose.

65. Barry
    Just to be specific. I know *exactly* what I am talking about, and there are reasons for that. Cook is getting people to do work for him. It is not for the present paper, but he will very likely use this input in a future paper. Also, going by the deliberately imprecise language used in the survey invitation, it is highly likely that the information derived from the survey will be used *in a way not immediately obvious* to casual takers of the survey. This is additionally virtually guaranteed given that the solar physicist Cook has now become a armchair psychologist, with the element of deception present almost in every psychology research effort.

66. …it’s difficult to believe crowdsourcing this is anything more than learning what different groups of people see in a rorschach test.

    Seriously?

    …I don’t think reviewing abstracts that contain no content to permit people to discern where the full content of the paper likes on the 1-7 likert index…

    Nowhere does Cook say that abstract ratings are a proxy for the ratings given the full versions.

    You’re invited to rate the abstracts of the climate papers with the purpose of estimating the level of consensus regarding the proposition that humans are causing global warming.

    Abstracts sometimes include a note on the paper’s conclusion/s, sometimes only describe the focus of inquiry. Abstracts can give opinions. Consensus is simply the weight of opinion. As long as distinctions of parameters and results between the public survey and the rest are clarified and discussed, then that’s fair play and we can form our own opinions as we see fit. If there is strong consistency between the survey results and the others, that would obviously be of interest.

67. Eliding davidmhoffer’s post,

    Mosher: “You are not getting the point of the survey”
    >>>>>>>>>>>>>>>>>>

    davidmhoffer: ….Cook’s survey is highly manipulative and clearly intended to produce a predetermined outcome.

    What that outcome is we don’t know for certain, but so what? Half the papers appear to be a study based on a predetermined outcome….

    The difference between those studies and this one is that this one may well have been debunked before it was even completed.

    hoffer doesn’t seem to have realized the disctinction. Here’s his post again to check if the elision has changed the meaning.

68. Thanks for the clarification, Shub.

69. barry

    Seriously?

    Yes. If you think something else can be learned from this, I suggest you specify what that is and explain why that particularly could be learned from this crowd sourced experiment designed as it has been designed. Otherwise, “seriously” is a pretty darn lame response.

    Abstracts sometimes include a note on the paper’s conclusion/s, sometimes only describe the focus of inquiry. Abstracts can give opinions.

    “Can” does not mean “usually do”. Have you read the ones presented in the study? Nearly none do.

    If there is strong consistency between the survey results and the others, that would obviously be of interest.

    This survey result and which other survey? And what if the results are inconsistent?

70. This survey result and which other survey?

    This is getting argumentative.

    Cook et al rates 12,000 papers according to their conformity with the consensus view on climate change. They also invited thousands of the authors to rate their own papers – a useful check. Separate from that study is the public survey assessing the consensus view from only the abstracts of papers that were rated by the original authors. That’s 3 different groups rating science studies according to their conformity with the consensus, with the public review limited only to their abstracts.

    Many papers don’t have the full version online. Getting the public to rate the full papers is a little onerous, anyway, so if you want as much participation as possible (larger sample is better), make the survey doable within 20 mintues, with as many studies as pssible – using the abstracts is an efficient way to get the most people participating.

    And what if the results are inconsistent?

    Then that can be discussed. The rating system might be different, too. We won’t know until the paper comes out. But there’s no need to write off the public survey attempt just because those results and/or rating system may be inconsistent with the study’s. We don’t even know if or how the public survey results will be compared with the study by Cook.

    If you think something else can be learned from this, I suggest you specify what that is

    As I explained, abstracts can provide content that relates to the survey. If a sub-set of abstracts provide opinion that can be rated, the result will likely be that the public survey results will be closer to neutral, presuming particpants give honest, well-considered reviews. If the public survey result is either consistent with the study results, or more neutral but consistently so, this would be corroborative. If not, then that can be discussed, too, or dismissed.

    I don’t see a problem with it. It’s an informal project at this stage. The ultimate purpose is a little vague. I’m not frustrated by that, and I don’t know why it’s caused such a ruckus. People can do it or not. I was content to participate and see what happened.

    I think comparing it to a rorschach test is way too unreasonable.

71. barry

    This is getting argumentative.

    I agree. When someone responds to a substantive discussions with “seriously” are being argumentitive — though in a rather incoherent- snotty- teenager way. If you don’t want the conversation to be argumentative, avoid indulging in that sort of surly behavior.

    As for what follows, you are discussing what you think you must get people to do but are failing to describe what goal you think will be achieved by that effort. If you want to tell us something is worthwhile can be accomplised by this effort by all these people, you are going to have to state what that is. Repeating what you are going to make them “do” is just avoiding the question I’ve asked you. You have claimed there is something worthwhile that can be accomplished by this. What is that wortwhile thing which you think can be accoplished?

    I suggest that if you cannot state what it is and persist in just characterizing it vaguely along the limes of “something”, then quite likely, you do not
    know what that “something”.

    But there’s no need to write off the public survey attempt just because those results and/or rating system may be inconsistent with the study’s.

    Write it off? I’m criticizing it as unpromising. My doing so has no effect on whether it gets implemented, written up and so forth. There may be “no need” for my criticism, but likewise, there is “no need” for you to complain that people who think the effort looks unpromising are expressing their opinion. Moreover, as far as I can tell there is “no need” for this survey at all– and yet it is being done. And I have been asked to invite my blog readers to be involved in it.

    I don’t see a problem with it. It’s an informal project at this stage.

    How informal is it? This so-called informal project involved clearing the project with the ethics panel at a university. It involved identifying 50 blogs. It involved sending emails out to the 50 blogs– and for all appearances sending them out in batches. (Annan’s like appeared some time after other. Though, admittedly, it may be he was out of town and didn’t see the invitation.)

    The ultimate purpose is a little vague. I’m not frustrated by that,

    Of course the purpose is vague. Although the ethics guidelines for that university seem to suggest that the purpose is supposed to be disclosed, it is infact vague.

    I don’t know why it’s caused such a ruckus

    What ruckus? SkS sent out invitations to bloggers. Bloggers blog. They did so. Discussion of the event ensued. There discussion at SkS. Do you consider that a “ruckus”? Or is it only discussion at forums not controlled by SkS that is the “ruckus”? You are participating in the discussion. Are your comments part of the “ruckus”?

    Many of these bloggers were shall we say “highlighted” in a recent “furious recursion” paper co-authored. So it’s hardly surprising much of the conversation refer back to Lewandowsky’s paper, how Cook and Lewandowski used that incident and so forth. I don’t consider any of this a “ruckus”. But if you don’t understand why the people invited to expressed their opinions of that, then your lack of understanding of why people do something merely shows you lack perception.

    I was content to participate and see what happened.
    Me too. And like you I am discussing it as it happens. I am under the impression people discuss sporting events too– some criticize the players performance. Some cheer them on. But no one suggest either the criticism or the cheering is evidence that one is not content to participate in the event and see what happens.

72. As for what follows, you are discussing what you think you must get people to do but are failing to describe what goal you think will be achieved by that effort.

    At the risk of repeating myself, a subset of the abstracts have information on the consensus question given in the survey. This subset may be considered a kind of proxy for the the full papers they head.

    A possible outcome might be:

    “While the public survey results are closer to neutral than the Cook et al review and the authors’ own ratings, the difference is fairly consistent. As we anticipate a more neutral score in general for abstracts-only, the comparison appears to corroborate the authors’ ratings and those of Cook et al, at least in inclination (qualitative) if not degree (quantitative).”

    There is plenty of debate on the skeptic climate blogs that there is a consensus at all. Broad participation from skeptics and others giving this result would provide data for that question.

73. ‘barry’,
    Are you a Skepticalscience regular posting under a different name here?

74. As for what follows, you are discussing what you think you must get people to do but are failing to describe what goal you think will be achieved by that effort.

    At the risk of repeating myself, a subset of the abstracts have information on the consensus question given in the survey. This subset may be considered a kind of proxy for the the full papers they head.

    This part is repeating yourself and is not a description of what would be achieved.

    “While the public survey results are closer to neutral than the Cook et al review and the authors’ own ratings, the difference is fairly consistent. As we anticipate a more neutral score in general for abstracts-only, the comparison appears to corroborate the authors’ ratings and those of Cook et al, at least in inclination (qualitative) if not degree (quantitative).”

    If this is the achievement, I would characterize the effort as having not been worth doing. Also note: If this is the result, the crowd sourced effort will have told us absolutely nothing about the consensus.

    There is plenty of debate on the skeptic climate blogs that there is a consensus at all. Broad participation from skeptics and others giving this result would provide data for that question.

    Broad participation from skeptics on rating abstracts would only provide data for that question if rating abstracts gave any information on this issue. Which. They. Don’t.

    The only thing broad participation in the abstract reading can tell us is: How do different groups of people rate the “tea leaves” on a likert scale of 1-7.

    We don’t need to see the final results to know this. We only need to read an assortment of abstracts and see that they do not contain the information required to determine whether the paper or author supports the consensus.

75. What ruckus?

    The conspiracy theorising, the outright rejection of the survey and promotion of bad faith of John Cook and SkS, the suspicion regarding the purpose, and the investigation of randomisation of the abstracts, that can be read here, WUWT and elsewhere.

    Write it off? I’m criticizing it as unpromising.

    I thought you were writing it off when you wrote:

    “Contribute what to what? Dung to the dungheap?”

    “it’s difficult to believe crowdsourcing this is anything more than learning what different groups of people see in a rorschach test.”

    “But I don’t think reviewing abstracts that contain no content to permit people to discern where the full content of the paper likes on the 1-7 likert index Cook provides can be useful for learning much of anything about where the consensus of science is on that likert scale.”

    I have already said I don’t think the public survey will inevitably provide a robust result on the consensus, but it may have some limited utility. We won’t know much until the survey is complete and the results given. It occurs to me that there is some merit in giving the least amount of inrformation as to the ultimate purpose of a survey, so as to avoid skewed results from surveyors – blind testing and all that.

76. We only need to read an assortment of abstracts and see that they do not contain the information required to determine whether the paper or author supports the consensus.

    Some do. More endorse the consensus implicitly. Some implicitly and explicitly reject the consensus as stated by Cook.

    I opened the survey to find 10 abstracts (all different from the first 10 I surveyed). Most of these gave implicit endorsement at the least.

    “…We find strong historical linkages between civil war and temperature in Africa, with warmer years leading to significant increases in the likelihood of war. When combined with climate model projections of future temperature trends, this historical response to temperature suggests a roughly 54% increase in armed conflict incidence by 2030, or an additional 393,000 battle deaths if future wars are as deadly as recent wars. Our results suggest an urgent need to reform African governments’ and foreign aid donors’ policies to deal with rising temperatures.”

    http://re.indiaenvironmentportal.org.in/files/Warming%20increases%20the%20risk%20of%20civil%20war%20in%20Africa.pdf

    “Unmitigated global warming due to the enhanced greenhouse effect could have significant impacts on the boreal forest in interior western Canada. Increases in annual temperature of 3 to 7 °C are projected for Alberta under a 2 × CO2 scenario by 2030 – 2050 A.D. Such an unprecedented rate of change has many short- and long-term implications for forest management and for industries.”

    http://pubs.cif-ifc.org/doi/abs/10.5558/tfc67342-4

    “A low-order physical-biogeochemical climate model was used to project atmospheric carbon dioxide and global warming for scenarios developed by the Intergovernmental Panel on Climate Change…”

    http://www.climate.unibe.ch/~plattner/papers/joos99sci.pdf

    “…The groups went on to hypothesise that the decrease in ionization dueto cosmic rays causes the decrease in cloud cover, thereby explaining a large part of the presently observed global warming. We have examined this hypothesis to look for evidence to corroborate it. None has been found and so our conclusions are to doubt it. From the absence of corroborative evidence, we estimate that less than 23%, at the 95% confidence level, of the 11-year cycle changes in the globally averaged cloud cover observed in solar cycle 22 is due to the change in the rate of ionization from the solar modulation of cosmic rays.”

    http://arxiv.org/pdf/0803.2298.pdf

    “With global warming, plant high temperature injury is becoming an increasingly serious problem…”

    http://xa.yimg.com/kq/groups/19147026/219445287/name/PNAS-2010-Sakata-8569-74%5B1%5D.pdf

    Some papers were neutral in the abstract, and swung to the consensus view in the body of the paper, eg:

    http://inderscience.metapress.com/index/A8L3R476M854521N.pdf

    My results for that survey were seven 3s (implicit endorsement with no quantification), and three 4s (neutral). Average rating was 3.3. I didn’t get an authors rating, because I’ve already done the test. I recorded the paper titles if you want to check.

    Warming Increases The Risk Of Civil War In Africa – 3

    Techniques For Estimating Uncertainty In Climate Change Scenarios And Impact Studies – 4

    Boreal Forest Sensitivity To Global Warming – Implications For Forest Management In Western Interior Canada – 3

    Global Warming And Marine Carbon Cycle Feedbacks An Future Atmospheric Co2 – 3

    Testing The Proposed Causal Link Between Cosmic Rays And Cloud Cover – 4

    Climate Induced Increases In Species Richness Of Marine Fishes – 3

    Feasibility Study On Smooth Shift From Lwr To Thorims-nes; A Mass Balance Calculation Of Fissile Materials And Minor Actinide – 3

    Auxins Reverse Plant Male Sterility Caused By High Temperatures – 3

    Negotiated Agreements As A Vehicle For Policy Learning – 4

    Global Climate Change Implications For Coastal And Offshore Oil And Gas Development – 3

77. What ruckus?

    The conspiracy theorising, the outright rejection of the survey and promotion of bad faith of John Cook and SkS, the suspicion regarding the purpose, and the investigation of randomisation of the abstracts, that can be read here, WUWT and elsewhere.

    Write it off? I’m criticizing it as unpromising.

    I thought you were writing it off when you wrote:

    “Contribute what to what? Dung to the dungheap?”

    “it’s difficult to believe crowdsourcing this is anything more than learning what different groups of people see in a rorschach test.”

    “But I don’t think reviewing abstracts that contain no content to permit people to discern where the full content of the paper likes on the 1-7 likert index Cook provides can be useful for learning much of anything about where the consensus of science is on that likert scale.”

    I have already said I don’t think the public survey will inevitably provide a robust result on the consensus, but it may have some limited utility. We won’t know much until the survey is complete and the results given. It occurs to me that there is some merit in giving the least amount of inrformation as to the ultimate purpose of a survey, so as to avoid skewed results from surveyors – blind testing and all that.

78. barry

    The conspiracy theorising,

    I haven’t seen any conspiracy theorising. Criticizing Cook’s survey isn’t conspiracy theorizing.

    the outright rejection of the survey

    The bloggers who didn’t post the links have every right to do as they please. Those who don’t want to take the survey have every right not to donate 15 minutes of their time to him. They aren’t John Cooks slaves or vassals. Failing to do him a favor is not a “ruckus”

    and promotion of bad faith of John Cook and SkS,

    Promotion?

    Look: A lot of people don’t like John Cook or SkS. Have a look at all the snide references to “fake skeptic”, or other feature. I personally don’t trust a guy who could be second author on this: http://www.frontiersin.org/Personality_Science_and_Individual_Differences/10.3389/fpsyg.2013.00073/abstract
    Maybe you have a different opinion. But I don’t consider saying we are dubious of the intentions when the 2nd author of that paper invites us to do him the favor of posting links to his survey or the favor of contributing time to fill out his survey. More over, given the obvious biases and blinders involved in that papers, we doubt his ability to objectively evaluate whatever data he might collect in this endeavor.

    You might call that a “ruckus”, but I’d say the stuff emanating from SkS could equally be called a “ruckus”.

    I thought you were writing it off when you wrote:

    If by writing it off you mean I have said I think it’s an exercise in reading tea leaves, yes.
    As for the dung to the dung heap, I gave a list of examples
    “Contribute what to what? Dung to the dungheap? Coals to newcastle? Food to the foodpantry?”
    Because you are being so nebulous and merely making claims like something would “contribute”, I was trying to get you to describe what might be contributed to what. I gave negative examples and positive ones. This is not “writing something off”.

    It occurs to me that there is some merit in giving the least amount of inrformation as to the ultimate purpose of a survey, so as to avoid skewed results from surveyors – blind testing and all that.

    If you want to fill out the survey, you are welcome to do so. No one is stopping you.

    FWIW: I’ve filled out the survey at least 20 times now. I want to read John Cooks section on throwing out data.

    the suspicion regarding the purpose, and the investigation of randomisation of the abstracts, that can be read here, WUWT and elsewhere.

79. Barry, Why are you pretending Cook has no history with those of us he would apply a label to? It is moronic to assume that he would suddenly turn around and become objective on AGW or skeptics for that matter.

80. Jeff, I think SkS has an ideological bent, like most other climate blogs. There has been confusion on the matter, and I’ve commented as to how I see things here, at WUWT and at SkS.

    There is a difference bewteen skepticism and suspicion. One can be suspicious of, but treat a matter with neutral skepticism. If anything, I’m trying to encourage the latter. There have been too many suppositions, stemming in part from the vagueness of the survey purpose, but also from the inclinations of commenters. I find the term ‘fake skeptic’ is all too often appropriate, and this applies to both ‘sides’ of the debate.

    A number of the SkS regulars have commented on the shortcomings of the survey. I don’t think it is going to have much of an impact with them, nor others in the climate blogs. I am still curious about what the results will be, and have no desire to argue or promote it as a robust test for consensus. If there is some coherence with other studies/polls on consensus, then that may have some limited value.

81. barry

    I find the term ‘fake skeptic’ is all too often appropriate, and this applies to both ‘sides’ of the debate.

    I think someone whose blog pretty much encourages conversations at his blog to label subjects of his study “fake skepctics” cannot be an unbiased investigator with respect to designing an experiment or interpreting the data. You are entitled to holding a contrary opinion. But I really don’t think it can be supported– and the fact that you say something like “I find the term ‘fake skeptic’ is all too often appropriate, and this applies to both ‘sides’ of the debate.” suggests you are not sufficiently unbiased to judge this matter.

    I honestly don’t think an unbiased investigator would have concocted the idea of even trying to gauge consensus by reviewing abstracts at all– and this is true even if this “study” find the same thing other not-unbiased investigators found. It is the case that people with the same bias are likely to see similar things in the tea leaves. A better study might look into that issue– but I somehow don’t think Cook is going to report this as a study suggesting that ‘both’ sides are just reading tea leaves.

    Of course we don’t know what he’ll report what he’ll report. But that doesn’t mean we can’t project — just as climate modelers project with their models.

    On those:

    There is a difference bewteen skepticism and suspicion.

    Of course. But so?

82. I find it interesting barry brings up the difference between skepticism and suspicion. I only became interested in this survey because what I saw didn’t make sense. And even then, I went to lengths to look for an innocent explanation. I didn’t say a word suggesting anything nefarious until John Cook e-mailed me basically saying, “The things I said about this survey were lies.”

    Maybe the WUWT crowd is different, but around here, this has been about skepticism not suspicion. barry shoud be praising us.

83. There is a difference bewteen skepticism and suspicion.

    Of course. But so?

    To my eye the distiction is blurred within the minds of many commenters at the blogs. Not with you on this thread, though, Lucia.

    the fact that you say something like “I find the term ‘fake skeptic’ is all too often appropriate, and this applies to both ‘sides’ of the debate.” suggests you are not sufficiently unbiased to judge this matter.

    I don’t see how. That is a non-partisan opinion, sincerely held. What is your reasoning here?

    A rigorous skpetic (who may hold any provisional opinion on people, politics or phenomena and act on them) is ultimately neutral in their assessments, foregoing argument of personality and absent personal predilection. You think that is a broad standard amongst the commenters at climate blogs?

84. Consider yourself praised, Brandon.

    But let me do that provisionally. Would you quote the full email where John Cook says what you claim?

85. barry, I already did.

86. Link?

87. Barry
    John Cook is the kind of guy that poached his fellow bloggers’ open comments to write a psychology paper.

    What exactly is it that makes you think his actions should be taken on good faith? Especially when they not what he says they are. What makes you think that being trustful of a person who has a prior record of deception, fabrication, and predatory behaviour, is a good idea? I am curious.

    You are aware, that the owner of this blog was named in the psychology paper as the originator of a conspiratorial idea? And the authors botched it up and credited someone else for the same?

88. barry

    That is a non-partisan opinion, sincerely held. What is your reasoning here?

    First: with respect to the issue or the ‘parties’ involved in this poll, it is partisan opinion. Second, a pre-existing negative opinion about a subset of participants invited to take the survey. There is circumstantial evidence that the survey is intended to discover/uncover/reveal something about one of the parties “i.e. fake skeptics” vs. the other “people John Cook likes fosters and tries to nourish and shelter at his blog”.

    So, for this reason, he is not a suitable person to be interpreting the data. Also, the fact that you think “fake skeptic” is a good lable suggest you may not be able to understand why John Cook is not the right person to be running a survey like this.

    A rigorous skpetic (who may hold any provisional opinion on people, politics or phenomena and act on them) is ultimately neutral in their assessments, foregoing argument of personality and absent personal predilection. You think that is a broad standard amongst the commenters at climate blogs?

    All blogs? Nope. But I think it’s odd those who embrace the term “fake skeptic” seem to suggest there is a camp of “real” skeptics out there. I certainly don’t think “real” skeptics are posting at Skeptical Science! That’s just hilarious. With respect to “Skepticism” that group is further from perfection that WUWT. Seriously!

89. Barry , you dont seem to be aware of the history. You say the regulars have criticised the survey and it wont have much impact. Well the previous fake survey was also criticised by the regulars but still went on to be published and have a big media impact.

90. barry-

    Link?

    Sheesh. You’ve been telling us all what you think we are all saying based on your having supposedly read the posts where we discuss the survey. Yet evidently, you haven’t been reading the posts or discussions!

    Now, before you continue telling us all what we supposedly think or have talked about, find the posts Brandon wrote and read and read them. While your at it, read the other posts discussing the survey.

91. lucia, to be fair, I only posted an except in the post itself. I only posted the full e-mail in the comments, a few days later. barry might have read the post but not followed the comments section. That might be why he asked me to post the “full email.”

    That said, I’m not sure why the excerpt in my post wouldn’t be enough. I can’t imagine what context he could hope to find.

92. Barry,

    Honest question. What is the definition of a “fake skeptic”?

    I (and others) have been called that in different contexts and it seems to mean different things at different times. Is is a skydragon or some form of person pretending to be open minded yet has already formed opinions on everything? Is it someone who posts things against the alarmist bent of the media regardless of the topic or does it apply equally both ways. For instance Skeptical Science only posts the most alarmist version possible of global warming, yet is self titled “skeptical”. Are they fake skeptics?

93. Barry
    How about this?
    .
    “I think that we (me too) need to grow a thicker skin and avoid the tendency to respond to every criticism on blogs like Lucia’s. Sadly, once you engage in meta-discussions there it appears to the detached or hostile reader that we’ve been rattled. Better, I think, to ignore comments there or to leave it to one of us (…) to respond very calmly that the citicism (sic) is misplaced (or not) and not get drawn in to those tit-for-tat discussions that Lucia revels in and amount to nothing.”
    .
    This is from the Skepticalscience backend. LOL.
    .
    So you see what the problem is? The Skepticalscience group has gotten used to hiding its dissent, disagreement, and doubts behind the scenes in their secret little forum, and emerging only to run around like wise little Al Gores (‘heh, don’t you know, the science is settled and the heat has gone into the ocean’). That’s why naturally occurring suspicion of untrustworthy entities looks like a ‘conspiracy’ to them. ‘Hey, why would these people openly voice their criticism and doubt publicly? Don’t they know that makes them look crazy? I know I wouldn’t say such things outside the safe confines of my secret hideout’
    .

94. Jeff, they do seem fake skeptics.

95. Brandon–
    Presumably if barry has been reading he knows you posted blog posts, discussion is going on there and can go there to look. The titles of your posts are still showing on the main page here at this post. Moreover, you said you were going to post John’s response before you actually did. So, someone in who had read your post would have know the full email was going to appear.

    It’s not like you posted deep in comments at an entirely different blog like WUWT– a location barry had to go to to hunt down evidence to support his claim that people were confused about how this survey interacts with the paper.

96. Shub,
    And gosh. To think now the head of that blog wants me to help him on his survey by inviting my readers.

    Of course, this is assuming that the point of his survey has anything to do with identifying the consensus based on evaluating abstracts. I tend to think that is quite likely the intended purpose. But given the fact that deception is allowed in these things, we can’t actually be sure, can we. But I suspect barry would suggest our pointing out that we are aware of the fact that the university ethics guidelines permits deceptions is somehow impugning John Cook. After all, we would be suggesting he might be doing something that the ethics committee permits!! OMG!

97. lucia, it’s remarkable how similar being fair and damning with faint praise can be.

98. Lucia (#112843) –
    Speaking of ethics guidelines, has John Cook ever gotten back to you with his application to the human-research ethics committee?

99. HaroldW,
    Nope.

100. Brandon,

     I found your post. I think your characterization is over the top. It is constructed from an elided version of what was said about the survey at SkS. Like others, you’ve exploited the vagueness of the survey purpose and parameters.

     At other climate blogs, I would have been provided a link (would have taken you a few seconds), instead of being scolded for not having already come upon it. I don’t frequent this blog, and only arrived because of the link to this thread by Lucia at WUWT.

101. It’s not like you posted deep in comments at an entirely different blog like WUWT– a location barry had to go to to hunt down evidence to support his claim that people were confused about how this survey interacts with the paper.

     Well, that’s a shame, Lucia. Your link to this thread was at the bottom of the thread at WUWT I got the quote from. I had read it all, and did not go hunting after the fact.

     Thanks for the discussion. The comments are now getting a bit too tribal for my taste. Adieu.

102. barry, if my characterization is “over the top,” why don’t you say in what way it is inaccurate? You don’t. All you say is it “is constructed from an elided version of what was said about the survey at SkS.” That is wrong in so many ways.

     First, I quoted three different descriptions of the survey, all from John Cook. Your argument that I used only one description, and misrepresented it, is false on its face.

     Second, I provided a full quotation for at least one description I referred to. This would have been impossible to miss if you read what I posted. Again, your claim is false on its face.

     Third, I discussed my characterization and the reason for it. You not only failed to contradict what I said in doing so, you ignored the fact I did it. If you were aware of what I said, you intentionally created a deceptive portrayal of my argument. If you were not aware of what I said, you created a portrayal you did not have the knowledge to justify.

     You can claim the comments are getting “too tribal” for you, but the reality is you are the one behaving unreasonably. Not only have you failed to offer any sort of justification for your criticism of what I said about John Cook, you’ve also leveled a criticism against me based on an untruth:

     At other climate blogs, I would have been provided a link (would have taken you a few seconds), instead of being scolded for not having already come upon it.

     It would not have taken me “a few seconds” to post the link. I was posting from a phone, and it would have taken me a couple extra minutes to post the link.

     That you didn’t even consider why I didn’t post a link (which was in stark contrast to my normal behavior) suggests you have little interest in reasonable dialogue. The same is true for the fact you didn’t bother to acquaint with basic information, readily available on this blog, before disagreeing with the views people had formed based upon that information. And to top all of it off, you’ve hand-wavingly dismissed a well-reasoned characterization of John Cook’s comments via a grossly untrue accusation against me.

     As far as I can see, you’ve done nothing to understand the criticisms of John Cook, even making untrue claims about them, all while insisting those criticisms are inappropriate. That is tribalism.

103. I just remembered barry earlier claimed there was conspiracy theorizing going on in regard to this survey. When called on it, he simply dropped the claim.

     His claim of tribalism seems to be like his claim things were getting argumentative: Blaming others for his own behavior.

104. ‘It would not have taken me “a few seconds” to post the link. I was posting from a phone, and it would have taken me a couple extra minutes to post the link.”

     I’m coming close to giving up posting from the phone because of all the issues it causes with being clear, providing links, remembering to quote.. etc.

Comments are closed.