I tried. I tried to be generous. I tried to find some technical issue for why John Cook’s latest survey would not produce a random sample of the 12,000+ papers in his database. I tried to find some innocent programming mistake we could all understand.
John Cook insists that isn’t the case. In response to an e-mail I sent him, he said:*
Q1 & 2: I use an SQL query to randomly select 10 abstracts. I restricted the search to only papers that have received a “self-rating” from the author of the paper (a survey we ran in 2012) and also to make the survey a little easier to stomach for the participant, I restricted the search to abstracts under 1000 characters. Some of the abstracts are mind-boggingly long (which seems to defeat the purpose of having a short summary abstract but I digress). So the SQL query used was this:
SELECT * FROM papers WHERE Self_Rating > 0 AND Abstract != ” AND LENGTH(Abstract) < 1000 ORDER BY RAND() LIMIT 10
In other words, when John Cook e-mailed people claiming to invite readers “to peruse the abstracts of” “around 12,000 papers,” it wasn’t true. When he posted on Skeptical Science saying people are “invited to rate the abstracts of” “over 12,000 papers listed in the ‘Web of Science,'” it wasn’t true. And when the survey says:
You will be shown the title and abstracts (summary) for 10 randomly selected scientific papers, obtained from a search of the ISI ‘Web Of Science’ database for papers matching the topic ‘global warming’ or ‘global climate change’ between 1991 to 2011.
It isn’t true. In fact, it seems John Cook lied every time he’s told people what the survey is. Only, he doesn’t seem to believe he’s lying. Even after telling me about how the samples are not randomly chosen from the 12,000+ papers, he says, “The survey’s claim of randomness is correct.” It’s like saying:
I have a hundred papers here. I’m going to have you read one at random.
(This randomly chosen paper will be selected only from papers with a prime number.)
But leaving off the parenthetical. John Cook apparently doesn’t believe that is lying. I think most people would disagree. But even if we don’t call it a lie, we must call it extremely deceptive.
*I don’t agree John Cook did as he claims as far as the programming goes. For example, I’ve previously shown results where some samples were sorted and others were not. His description doesn’t explain that. But I’ll leave discussion of technical issues like that to comments of the previous post.
Tom Curtis is already defending John Cook on this issue, saying:
He conveniently overlooks how the survey describes the samples:
This description is exactly in line with how the Skeptical Science post he’s posting on describes the paper submitted to ERL:
In other words, while the survey didn’t explicitly say samples would be drawn from 12,000+ papers, it clearly referred to a population we know to be those 12,000+ papers. Tom Curtis’s defense is just ridiculous semantic parsing he uses to claim people “fail at reading comprehension.”
And the Trojan Number is 12,000. We have a winner!
http://www.numberwatch.co.uk/trojan_number.htm
Trojan Numbers: good term, an infiltration and overthrow by clandestine means.
That means that those who push the TNs, as the MSM does with the “97%” consensus, would be Trojan Soldiers, those who follow up the infiltration when the gates are opened.
The TS don’t know the reasons they are at war other than what they were told, they don’t know who is going to gain or lose by the war, and they don’t know where they will be standing when the last ship pulls away from the shore, but they rush to take advantage of the breach because it is so exciting and profitable to ravage the city.
Is John Cook trained as a scientist of any sort? Because if he is, then some educational system somewhere has failed spectacularly.
If not, then he really should stop trying to do “science.”
Don’t want to say I told you so, Brandon, but I did. The papers being presented are, in all likelihood, not the 12,000. They are just as likely not the ones where scientists rated their own abstracts. They are likely ones where the skepticalscience team had difficulty/discrepancy classifying.
Shub–
I still don’t buy into your theory. I suspect it’s more likely that Cook wants to compare the ratings of the abstracts by members of the public to ratings of scientists/authors. That would explain the down selection. However, the blog announcement and survey invitation do not explain that clearly.
Tom Curtis’s claim that saying this is supposed to “mirror” a paper that has been submitted but is not yet published and whose text is not available for people to read does not mitigate the misleading nature of the text actually provided to those evaluating the survey which does mention the 12,000 papers and says the 10 are selected from that batch. But I don’t think that means that Cook is trying to crowd-source work his team would otherwise be doing. I think he is trying to get bloggers audiences to take the survey.
I haven’t seen the link appear at Real Climate. Planet 3.0 seems to have linked to SkS. He supposedly invited 50 blogs with high alexa ranks.
julio (Comment #112507)
May 6th, 2013 at 8:57 am
Is John Cook trained as a scientist of any sort? Because if he is, then some educational system somewhere has failed spectacularly.
If not, then he really should stop trying to do “science.â€
Shub (Comment #112512)
May 6th, 2013 at 9:49 am
Don’t want to say I told you so, Brandon, but I did. The papers being presented are, in all likelihood, not the 12,000. They are just as likely not the ones where scientists rated their own abstracts. They are likely ones where the skepticalscience team had difficulty/discrepancy classifying.
#############
the point is to show how readers differ in their assesments from authors. By doing this you will see whether skeptics can read or not
May I suggest an alternative SQL query for John Cook?
This will tell us how many papers are actually being considered. It is an important question. If it turns out that 12,000 papers are in scope, then all is well. If it turns out that 12 papers are in scope, then we’ve all been wasting our time.
Mr Cook – over to you.
Sincerely – please.
SELECT Count(*) as PapersInScope
FROM papers
WHERE
Self_Rating > 0
AND
Abstract != â€
AND
LENGTH(Abstract) < 1000
It looks like this individual, James Powell, already undertook the same methodology that Cook 2013 “Quantifying the consensus on anthropogenic global warming in the scientific literature” is using (Powell just extended it to 2012 so has more papers). 98% support AGW.
Warning, my computer froze up on one the pages here and I had to unplug it to shut it down.
http://www.jamespowell.org/PieChart/piechart.html
I don’t think the clause ORDER BY RAND() has the effect the author intended. If he is using Microsoft SQL Server, he should try ORDER BY NEWID().
I expect Cook to write the above details into the paper. I see no need to burden the volunteer participants with such unnecessary inconsequential information.
For the research question involved, there seems to be no need whatsoever to randomize the abstracts. One could also have presented the abstracts one after the other alphabetically. And the term “random” does not mean that every item has the same probability. I guess some here may be familiar with a random number generator that generates normally distributed numbers: randn().
Another tack we can all take is to crowd-source the identifiers of the papers that actually come up in the survey, and then count them up….
If you click to participate, and then “View source…” on the page that comes up with the paper abstracts and dropdown rating lists, you can see what looks like the identifier for each paper.
There are lines like [select name=’3764′ id=’3764′] follow by the [option] tags for the ratings.
My ten IDs were:
3764 Climatic-change – Effects Of Parameter-estimation And Choice Of Weather Patterns On The Reliability Of Projections
4151 Implications Of A New Eddy Parameterization For Ocean Models
5401 Effects Of Stabilizing Atmospheric Co(2) On Global Climate In The Next Two Centuries
5631 Modelling The Variability Of The Sea-ice Conditions In The Baltic Sea Under Different Climate Conditions
8546 Global Warming, Elevational Range Shifts, And Lowland Biotic Attrition In The Wet Tropics
9127 Genes For Greens
9539 Zeitmop Concept – A Polygeneration System For Municipal Energy Demands
11416 Fossil-fuel Constraints On Global Warming
11499 Phenology, Ontogeny And The Effects Of Climate Change On The Timing Of Species Interactions
12376 Enhanced Modern Heat Transfer To The Arctic By Warm Atlantic Water
Interesting that we now have a climate scientist saying that it’s OK for Cook to mislead participants in this way.
“But leaving off the parenthetical. John Cook apparently doesn’t believe that is lying. I think most people would disagree. But even if we don’t call it a lie, we must call it extremely deceptive.”
huh. I wouldn’t call it a lie or deceptive. It’s definately an incomplete description of how the papers were winnowed down before applying the rand() function, but then again he didnt tell you what the seed was for the rand() function.
Like Victor I would imagine the full description of the selection process to be published with the paper, but as a notice to users he tells you everything you need to know– it’s random
If he told you that he winnowed the selection to abstracts that had less than 1250 words would that make a material difference to you?
would you think it less random?
Just because you thought X when Y was the case does not mean he was deceiving you. His description was inadequate for the purposes of reconstructing what he did. I would hope he describes it adequately in the paper. His description of “randomly selected” is adequate for a subject of the test.
To understand a lie or a deception you need to understand the purpose of the statement. His description of “randomly selected” is adequate to the purpose of informing a subject. Its not a lie nor is it a deception, but you clearly came away from the sentence with a mistaken idea. That’s because you look at the sentence as an instruction to replicate. its not. Hopefully he would put that in the paper. In short, simply because you think X when Y is the case does not mean you were deceived.
Paulm–
It might not be up to climate scientists.
There are the guidelines for ethics.
http://www.uq.edu.au/research/rid/human-guidelines
Within that it states:
At the survey we read
I think it is a stretch to claim that this describing both clear and precise about the procedure. Not withstanding the reference to the “mirroring” in the paragraph preceding, the 10 papers are not randomly selected from a database created from search of the ISI ‘Web of Science’ database matching the topic ”global warming’ or ‘global climate change’ .
I don’t know if the ethics committee would give a hoot about this. Maybe they do. Maybe they don’t.
But if procedure a participant will be involved in can only be fully understood by reading an as yet unpublished and not available paper, then it’s obvious that procedure was not clear and precise.
RE: julio (Comment #112507)
May 6th, 2013 at 8:57 am
Is John Cook trained as a scientist of any sort? Because if he is, then some educational system somewhere has failed spectacularly.
If not, then he really should stop trying to do “science.â€
################################
Haha thank you julio, I agree. Making stuff up to scare people about climate change is not doing science.
Mosher
You may be correct that’s what Cook wants to learn. But you can’t learn that from this study. To learn that you need the assessments to differ only by characteristics of the population taking the survey. But instead:
1) the one group is reviewing abstracts and the other is self assessing the full paper. This by itself means that differences in meaning can spring from the fact that the contents of the abstracts very often do not provide information relevant to answering the questions Cook post even when the papers do contain that information.
2) The group reviewing abstracts is answering this particular set of odd question immediately after reading the abstract. The group reviewing full papers is invited to diagnose without necessarily rereading their papers to refresh their memories of the contents. (I put it to you that a person trying to give precisely correct answers about the stance vs. climate change of their paper discussing responses of butterflies to winter written 20 years ago (e.g. 1992 at the time Pielke wrote his blog post) might need to re-read that specific paper with these particular questions in mind.We can’t begin to guess how the authors approached answering these questions. Some might merely answer based on how they think about climate change today.
3) It appears the authors were presented with a pre-question prior to the set presented to those reviewing the abstracts. Those reviewing abstracts are not presented these pre-questions. Pre-questions do affect how people answer follow on questions.
Of course I can’t be sure of all these things. But this is the situation that the survey seems to fall in based on information we do have available at Pielke Sr.s blog.
By the way: in cases where deception are involved in obtaining consent, the subjects can request to have their data removed from the study:
the subject will be permitted to withdraw along with her or his data when the deception is revealed
If the mismatch between the description and the true procedure is considered deception a participant could request their data be removed. Unless the person requesting their data be removed entered their email or knows their IP on the date they entered data, have no idea how John Cook would be able to go about identifying the data to remove.
I think we have two different conversations going on here: first we have a discussion about what the participants are required to know to make the survey/study meaningful and the second is essentially an analysis and anticipation of how the survey/study will be used.
I do think that the choice of survey replies as presented are far too vague to have much meaning for an analysis in whatever manner that is done. In its most simplistic interpretation the survey is simply an attempt to confirm a consensus that GHGs lead to some warming which in and of itself is not new nor is it particularly useful information.
I suppose one could question why the survey is as vague as it is and could it lead the survey takers to misinterpretations and thus not providing reliable information to those who will analyze it.
I have not followed this thread closely but do we indeed know how many abstracts are available for a random draw now that the words per abstract have been limited. Can another simulation of random draws be made and compared to what we observe being drawn based on a new total number of abstracts available or are the observed compilation of survey draws somehow limited by your IP address? Can you work backwards based replicate draws to determine how many abstrcts are available?
I await another defender to appear and wonder how the posters here can spend so much time analyzing this survey and be so suspicious of the survey taker(s). I personally do it because it is fun and I find it important to put these works into perspective. So far there are lots of questions and very few answers.
Steven Mosher
No I didn’t. I took it as a sentence that provides “a clear and precise description of procedures for their involvement ”
That is not a clear and precise description of my involvement. I thought I was evaluating 10 papers selected radomly from the database of 12,000. But the screening process to first downselect is not “random”.
Selecting only the short ones is certainly not random. That is deterministic no matter how you slice it.
Your arguing about this may tempt me to write the ethics panel, explain what I thought and request that Cook remove my data!!
Cook gives two reasons for making people the survey. One, “to replicate the experience of rating a random assortment of climate papers, gaining appreciation of the diversity of climate research available”
Can anyone decipher what this unscientific, grammatically incorrect gobbledygook means? Cook is trying to replicate *for the taker of the survey* the experience of looking at climate abstracts, in order to make them appreciate them about the ‘diversity’ of climate research? Why would I want to know how a researcher felt like looking at lots of climate-related abstracts? Why would I care to gain appreciation of the diversity?
If the purpose of the survey was to “compare the ratings of the abstracts by members of the public to ratings of scientists/authors”, he would have said so. Cook doesn’t say so because the purpose is not to obtain such comparisons. The purpose likely is to use portions of audience-derived abstract ratings to increase numbers of papers ‘supportive of the consensus’.
Shub,
Yes, that is how I read that clause. He’s stating that by looking at various abstracts you get to experience the variety of research. It’s not a survey, it is performance art for the participant. You get an EXPERIENCE. Ooh, aah.
Shub (Comment #112535)
May 6th, 2013 at 12:09 pm
“If the purpose of the survey was to “compare the ratings of the abstracts by members of the public to ratings of scientists/authorsâ€, he would have said so. Cook doesn’t say so because the purpose is not to obtain such comparisons.”
I think one has to judge Cook’s involvement in this survey by what he is not saying and particularly since many questions have been presented. He could be clear as a bell on this matter but he obviously is not.
I get a particular kick out of those who would choose to defend Cook’s failure to reveal upfront that the abstracts were word limited and there were not 12,000 abstracts for random selection. I do not believe any of the defenders ask the question of Cook: If not 12,000 then how many? It was important evidently to reveal the number 12,000 so why not the actual number?
Lucia: “I don’t know if the ethics committee would give a hoot about this. Maybe they do. Maybe they don’t.”
I am sure the committee does not care. The participants have to do what they expected: rate 10 scientific abstracts and they do not have to stab 10 Guinea pigs to death. In you interpretation of the ethics rules, psychological research would be nearly impossible. In those experiments, you can tell the participants what they will have to do, but you cannot give them the details of what is actually measured as that would influence the results in many cases.
Lucia: “Selecting only the short ones is certainly not random. That is deterministic no matter how you slice it.”
That is stochastic no matter how you slice it. A conditional probability is also a probability.
With the added problem that Cook was “lying” about the random numbers, they are pseudo-random numbers generated by a deterministic algorithm. 🙂
I understand that you guys do not like Cook, he clearly has an agenda that opposes yours, but here you are being extreme unreasonable. That is a pity, from my last positive interaction, I came away with the impression that this place is more interested in science as in rabble.
lucia (Comment #112530)
May 6th, 2013 at 11:51 am
Mosher
the point is to show how readers differ in their assesments from authors. By doing this you will see whether skeptics can read or not
You may be correct that’s what Cook wants to learn. But you can’t learn that from this study. To learn that you need the assessments to differ only by characteristics of the population taking the survey. But instead:
##################
well you can purport to learn that and you may be able to learn that from the study, but proving that is a different thing. For example, he could also show just the abstracts to authors, that is, authors read a selection of abstracts in addition to their own papers. Spencer or Christy participated perhaps wew can ask them
So, you have
Authors view of his paper: 3
Other authors view of his abstract 3.
WUWT reader of abstract 4.
And in this case you have the data you need to draw a conclusion.
‘If the purpose of the survey was to “compare the ratings of the abstracts by members of the public to ratings of scientists/authorsâ€, he would have said so. Cook doesn’t say so because the purpose is not to obtain such comparisons. The purpose likely is to use portions of audience-derived abstract ratings to increase numbers of papers ‘supportive of the consensus’.”
I’m not sure, I believe in the past I’ve taken tests where I was told the purpose was one thing while the real purpose was something else, so as to keep me , as subject, from gaming my answers. but that was a while ago when it was ethical to shoot me up with radioactive substances. hey I made 75 bucks and needed the food money.
haha. I get to play Nick Stokes
‘That is not a clear and precise description of my involvement. I thought I was evaluating 10 papers selected radomly from the database of 12,000. But the screening process to first downselect is not “randomâ€.
the screening process.. at some point in the past 12000 artcles were selected. Then assume that authors are randomly selected to assess their own papers.. they respond and agree or not randomly.
When its all done, Cook pull a number our of his butt: 1000. not 1001, not 435. Then he selects randomly from these.
I would not call that deterministic.
also, dont most abstracts have a word limit.
@ Victor.
Its not that we dislike Cook. Its that we like to argue and Lucia is a tiger.
Steven,
So maybe we can resolve our little argument by my contacting humanethics@research.uq.edu.au and sending them this:
To whom it may concern,
I participated in John Cook’s survey available at links similar to these:
http://survey.gci.uq.edu.au/survey.php?c=II7WP4R4VRU7
http://survey.gci.uq.edu.au/survey.php?c=NRYZLL5GEGZU
http://survey.gci.uq.edu.au/survey.php?c=QQ5LENROSHQM
http://survey.gci.uq.edu.au/survey.php?c=1OBJM2FVTG68
http://survey.gci.uq.edu.au/survey.php?c=GGB5IS4BFOO0
http://survey.gci.uq.edu.au/survey.php?c=T483TT7NXPJA
http://survey.gci.uq.edu.au/survey.php?c=1R9YT8YMZTWF
http://survey.gci.uq.edu.au/survey.php?c=A5YD100J37CL
http://survey.gci.uq.edu.au/survey.php?c=YG6T1272VU5M
http://survey.gci.uq.edu.au/survey.php?c=Y368342UF93D
Because I am concerned about anonymity of online surveys, I collected together a number of query strings and do not know which particular query string I used to enter my survey.
Prior to participating clicking the button granting my consent to participate, I read this description:
Based on that text, I believed that I would be presented with
“10 randomly selected scientific papers, obtained from a search of the ISI ‘Web Of Science’ database for papers matching the topic ‘global warming’ or ‘global climate change’ between 1991 to 2011”
which I could then evaluate. Moreover, the wording was such that I thought that each of the 12,000 or so papers corresponding to such a search was equally likely to be selected, thereby giving me the opportunity to gain an appreciation of the diversity of climate research available without any other filters that might have been imposed by the research investigator.
It has come to my attention the 10 papers I was presented were not randomly selected from a database consisting of roughly 12,000 papers obtained by searching the ISI “Web Of Science’. Rather, the papers were selected from a smaller subset of papers that underwent to forms of deterministic screening. One screen was that only papers previously evaluated by the authors who wrote the papers were included. The other screen to exclude the more informative abstracts with 1,000 characters or more. Because the screening could be done by the authors prior to initiation of the survey, I consider both forms of screening to be deterministic, not random. The screening by number of characters in the abstract is certainly a purely deterministic.
While I understand the researchers may not consider these two screening detrimental to their true underlying research objective, I believe that the description provided participants was neither precise nor accurate.
I should note that I also don’t consider that these two undisclosed random proceedures necessary or beneficial to the purposes suggested to participants. These purposes are either:
a) ‘estimating the level of consensus regarding the proposition that humans are causing global warming’
b) ‘replicate the experience of rating a random assortment of climate papers, gaining appreciation of the diversity of climate research available’ pr
c) ‘getting feed back from ‘ a wide range of climate blogs in order to analyse ratings from a diverse range of participants’.
The first appears to be the actual purpose of the research project. The final two are stated as the purposes of the survey.
All three could better achieved by presenting 10 papers from the full selection of 12,000 which based on the researchers description of the proceedure I understood to be the procedure for this paper.
As I see it:
My agreement was given when I was under jthe impression (a) the purpose of the study was to estimate the consensus on AGW based on reviewing 10 papers chosen randomly from a set of 12,000 papers ‘obtained from a search of the ISI ‘Web Of Science’ database for papers matching the topic ‘global warming’ or ‘global climate change’ between 1991 to 2011′ and (b) the purpose was to identify the consensus based on that full set of papers rather than a screened subset.
I now believe this was not the purpose of the study and believe I was decieved. I light of this, I request that John Cook delete my entry from his dataset.
Thank you,
Lucia Liljegren
=======
Note sure I care enough to cause this much trouble…… (Knowing you, I imagine you may be trying to taunt me into it.)
Yes. Most abstracts have a word limit. Why Cook felt the need to impose another one, I do not know. It might have been interesting to see the long ones. (Heck, I think I have the whole database. I could look.)
Victor
Sending the letter I put in comments to the ethics committee might be unreasonable. Saying I think Cook did not properly describe the procedure? I think he didn’t do so. I don’t think my expressing my opinion in blog comments is “unreasonable”.
Victor
BTW: I tend to agree with you. One of the reasons I’m not going to zing off a letter like the one I wrote is that I don’t think the committee would care. But that doesn’t mean the procedure is describe clearly and precisely. It means that even though the rule claims that’s required the committee often won’t care if the connection to precision and clarity is … uhmm.. not present.
If I’d clicked that button on a web page and found myself with a knife being asked to stab Guinea pigs, I would have run shrieking out of the room and sworn off the web for good!
looks good lucia, I think their answer would be instructive and would alter my position accordingly.
one nit:
“I now believe this was not the purpose of the study and believe I was decieved. I light of this, I request that John Cook delete my entry from his dataset.
Thank you,”
how about
“I now have reason to doubt this was the purpose of the study and that I was misinformed.”
‘If I’d clicked that button on a web page and found myself with a knife being asked to stab Guinea pigs, I would have run shrieking out of the room and sworn off the web for good!”
in south america guinea pigs are food.
Steve Haffner:
If seeded properly, ORDER BY RAND() will work pretty well at providing randomness. The biggest problem with it is just efficiency. If Cook had used the command without restricting it to just a few hundred entries, he’d have created a massive drain on server resources.
Victor Venema:
Thanks for telling us you have such low standards. I hope you’ll hold to them in the future when assessing the work of people you don’t like.
Random means lacking a pattern (we can discern). Randomly distributed data is not random. It is data fitting a particular pattern with randomness added to that pattern.
Perhaps Cook’s sample selection is random within the constraints of a specific pattern. That doesn’t make it random. That makes it have a random component.
Lucia, you really send that letter? 🙂 Do show the answer you will receive. I am curious if they find a polite formulation.
We really life in two different worlds. Let’s agree to disagree. This must be the most unimportant discussion of the year.
Anyway, as Steven Mosher, I expect that the main reason for the public survey was to compare the answers from different blogs. By mixing the links for various blogs and by linking to SkS you already effectively destroyed the data quality.
Lucia,
The intro page to the survey says the purpose is to give you an experience, so it’s actually an art project. 🙂
“Firstly, to replicate the experience of rating a random assortment of climate papers, gaining appreciation of the diversity of climate research available.”
Dermot O’Logical
I’ve actually compiled a list of paper IDs I’ve seen. I didn’t catch all of them because I didn’t realize I’d have a reason to, but I’m up to 231 papers. It should be possible to estimate how many papers there are by looking at how many duplicates there are within those 231, but I don’t know the math well enough to figure it out.
Steven Mosher:
Thanks for acknowledging your unreasonableness 😛
Victor Venema:
If that was the main purpose of the survey, John Cook ruined it by screening the papers as he did. If the main purpose was to compare answers based on originating blog, there was no reason to limit papers based upon authors having rated the papers.
Cool. This should mean can we can count on your support in the future if we criticize whatever results John Cook’s publishes if he can’t find a way to “fix” this data quality problem.
‘ If the main purpose was to compare answers based on originating blog, there was no reason to limit papers based upon authors having rated the papers.”
I dont know what cook wanted.
I will say what I would do.
I would take all 12K abstracts and send them to authors of science to score 1-7.
I would ask the some authors of these papers to score their own papers by the same scale.
I would ask skeptics to score the abstract and non skeptics to score the abstracts.
That would give me data to do an interesting study about the framing that people bring to their reading.
Victor Venema
Nooooo!!!!!! I told Mosher I was tempted to send it. Sending it would bring an objective third party into the academic exercise of arguing about whether Cook was or was not sufficiently clear.
I assume if I sent it I would receive a letter saying they are looking into it. I don’t quite know what would happen afterwards.
Victor:
(a) Did you check out the link at Comment #112491 about the “Trojan number”? It makes an important point.
(b) Why would you want to choose randomly from a large sample? To minimize systematic effects. Suppose that there is one particular paper that keeps popping up and that happens to be especially contentious. Just like that, your whole survey is unreliable.
We know the algorithm Cook uses does lead to frequent repetition of certain papers. We do not know the real size of his sample. You do not think these are valid grounds for criticism?
StevenMosher
I would also want to do this. I don’t think I would limit the papers to those scored by their own authors. I think that after a paper was selected and given to a user from one particular query string, I would put that in memory and try to be sure to pass that out to all other query strings.
That said: If the ethics guidelines say you must describe your purpose, then I would mention was that part of the purpose was to get multiple evaluations of the similar papers and state that we have down selected from the full set of 12,000.
I suspect Cook does want to try to discover if there is a difference in answers from c=querystringforWUWT vs c=querystringforSKS and so on. That said, he may merely wish to know how many people can from WUWT vs SKS. Knowing people can strip or spoof referrers he came up with the query strings. There may be some other reason for the query strings. I don’t know. But… well… given how his survey emphasizes how he is preserving our privacy, I think people who take his survey are justified in taking any steps they might to do similarly.
I’ve never used Tor… but heck. Someday, there might be some other Lewandowsky or Cook things I want to do, and I’ll have to fire that up!
Steven Mosher, that is what I would do exactly. And it is thus not a long shot to assume that this is what Cook had wanted to do.
Brandon Shollenberger, nice to see you happy. Maybe Cook was smart enough to use the referrer information from the web-browsers to determine where people came from. Otherwise, he can only use those links that were not mixed up, which would seriously reduce the amount of data he can analyse.
“whether Cook was or was not sufficiently clear.”
hmm I wonder if he needs to be clearer to you ( as a host) than he has to be to subjects.
i suppose by hosting his survey you have the right to demand certain answers before you give him access to your users and I might even argue that it would make you a co author of sorts.. some co authors I know have done less..
so for subjects one level of detail may be enough, you are taking the test of your own free will, for blogs that host it, i would expect more clarity, for fellow researchers to replicate or assess, full disclosure.
After all, when Gallup calls you they do not explain their entire selection process..
Victor –
“…you already effectively destroyed the data quality.â€
It’s an easily gamed on-line survey shopped around to blogs dealing with a highly contentious set of issues, in a thoroughly poisoned atmosphere in part exacerbated by the author of the survey. What data quality could it possibly have delivered?
Yes Victor,
I cannot count the number of times people have shoved an abstract under my nose and said “See! proves it” and I have to wonder if we are reading the same document.. and then you read the paper and woaaa..
lots of disconnects in our reading, framing, expectations, exegesis.
As much as I appreciate some peoples’ literalist approach, it is only an approach. very often when you take the literalist approach you end up thinking the other party is speaking nonsense or lying or mis representing.
BTW, looking forward to you next piece of work.
Victor
I’m sure Cook is smart enough to use the referrer information. In fact, I think he’s smarter than that. I think he’s smart enough to know it’s unreliable. I have a browser add on that let’s me spoof referrers. It lets me:
1) Send normal referrers. I do that most of the time for most things.
2) ‘Standard forge’ by always sending the top of the domain name. So, when at ‘realclimate.org/whatever_blog_post’, I could send ‘realclimate.org’ even though I came from google, WUWT or whatever.
3) Custom forge. I used this when fiddling with things I have coded to reduce the amount of image scraping. My method relies on recognizing that most of the scrapers forge referrers, but I want to be able to set up different conditions depending on what’s past. I also spoof googlebot to see how my filters deal with that.
4) Leave no referrer at all.
Lots and lots and lots of people know this. Lots of people have their browsers set to “standard forge”.
Regarding :
julio (Comment #112507)
No julio, he’s NOT a scientist.
John Cook’s own words from his own website-
“About Skeptical Science”
“This site was created by John Cook. I’m not a climatologist or a scientist but a self employed cartoonist and web programmer by trade. I did a Physics degree at the University of Queensland and while I achieved First Class Honours and could’ve continued onto a PhD, I instead quit academia and became a professional scrawler. Too much doodling in lectures, I think. Nevertheless, I’ve pursued a keen interest in science and if anything, found my curiosity about how the world works increased once I wasn’t forced to study for impending exams.”
Which is why I can’t fathom anyone with any degree of intelligence wasting any time on this man’s “survey” or referring to it as “science” of any kind.
No. But then they don’t have to follow the ethics requirements of University of Wherever (Queensland?)
kch
“It’s an easily gamed on-line survey shopped around to blogs dealing with a highly contentious set of issues, in a thoroughly poisoned atmosphere in part exacerbated by the author of the survey. What data quality could it possibly have delivered?”
the resistence to being surveyed is interesting in an of itself.
most of us realize that the person asking the questions is perceived to have power. Small children discover this and when they do “because I said so” is the proper response
Victor Venema:
Not only is there the possibility of people having referrers that don’t match the link they used, there is the possibility of people not having referrers at all. lucia gave this advice:
I often follow this approach in normal browsing, and it (usually) has nothing to do with privacy. Clicking on links without checking where they go first is a good way to wind up with trouble. Copying the link’s destination then pasting it in the URL bar ensures I pay attention to where it goes.
And there are other reasons empty referrers might happens. I’ve even seen browser configurations that automatically strip referrers. That’s not common, but the point is this is yet another filter he’d have to use on the data.
Of course, there’s a more difficult problem. People who hear about this survey might be curious and go to the Skeptical Science article to read about it. Once they’re done reading, they’d likely click on the link there rather than go back to whatever page they came from to find the link. Other people might get sent a link to a page hosting the survey on a site they normally don’t visit.
There’s no way to know a person following a link to this survey from a site is a normal member of thet audience for that site.
julio, randomness does not matter for the question whether a specific paper is judged differently by the authors as by readers of various blogs. Also in the uniform random case, some abstracts will be judged more often than others. That is something you have to take into account in the statistical analysis.
In fact, it would be a good idea to systematically send the same abstract to multiple people. The abstracts that were only judged by one person are not interesting.
“I have a browser add on that let’s me spoof referrers.”
That is a nice idea. Should install one as well.
Still that should not be a problem for the study. The spoofed referrer will not be the URL of one of the other blog posts. And the referrers that do not fit can simply be ignored in the analysis.
The people coming from SkS should be ignored as well. If only because WUWT send his people there.
I suspect Cook avoided mention of final size of the database because the filtering reduced the abstract count to such an extent it wouldn’t be very impressive.
I just took a look in google scholar, doing separate searches for “global warming” and “climate change”. What I see is thousands of papers with those terms in the title of the paper. I’ve only seen a couple different versions of the survey, but I don’t remember seeing either of those search terms mentioned in the abstract titles. Were all those abstracts also filtered out?
Victor Venema:
I’m not sure why you’d say this. This survey isn’t about “a specific paper.” It’s about a collection of papers. And how those papers are chosen do matter.
For example, let’s suppose skeptics have a shorter attention span than non-skeptics. This means they’re less likely to read through an entire abstract if that abstract is long. When presented with such an abstract, they’re more likely to pick an answer that doesn’t affect their real view. What would happen if we limit ourselves to shorter abstracts? Skeptics would give more accurate results than if we didn’t limit ourselves. That’s a systematic bias.
Alternatively, let’s suppose serious scientists are less likely to participate in a survey of self-rating papers than less serious scientists because they find such not very serious. The result might be that lower quality work which is less likely to use critical thinking and more likely to pay lip service to global warming would be unduly represented in the resulting population. Skeptics then rate the papers differently because they have higher standards.
There are dozens of potential biases John Cook’s deterministic filtering could introduce.
Victor–
Do be aware that some (not all) comment filters don’t like spoofing referrers. The reason is that comment bots frequently spoof referrers.
Bob Koss:
That’s my guess as well. 12,000 sounds so much more impressive than “a few hundred.”
Victor Venema:
I think you missed the part where lucia said, “Custom forge.”
If the SQL has “order by rand()” and also “limit 10”, then Cook might think the statement will sort the whole 12,000 randomly and then pick the first 10, which will give a different result each time. I think it will pick the first 10 and then sort randomly, which gives the same 10 each time but in different orders.
Aetheressa,
Your comment above re degrees and following careers, are you not tripping the trope fantastic and using the arguments of climate scientists against those non climate scientists who challenge them?
Not sticking up for the bloke, but mirror, mirror.
Steven Mosher (Comment #112570)
“the resistence to being surveyed is interesting in an of itself.”
Can’t disagree with you here, but I can’t imagine how anyone could go from ‘interesting’ to ‘meaningful’ with this mess. the water’s too thoroughly muddied…
“most of us realize that the person asking the questions is perceived to have power. Small children discover this and when they do “because I said so†is the proper response”
I’m not sure what you’re getting at here. How does Cook asking questions indicate that he has power, and are you further implying that the poor sceptics are like children? Or are you saying that this is how Cook sees this?
Bob Koss (Comment #112574)
May 6th, 2013 at 3:20 pm
I suspect Cook avoided mention of final size of the database because the filtering reduced the abstract count to such an extent it wouldn’t be very impressive.
$$$$$$$$
Many skeptics have seen this study as a study on the existence of a consensus.
I took the survey. After seeing the neutrality of the abstracts
and after seeing that my score was constrasted with the authors score, i could form a reasonable hypothesis about the intent of the study. It is not about the existence of consensus.
It is about how you process the abstract versus how other people process the abstract.
Question; How can so many skeptics not get the consensus?
Answer 1. They dont know the science
Answer 2. When they read the science they dont get it.
http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
You see some of that as people with no expertise try to explain what Cooks study is about and how such studies should be conducted.
“I’m not sure what you’re getting at here. How does Cook asking questions indicate that he has power, and are you further implying that the poor sceptics are like children? Or are you saying that this is how Cook sees this?”
what I am getting at is this. folks who dont like cook dont like answering his questions because at some fundamental level we recognize that the person asking the question has assumed some power or is ceded some power. And Im noting that even small children recognize this.
ever notice how annoying it is when somebody doesnt answer your questions or tell you you have no right to ask them..
ANSWER MY QUESTION dammit.
Carl:
Nope. The Limit part is the last thing applied.
In fact, I don’t think it will do either thing you listed. He’s limiting his query to a few hundred papers so those few hundred entries are the only ones that’ll get assigned a random value/get sorted. Or so I think. I’d have to check to be sure. I haven’t used his approach in ages because of its inefficiency.
I tweaked the range of my random number generator numerous times in an effort to approximate the 26 duplicates Brandon found out of 160 abstracts. My current best estimate is Cook is actually only using between 375-475 abstracts out of the original 12000.
Re: Comment #112575
Victor, basically, Brandon is right. I can’t believe we have to explain this to a fellow scientist. Basically, you introduce randomness when you don’t think you can control for all the potential sources of bias in your experiment, and so hope that by sheer sample size they will cancel each other out (“law” of large numbers).
*If* you knew everything that could potentially bias the response of your test subjects *and* could control perfectly for all those variables, then yes, you could get accurate results by showing everybody the same subset of 10 papers–a very carefully chosen one! But, basically, if you already knew so much you wouldn’t even need to run the experiment at all.
My point remains: Cook clearly has no idea how science is done (not surprising if he has learned from Lewandowsky!). His response further confirms that. Any real scientist who had the problem pointed out to him would try to take care of it–even if it is unlikely to affect the final result very much, just for the sake of eliminating as many sources of potential error as possible. And he would also disclose the real size of his sample, incidentally.
Victor Venema (Comment #112541)
Victor, you show your prejudices by your failures to question what Cook is attempting to do here. I suspect you might be taking more seriously Cook’s “study” than others here (rightly so) are.
If, as has been suggested, Cook is attempting to show that there are (more) dumb a__es (DA) on one side of the AGW debate that in and of itself would be as useless a pursuit of knowledge as what any politician or partisan supporter would do in a similar situation in order to garner support for their causes. You attempt to show that the opposition has DAs and thus imply that their view of things is misinformed without making any attempt to show how many DAs are on your side.
From a strictly scientific standpoint on AGW these opinions of misinformed laypersons means absolutely nothing. From a political standpoint it means nothing other than to show the limitations of a democracy with an uninformed public and that if a study is truly honest and non partisan that an uniformed public is never just on one side of an issue.
‘I’m not sure why you’d say this. This survey isn’t about “a specific paper.†It’s about a collection of papers. And how those papers are chosen do matter.”
assumes facts not in evidence. The survey is not about a collection of papers, the survey uses a collection of papers.
You are presented with a subset. the order of that subset is randomized.
Victor and I both believe the survey is about the difference between how authors rate abstracts and papers and how readers rate them. Not about the papers. Not about the science. It’s about your reading comprehension and your biases.
Of course, this is my opinion, not a fact, but it makes sense of more facts than alternative opinions, and I dont have to accuse Cook of lying to make my points. Always a bonus in my book. Even though, I dont much care for his view of certain things.
Kenneth
“If, as has been suggested, Cook is attempting to show that there are (more) dumb a__es (DA) on one side of the AGW debate that in and of itself would be as useless a pursuit of knowledge as what any politician or partisan supporter would do in a similar situation in order to garner support for their causes. You attempt to show that the opposition has DAs and thus imply that their view of things is misinformed without making any attempt to show how many DAs are on your side.”
It could backfire. Imagine if SKS readers saw MORE EVIDENCE for consensus than scientists did. That would say that the meme of ‘consensus’ has been oversold.
one can well imagine a situation in which the scientist come out in the middle.. conservative skeptical, whereas blog readers both sides were biased.. one side seeing more support for CAGW and the other side reading falsification whereever they could.
Hell, look at the damn temperatures for 1900 to current.
A skeptic “sees” the peak in 1930s, sees the platuea of the last 15 years, while CAGWers ‘see’ the steady rise.
Same pixels, different message, rorshack test.
How can we see the same thing and come to different conclusions
what do you see
http://photos1.blogger.com/blogger/5639/2020/1600/chica_o_vieja.jpg
and once you have becomed fixated on one interpretation it is very hard to see something different.
lucia (Comment #112569)
May 6th, 2013 at 2:58 pm
After all, when Gallup calls you they do not explain their entire selection process..
No. But then they don’t have to follow the ethics requirements of University of Wherever (Queensland?)
##########
bam. got me.
Steven Mosher (Comment #112584)
Ok, now I see what you’re getting at, but I’d have to disagree – in this case, I think the disinclination to answer (or outright screwing with the responses) aren’t a reaction to any power issues, but rather to trust/respect issues.
Steven Mosher (Comment #112589)
“…the survey is about the difference between how authors rate abstracts and papers and how readers rate them.”
FWIW, I’d agree. I’d also say that this would have been a worthwhile endeavour if conducted differently and by someone else. Cook’s leadership virtually guarantees failure…
I suspect Cook is trying to manufacture a story along the lines of “deniers are so prejudiced that they see AGW-ghosts in this huge 12,000!!! item database of elegant peer-reviewed gold-standard science papers when the authors themselves disagree.”
Steven Mosher:
Uses stupid and uncharitable semantic parsing. I said the survey is “about a collection of papers” to distinguish it from something dealing with a single paper. I wasn’t trying to say what the aims of the survey was – just a population used in it.
Good job with the Stokes routine!
Brandon Shollenberger: “I think you missed the part where lucia said, “Custom forge.—
No, I did not. Not many will go to the trouble of searching for a pro AGW blog with a link. And if, you can just click on it.
.
Brandon Shollenberger: “For example, let’s suppose skeptics have a shorter attention span than non-skeptics.”
That is why Cook should mention in his article, that he selected the short abstracts. Even if your theory is very unlikely, it is hard to know in advance which details are important for future researchers. Thus it is best to err on the side of too much information in a scientific article.
.
Brandon Shollenberger and julio: It sounds like you are assuming that the scores of all WUWT-fans will be averaged and of all SkS-fans will be averaged. Then perfect randomization is important. What I was trying to explain with the term “specific abstract” is that you will compute the difference in score for a specific abstract and then average over over all these differences. If the number of participants you had to compute the difference varies, you will also have to take the uncertainty of the difference into account when averaging over the differences.
.
Kenneth Fritsch, that could go both ways. Let’s see how biased, for example, the SkS readers are. For your research question it would be better to use an IQ test. 😉
You are right, the results of this survey tell us nothing about the truth of climate change.
I cannot judge whether a social scientist would find it interesting to see whether different groups judge an scientific abstract differently. That is where the scientific interest should be. Thus the scientific interest is likely a question such as Steven Mosher posed: “Question; How can so many sceptics not get the consensus? Answer 1. They dont know the science. Answer 2. When they read the science they dont get it.” I have trouble understanding why there are so many climate “sceptics”. Thus in principle such questions are interesting.
The real interest of Cook is naturally not such a scientific question, but to bash on the “sceptics”. As he himself is an active participant of that debate, I would have preferred that someone more impartial would write these kind of papers.
From its construction, specifically the multiple referral codes, the survey seems to be trying to draw conclusions about the surveyed population, not about the papers. [As Mosher hypothesizes above, and Lucia suspects in this comment.] This would also explain the reluctance/inability to reveal the ethics application, which would in that case indicate that the experimental subjects will be misled as to the study’s purpose.
It might not be exactly as Mosher suspects [he mentions Dunning-Kruger], but a more basic test of how one’s alignment affects one’s views. I’m sure Cook would be delighted to be able to “prove” that skeptical/lukewarmer types exhibit more interpretive bias, just as Lew was delighted to “prove” that they harbor more crazy conspiracy theories.
I don’t think it actually matters whether the study is constructed with proper randomization, or to exclude gaming. Nor whether the exercise of reading an abstract provides any useful indication of bias. It doesn’t take a crystal ball to guess that Cook will come up with some interpretation of the data to denigrate skeptics, just as Lew did. This isn’t a “nefarious intent” fantasy; it is the logical result of a partisan making an assessment — the conclusion will be partisan, as we have seen time and again in political polling.
[Edit: I noticed Victor’s comment above after posting. Especially the last paragraph. Is there a consensus here?]
Victor Venema:
This isn’t in line with what you said before:
You didn’t say, “It won’t usually be the URL.” You said it “will not be the URL.” That is, it will never be. That’s why I said what I said.
Besides, we know people have reasons to cheat on this survey. We don’t know how much trouble they’re willing to go through to cheat. It’s perfectly possible people will spoof referrers like you say they wouldn’t just to cheat. And we can’t know how many people will do so (or in what quantity).
Victor Venema (Comment #112599)
May 6th, 2013 at 5:00 pm
“I have trouble understanding why there are so many climate “scepticsâ€. Thus in principle such questions are interesting.”
Victor, I have trouble understanding why the consensus is defined so loosely that just about every reasonable participant in the AGW discussion would be a member.
I do not know how well informed you are on the skeptics side of AGW, as I have heard you make note that in your country that skeptics are rare, but skeptics are skeptical about the quality of the science as practiced by climate scientists and the fact that GHGs cause warming is not an issue. Cook’s survey appears to miss that point and in my interpretation (which no one here has refuted) of the potential replies it would appear that Cook really has not gone beyond the premise that GHGs can cause warming. Perhaps you could chime in here and give your opinion of what the replies to survey mean.
HaroldW:
I don’t think it “matters” in the sense of affecting what their conclusions will be. I do think it “matters” in the sense of it may affect how they justify those conclusions. The more one has to stretch to reach a conclusion, the less people exercising critical thinking will believe it.
For example, if you removed results from people who believed in conspiracies from Lewandowsky’s data set, what do you think would happen? You’d likely expect there to be no correlation between skepticism and conspiratorial ideation? After all, how could there be a correlation when no conspiratorial ideation exists in the data?
Turns out, there can be. The analysis method used by Lewandowsky will find a correlation. The reason is the level of disbelief people listed for conspiracies is the primary source of the correlation his method found. It’s not that people believe in conspiracies. It’s that some disbelieve them more strongly than others.
Brandon Shollenberger, it will not happen in reality. Such a person would click on the link and not spoof an URL. Mathematician would say, almost surely.
.
Kenneth Fritsch, I mainly know the sceptics from WUWT. Thus I have a very bad impression about them. Before I found that site, I had never imagined something like that being possible.
How do you know the quality of climate science is so bad? Do you read the scientific literature yourself or do you get your information from the press and blogs?
Journalists are interested in spooky stories and will select the works of those scientist that makes the nicest headlines and make it a little stronger.
Most of the “bad science” on WUWT is misunderstanding the articles at best and more likely deliberate misinformation.
.
Have to sleep now. Good night.
Steven Mosher (Comment #112592)
May 6th, 2013 at 4:19 pm
“It could backfire. Imagine if SKS readers saw MORE EVIDENCE for consensus than scientists did. That would say that the meme of ‘consensus’ has been oversold.”
My point is that overselling will always happen on both sides of a partisan issue. Those people with a problem in this matters, in my mind, are those that see overselling confined to the other side.
Obviously Cook can always claim that he could not reveal his initial hypothesis in order to prevent influencing and gaming responses and thus if the results merely show what I would expect from such a survey (that it can find oversold participants on all sides of the issue) he can change it to something like we need to follow better what the scientists conclude, i.e. the consensus position. Of course then we are back to a very loosely defined consensus position.
Steven Mosher (Comment #112544) May 6th, 2013 at 1:33 pm
“haha. I get to play Nick Stokes”
Thanks, glad of the help. But standards of hunkiness are high.
Whatever happened to double-blinded?
You have an investigator who is questioning the readers of his own blog and blogs he is openly opposed to and has arranged the survey so that he can distinguish between the two groups.
And Mosher’s suggestion is the purpose of the survey is to see jf skeptics can go beyond their bias and read an abstract. The real question might be whether the investigator can protect us from his own biases. The methodology is going to have to be clear and very precise on this.
the ‘conflict of interest’ statement at the end of the paper is also going to be a fascinating read.
Victor Venema:
Say what?! There have already been people who openly admitted to cheating the survey. How can you possibly say this “will not happen in reality”? Are you telling us you know people who want to cheat the survey will just choose not to incorporate this into their efforts?
HR:
It got blind-sided by a mob of Politicization, Messiah Complex, Laziness, and others.
And this investigator believes in a cause so much that he believes not agreeing with it could send people to Hell:
“How do you know the quality of climate science is so bad? Do you read the scientific literature yourself or do you get your information from the press and blogs?”
I actually read the papers and do my own analyses. How about you? Actually being skeptical does not preclude finding good climate science. Does it? Perhaps we could discuss a few important climate science papers in detail. I also get some feel for the quality of climate science by having discussions with people like yourself. Perhaps you remember our discussion about using breakpoint algorithms for adjusting temperature series and detecting many smaller breakpoints that could lead to an extended trend in a difference series.
Brandon (#112604) –
Agreed. And I know that it can be intellectually challenging to think about the methodology — how can it be gamed? how could a survey be designed which would prevent that method of gaming? how strong are the conclusions, and how much do they depend on the author’s data processing choices?
At another, less busy, time, I might join in. But the “bottom line” for me is this — if I see a poll by the Republican (Democratic) National Committee which concludes that 60% of the country hates the Democratic (Republican) policy choices, I know it’s BS. They’re selling something. Same with Lew and Cook. “By all means let’s be open-minded, but not so open-minded that our brains drop out.”
Cook is an advocate first and foremost. He is only ever going to publish results that confirm his position. It would be naive to believe otherwise.
What does this matter. The result is given. It like putting Donald Trump in charge of an investigation of Obama’s birth certificate.
You publish the bits you like and leave out the rest. If you don’t like any of it then you don’t publish
“Steven Mosher (Comment #112544) May 6th, 2013 at 1:33 pm
“haha. I get to play Nick Stokesâ€
Thanks, glad of the help. But standards of hunkiness are high.”
very funny.
You tried, you tried . . . did you try to include a link to the research you’re so very angsty about?
Not every reader is familiar with the ins and outs of your latest Polish cavalry charge against the dastardly warmists.
“the road we are on takes us to a planet that is 5 °C hotter, also known as Hell.”
That is fodder for a 2 part unambiguous survey question:
The planet will be 5 degrees C hotter if we do not mitigate (successfully). True/False
Hell is 5 degrees C hotter than the planet currently is. True/False
“And Mosher’s suggestion is the purpose of the survey is to see jf skeptics can go beyond their bias and read an abstract. ”
Note that I also argue that SkS readers could show their bias.
For example, if a paper was rated by an author as being “against” AGW and SkS readers saw it as neutral, that would argue they have a frame that is somewhat tainted.
I find such notions totally uncontroversial. Look, Nick and I are both smart. We have both been to school. There are plenty of occassions where our readings of the same signs differ in material ways. Same goes for me and Brandon. Both educated, both able to read, but we do not always see the same thing even when we are looking at the same thing.
This fits with my general view of perception
http://en.wikipedia.org/wiki/Jerome_Bruner
“Bruner is one of the pioneers of the cognitive psychology movement in the United States. This began through his own research when he began to study sensation and perception as being active, rather than passive processes. In 1947, Bruner published his classic study Value and Need as Organizing Factors in Perception in which poor and rich children were asked to estimate the size of coins or wooden disks the size of American pennies, nickels, dimes, quarters and half-dollars. The results showed that the value and need the poor and rich children associated with coins caused them to significantly overestimate the size of the coins, especially when compared to their more accurate estimations of the same size disks.[5] Similarly, another classic study conducted by Bruner and Leo Postman showed slower reaction times and less accurate answers when a deck of playing cards reversed the color of the suit symbol for some cards (e.g. red spades and black hearts).[6]”
I think it is ok to observe these differences, to comment on these differences and to celebrate these differences and yes to poke fun at each other over these differences. I don’t think its harmless to demonize folks over these differences, and of course I am no angel.
Victor
“The real interest of Cook is naturally not such a scientific question, but to bash on the “scepticsâ€. As he himself is an active participant of that debate, I would have preferred that someone more impartial would write these kind of papers.”
I think we can all agree on this. There is an interesting communications/cognitive psychology question here. How can the different people look at the same evidence and draw such different conclusions. Citing Bruner above I have to say that I do not believe that “perception” is a passive process, but rather we always come at data, at observations, with theoretical constructs. There are no raw feels and there is no raw data ( to be overly extreme just to make a point and upset card carrying realists ) and the most interesting situations for me are those like the old lady/young lady illusion..
Where things go off the rails I think is when people conclude that this fact about perception somehow ends us up in subjective relativism. and it goes off the rails when some insist there is position of purity free from needs and values that can be achieved in all cases.
Brandon,
If I may follow up on our exchange in the previous thread:
http://rankexploits.com/musings/2013/a-random-failure/#comment-112482
(and apologies if I am laboring past something obvious I have missed)
Cook’s instructions to the survey-taker seem to make no distinction between (X1) abstracts which indicate the paper will contribute new evidence to support an AGW hypothesis or theory (however unspecified by the terms of the study!), and (X2) abstracts which merely assume AGW in order to announce a paper studying the alleged impacts of (assumed) AGW on some part of the earth’s surface.
Yet, some participants are likely to think Cook means the former and some will think Cook includes the latter. Does everyone but me know which version Cook intended? Is the user supposed to judge whether the paper adds new evidence for AGW? Unless all survey-takers know somehow whether they are supposed to judge X1 or X2 it seems to me that the instructions are fatally ambiguous.
Wouldn’t this be an uncontrolled ambiguity in how survey instructions are interpreted? Could that be something Cook means to study? But I don’t know how he could, since he doesn’t have any questions which could probe that issue of interpretation.
I imagined (before seeing any abstracts) that I would be looking for abstracts of X1 type, but then found that many were of the X2 type, studies of limited eco-systems to estimate impacts from ASSUMED climate change.
The latter kind of papers do not add anything to the case FOR AGW, they assume it.
Steven Mosher:
Sure. Nobody sees all the possible meanings of everything they read. And even when people see the same possible meanings, they won’t always pick the same ones as most likely. That’s unremarkable. That bias can influence which interpretations are seen/picked is also unremarkable. Specific examples of such happening, like Cook seems to be trying to find, can be remarkable.
On a related note, those sort of difference are easy to combat. People should just say what their interpretations are and explain them. With reasonable discussions, that’s all it takes. Which is why I’m offering this scenario:
Did I lie? Technically I did have a hundred papers. Technically I did do my picking via a random coin flip. Does the fact I made technically true statements mean I told the truth even though I made the statements with the knowledge they’d mislead?
Can we have a reasonable discussion when lies of omission aren’t considered deception?
Could I just note that Steve Moshers question;
“Question; How can so many sceptics not get the consensus? Answer 1. They dont know the science. Answer 2. When they read the science they dont get it.”
Contains the implicit assumption that the “Consensus” is correct.
In a similar line (and people have been studying ulcers for far longer than worrying about climate): Question; How can a fully trained Doctor not get the consensus that stomach ulcers are caused by stress?
Answer 1. He doesn’t get the science.
Answer 2. When he reads the science he doesn’t get it.
Unfortunately the correct answer is;
Answer 3. The Consensus was wrong in every respect and Barry Marshal recieved his Nobel prize in 2005.
Assuming that only one side has fully correct answers is not a path to truth.
Robert:
Nope. I intentionally didn’t include a link to “the research” because I wanted to ensure people like you were excluded. I knew:
And figured they wouldn’t look at any of four of the five previous posts on this site. Also, I intentionally didn’t link to “the research” because I figured if anyone read the published results, they’d know I was full of it.
You caught me.
Mosher
You’re looking for this study to shed light in the wrong place.
On another matter here is a link to the UoQ ethics guidelines.
http://www.uq.edu.au/research/rid/files/human/uq_interpretations_national_statement_20111010.rtf
And from it the guidelines definition of deception.
“Deception
This occurs when research participants have essential information withheld and/or are intentionally misled about procedures and purposes, including studies where participants are deliberately given misleading information about the purpose of a research study. ”
and on what defines consent
“Consent
The voluntary authorisation by a competent person or group, based on adequate knowledge and understanding of relevant material, of a researcher to conduct research which involves them as participants.”
Who knows if what Brandon has unearthed here constitutes deception?
ok, I think I see the phrase (of Cook’s) that give me trouble: in the instructions when you enter the survey form, he refers to judging “the level of endorsement that is expressed in that paper….” (emphasis added). I suppose it does not seem to matter what the source of “endorsement” is for this purpose, whether it is justified in the paper itself or whether it is merely a background assumption. Yet, to my mind a paper which offers “endorsement” without any pretense of added evidence and argument for AGW does no more for the case for AGW than anyone’s blog opinion etc.
Cook can say he’s only trying to assess how people review the degree of “consensus” that exists without regard to how it comes to exist, whether or not a particular abstract/paper adds anything to the evidence and reasoning for that consensus. Yet, this is one of the weak points for strong assertions of CAGW, conflating all the papers and scientists which assume without argument some degree of AGW (which all of us “lukewarmers” can accept anyway). It’s not the point of Cook’s survey, but the casual, lazy conflation of ALL degrees and kinds of “AGW’ is one of the biggest problems with both scientific and public policy discourse about climate.
Steven Mosher (Comment #112619)
May 6th, 2013 at 6:49 pm
Victor
“The real interest of Cook is naturally not such a scientific question, but to bash on the “scepticsâ€. As he himself is an active participant of that debate, I would have preferred that someone more impartial would write these kind of papers.â€
I think we can all agree on this.
But to what end? This isn’t just the satisfy his own ego that he is right and sceptics are deluded. Or just for your amusement. This is his attempt to suggest that through science he can prove anybody that is opposing the IPCC’s version of the consensus is suffering from some type of mental block or worse. We’ve been down this sort of road many times in history. It never ends well.
p.s. I think I am reading “level of endorsement” in the context of a scientific paper to include a scientific **justification** and evidence-bearing argument, when in fact many of the abstracts/papers trumpeted by these mass literature surveys do not offer any added evidence or justification, merely assumptions.
This may not in fact be a problem for the limited aim of Cook’s survey, but I think it is an enormous problem for how mendaciously the public debates have been conducted. Since I am not one who has trouble with the idea that humans may be contributing to some amount of warming (my doubts are over whether the magnitudes have been accurately assessed and whether policy proposals are reasonable), this notion that it is helpful to count how many scientific abstracts may ***ASSUME*** some amount of AGW is what seems utterly silly to me. It’s like a high school popularity contest, not a scientific argument.
Skiphil, yup. What John Cook is doing isn’t necessarily wrong. It’s just the sort of thing that is pretty much guaranteed to get misinterpreted. Possibly even by the people behind it.
HR, part of me hopes it isn’t considered deception. I love wordplay, and I have nihilistic flirtations. If the sort of behavior Cook engaged in is okay, I can use similar tactics to offer false support for any number of arguments. There’d be practically no limit to the nonsense I could pull off.
“Who knows if what Brandon has unearthed here constitutes deception?”
indeed. who knows if there are an odd number of stars or even number.
The Key is
“essential information”
Now, Victor and I look at that and say the withheld information is
not essential. That is, if I knew the abstracts were limited to 875 characters I still would participate. Now, Brandon will say the information was essential.. essential to what? essential to answer the questions? essential to reproduce the study? essential to audit it? If he knew that all 12K papers were used would that change his decision to participate? Was he harmed by with holding this information.
who decides essential? essential is not obvious on its face. essential is conditional, requires judgement, exists in a context of past practices. we dont know what this community of researchers deems essential.. u say tomato, I say tomato.
Now, if you want to prove via syllogism what is essential and what is not essential, then have at it. See how literalism works in debates..kinda annoying.
Steven Mosher wrote:
the point is to show how readers differ in their assesments from authors. By doing this you will see whether skeptics can read or not
Or whether authors can write or not 🙂
OT but on the Cook-Lewandowsky propaganda efforts more broadly speaking:
This can use some serious ‘Fisking’ for those so inclined:
(h/t Ruth Dixon at Bishop Hill)
Scientific American blog spouts Lewandowsky propaganda
Steven Mosher says:
What I find “kinda annoying” is people creating strawman, especially when they’re used to make false claims about me. Mosher acts as though the only way Cook’s writing on this survey could be deceptive is if “essential information” was withheld. That is wrong. Even if we limit ourselves to this one particular definition, it is wrong because we cannot limit ourselves to one part of the definition. The definition was:
Notice what I made bold. Mosher acts as though what preceded the “and/or” is the only thing that matters. In reality, something could be deceptive despite failing to meet the standard Mosher refers to. All it has to do is meet the second standard listed.
Mosher claims, “Brandon will say the information was essential.” This is untrue. What I will say is we were intentionally misled about the procedures and purposes of this survey. As such, deception occurred even by the limited definition offered by the University of Queensland.
Eli is with Victor on this one. Youse folk are being foolish. However, when the first paper is published you will have an idea of the number of papers that were rated by their authors. Sou has it nailed
Mosher writes “at some point in the past 12000 artcles were selected.”
And at some point in the future, the results of this study will reference the 12,000 papers that supported it and how they are “crowd source rated to be AGW friendly”. Just you wait and see 😉
There appears to be a significant ignorance here about informed consent for surveys. The major requirement is that the subjects (that’s you and Eli) be told of any risks and benefits to them. It is not necessary to provide all details
If the survey instrument was constructed it should not be possible to associate a name or email address with a set of answers, so Lucia’s letter had damn well better have gotten the no can do answer. John’s preamble appears to have met those conditions. True Australia and the US may differ, but then it would have been up to Shollenberger to go find and read the rules before whining.
Mosher I would read it differently.
I’ve had limited experience preparing information sheets and consent forms for ethically approved medical research projects where the wording and clarity is extremely important. Although different procedures exist to fast-track ‘non-risky’ projects like Cook’s, the UoQ appears to hold both to similar standards
The intention with this type of material is to deliver clear, simple and straightforward info so that the reader can in no way be misled. The assumption is you are talking to a person uneducated on the subject and so must provide the clearest, most straightforward information.
So in general terms of maintaining good standards of ethical approval it seems to fail because it makes statements which can only be truly understood by further explanation. That would presumably be important to the ethics board that approved this.
I agree that it makes little difference to the people who take part in the survey, it does no harm. Cook could probably have provided less information and that would have been OK. It’s providing false information or at least information that is open to misinterpretation which is the problem. If Brendan is right then this looks sloppy, not a word you want associated with an ethics approval process.
I think the deception is not to do with the intentions of Cook it’s more about whether somebody who takes part in the survey is truly giving consent if they are provided with information that is difficult to understand.
If non of this matters because it’s a harmless online survey then really it shouldn’t go through ethics approval. But it is going through ethics approval so it should meet some standard set by that process.
Eli
” True Australia and the US may differ, but then it would have been up to Shollenberger to go find and read the rules before whining.”
na, that would spoil all our fun.
see what I said above
‘http://en.wikipedia.org/wiki/D…..ger_effect
You see some of that as people with no expertise try to explain what Cooks study is about and how such studies should be conducted.”
so I beat you to accusing Brandon of
http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
Alex Heyworth (Comment #112630)
May 6th, 2013 at 9:03 pm
Steven Mosher wrote:
the point is to show how readers differ in their assesments from authors. By doing this you will see whether skeptics can read or not
Or whether authors can write or not
####################
+1 for thinking logically and considering all possibilities
Brandon
“Uses stupid and uncharitable semantic parsing. I said the survey is “about a collection of papers†to distinguish it from something dealing with a single paper. I wasn’t trying to say what the aims of the survey was – just a population used in it.”
Then you should have said, USES a collection of papers. writing it was ABOUT a collection of papers was either a lie or deceptive or misleading.
hehe
Eli,
“Eli is with Victor on this one. Youse folk are being foolish.”
some of us are having fun, so go away mr poopy. I mean seriously by now you should get that we enjoy these types of arguments, call them scholastic. Sometimes they matter, sometimes they dont, and sometimes we argue about whether they matter or not. Its a slow week in climateball, what do we have
At WUWT Willis discovers the land ocean contrast
At Judith’s its more sociology.
At CA.. the end of a Pages2K discussion and fun with Cook
At ELi’s? New links? wow, you amaze. At least you were honest about being lazy
At Tammys: Boston? WTF
At RC: an open thread which has more comments than the usual science thread.
oh Climate Dialogue was good, you should have kept your mouth shut
Thats why…
Best of the climate blogs this week is Neven. with a new look
and boffo charts from winepus and others.
http://neven1.typepad.com/
HR:
Indeed. John Cook could have given an accurate number of papers being used (“a few hundred”). He could have given no number at all. Either would have been acceptable. The problem is he gave a totally inaccurate number, 12,000+.
If I’m right? John Cook told people the survey covers 12,000+ papers. He has admitted he used a subset of the 12,000+ papers. That seems to be indisputable to me.
Steven Mosher:
It wasn’t wrong as what I wrote could reasonably be interpreted multiple ways, but even if it were, a mistake is not a lie or an attempt to deceive.
Stokes would know better. You’re slipping.
Steven Mosher (Comment #112644)
May 6th, 2013 at 11:31 pm
An unwarranted nasty slur, mosh, and quite unworthy of you.
I pointed out what I thought were some interesting questions about the land-ocean contrast that you had no answer for. You claimed people had discussed the questions I raised … but the citations you gave didn’t discuss the questions at all.
In other words, you struck out completely over on my thread.
So in response you come here to throw mud on me, where I only saw it by chance? And your schtick here is to claim that I’d never heard that the land and ocean warm at different rates?
Really? That’s the kind of pyrites you’re trying to pass off as gold?
Let me invite you to screw up your courage and come over and tell lies about me on my own threads, Steven. Don’t hide your light over here. It makes you look pusillanimous as well as unpleasantly wrong, and that’s a bad combo. If you have complaints about me, bring them to my threads where we can discuss them and not bother Lucia, there’s a good fellow.
w.
I must admit that I haven’t read all of the comments, but does the rand() function need to be initially randomized in order to produce truly random results?
For example, in the case of Visual Basic, the rnd() function will always produce the same results when a program is run, unless it is first randomized.
“the road we are on takes us to a planet that is 5 °C hotter, also known as Hell.”
That reminds me of a Chapter heading in Björn Kurténs classic “Age of Mammals” (1971): “Eocene – Paradise Lost”.
The view on whether a warmer climate is a good or bad thing seems to have changed a bit since the 70’s
Willis Eschenbach (Comment #112647)
Look down Willis. Is one leg a little longer than the other?
Eli
1) I said before that I did not send the letter.
I’m am pretty sure that John Cook’s survey instrument could permit him to associate names with answers. In the first place, he requests people volunteer emails and comments. In the second, his survey includes code to fingerprint the screen:
function survparam(){
var s_width=screen.width
var s_height=screen.height
var d = new Date()
var timezoneoffset = d.getTimezoneOffset();
var pluginstring = '';
numPlugins = navigator.plugins.length;
for (i = 0; i < numPlugins; i++) {
plugin = navigator.plugins[i];
pluginstring = pluginstring + plugin.name + ',';
}
if (localStorage) {
if (!localStorage.getItem('survparam6')) {
localStorage.setItem('survparam6',new Date().getTime() + ":" + Math.floor(Math.random() * 10000000));
}
document.getElementById('survparam6').value = localStorage.getItem('survparam6');
}
document.getElementById('survparam1').value=s_width
document.getElementById('survparam2').value=s_height
document.getElementById('survparam3').value=pluginstring
document.getElementById('survparam4').value=getFonts();
document.getElementById('survparam5').value=timezoneoffset
}
This code should make it very easy for him to detect multiple entries by the same person. He’s going to need to because when I saw it I changed my mind about not adding any and added a bunch.
I’ve always been very interested to know what they do. I’ll be interested in seeing the discussion about the method used to detect the multiple entries in the paper.
How are the authors selected? Are they the lead-author?
How many of them/what fraction agree with the proposition about ‘global warming’?
Questions, questions….
michael hart–
Do you mean the authors who rank the their own papers? Presumably that will be fully and clearly explained in the as yet not published Cook paper that has been accepted by some journal or other and which the survey says this survey is intended to “mirror”.
Now some, not Eli to be sure, might think that Brandon has missed the purpose of this survey.
Eli
Fell free to tell us what the purpose is. When doing it, elaborate point to the specific language in the consent form and explain precisely how we know the purpose based on the language in the consent form.
If you learned from some other source, please link and elaborate on precisely what the words at the other source say the purpose is.
“I must admit that I haven’t read all of the comments, but does the rand() function need to be initially randomized in order to produce truly random results?”
Ray, in the preceding post it is pointed out that the MySQL server is handling the random numbers and not the web server. Initialization of the database server also initializes the random number sequence, so this bit of SQL does not need to do that. This SQL looks like a MySQL incantation, so we’re assuming that is the type of database server which is being used.
So do we have an idea of the number of papers that are the basis for random draws? And can we test with multiple draws to test for randomness? Can we assume randomness and back out the number of abstracts in the population being drawn from?
Is that number important to the people taking the survey? I would doubt that and I say that because taking the survey would appear to be more a game for the people to whom it is aimed. Is that number important to those who want to analyze the methodology? I would think it is. My first thought was that a pool of 12,000 abstracts is too big and would require a huge response in order to obtain meaningful replication of responses for given abstracts.
I suspect that the survey was(is) a work in progress that was amended several times and what we see is a less than a finished product. The publication of the 12,000 abstract number could well have been written up (and perhaps forgotten) before the author determined what I pointed to above plus the fact that a long abstract requires a longer attention span – than many survey takers might possess. These facts could have been, at least partially, revealed if the author made an initial test run.
lucia (Comment #112655)
May 7th, 2013 at 7:29 am
Lucia, I think the wily wabbit is saying that he does not think Brandon has missed the point. I do not know if that is an endorsement Brandon might have sought, but the way I read 1st/3rd person disconnects , the wabbit has endorsed.
Kenneth
The person who would know for sure is John Cook. You could ask him. Others could estimate by loading the survey over and over and tallying the papers that appear including how often they repeat. Maybe someone has done that; if they have it’s buried in comments somewhere.
This depends on the purpose of the survey.
If the purpose of the survey is to estimate “the level of consensus regarding the proposition that humans are causing global warming’ as diagnosed by a bunch of people reading abstracts, there is absolutely no valid statistical reason to down select from the 12,000. It would be better to just pull 10 papers randomly from the entire set, and get whatever you get.
If the purpose is to “mirror” the as yet unpublished paper, then there might be a reason to downselect. If the purpose is to “replicate” and “experience”, then maybe there is a reason to downselect to the group of papers. But that reason isn’t statistical, it is merely that to replicate an “experience”, you want to match that experience as much as possible. If the purpose is “analyse ratings from a diverse range of participants” one doesn’t necessarily need to downselect. It all depends what “analyze” means. If it means: To compare the responses form c=Blog1 vs. c=Blog2, then downselecting might be useful because people from the two blogs are more likely to evaluate the exact same papers. Even so, it’s not entirely clear hat it’s necessary. Even without downselecting, you could do some analysis and learn people on Blog1 tended to give higher or lower likert scores than people at Blog2.
BTW: The query string after c=…. is passed in the post on submission. So that is likely logged. As it is logged, it is quite likely intended to be used.
“Mosher: It could backfire. Imagine if SKS readers saw MORE EVIDENCE for consensus than scientists did. That would say that the meme of ‘consensus’ has been oversold.
one can well imagine a situation in which the scientist come out in the middle.. conservative skeptical, whereas blog readers both sides were biased.. one side seeing more support for CAGW and the other side reading falsification whereever they could.”
It can’t backfire. Best case he doesn’t publish it saying that somehow his survey data was ‘corrupted’ by nefarious people. Worst case he filters the data – there’s no such thing as an unpickable cherry.
scott:
“It can’t backfire. Best case he doesn’t publish it saying that somehow his survey data was ‘corrupted’ by nefarious people. Worst case he filters the data – there’s no such thing as an unpickable cherry.”
It can backfire, but you might never find out about it
Willis
‘An unwarranted nasty slur, mosh, and quite unworthy of you.”
how is what I said a “slur”?
quote my words exactly and explain what the slur is and where I said it?
“And your schtick here is to claim that I’d never heard that the land and ocean warm at different rates?”
Quote my words exactly willis, where do i say this.
Brandon
‘It wasn’t wrong as what I wrote could reasonably be interpreted multiple ways, but even if it were, a mistake is not a lie or an attempt to deceive.”
Anything, including what Cook wrote can be interpreted many ways. Meaning is not inherent in the sign. The fact is I interpreted what you wrote one way, and was mislead.
j ferguson (Comment #112650)
May 7th, 2013 at 4:59 am
Willis Eschenbach (Comment #112647)
Look down Willis. Is one leg a little longer than the other?
###########
hehe. I don’t think Willis has read the whole thread. Brandon
gets it and is playing along nicely. sheesh.
Here Willis,
you get the same treatment as Eli.
‘some of us are having fun, so go away mr poopy.”
It’s not entirely inherent in the sign. But there is a limit.
Sometimes language is accidentally misleading, sometimes it’s written with the intention of creating a false impression without actually lying. People eventually realize that some people do the latter quite often. Like a particular former president who ‘did not have sexual relations with that woman” and who in grand jury testimony resorted to saying “”It depends on what the meaning of the words ‘is’ is.”
Generally speaking, sloppy language in blog comments are can be confusing and people figure that others can ask for clarification. Incomplete or misleading answers in response to direct questions? Misleading information in highly proof-read formal stuff? People tend to suspect language that conveys a false impression in those case may be misleading intentionally. Of course these suspicions may be wrong. (And of course expressing them might get your words written up as evidence of “conspiracy ideation”. But there you go.)
Mosher says
“It can’t backfire. Best case he doesn’t publish it saying that somehow his survey data was ‘corrupted’ by nefarious people. Worst case he filters the data – there’s no such thing as an unpickable cherry.”
Presumably the earth would boil before Cook authored a paper that concluded that WuWT readers were more considered in their reading of climate papers than SkS readers. That very fact invalidates the study
“Worst case he filters the data – there’s no such thing as an unpickable cherry.”
Exactly. Just select-out based on the authors (opinions). Tell (‘non-aligned’?) authors that insufficient numbers got randomly assigned to their works, or something.
A bit like the way trees are selected by some dendrochronologists.
Lucia
I recommend the following
http://www.amazon.com/Explanation-Power-Control-Human-Behavior/dp/0816616574
“Explanation and Power was first published in 1988. Minnesota Archive Editions uses digital technology to make long-unavailable books once again accessible, and are published unaltered from the original University of Minnesota Press editions.
The meaning of any utterance or any sign is the response to that utterance or sign: this is the fundamental proposition behind Morse Peckham’s Explanation and Power. Published in 1979 and now available in paperback for the first time, Explanation and Power grew out of Peckham’s efforts, as a scholar of Victorian literature, to understand the nature of Romanticism. His search ultimately led back to—and built upon—the tradition of signs developed by the American Pragmatists. Since, in Peckham’s view, meaning is not inherent in word or sign, only in response, human behavior itself must depend upon interaction, which in turn relies upon the stability of verbal and nonverbal signs. In the end, meaning can be stabilized only by explanation, and when explanation fails, by force. Peckham’s semiotic account of human behavior, radical in its time, contends with the same issues that animate today’s debates in critical theory — how culture is produced, how meaning is arrived at, the relation of knowledge to power and of society to its institutions. Readers across a wide range of disciplines, in the humanities and social sciences, will welcome its reappearance.”
It’s sure to annoy.
Let’s try to zero in, for a minute, on the Ethics Committee issues posed by Cook’s study.
Be forewarned, this stuff gets casuistical.
And I’ll use terminology from the USA, in the awareness that Australian practices sometimes differ.
(From the history of Lewandowsky, Oberauer, and Gignac’s opus, I’ve learned that Oz regulations allow wholesale changes to a previously approved study to be approved as amendments. Under US regulations, an entire new application would be mandatory.)
The “informational letter” that participants in Cook’s study see includes the following language:
Under the American rules for this kind of study, Cook would not be obliged to disclose his hypothesis to the participant. He would be permitted, if not encouraged, to leave other details out—so long as he is not withholding information about risks from participating, or actively deceiving the participants.
Now it would be different if Cook had applied for permission to use deception and explained to the committee why he considered it necessary. Then he might get authorization to employ an actively misleading cover story.
But he would have had to make a special case for it. And under the American system, his application would have been kicked to a higher level of review (implying more forms to fill out, longer delay before receiving approval, greater likelihood that someone would object and make him change stuff, need for the committee to approve any deceptive language word for word, etc.).
Now maybe Cook did designate the above language as deceptive and got approval to use it.
(In which case, he definitely wouldn’t want site owners to see his Ethics Committee application, because the deception and the rationale for it would be spelled out there.)
Or maybe he didn’t designate the above language as deceptive, but he put it in the informational letter even though his actual procedure is not what he is telling participants it is. In which case, he would be misleading participants without permission from the Ethics Committee.
IMHO, the manner in which abstracts were selected for presentation to the participant is not the kind of thing over which a researcher would usually be requesting permission to use deception…
JunkPsychology:
So are you saying:
1) In the US, Cook could have left all that prefatory material explaining the purpose of the study and so on or he may put in it, his choice.
2) In the US, if he chose to put it in, what he elects to put it in, he is prohibited from making what he includes deceptive unless he gets permission from the ethics board. (If he gets permission, then it’s permitted to deceive.)
That’s what I think you are saying (among other things) and it’s what seems reasonable to me. Am I getting this right?
Mosher
Why?
Lucia,
Right.
An Institutional Review Board (as the committee in question is called in the US) would not require him to give so elaborate an explanation of the purpose of his study.
But the IRB sees the informational letter with the application, has to approve it—and has to see and approve any changes the researcher may want to make after the application is approved.
Right.
But the kind of deception an IRB will allow (after a special case is made, and extra hoops are jumped) is usually giving a completely misdirecting cover story about the purpose of the study.
For example, the experiment is really about what you will do when the guy who’s been stationed in the narrow corridor is pulling out a file drawer that blocks your path, refuses to get out of your way, and then insults you.
But if you know that the experiment is about your reaction to an insult, you’ll watch how you behave.
So the researchers are allowed to tell you instead that the experiment is about doing memory tests, or filling out some surveys, or doing a usability test on software. And that lab suite with the conveniently narrow corridor is the place you go when you think you’re going to be doing the memory test, or whatever.
Steven Mosher:
I just interpreted your comment here as, “I’m a pink elephant.” The fact is I interpreted what you wrote one way, and I was mislead.
lucia:
Because it’s basically pure relativism. Even worse, it takes control of meaning completely away from the speaker.* It says what you intend has no bearing on the meaning of what you say; all that matters is how other people interpret your remarks.
In effect, it makes it so there is no wrong interpretation for anything anyone says. You can interpret, or even misinterpret, anything you want however you want, and it is defensible.
Steven Mosher,
I see from the blurb that Peckham’s starting point was Charles Peirce’s theory of signs. But Peckham evidently didn’t end up in the same place as Peirce.
How would you describe Peckham in terms that today’s academics use: Deconstructionist? Postmodernist? Reader response guy? Or something else…
And, as best you understand Peckham, would he have endorsed or rejected the doctrine that science is just another hegemonic discourse?
If you decieve your participants, you are supposed to debrief. Cook did no such thing with his limbo Recursive Fury paper.
Yes, if you deceive you are supposed to debrief.
This principle goes back to Stanley Milgram’s experiments on obedience to authority, in the days before there were any IRBs or even any ethics committees.
But “Recursive Fury,” at first glance, is simply an analysis of comments made in public, by people who, you might naïvely presume, would have said the same things whether they had been recruited for a study or not.
Therefore, on the face of it, it was not what would normally be considered a human subjects study.
Did Lewandowsky ever run it past his institution’s Ethics Committee?
Unfortunately for Lewandowsky and his collaborators, he was involved in private communications with some of the same people whose comments he planned to analyze—and some of his private communications, in turn, influenced what his “participants” ended up saying in public.
And all the while he and two of his collaborators (Cook and Hubble-Marriott) were publicly ripping these same critics—a fact not disclosed in the manuscript he submitted for publication.
In psychology, you reply to your critics.
You don’t pretend to have conducted a study with your critics as the participants.
Mosher
If that’s what it claims, it’s view of communication is totally idiotic. That said, an awful lot of people try to claim the opposite which is that the person who said something gets not only to tell you what they said literally meant, but to be able to claim that any misinterpretation is on the side of the listener.
Obviously, communication is a two way street. People grok this even if they can’t phrase it in some scholarly language.
lucia, that was my description of it, not Mosher’s. I don’t know that Mosher would agree with it. I think my description takes the ideas Morse Peckham expresses to an extreme, but is still fair (reductio ad absurdum). The biggest issue I think one could raise with what I said is people and society don’t just pick arbitrary interpretations. That may be (mostly) true, but it not happening in no way means it couldn’t happen.
As for phrasing things “in some scholarly language,” believe me, you phrase things far better than Peckham does. Look at this excerpt:
That is only half of one paragraph. Does anyone think that is a clear form of communication? The redundancies in his writing alone are nightmarish. We could rephrase everything he writes in a quarter the words. And if we did that, it’d be a lot easier to see his arguments for what they are.
(There is humor in a person with bad communication skills telling us what we say has no meaning.)
“JunkPsychology (Comment #112689)
May 7th, 2013 at 3:07 pm
Steven Mosher,
I see from the blurb that Peckham’s starting point was Charles Peirce’s theory of signs. But Peckham evidently didn’t end up in the same place as Peirce.
How would you describe Peckham in terms that today’s academics use: Deconstructionist? Postmodernist? Reader response guy? Or something else…
And, as best you understand Peckham, would he have endorsed or rejected the doctrine that science is just another hegemonic discourse?
###########
yes, he started in the same place as Pierce. He preceeded the deconstructionists, and is very different from them. I spent a fair number of years studying Derrida back when he first came on the scene in southern california ( charming man) and the vocabularies are incommensurate. Not at all like Stanely Fish ( reader response ) although I can see how you would make the mistake. The best way to describe him would be as a radical empiricist, or perhaps pragmatic behaviorist. Science is the most powerful form of discourse.
Short course: The meaning of a sign is the response to the sign. This signifies that meaning is not “in” the sign, not inherent “in” the sign in such a way that it determines the response. A sign doesnt mean what the signaller “meant” because we have no access to what they “meant”. And their testimoney as to what they “meant” has no inherent priority. If you ask them what they “meant” all you get are more signs. And referring to dictionaries just gets you more signs. The meaning arise through interaction, and while any response to a sign is possible ( hey you ran the stop sign) culture and society and other speakers work to regulate ,channell, or control the responses to signs. So, you run a stop sign ( the sign does not determine the response ) and the cop explains to you what it means, what kinds of responses are allowable and what kind of responses are not allowed. Now you can disagree with the cop. And then you get to see the judge and he will tell you what the accepted response to the sign is and he may fine you. And you can still disagree and argue, and then the judge will exercise what peckham calls ultimate sanctions: he punishes you and throws you in jail. Ultimately meaning in enforced by power.
How would deconstruction differ? Derridean deconstruction is more about the play of the signifier, a deconstructionist exploits specific forms in discourse to turn a text against itself, where it unravels its own authority. Simple example, would be to note that a text which valorizes speech over writing happens in writing. Peckham is serious and Derrida is playful.
Lucia
“Obviously, communication is a two way street. People grok this even if they can’t phrase it in some scholarly language.”
Err. No, Peckham would argue that it is not a two way street. Its a multiway street.
Consider the problem you had with John Cook.
John reads the guidelines for ethical behavior. He thinks they mean X. You read them and think they mean Y. Nobody goes to the author of that text to ask them “what was in their head” or what they “intended”. You cant just stomp your foot and say “It means what I say it means”. well, you can..There is a third party who has a say in what it means. The ethics board. So they interpret the text and they explain what it means. Of course, you might not like that interpretation, so you go through a whole appeal process.. In in the end “somebody’ or some institution tells you ‘what it means’. The meaning wasnt “in” the sign. The sign didnt determine the meaning. The meaning was determined by an interaction and in the end by power.
“A proof only becomes a proof after the social act of “accepting it as a proofâ€.â€
( very nice quote in a book on mathematical logic, google it )
That is what peckham means by the meaning is not “in” the sign. the sign doesnt determine the response. Now of course you learn how to respond to signs, you explain ( use words ) to justify why you responded the way you did, but in the end meaning is determined socially, culturally, and enforced via power.
Sure to annoy.
I can’t help but think, “niggardly.” Lots of people may remember the David Howard incident involving that word, but the one that stuck in my mind happened shortly after. A black student was offended by a teacher’s normal use of “niggardly” because she thought it was a racial slur. Even after she was informed her offense was baseless, she defended it, saying:
Sure, and perhaps water buffalo really is a racist term.
Steven Mosher gives a great display of the tautology of this belief:
He says the “meaning” was determined by power. This is only true if we accept the interpretation given by those “in power” is the true meaning of the word/sentence/sign/whatever. If we don’t accept such, his statement is false.
People “in power” can make mistakes. People “in power” can intentionally deceive. People “in power” can be wrong. The fact they have power doesn’t magically make them right.
Or maybe it does. Maybe if the courts make a mistake interpreting the law, the law changes its meaning to match their mistake, and thus, they didn’t make one.
I’m starting to see the beginnings of a novel set in a dystopian future where…
In summary, John Cook presented people with 10 random papers, excluding ones that were too long and others that were not rated by their authors.
Seems fine to me.
I quite enjoyed reading them, personally. On of was virtually incomprehensible – I had to read through it about 5 times in order to assign some sort of meaning to it. I suspect the long hand of Lacan….
Anyway, I don’t understand why the shrill complaints about this survey. Its intention is obvious, and the sampling method for selecting the papers is not in the slightest bit important.
Craig
I don’t know which things you consider to be “complaints” vs. “observations” nor which things you consider to be “shrill” vs just “said”.
I tend to suspect that you are diagnosing “shrillness” merely on the basis that people are discussing things that either don’t interest you or that, for some reason, you think ought not to be discussed. If this were a cocktail party, the correct response would be to go find the group that discusses the subject that interests you ( e.g. mating habits of boll weevils) vs. the one that does not interests you (e.g. the relative merits of interval training vs. sustained running. ) Standing up and telling the people involved in the discussion you find uninteresting that you don’t understand their “shrill complaints” is a bit silly.
As for this:
It is observed that the survey tells people they papers will be randomly selected from a batch, and the wording suggests that batch will be from the 12,000 corresponding to a particular search of a database. Brandon is observing this does not seem to be the case. It appears Brandon is correct.
Given that Brandon is correct people are discussing whether the wording of the survey introduction complies with the panel of the ethics board. It may or may not comply. This is a discussion. Some people may or may not care about whether it does or does not comply. But that discussion doesn’t become a “complaint” nor “shrill” merely by the fact of taking place.
Also, even after accounting for the non-random preselection of surveys, some are debating whether the 10 are even selected randomly from the subset. Some people think the search doesn’t operate randomly, some do. That discussion is taking place. I don’t see how the fact of the discussion is someone “shrill”. Admittedly, if the intention is to get a random distribution, it might be called a “complaint” in the sense that it is a “criticism of the details of the implementation”. But really– that’s pretty normal in evaluating a method of conducting and experiment.
I get that you might not ‘understand’ why some people might discuss their understanding of how MySQL works. Many find that pretty boring. But it affects whether the abstracts are selected randomly;evaluating this sort of thing is required if good methods are to be preferred over bad methods.
As for filling out the forms: Yes. It’s fun. It’s especially fun to see the blurb a the end that tells you how the authors rated the surveys. Tempts people to enter more.
Craig Thomas magically writes “On of was virtually incomprehensible ”
And I have nothing to add to that part 🙂
He goes on to say “Its intention is obvious, and the sampling method for selecting the papers is not in the slightest bit important.”
But IMO the new statement will be “Everyone including skeptics agrees that AGW is happening according to their survyed results rating evidence from over 12,000 papers”
And in time the insideous meme will be simplified to “Over 12,000 papers support AGW as agreed by everyone”
Once its out there it will be as hard to dislodge as the 97% of scientists… meme
Nothing but words all the way down—until you get to someone who can coerce the other parties to the communication.
The first part sounds pretty Derridean to me. Even if Peckham doesn’t appear to be having fun.
The second part says, among many other things, that the US Constitution means precisely what 5 members of the present Supreme Court want it to mean—unless the President tells the Supreme Court that the Executive Branch will not comply, and what troops does the Chief Justice propose to send.
And that science is a hegemonic discourse.
Re: JunkPsychology (May 9 08:45),
Yes, sort of. Look at the history of continental drift or Lysenkoism. But in science, observation eventually trumps hegemony. The movement of the continents can now be measured. The structure of DNA and the mechanism of its replication was discovered.
JunkPsychology, this argument can be shown to be false by counter example:
“pi” is a word.
It has a unique meaning. So in fact some words have a determinate meaning, just not all words.
In the case of pi, it’s a unique series of digits 3.14159… but there are an unlimited number of methods for calculating it.
I would go on to argue that measurables also have a determinate meaning. Words like “pressure” have a very specific meaning, and include prescriptions (“ANSI standards”) for how to measure them.
However, theoretical frameworks used to interpret the results of the measurement themselves are malleable… they are modified as more observations are added.
Also multiple frameworks may exist to explain the same data. The decision on which framework does a better job in the short term probably resembles your “hegemonic discourse”.
The caloric theory of heat versus kinetic theory of heat energy is probably a nice example of this…
Laplace’s heat equation can be interpreted by the caloric theory equally well as with the kinetic theory. In fact, many people still talk of “heat flowing” between bodies, so at a lay level, the concept is still very much alive (and a useable framework).
The caloric theory breaks down when you “look under the hood”. Spectroscopic measurements demonstrate the supremacy of the kinetic theory.
People who want to argue that everything in science is purely subjective need to come up with an explanation for why we had abacuses 1000 years ago and now we have iPads. I presume nobody is foolish enough to suggest that this progress is purely subjective too.
Cook has a set of papers whose authors described their ranking. He is asking another set of people to rank them. If there is a null hypothesis it is that he will get a bra curve. He does not need this survey to rank the papers, he got that from the authors. The original design had different URLs for different blogs. Several blogs have blown that up. That is data.
When you have to ask who the mark at the table is it is you.
The better survey would have been to have the paper’s authors rank the paper (and not the abstract) and to give reasons for the ranking and then have others in the field do the same and then finally have have interested laypersons do the same. The key here would be to analyze and scrutinize the reasons given for making the ranking. It would take a much bigger effort than the Cook survey and potential analysis appears on course to take, but it could take the dumb ass factor out of the replies and the analysis might reveal over/under selling on the part of the authors compared to others in the field and the lay public.
[quote]Cook has a set of papers whose authors described their ranking. He is asking another set of people to rank them. If there is a null hypothesis it is that he will get a bra curve. He does not need this survey to rank the papers, he got that from the authors.[/quote]
Gosh, where would we be without the rabbit to explain things to us?
Cook is not asking people to *rank* anything. He is sort-of asking them to “rate” them, although it’s questionable whether he’s even using a scale as opposed to a set of discrete responses. As such, the reporting of means is dubious.
Also, as Lucia has pointed out, the response set is not identical for the authors and the blog-sourced respondents.
Eli
I believe I was the first to post other urls at my blog withholding mine. You can call people’s reactions data if you wish. It is data that shows what happens if a partisan blogger first writes a paper trashing a group of blogs and their visitors and soon after requests those bloggers and their visitors to do him the favor of helping him with his so-called research.
I think I’ve waited long enough. I e-mailed John Cook several days ago, and as shown in this post, he responded. In fact, he responded within twenty minutes, saying:
I think the last sentence is rude, especially given he could have saved everyone time and trouble by just not deceiving us. However, I ignored that and responded about two hours later:
Realizing I had forgotten an issue, I sent a second e-mail shortly after:
It’s been several days, and he hasn’t responded. I was polite, and my questions were reasonable, and it seems he was willing to respond when he felt he “had an answer.” As soon as that wasn’t the case, he quit.
It seems that’s the sort of “science” we can expect from Skeptical Science.
Is is too early to affix a “-gate” to this situation?
Boris,
It is never too early to affix “gate” to any situation. But you need to think of the correct word to affix. Cook-gate? Survey-gate? Nonrandom-gate? None sound very catchy. Do you have catchy sounding name?
Lucia,
How about we call all the psyco-babble efforts to define skeptics as either irrational or derranged (which is what this survey nonsense all comes down to) “Lewgate”? Sounds like the name of a semiprecious stone; has a nice ‘ring’ to it I think.
How about, “Lewgie”?
As in, “Cook hung a Lewgie”?
http://www.urbandictionary.com/define.php?term=loogie
I think the term covers both the intentions and qualities of what Lewandowsky, Cook, etc. have performed.
Now some, not Eli of course, might consider Brandon, what’s the word, oh yeah, qvetching, about how rude John Cook is a bit much.
Of course, Eli knows this is just the long con. Bunnies have to admit that it is impossible to figure out the dysfunction of denial, however, we often get a taste of it at Easter dinner, after the chocolate eggs have been delivered and consumed, when Aunt Lucia and Cousin Brandon begin to hold forth over the creamed carrots. Uncle Steve, of course is outside smoking a ciggie. The scams that our nearest and dearest create are scary, they are meant to be scary, scary separates the marks from their money. Rick Perlstein called it the long con and was eerily accurate (as you shall see and hear to your cost if you dare start the video at the link )
A rabett rambling on about a wicked deception regarding carrots is pretty funny.
The implications on his view of family life and the intentions of ‘the nearest and dearest’ give a lot of insight on the sad state of the rabett’s world.
And the link provided is sort of finding out that the rabett, who seldom misses an opportunity to remind all of how clever he is, is really lost in his own maze and has no idea he is even lost.
Long con? When tempted to say that the “long con” of climate alarmism is collapsing into irrelevance, we can always find another pile of fresh bunny droppings to wade through. Also irrelevant, but hard to avoid in Climate World.
I hesitate to post on this, but in support of a lost cause:
This survey can’t produce any scientific results, even by the low standards of psychology, social science, or education, for a number of reasons.
1) The terms involved are not defined in anyway that provides for a common basis for response among the attendees. That means the input(ratings by the authors), the answers by attendees, and the any results put forward by the surveyor are simply opinions with no relationships to each other. These opinions can be segregated. correlated, analyzed, parsed, graphed or whatever in any way and not generate any meaningful insights. The only real output will be an complilation of the surveyor’s biases and opinions.
2) The survey is very poorly constructed. The question(to what degree is a paper supportive of anthropological global warming theory) and the ranking given doesn’t allow any kind of measureable response. A simply 1-5 scale from 1(Strongly supports Anthropogenic Global warming) to 5(Strongly supports Natural and Unknown causes for Clmate Changes) would at least give a relatively unambiguous scale of response.
3) The same question should be asked of a random sample of trained scientists(physicists, chemists, biologists, mathematicians, statisticians, geologists, meteorologists, climate scientists, economists, probablilty, mathematical modelling, and others).
4) The survey must include metric questions about the attendee’s knowledge and skill in the scientific method.
5) General respondents should be chosen by a random method, not self-selected. If responses are solicited through web blogs or similar means the sites must be randomly selected in a balanced manner.
Probably the best method to do this survey would be to contract with Ramussen or some other competent polling company to construct and run the survey. Given the contentious subject matter and the wide variety of opinion and skill out there a sample of 10,000 plus would be appropriate. It might only cost $3-4 million, but the results would be believeable.
Based on the history of the current surveyor and his previous co-authors and co-workers any kind of scientific or meaningful result from this poll is highly doubtful.