Some of you will enjoy this interesting tidbit. Jose Duarte reports that a 35,757-year-old participated in the survey underlying the results reported in “Lewandowsky, Oberaurer and Gignac authored “The Role of Conspiracist Ideation and Worldviews in Predicting Rejection of Science”. I realize many don’t wish to be ageist and suggest the opinions of the elderly should be discounted in attitudinal surveys. But I might suggest this degree of elderly might suggest some level of survey error. (Not that anyone could seriously imagine a drunk might fill out an online survey during happy hour. 🙂 Certainly no one would suggest such a thing twice. )
Update 1/6/2014: It has come to my attention that it was a 32,757 year old participant. So not quite the fossil one might expect of a 35,757 old participant.
Evidently minors also participated. Jose Duarte reports
There are also seven minors, including a 5-year-old and two 14-year-olds.
Sharp reading skills for that 5 year old! I’d be surprised to learn a 5 year old had an opinion on whether the moon-landing was a fraud or that they knew who Princess Diana was. I’m pretty sure I didn’t even know who the Queen of England was when I was five.
Anyway, I encourage people to read more at Jose’s blog.
BTW:
The sampling method in the Stephan Lewandowsky Gilles E. Gignac, Klaus Oberauer is
Visiting uSamp and Qualtrics.com isn’t very informative.
http://www.qualtrics.com/research-suite/
http://www.usamp.com/
In particular, I would hardly say one can find “Details about the panel and the sampling method can be found on the uSamp.com web page.” for this particular study at usamp.com. Usamp.com advertises a range of services. A customer can use it in a large variety of ways. One might, for example, find “mothers of infants” or “democrats” or any number of subsamples. But that doesn’t tell us the details of the sampling method for Lew’s paper.
That said: he does say “bi-partisan”. So, presumably the 5 year old and 32,757 year old were draw from samples that included democrats and republicans.
Definitely not true in Canada when I was five. Queen Elizabeth’s coronation was an enormous event for that Canada that even five year olds were aware of. I think that we had kindergarten projects about the coronation. 🙂
This is an interesting post. I pointed out the problematic age data quite a while back (I think I even mentioned it to Duarte), but I never checked the correlation results with those entries excluded. I just took the authors at their word when they said age showed no correlation.
It just goes to show, the more you look at these things, the worse you find they are. It’s difficult to imagine how they could have done more things wrong.
Oh, on the survey methodology, I did comment on the lack of information before. There’s another issue though. Even if we could find the survey’s methodology on a website, there’s no guarantee that website will be around in the future (in its current form or at all).
We can’t even assume any archived copies of it will stick around as archiving serviced allow people to remove content for their domains (they probably have to for legal reasons). As such, archived pages could be removed by people uninvolved in the survey who simply happen to have acquired the rights to that domain.
At a bare minimum, if a researcher is going to use the description on a website as an explanation of its methodology, he or she needs to make a copy of it and store it in his SI.
Brandon
Well… yes.
I also suspect uri corresponding to the top level domain of a commercial site never provided information that describe the specific implementation in any academic research article. Any commercial site is going to use their top-level domain uri to promote their services– which both those commercial companies do. Both companies appear to provide potentially valuable services to all sorts of groups including marketers of various sorts. If you want to find out how to market your new improved baby formula to new mothers, it looks like that company might be able to link you up with new mothers more cost-effectively than many other means you could try. But in such cases, a baby formula vendor might not need to explain the methodology in great detail to anyone outside the company. They might just use it to decide on a marketing strategy (e.g. soprano vs. tenor voice in commercials etc.) and then hope for the best.
When I was five, England still had a King. And the coronation of Queen Elizabeth II was a fairly big deal in the US too, as I remember. There was even same day television coverage on the relatively tiny screens of the day.
lucia, that URL definitely did not ever contain any sort of explanation. I visited it not long after the paper came out. I looked everywhere I could on the site, and I couldn’t find relevant information anywhere on it. I found some general information on an entirely different page, and I could probably hazard guesses for most questions based upon what it said, but that’s it. I don’t even know if you could have gotten all the relevant information while hiring them to perform the survey for you.
Strangely, untraceable references seems to be a under-discussed problem in science. It may just be the fields of science I’ve paid attention to, but I’ve encountered similar issues even in things completely unrelated to the global warming debate. When I first got interested in high-energy physics, I tried tracking down references used in some papers I read, and it was bizarre. In one case, I found a reference to a paper which didn’t exist (as confirmed by the supposed lead author of the paper). In other cases, I found references to papers which didn’t even discuss the topics they were referenced as discussing. In one case, I even found an equation said to have been taken from a particular reference which only vaguely discussed the subject. I later found the equation in the author’s PhD dissertation, described as a new innovation. Apparently the author decided, five years later, to give credit to the equation to someone else so his work seemed like it was built upon a broader body of work.
I’ve talked to people who say they generally won’t check paper’s references when reviewing them. They told me the only time they do so is if they think there may be a problem, or they just think the reference might be interesting to read. I find that baffling, even though I understand their reasons.
Damn! DeWitt Payne beat me to it.
When I was five our Queen was definitely a King.
And like in Canada, Down Under, the coronoation of QEII was a Big Deal.
I vividly recall the ancient globe at primary school being painted gold to become the base of the orb for our crown jewels. This was a great pity. That globe was probably a collector’s item; Tsarist Russia was coloured bright green.
A bit wrinkly, but marvellous for her age!
here is the point blank refusal by the Vice Chancellor the University of Western Australia, to provide data for the similar survey for LOG12 – Lew et al (The earlier – NASA Moon Hoax – paper)
This request for the data was for the stated purpose of submitting a comment to Psychological Science (which had been suggested to me as the thing to do, by the Chief Editor – Prof Erich Eich)
From: Paul Johnson
Sent: Friday, March 28, 2014 8:08 AM
To: Barry Woods
Cc: Murray Maybery ; Kimberley Heitman
Subject: request for access to data
Mr B. Woods
Dear Mr Woods,
I refer to your emails of the 11th and 25th March directed to Professor Maybery, which repeat a request you made by email dated the 5th September 2013 to Professor Lewandowsky (copied to numerous recipients) in which you request access to Professor Lewandowsky’s data for the purpose of submitting a comment to the Journal of Psychological Science.
It is not the University’s practice to accede to such requests.
Yours faithfully,
Professor Paul Johnson,
Vice-Chancellor
http://climateaudit.org/2014/03/30/uwa-vice-chancellor-johnson-circles-the-wagons/
—————————————-
yes – I was surprised he put that in writing..
Disappointing for me? (yes)
Devastating for the credibility of the University, journal and the field of psychology?
I leave Lewandowsky et al out of that list as it appears that they cant help themselves (activists) but the academy should protect it’s own credibility?
My request, that prompted this response is here:
http://unsettledclimate.org/2014/04/05/i-requested-data-from-the-university-of-western-australia/
From the earlier and similar survey- LOG12 paper – Lewandowsky et al (Psychological Science – APS)
——————————–
“An additional 161 responses were eliminated because the respondent’s age was implausible ( 95 years old), values for the consensus items were outside the range of the rating scale, or responses were incomplete. This left 1,145 complete records for analysis.” – LOG12
———————————–
Thus, in the flagship journal of the APS (Psychological Science) , it seems that 11-17 yr olds are both plausible and appropriate…..
How many of the tiny number of respondents that believed in the conspiracy theories, were minors?
We don’t know, Lewandowsky et al, UWA, APS, Psychological Science will not release the data (raw, all respondents, including age, gender, referring url). If they can’t and they’ve lost it, well that is just as bad.
“Median age of respondents retained for analysis was 43.0…”
But the mean was probably around 75 !
Brandon, Lucia, I have had the same experiences tracing IPCC claims. For a most illuminating example, see essay No Bodies (concerning AR4 extinction prognostications) in ebook Blowing Smoke. You were both gifted copies.
A personal anecdote about where I learned this. My required short thesis at the Harvard Law/Harvard Business joint program was on an obscure issue in antitrust law. Typed on a manual Smith Corona using actual carbon paper ( electrics and computers existed, but I could not afford them then). My supervisor, an old HLS prof, took my draft, went into the Langdel law library, and checked every one of the about 100 case citations. He found them all, but noted back to me (with major minus marks) that two of the typed citations were incorrect and had caused him to waste valuable time fixing/finding. A life lesson.
Forget the ideation issue, how could Lewandowsky not recognize the climatic opportunity he had before him?
I am sure this 32,757 year old participant would at least remember some of the more substantive climate changes over his lifetime.
See the twitter discussion at https://twitter.com/STWorg/status/552601648523382784.
I have a belief that the quality of a paper is directly related to the amount of effort one puts into it.
IMO starting with an internet survey immediately indicates what kind of person is producing that paper and what level of quality is likely to be achieved.
Someone (an American) born on 27th March, 1957, entered date of birth instead of age.
Must be an American as 032757 uses the illogical leading-month format that is neither big- nor little- endian.
What information could you hold about participants to allow you to perform propensity weighting without knowing the participants age?
Steve Ta,
Out of curiosity, what does ‘propensity weighting’ mean? I looked up wikipedia.
http://en.wikipedia.org/wiki/Propensity_score_matching One of the sentences said:
But there is no “treatment” in this study… right? So how does one do “propensity” weighting.
I’m no expert, but I understood this to mean that the “random” sample was drawn with the aim of ensuring that the known overall variables of the population (age distribution, sex ratio, urban/rural, red/blue, etc) were reflected as closely as possible within the sample set used for the survey.
I don’t see how that pre-selection can happen without already knowing details of the participants.
See here for a medical example:
http://www.jds-online.com/file_download/94/JDS-233.pdf
SteveTa
I figure they did what you claim. I’m just not sure that’s ordinarily called “propensity matching”. (I don’t work in that area. So I can’t claim my not knowing means anything.)
The paper still you link discusses ‘treated’ and ‘control’ groups– just like wikipedia.
Later one:
Seems to me even if there aren’t any “treated” or “control” groups, there needs to be a formal definition of two groups whose “propensities for X” the researcher tried to match.
Who are the two groups? Men/women? Dems/Reps? Old-timers/Young-uns? What “propensity for X” matched between the two groups? (Love of chocolate? Attraction to exercise? risk of diabetes? Height? Hours watching tv or internet use? Hours standing in grocery store lines reading headlines about Princess Di? )
It can’t help but think that if you say you “propensity matched” you need to say quite a bit more so that people know what you tried to match.
I read the “propensity weighting” sentence the same as Steve Ta (#137272) above, that the sampling organization ensured that the sampled population was representative. “Participants were drawn from a completely bipartisan panel of more than 5.5 million U.S. residents (as of January 2013), via propensity weighting to ensure representativeness. The panel from which participants were sampled is maintained by uSamp.com. Details about the panel and the sampling method can be found on the uSamp.com web page.”
In other words, having demonstrated their incompetence at running a survey themselves, they outsourced, and then repeated some marketing claim.
.
There’s nothing about control groups in the paper, so the Wiki definition is not what LOG13 intended by the term.
.
Do you really think it credible that 2.6% of Americans believe “the Apollo moon landings never happened and were staged in a Hollywood studio”?
[That’s a rhetorical question, so I provide my answer: no. I know my acquaintances are not a representative sample of America, but they cover a fairly wide range of (to my mind) outlandish views, and I don’t know one who would answer “yes”.]
Off-topic to Lewandowsky, but relevant to the believer/skeptic divide (and more convincing than conspiracy ideation) is this post mentioned at Bishop Hill.
Using the internet to obtain survey results where participants are 32,757 years old reminds me of this story (this is satire):
http://www.theonion.com/video/teen-boys-losing-virginity-earlier-and-earlier-rep,35906/
Carrick,
Yeah. Survey’s in general are difficult.
The cited commercial services describe their methods and clearly try to impose quality control. Usamp tries to recruit useful participants, but really how do you do that properly? http://www.usamp.com/panel/#recruiting
Here’s an article where they are trying to get appropriate recruits for B2B surveys
http://smallbiztrends.com/2012/03/usamp-announces-thewhiteboard-b2b-panel.html
It sounds like they are set up to over meaningful ‘rewards’ (virtual visa cards, frequent flyer miles etc.) I suspect they probably also verify identities with something like that.
But how do you create meaningful but affordable ‘rewards’ for people to fill out academic surveys. How much were Lew and crew going to pay out? $20/person? That would be $20* 1000 to uSamp; less to the participants. $1/person? Who are you going to get any sort of “unbiased” sample for that. No American with any sort of real job is going to sign up for gigs to fill out surveys at $1/person.
Meanwhile, low income people might want to figure out a way to set up 10 sock-puppets and get 50 invites a day to make $50/day. Given the reality of the internet, it’s likely a group of people could figure out how to set up all the fake identities, keep track of them and so on. I wouldn’t be surprised if someone wrote an app for that! It might even “auto submit”. Who knows? And if a few sockpuppets got caught, who cares. They just make a new one.
Carrick,
Some pretty impressive numbers in that Onion survey, but I don’t think I saw 97% anywhere.
Looks like they may have some more work to do.
I googled a bit:
http://freemoneyoffersonline.com/?p=407
This is an October 9, 2014 blog that suggest one join “Points2Shop” which an interface to permit you to apply to fill out surveys for “USamp”. I don’t know what “Points2Shop”‘s business model is, but presumably, they make money as a result of people signing up for USamp surveys through “Points2Shop”.
I also found
http://mysurvey123.com/surveyhead/
Which discusses a survey system associated with USamp.
Based on the 2nd ones the fees look high enough to attract some people: better than flipping burgers. So I imagine some would sign up and be motivated to not risk getting dumped by joking around and saying they were 32,757 yo.
Whether it’s possible to scam using sock puppets…. this is the internet. I would never want to suggest the answer must be “no”.
I take back the idea ‘http://mysurvey123.com/surveyhead/’ might be better than flipping burgers. To check I:
1) Signed up.
2) Was accepted.
3) Visited their dashboard to see survey offers. Most surveys pay tiny amounts. (e.g. one is $0.75 for 20 Minutes). Some appear to pay ‘better than flipping burgers. I clicked for one that was something like $3 for 10 minutes, if you could get 6 of these that would be $18/hour, which is definitely better than flipping burgers.
4) Clicked to take a survey I “might” qualify for. It turns out one first fills out a survey to qualify to take a survey. You are paid nothing for the pre-survey. I spent a few minutes on that and did not qualify. (Why? Dunno.) Anyway, the pay out was $0/n minutes. I tried a second one– same deal. $0/n minutes. Clearly, real hourly rate needs to be discounted based on the time required to fill out the pre-qualifying surveys and likelihood one would be accepted. My guess is that people who have access to money through other means and/or who are in demographics that are not in high demand will avoid these places.
What the heck, I’m going to try the $1 for 2 minutes.
John M:
Good point. The science is clearly not settled here.
Based on Lucia’s results of trying to make money taking surveys, I would suggest that the survey audience consists of people with too much time on their hands and who can’t tell that they are making almost no money doing this. That is, likely not the employed or busy moms, likely not blue-collar who would be unlikely to spend that much time online, and likely not older than 65 for the same reason. This is therefore far from a balanced survey audience. Maybe ok for trying out T-shirt designs or the latest music, but science and policy opinions? No way.
Craig,
Depending on the competition between script-kiddies and those running the survey, a sizable fraction could be script-kiddies (that is ‘bots’). The issue will come down to balancing whether any ‘real people’ are rewarded vs whether any ‘script-kiddies’ are rewarded.
On the “real people” end: Given how annoying the ‘pre-survey’ before the ‘pay survey’ issue is, and how it can potentially really cut into the effective pay level, how many people are attracted?
On the “script-kiddie’ end, the question whether script-kiddies find spoofing people ‘do-able’ and at what level of scripting and resources.
I have no idea on the latter. But I would never confidently claim script-kiddies can’t trick an online system, and I would tend to suspect they can.
My conversation with Cleverbot:
me:Did america fake the moon landing?
You can drink in America too, you just have to be responsible about it.
me: Is climate change real?
No.
Me: Are sceptics funded by the fossil fuel industry?
I have no clue what that means.
don’t know if script-kiddies are smarter than cleverbot, but it DID get the second question right!
Maybe gold farmers in China and India could take this up.
Maybe Lewandowsky’s new research partner is Mel Brooks?
https://www.youtube.com/watch?v=XOTKDgrdvdg
I looked closely at this data set last year and earlier this year and drafted a comment on the Lewandowsky Hoax, which I’d been meaning to tidy up, but found the whole project so annoying that it’s hard to finish.
Obviously one of the large issues in the original Lew survey was fake responses. But there are fake responses in other surveys and to get a rough benchmark, 4% of the respondents to the 2013 PPP survey in the U.S. purported to believe the “reptilian” conspiracy.
This number is much higher than the percentages of nonsense responses in the climate blog surveys – either Lew’s survey of green faction blogs or the WUWT survey.
Isn’t Mel Brooks the 32,757 year old man?
Jeff,
Please see the link at my (Comment #134329)
Even at a mere 2,000 years, Mel Brooks offers more insights than Prof. Lewandowsky will ever get from his 32,757 year old man. And Mel Brooks is of course vastly more entertaining.
https://www.youtube.com/watch?v=XOTKDgrdvdg
Lew came out of the extremist left-wing closet on that one didn’t he!? Msnbc direct wired to his brain.
Lucia, I forgot to wish you a happy 200th anniversary of the Battle of New Orleans. Guess you guys got the last laugh, even though technically it was after the war ended. Have a good weekend!
AJ,
Thanks! Well… winning a battle after the war is over is definitely the last laugh!
I hear we had help from an alligator.
AJ “… even though technically it was after the war ended.”
Actually, it WASN’T after the war ended. The treaty had to be ratified by congress to be official. It was finally ratified February 16, 1815, right after the proposed treaty reached the U.S.
A few years ago my wife and I took a cruise up the East Coast from Boston to British Columbia. It was during that cruise that I learned of Laura Secord being the Canadian equivalent of Paul Revere- from a Secord chocolate candy display.
Reading up a little on the war lately, I see there were two parts.
In Part I, we, the US, figured we could annex Canada while Great Britain was preoccupied with France. The home team Canadians and British won. Later, when the Canadians and British had suppressed the French, and sent 15,000 troopers to North Amereica. This time, when the Canadians and British attacked the US, the American won. I guess the war was somewhat like pro sports, where the home team usually wins.
“A few years ago my wife and I took a cruise up the East Coast from Boston to British Columbia.”
Through the Northwest Passage?
Sorry, I meant Boston to Nova Scotia.
I was conflating this in my memoy with another trip to Vancouver Island.
Usamp is actually one of the better commercial internet access panels. A company I worked with a few years ago rated the major panels and found Usamp as one of the ones that were most consistent.
I can’t imagine that they would ever publish the details of a survey such as this. They would consider it confidential.
Because you cannot get a random sample for internet surveys, when you want a survey to reach what you think is a representative group you have to create what used to be called a sample frame, a matrix of quotas starting with age, gender, ethnicity and moving as far as you like to include income, household size, religious preference, etc., etc. Evidently they’ve changed the name in the never-ending quest to make jargon more complex…
The more complex the matrix the higher the cost, of course. Lewandowsky, not Usamp, would be the one to disclose what he paid for.
Qualtrics is the low cost survey engine of choice for those who recognize the problems with Survey Monkey but don’t want to pay for the high cost, high time requirements of Confirmit or Askia.
Qualtrics doesn’t fall over in the field and you can learn how to use it fairly quickly. Like all survey engines it takes some effort to get familiar with, but there is no excuse for what Duarte noticed in Lewandowsky’s survey. That is to say I can easily believe Lewandowsky would not set the age requirements efficiently. I have yet to see him do anything efficiently. However to ignore the consequences and continue to report on the age of respondents as if all was well is hilariously bad judgment and bad practice.
The research efforts of various social sciences generally have a poor reputation for due diligence, competence and even relevance. Lewandowsky is providing a case study on why that is so.