Today, Rasmus Benestad explained why Nicola Scafetta should share code to reproduce the results of Scafettas 2006 GRL paper. Rasmus gives some background, explains why he wants the code and relates:
…So I asked him to post his code openly on the Internet so that others could repeat our test with their code. That should settle our controversy.
After repeated requests, he told me that he doesn’t really understand why I’m not able to write my own program to reproduce the calculations (actually, I did in the paper together with Gavin, but Scafetta wouldn’t accept our analysis), and keeps insulting me by telling me to take a course on wavelet analysis. Furthermore, he stated that there “are several other and even more serious problems†in our work. I figure then that the easiest way to get to the bottom of this issue it to repeat our tests with his code.
A replication in general doesn’t require full disclosure of source code because the description in the paper should be sufficient, though in this case it clearly wasn’t. So to both save having us do it again and perhaps miss some other little detail – in addition to using an algorithm that Scafetta is happy with – it’s worth getting the code with which to validate our efforts.
It should be a common courtesy to provide methods requested by other scientists in order to speedily get to the essence of the issue, and not to waste time with the minutiae of which year is picked to end the analysis.
Rasmus writes more. But the long and short of it is: If Scafetta should share the code. If Nicola needs a server to host the code, I would be happy to upload it to my server and provide a link here on the blog. If Nicola remains unwilling to post his code on the internet, I would also be happy to let Dr. Scafetta write a blog post explaining why he thinks the code should remain a private secret.
I should note that in the previous bout on this issue, Rasmus sent me his code in R. So, Rasmus is expecting no more of Scafetta than he expects of himself. Rasmus is a scientist who is willing to send almost complete strangers code.
Is there something missing there? Why is Rasmus expecting more of Scarfetta than he expects of himself?
I’m guessing that Lucia meant that he “expects no more…”
Ouch! That should read “no more”. Rasmus is not expecting more of Scafetta than himself. When I asked Rasmus for code, he sent it to me. No questions asked.
hi Lucia,
I posted a message to RealClimate stating, as a climate change skeptic, I fully agree that Scafetta et al. should release their code. I then asked, does this mean that RealClimate scientists will also release their code in future, even to Steve McIntyre?
That post was deleted, so perhaps the message can be made here.
Best,
Alex
While I agree in ‘open code’ and ‘open data’ there is a little more to the story – there is a history between them. Maybe if Rasmus had been a little nicer at RC, Scafetta might have been more willing to share (maybe not too).
http://www.realclimate.org/index.php/archives/2007/11/a-phenomenological-sequel/
Scafetta was a tool on CA when we discussed this in early august. Junk science. People just need to ignore papers that dont post code and data.
I have to think there is a certain amount of irony in someone at RC asking for openness in code and data!!
Still, he needs to give it up!!
Nice job. Don’t piss off the Lucia.
I emailed him Scafetta several times explaining what it would do for the credibility of his work. He told me it was too simple or something like that.
Ya know, I’m sick of that sh..
Steig was wrong, Mann was wrong an Nikkie was wrong, CRU was wrong, Santer was wrong, Jones was wrong, Comiso was wrong, on and on.
When people need to hide their work, it should be pulled from publication.
Comment#28564Are the first two sentences really necessary?
I second what Mosher said. I also recall him being immature and playing games rather than answering a question straightforwardly on CA.
Thanks Calvin.
I reread that thread and just confirmed my impression of him. Most everyone was over polite to him.
jeffid,
ya, scafetta’s afraid to share his code with a girl. he needs to pull his big boy pants on.
[;beginsnark]
Well, gee, why should Dr. Scafetta reveal anything to people who just want to find something wrong with his work?
[;endsnark]
Pardon me but could someone be kind enough to give a brief overview of the issues here? I am unfamiliar with Scafetta et.al. Thanks
I agree – Scaffetta should provide it all. Now.
Yes, Scafetta should provide his code ( I also spoke out on this when it arose previously and emailed Scafetta to do do so). But, Lucia, surely it is totally hypocritical for a realclmate author to make such demands without acknowledging and apologizing for past sins by realclimate coauthors. I requested source code for MBH for precisely the same reasons as Rasmus and Gavin now want Scafetta’s source code and have been subjected to much abuse for this. The reasoning in my original correspondence reads almost exactly like Rasmus’ now.
Last summer when this incident previously arose, I wrote to Gavin Schmidt, then fresh from his conversion to the merits of source code as a guide to obscure methodology, and asked him to request clarifying code for MBH retention of principal components, a battleground issue in Wahl and Ammann 2007. Gavin’s conversion was not complete enough to get the clarifying code from Mann or, for that matter, to even acknowledge the email.
And surely it is time for realclimate to now take a lead in abandoning the untrue meme that the work programs of legions of climate scientists were ever disrupted with unreasonable FOI requests for clarification of methodology.
Yup, he should release it, as should Mann (MBH), Steig, Jones, Santer, and others. If they have nothing to hide, then don’t hide it (even if it’s ugly spaghetti without adequate explanatory comments).
SteveM–
Oddly enough, I evaluate each RC author and each demand for code individually.
Rasmus is not Mann. The code he requests is recent, evidently simple and can lay no claim to being the sort of IP that Scafetta might later sell and commercialize for big bucks. Scafetta should hand it over.
If Mann were the one asking for code, I would make snide remarks about Mann.
I know Rasmus wants this for work being done jointly with Gavin. I agree that many of the ‘free the code’ people would be more impressed if Gavin were to come out and say that Mann should fork over his code for the same reasons Rasmus outlines in his code request.
That said:
1) Scafetta should fork over the code and
2) I don’t think Rasmus is hypocritical.
Is there a group more emotionally underdeveloped than the group of hairdressers we call Climate Scientists? Why, they’d ruin Christmas Morning for Everyone if they didn’t see a big fire engine under the tree! Dysfunctional, anyone?
(Rhetorical) lol
Andrew
Andrew_KY–
You need to think up better names to call people. This is not a hairdresser:
Reproducibility of analysis and modeling methods using complex software systems raises the core issue of how to keep the objectivity of science in the computer age. This spans both academic and commercial worlds.
Posting the data and code is one method of open peer review. The other method is to provide objective test data and for the contending software codes to demonstrate how well they model that data.
The latter method is being used in testing commercial CFD codes. There are numerous papers addressing “CFD code validation”. e.g.
Comprehensive Approach to Verification and Validation of CFD Simulations – Part 1: Methodology and Procedures
J. Fluids Eng. — December 2001 — Volume 123, Issue 4, 793 (10 pages) doi:10.1115/1.1412235
Comprehensive Approach to Verification and Validation of CFD Simulations – Part 2: Application for Rans Simulation of a Cargo/Container Ship J. Fluids Eng. — December 2001 — Volume 123, Issue 4, 803 (8 pages) doi:10.1115/1.1412236
Similarly, see NIST on CFD validation.
NIST has established an experimental combustion database and methodology to verify CFD codes:
Widmann, J. F., Charagundla, R. , Presser, C., Heckert, A. N.
Benchmark Experimental Database for Multiphase Combustion Model Input and Validation: Baseline Case February 01, 1999 NIST Interagency/Internal Report (NISTIR) – 6286
A Benchmark Experimental Database for Multiphase Combustion Model Input and Validation. A Users Follow-up
Cary Presser*, NIST Gaithersburg.
In this present case, Rasmus says he can’t reproduce Scafetta, and Scafetta appeals to textbook wavelet methods.
Some first steps to address this objective performance issue is to apply the CFD validation methods. Thus have both codes demonstrate their capabilities of reproducing frequency from open data. e.g.,
* 1) Demonstrate established benchmark tests.
* 2) Demonstrate analysis of synthetic data created by open code simulating the sun.
* 3) Demonstrate analysis of posted real solar data.
Let the software duels begin.
Lucia,
Wow. If I had a head of hair like that… look out, ladies… Dan Haggerty II is on the prowl, and I been lonely out in the mountains a looong time… 😉
(Truly Awful, I know)
Andrew
The other side of this story is the ClimateGate saga and the difficulty Steve McIntyre and others have had in reproducing results by “the Team” who have systematically refused to provide either data or code. So I sympathize with the demand for code and data when not able to reproduce the results. In this case there is the issue of reproducing Scafetta’s paper. Is the description and data availability sufficient? If not, is a corregidum or supplement needed to fill that in? Then with a reproducible method, does Scafetta’s code do what he claims?
Officials in New Zealand are stonewalling efforts to get answers to requests for info on the adjustment process for their temperature data. The foot dragging will not be successful and will only make them look guilty of trying to hide something unsavory.
I predict that we will see some politicians somewhere issue subpoenas and force some folks not only to release all their records, but also submit to careful cross examination about precisely what adjustments have been made and why. The genie is out of the bottle and there’s no way to put it back. The momentum is too great. It’s all going to come out.
Free the data, free the code. It’s fun to watch climate “science” dragged kicking and screaming into the realm of real science. Don’t be surprised if some “scientists” end up having all their work examined under a microscope down to the smallest nit. For some, disaster awaits. The sense of foreboding must be terrifying.
Hmm. Anyone else note the backtracking and hedging by Gav in his responses at RC (including in his reply to Mosh). The problem is being redefined as not that the code is not available but that the description in the original article was not detailed enough to allow someone to produce their own code. Gav is being a little sneaky here and is holding onto an out for the next time a mainstream climate scientist is criticized for not releasing code. I assume he is also trying to immunize himself from charges of hypocrisy for past behaviors (so where exactly is that calculation of the MBH confidence interval, Gav?) Way to have your cake and eat it too.
brid (Comment#28597),
As expected, Gav et al want the right to decide when release of code is appropriate… releasing code would of course NEVER be required of them. Goes along the same lines as UEA emails showing concerted efforts to block publication of “bad” research (AKA what the Team doesn’t agree with)….. they get to decide what is “bad” and what is “good”, and to hell with everyone else. Gav probably just doesn’t understand how bad his support for not requiring disclosure of all code and data makes him look.
Lucia,
“This is not a hairdresser”
No, but he could use one.
brid (Comment#28597)
I’ve advanced this precise argument with stevemosher: Sometimes discussing the algorithm is sufficient. But if there is any level of complication, the discussion in papers in often insufficient from the point of view of a reader who really wants to implement. This appears to be the case with both Mann papers and Scafetta papers.
It’s worth nothing that SteveM has given precisely Gavin’s argument as the reason Mann should fork over his code. It has also been the precise argument for people asking for the CRU code.
SteveM has tried to follow the discussion in various Hockey stick papers and replicate. He found insufficient detail to be certain what they did. He asked for code, and was refused. Mann should not have refused on the basis that he or even those in his tribe have decided they discussion is sufficiently detailed. Mann should just fork over the code.
Now Gavin and Rasmus have tried to to implement Scafetta’s method. They can’t and find insufficient detail to be certain what Scafetta did. They asked for code and were refused. Scafetta should not have refused on the basis that he or even those in his tribe have decided they discussion is sufficiently detailed. Scafetta should just fork over the code.
It would be nice if Gavin could be a bit more objective and see this. But, evidently, for some reason, he’s got blinders on with respect to the Mann and hockey stick papers. What Rasmus thinks of the Mann situation, I do not know. I guess I could ask.
But the fact is: Scafetta should give his code to Rasmus, period.
lucia (Comment#28603),
Since nearly all climate research is pubically funded, the IP rights already belong to the public. Would you not support a simple law that requires complete public disclosure of all code and data that results form pubilcally funded research? After all, in preparing a paper all the work is already done, all you would have to do is include the code/data as part of the supplemental information. I don’t even see this automatic disclosure as a burden.
SteveF-
I think it’s more complicated than that. You would need to consult an IP attorney. However, I think the Army who funded the work should stipulate that short data processing codes for papers published in peer reviewed journals or cleared reports filed as reports to the sponsor should be made available on request. Also, the journals should require that authors comply with request. If necessary, the journals should set up a database where outsiders can make a formal request, and the author can either post the requested information publicly, or explain the reason for denying the request publicly. Some sort of rating system for turn down should be ginned up. (After all, if no one has requested code during the first 30 years, and several authors are now dead, it’s not too surprising if the answer is “Code not available because computer jockey graduated, moved, was hit by a car and his advisor lost his set of floppies from back in the 80s. On the other hand, Rasmus initially asked for code within… oh.. 1 month of the paper’s publication. Scafetta has not claimed the code is lost.)
Of course, if the journals are going to require sharing code, they should be even handed. If GRL doesn’t require Mann to behave well and share code with SteveM, then we can’t expect GRL to force Scafetta to behave well and share with Rasmus.
What hypocrites.
When is RC’s Steig going to release his code? (when Jeff asked, they just sent him the Regem manual).
When is RC going to call for CRU to publish their code?
And they are still deleting my comments.
NASA / GISS ‘s Gavin Schmidt is playing word games again.
There is not sufficient documentation of the ModelE models, methods, and code anywhere to support replication.
And the constant repeating of Go read the source code is disingenuous to the max. There is not sufficient documentation of the code to support correct and complete understanding.
And so too is Go use the code. Such complex software relating to complex phenomena and processes requires training and months (years?) of experience to become a reliable user sufficient for getting correct results.
First off, to prevent flaming:
MBH should release their code and data too.
—
Dan Hughes:
The Model E and GISTemp code is available. I don’t think that complaining that the code is too hard to read and use is valid. It’s not an office application or game that your grandparents are supposed to install and use.
Have you ever tried to install and use SQL Server, Octave, or R? They are all complex and require months or years of experience to become a reliable user. Sometimes the steep learning curve can not be avoided.
—
David L. Hagen:
I agree with your statements about benchmark testing and testing with synthetic data where the real answer is known. Unit testing is a required practice in good software shops.
Lucia,
I agree that Scafetta should hand over his code, period. I don’t think you will hear much argument on this here. However, I take Mosh’s side on the complexity question. If someone is publishing in a serious peer-reviewed technical journal, then almost by definition there will be complexity. But on a more philosophical basis, why leave this up to judgment? In one of his responses, Gav says: “However, replication in general doesn’t require turnkey code for every random paper. If someone says they used linear regression, you can do that yourself.” Sounds reasonable, no? But we know in practice that climate scientist have used this excuse for items far more complex than a linear regression. (And even in linear regression, there can be issues: remember the kerfuffle between Hugh McCulloch and Eric Steig over correcting for autocorrelation.) As soon as you open the door a crack, people will drive a truck through it.
And if something is a really simple algorithm, then it should be trivial for the author to provide code. It makes one wonder why people fight this so vehemently.
I come from the economics / finance world where disclosure of code is standard. The American Economic Review rule has been cited here (and at CA) numerous times, but I will repeat it: “For econometric and simulation papers, the minimum requirement should include the data set(s) and programs used to run the final models, plus a description of how previous intermediate data sets and programs were employed to create the final data set(s).”
Let’s hold climate science to the same standard.
Shop rights. As a general proposition, people employed to do work don’t have a claim on the work product – although most anything is changeable by contract, and I’m not exactly sure how things work in the academic setting, but I see that MIT has a “technology licensing office.” Has anyone ever taken up these issues with the offending academic’s department head or whatever dean controls?
Lucia,
“I think it’s more complicated than that. You would need to consult an IP attorney.”
I don’t think it is. Congress passes a law that says something like:
“Any person or organization that accepts funding from the United States or from agencies funded by the United States, for the conduct of research in any field, shall, as a condition of receiving such funding, agree to make available to the public all raw data and software source code used in the preparation of any and all publications or presentations based on said research. Disclosure shall be concurrent with any publication or presentation of research results, and all disclosed raw data and software source code shall thereafter be freely usable by any person or organization.”
Easy, and no IP attorney need be involved. You may need to add a bit more to protect the privacy of individuals in human studies, but that’s about it. If some researcher (say Dr. Sander or Dr. Steeg) wants to treat his data and code as private property, then this option remains available… with private funding. Taxpayers should get what they pay for, nothing less.
JohnV
Here is my standard question that has yet to be answered by anyone.
Where is the documentation that reports the final form of the continuous equation for the momentum balance in the vertical direction for the ModelE code.
Where is the documentation for the discrete approximation for that equation reporting the time levels for each term in the approximation.
What is the real-world-application truncation order for the ModelE code.
If you can’t provide papers and reports maybe you can point me to the source code routines that supplies answers. No one has provided this information. I’m certain that you will be the first.
It is not necessary to attempt to reverse engineer source code, 100% probability that the outcome will not be correct, to understand the soundness, or lack thereof, of the models and methods. Written documentation is much more efficient and accurate.
Pointing to the source code as a source of information about the continuous equations is complete stupidity.
Apparently you have no problem accepting calculated results the basis of which you have absolutely no idea about.
ps
It requires less than 5 minutes to install and correctly operate R.
Not much. With the single exception of national security (which no one can claim with a straight face has anything to do with climate science), there’s no good reason to keep any of the research secret. The only place the two overlap is in the IR codes, but that’s a rare exception to the rule.
Dan Hughes and JohnV,
I did download “ModelE1” from
http://www.giss.nasa.gov/tools/modelE/
I am taking a look at it just for my edification. You might want to try the online viewer:
http://www.giss.nasa.gov/tools/modelE/modelEsrc/
which will let you do some structured code viewing without even downloading.
As to running this beastie, I think I would need a severe upgrade and the data (at same site) is pretty huge for a PC – 182 MB zipped, From what I have looked at, (not much yet) I have some questions as to file formats (not all text, no file suffixes) and provenance (latest files are Jan 2004).
And here is another mystery:
1. Did you want wheels with your truck?
2. The clouds thing again.
And another:
However, that link leads to a blank page.
In any case, there is over 4 megabytes alone of fortran source code (might take good analyst a year to learn if said analyst was well versed in the science). And then there is the matter of the data used for starting conditions.
Now wait for it…
I’m Your Density
Do you guys remember the story of Archimedes in the bathtub when he yelled, “Eureka!”?
As far as I can tell, there has been no such equivalent moment in Climate Science (they are not running naked through the streets)… but there are many squiggly lines.
What I see is a tribe of world-class criminals trying to sell the soft-brained.
If you disagree please tell me where my analysis is incorrect.
Andrew
Dan Hughes:
Installing R is easy, I agree. Correctly operating it without the appropriate background — not so easy. But back to the real subject at hand…
.
My comments about complaining that the code is “too hard to read and use” were directed at your statement:
.
I have no problem with complaints about lack of documentation in the code or other publications. Those could be valid complaints. Just don’t complain that it’s hard to use and has a steep learning curve.
Lucia, the Climategate Letters cast new light on Mann’s refusals.
As to Osborn or Briffa (“trusted colleagues”), Mann did not take the position that the descriptions were sufficient to permit replication. He sent code for Mann and Jones 2003 to them when they couldn’t understand what he did. He provided MBH residuals (“dirty laundry”) to Tim Osborn that he refused to provide me and lengthy explication of the MBH confidence interval calculations, including statements that certain columns of online information (ones that puzzled Jean S, UC and me) were unintelligible and irrelevant.
However, if you are not a “trusted colleague” who can be relied upon to handle “diry laundry” discreetly, then you are told a different story.
Andrew_KY:
The anonymous accusations of fraud and criminal activity get a little (lot) tiring. If you’re going to accuse someone of fraud, have the courage to put your real name behind it.
.
Are you really complaining now that climate science is not based on a single eureka moment? It seems to me that a slowly growing and evolving body of evidence over 100+ years of research is a lot more convincing than a single flash of brilliance.
JohnV,
Why can’t I complain? And Andrew is my real name. Do you want my credit card info too?
Since you admit there has been no Eureka Moment, has there been anything close to one in Climate Science? What was it?
Andrew
Andrew_KY:
I’ll keep it simple:
– Complaining is ok
– Libel is not
.
I’m ignoring your eureka tangent unless you can explain why you think eureka moments are more important than the slow, steady, and patient work of science.
JohnV,
Don’t throw accusations of Libel around unless you are ready to put your real name behind it.
You weren’t ignoring my Eureka Tangent before. It’s a problem now, I guess, since you won’t answer.
Anyway, I’m looking for the major discovery that you think supports AGW theory. Do you have the reference handy?
Andrew
JohnV, OIC now, that was Yet Another Naked Strawman (YANS).
I said nothing whatsoever about “too hard to read and use†and I did not “complain”. As your lack of providing specific information indicates, apparently I simply stated the correct state of the situation.
Those are your words, not mine. Kindly remove the quotes when you refer to my comments.
And YANS is the universal response whenever I’ve asked the questions.
I think we can question if anyone knows the continuous form of the equations used in ModelE. And safely assume that the discrete approximations and the solution methods applied to these are black boxes.
“I’m looking for the major discovery that you think supports AGW theory. Do you have the reference handy?”
CO2 is a GHG?
CO2 is rising?
CO2 is rising due to fossil fuel burning?
Current temps are rising?
As temps rise, there is more water vapor in the atmosphere?
As temps rise, more ice melts, decreasing albedo and increasing surface temperature?
The upper stratosphere is cooling as predicted by an enhanced greenhouse effect?
Major discoveries: take your pick.
SteveF–
But they haven’t passed such a law. If you want them to do so, you will have to contact your congressman and suggest they do so.
SteveMcIntyre:
I agree that Mann has no reason to not send code. The fact that his reason isn’t even the notion that you could replicate based on the contents of the paper makes thing even worse.
Mann has behaved badly. Abysmally in fact.
Gavin looks very bad suggesting Scafetta should hand over code while simultaneously explaining that Mann need not.
It would be nice if Rasmus came out and said that he thought Mann should hand over code–but at least, as far as I’m aware, Rasmus is not posting explanations why Mann is exempt from providing code. We don’t know Rasmus’s opinion on this because he hasn’t told us.
Off thread, and my apologie$ for that but talk about from the ridiculous to the sublime:
http://www.crookedbrains.net/2009/01/knitting.html
Vandalism with the woman’s touch!
Today, Rasmus Benestad explained why Nicola Scafetta should share code to reproduce the results of Scafettas 2006 GRL paper
The question should be asked why is Rasmus asking for the code at this present time ?(at the “peak” of the Copenhagen conference)
Machiavellian intrigue,would suggest a preemptive strike (poisoning the Well) is the reason.
The review by Lockwood 2009 (proceedings of the Royal Society) seems to bring substantive questions (and indeed paradox) to the TSI reconstructions, and here R&S seemed to have painted themselves into a corner, where the downturn in TSI eg Froelich 2009 would allow the Agw community to explain the T plateau of the 21st Century,but this would presume a higher sensitivity to solar forcing .
However PMOD has diverged from Acrim (and other products) and which show a lesser gradient then PMOD .
Irony.
Boris, you missed one that I think is key:
Rising sea levels.
You really don’t need to statistically hash all the temperature data if you believe that one. The ocean is the earth’s thermometer…. I’m still skeptical though and always will be. There are always plenty more questions to be asked.
Boris,
I think you are missing the point. Who was the first person to pull your list of assertions together, at what point in time, to conclude AGW was true?
Andrew
Some participants in this thread have raised questions about IP rights in gov’t funded research. I am an entrepreneur who has licensed technology from universities in the past. I have also participated as a panelist at an AUTM conference on the topic of commercial licensing of university intellectual property. I am not an attorney.
The relevant law for US funded research is summarized here:
http://www.autm.net/Bayh_Dole_Act/3143.htm
If you are interested in the bottom line – researchers do retain certain proprietary rights to their research even when it is funded by the U.S. Gov’t.
Lucia,
I replied to your comment. Still a little confused. Did you mean that the Russian gridpoints should be zeroed out?
Lucia, you need to expand your complaint beyond Scafetta to include Gav, Mann, and the rest of the code – hogs. Not in comments, but on the main page.
Don’t target one individual, target everyone who commits the same type of climate malpractice.
These grifters have collected far too many millions of tax dollars to hide behind this kind of smokescreen. Let them all have it with both barrels, please, Lucia.
Lucia, I agree that Scafetta should make the code and data public, as should all scientists.
However, your omission of the history is scandalous. You say:
However, Rasmus is expecting a whole lot more of Scafetta than he expects of his friends. If I were Scafetta, I’d be sorely tempted to say “You can have all of my code and data made public as soon as you publicly make the same request of Mann and Thompson and all the rest.”
I know that Mann and Thomson wouldn’t do it, but I’d like to see Benestad forced to ask. The fact that he has not done so, but is now abusing Scafetta for doing what he hasn’t asked his friends to do, is disturbing. The fact that your head post makes no mention of his past omissions, however, is much more disturbing. You’re giving Rasmus a free pass. Perhaps you could explain why.
What is the timeframe for these code requests?
Scafetta’s demand that they take a class in wavelets is maybe deliberately similar to Steig.
Some thoughts from frequent commenters at Tamino and RC:
Jim Eager // June 29, 2009 at 6:57 pm | Reply
dhogaza wrote: “Write your own program to implement the algorithm.â€
Exactly. Running the exact same data through the exact same algorithms using the exact same code will give you the exact same errors, if there are any, so you would not even know if there were any errors or not.
But at least it would keep some trolls busy for a while so they wouldn’t be cluttering up threads.
t_p_hamilton // July 2, 2009 at 9:39 pm | Reply
michel saysâ€Some of you think that no, they should not, and anyone having any doubt about it should be obliged to recode himself to see if he gets the same results. You are not explicit what he should do if not. Punt?â€
Yes. If they don’t have the ability to reproduce the results, they don’t have the understanding to make a scientific contribution. In lieu of actually writing a complete code, perhaps the “skeptic†could show intimate familiarity with coding and models, in order to make requests that the original group would not think would be misused by GIGO.
“Nor are you explicit about what it would prove if so. That he has made the same errors?â€
The same coding errors, or the same error in a theoretical derivation? You clearly have no experience in this area.
#
MikeN also shows his lack of experience:â€So what happens if someone tries to recode and fails? He is attacked for poor coding.â€
Maybe because he is not competent. Has that thought ever crossed your mind?
†How would we know if the first guy did it wrong?â€
I suggest choosing smarter “skepticsâ€. Here’s how:
Send them through graduate school and post-docs in climatology-related fields, let them learn these codes, data sets and theories inside and out. Put them in a position where their reputation is adversely affected by errors, and see what happens.
Let us know how that works out for you.
Andrew_KY (Comment#28624) December 18th, 2009 at 11:39 am
Some light reading, a condensed history http://www.aip.org/history/climate/
Isn’t this the site where JeanS pointed out Rahmstorf’s error? What do you guys think of this, also from Tamino’s Breaking Records post?
#
MikeN asks:â€Personal insult? Dr Rahmstorf has admitted on RC the caption is wrong, and that he changed the smoothing method to generate the graph in the Copenhagen Report.
How was this error supposed to be found, when code is not revealed?â€
This is an example where an “error was found†without “releasing codeâ€! MikeN asks, how could it be found without releasing code. Maybe he should listen to the person who found that N=15 is better than N=11 for a clue how to find this “errorâ€: “[Response: Almost correct: we chose M=15. In hindsight, the averaging period of 11 years that we used in the 2007 Science paper was too short to determine a robust climate trend. The 2-sigma error of an 11-year trend is about +/- 0.2 ºC, i.e. as large as the trend itself. Therefore, an 11-year trend is still strongly affected by interannual variability (i.e. weather). You can tell from the fact that adding just one cool year – 2008 – significantly changes the trend line, even though 2008 is entirely within the normal range of natural variability around the trend line and thus should not affect any statistically robust trend estimate. -stefan]â€
Standard operating procedure for science.
Standard operating procedure for “skeptics†– OMG somebody found a typo in a figure caption! Science can’t be trusted!
bugs,
Why don’t you just type when you think the pivotal moment was, and who it involved, and then put a quote and link underneath it for reference? Just throwing a link at me with 30 more links in it, is pointless. Can we focus a little please?
Andrew
Andrew_KY:
No single pivotal moment.
Lots of slow and steady progress towards understanding.
If you’re trying to make a point then just make it.
Dan Hughes says:
“JohnV,
Those are your words, not mine. Kindly remove the quotes when you refer to my comments.”
.
John V never learns. He’s apt to incorrectly paraphrase and misattribute. [Did I miss the apology for this one?]
JohnV,
That’s the point. You mentioned that Archimedes’ Eureka Moment was an event of brilliance, when a man put 2+2 together to discover an answer.
AGW has no such brilliant moment. I have seen a long list of assertions, but I have yet to be shown who compiled all of this science into it’s now historic conclusion. Who had the moment of revelation when the 2+2 finally manifested itself with the = sign. Who gets the credit for recognizing what we are being told is the sound science of AGW and when did it happen?
It didn’t happen, evidently.
Andrew
I forgot to include this with my earlier comment:
http://www.youtube.com/watch?v=J8bRciuMLqQ
Andrew
Do you want a pony for xmas too? Read it understand the gradual progress of science over a century.
bugs,
I see you have no intention of telling me what you think.
And for what I want this year… I want all of us to give something to the poor before we accept any gifts and…
“to remember upon Christmas Day, who made lame beggars walk, and blind men see.”
That’s what I want.
Andrew
JohnV:
I can think of a few key benchmarks.
Tyndall’s experiments.
Keelings CO2 curve.
The discovery of the water vapor feedback mechanism (anybody know who is the proper attribution here?)
The politicization of climate science by it’s more ardent advocates.
“discovery of the water vapor feedback mechanism”
.
discovery or invention?
Carrick, can you elaborate a bit on the “discovery of the water vapor feedback mechanism”? Seems that one is still pretty much up in the air, at least insofar as it operates in the real world. My understanding is that we don’t even know the sign, much less the magnitude. That being the case, probably no surprise that there isn’t a single name that pops out for attribution . . .
The pivotal moment was when they all got together and realised that – “hey – maybe we can all make a bucketload of money out of all this garbage!”
Carrick (Comment#28666)
Spencer Weart” is the chronicler here. He attributes the appreciation of WV feedback to Arrhenius, whose paper and book should certainly rate on your list. Arrhenius used the assumption that the relative humidity would remain constant, which gave the feedback.
Today many of these papers are publishing results of computer runs which to me is equivalent to a mathematician publishing results of a derivation but refusing to show the derivation itself! Without the code you can’t see what the ‘derivation’ is.
I agree with Willis Eschenbach though. When Rasmus makes the same request of his friends then he has standing to make the request of Scafetta. Otherwise it’s just hypocritical.
I visited Spencer Weart’s site. He gives glowing reviews as being the harshest one’s he’s received (and reasons to buy his book). Then admits lower down on the page that he ignored all skeptical comments and claims that this choice to leave all contrary information out was “vindicated” in 2006. Such a wonderful resource for somebody who truly wants to understand the issues.
By the way, I think Scafetta should ask Rasmus to wear a “Free the data, free the code” T-shirt for a month.
BarryW–
I do agree with Willis that Rasmus has not made public requests of his friends. We can’t know if he has done so in private. One hopes. . . (maybe in vain, but one hopes.)
MikeP–
Lots of people would back Scafetta if his response was “I’ll hand over code when Mann and Steig do”. That would at least indicate that he accepts the notion of sharing in theory but doesn’t see why sharing should be one way. But he’s not saying that.
Nick Stokes:
I thought Arrhenius got the physics wrong?
Going through Weart’s SI, I think this may be the “seminal” paper:
Manabe, Syukuro, and Richard T. Wetherald (1975). “The Effects of Doubling the CO2 Concentration on the Climate of a General Circulation Model.” J. Atmospheric Sciences 32: 3-15.
“Do you want a pony for xmas too?”
Bugs, how did you feel when your mother told you that Santa Claus didn’t exists? Don’t you have that feeling a little bit now?
bender:
As I explained before, “too hard to read and use” were my words. I was clearly quoting my previous comment. Dan Hughes thought I was quoting him. His mistake and now yours too.
—
Andrew_KY:
Science just does not work the way you seem to think it works. AGW is not a theory — it is the natural conclusion from tons of science by thousands of researchers over 100+ years.
.
If I understand you correctly, your point is that “AGW has no such brilliant moment” ( comment #28660). Can you explain why you think that’s important?
JohnV,
Please refrain from being deliberately obtuse with me, if you can.
I know you warmers have to gloss over things that don’t support your beliefs, but I would like to have a serious discussion when people respond to my comments.
Let’s try this again. AGW is an important conclusion. Nations worldwide are making arrangements based on it. And I’m personally supposed to believe it, if I’m smart.
Who first made the famous conclusion and when? Who gets the credit for it? Someone must have done it. Let’s find out who it was so we can give that person the praise they deserve.
Andrew
Andrew_KY:
It doesn’t matter how many times you ask me the question. My answer will stay the same. There was no single person or eureka moment for AGW. Like most science the “famous conclusion” is based on slow progression of knowledge.
.
I have answered your question a few times. Now it’s time for you to answer mine:
Do you think science progresses on eureka moments?
Whad do you think the lack of a eureka moment means?
.
If you refuse to answer again then I’ll know for sure that you’re just trolling.
Discovered and confirmed.
One cite among many.
Deleting questions about Mann and Steig not releasing code are definitely signs of bad faith. I see Gavin & Rasmus in comments. If both agree that Scafetta should release code, then they should at a minimum permit people to ask why Steig or Mann will not.
Andrew_KY, JohnV has a point here.
There are certain scientific principles that get agreed to over time, simply because we know they work by continued repetition.
The idea that GHGs warms the atmosphere is one of those. The seminal experimental work was due to Tyndall, so we have known since his day that if humans increased atmospheric CO2 that we would be warming the climate.
Beyond that, I included Keeling in my list because he is the first one to instrument and measure the influence of human activity on atmospheric CO2 content.
We know the earth is warming (GW is not a theory but a generalization from observation), we know that humans are changing CO2 concentration, so we know that AGW is a fact of science in this sense of generalization from observation. It doesn’t require CRU, GISS or anything else
What is left undiscussed in my comments is how much of GW can be attributed to human activity and how much the future climate will change as a result of human activity. Both of these issues are bigger than just the raw CO2 concentration number that alarmists simplistically glom onto.
“There was no single person or eureka moment for AGW”
That’s all I need to know.
The the AGW conclusion just appeared to a group of people when they were on a pilgrimage or something.
Andrew
Lucia:
I agree.
There is a lot of capriciousness in their commenting policy, and the science would benefit from a much more rigorous and rigid policy for when comments should be removed. As things stand with their blog, it is clear these authors will not allow an honest discussion on almost any given topic.
John V, I find that the opposite “is the natural conclusion from tons of science by thousands of researchers over 100+ years.” You need to do better than that if you truly want to convince anybody.
I also think you must be including in your total thousands of researchers who’ve started with the conclusion that AGW exists and with better or worse mathematics tried to predict the consequences of it. Or came to whatever conclusions their study came up with and included the obligatory mantra “possibly due to AGW”.
Carrick,
Thanks for the info.
“There are certain scientific principles that get agreed to over time, simply because we know they work by continued repetition. ”
This is a generalization, too. I’m not sure where the ‘repetition’ happened in regards to the world’s climate. When did the global climate ‘repeat’?
Andrew
“The the AGW conclusion just appeared to a group of people when they were on a pilgrimage or something.”
Your “no eureka moment” argument is one of the dumbest I have seen against AGW. Given the multitude of dumb arguments against AGW, you should be quite proud.
Boris,
A fact is a fact. That you think it’s ‘dumb’ is entirely your choice, and you are certainly entitled to your opinion.
Andrew
‘Boris’,
If that’s your ‘REAL’ name. 😉
Andrew
Re: water vapor feedback,
Boris appeals to authority:
“Discovered and confirmed.”
Sorry, Boris, I missed your analysis of Lindzen & Choi 2009. Was it in your linkie?
Boris,
re; 28691
(quote)Thus, we conclude that the water vapour feedback is not overestimated in models and should amplify the climate response to increased trace-gas concentrations. (end quote)
Probability again, as good scientists should word their papers – “should”
I also note that this paper is from 1991, and of course over the last 18 years, more data has been collected, more analysis too.
Wondering if you have seen Dr. Spencers’ latest work in regards to this subject?
http://www.drroyspencer.com/
http://www.drroyspencer.com/wp-content/uploads/Spencer-Forcing-Feedback-AGU-09-San-Francisco-final.pdf
Could be that water vapour actually helps cool the system.
“Hmm. Anyone else note the backtracking and hedging by Gav in his responses at RC (including in his reply to Mosh). The problem is being redefined as not that the code is not available but that the description in the original article was not detailed enough to allow someone to produce their own code. Gav is being a little sneaky here and is holding onto an out for the next time a mainstream climate scientist is criticized for not releasing code. I assume he is also trying to immunize himself from charges of hypocrisy for past behaviors (so where exactly is that calculation of the MBH confidence interval, Gav?) Way to have your cake and eat it too.”
The groundwork is being laid for the rewriting of history.
Gavin confuses a couple things. First and FOREMOST I think code and data ought to be produced NECESSARILY to prove the most basic claim made by any paper. Namely, that the figures present RESULT FROM the data and code the author purports to use. This is not a matter of replication, this is a matter of QA and fact checking. Any one out there ever re run your analysis and find out that your figures changed? I thought so. There is no coherent logical rational objection to providing the code and data. NONE.
Reasons they have tried:
1. There is IPR to consider:
A. There is public funding to consider
B. License it. I’ve signed many licenses to see and use code without the rights to modify it and redistribute it.
C. What commercial interest is there? ( hehe)
2. You can theoretically do your own code from our description.
A. Experience shows otherwise.
B. So what? This has nothing to do with the obligation to share
data and code so that other scientists can build on your
work.
C. If results of my emulation dont match yours then What?
1. Your instructions were bad
2. Your instructions were good and I made mistakes
3. Your instructions were good and YOU made mistakes
D If results of my emulation match then what?
1. They code may still function differently under different
test circumstances.
3. We didnt comment our code very well.
A. Been there. used to it
4. The code was not meant for distribution
A. Ben there. used to it
5. You’ll bug us with support questions
A. Nope. provide the code as is.
6. We dont have the budget
A. The cost of your denial exceeds to cost to post it up.
7. You’ll take our work and beat us out in bids for future work
A. Competition is a good thing
8. We’ve got 25 years in this and your motive is bad. you want to prove us wrong.
A. duh.
“But on a more philosophical basis, why leave this up to judgment? In one of his responses, Gav says: “However, replication in general doesn’t require turnkey code for every random paper. If someone says they used linear regression, you can do that yourself.†Sounds reasonable, no? But we know in practice that climate scientist have used this excuse for items far more complex than a linear regression. ”
gavin’s statement is an empirical hypothesis. I said the written word “rarely” provides enough information to construct the algorithm” Gavin says “generally”
A couple points.
1. I tried to compile a written description the compiler barfed.
2. That one COULD code it by onesself, is a non sequitor. The question is SHOULD ONE HAVE TO. And please dont tell me to do it for my “own good”
3. Empirical research that I have seen indicates that Gavin is overstating the case when he says “generally”
4. How does a journal handle cases where a researcher cannot replicate the code from the description? who do you trust?
“Sorry, Boris, I missed your analysis of Lindzen & Choi 2009. Was it in your linkie?”
No, But here’s James Annan:
http://julesandjames.blogspot.com/2009/08/quick-comment-on-lindzen-and-choi.html
Don’t neglect the comments. Also, I believe Spencer has shown that L&C is pretty bad due to using model simulations forced by historical SST instead of fully coupled models.(link in comments at Annan’s place).
“7. You’ll take our work and beat us out in bids for future work
A. Competition is a good thing”
Um, steven, stealing someone’s work is pretty much the opposite of competition, unless you are one of those anarcho-capitalists who don’t believe in any IP laws at all. I sure hope you aren’t.
I recall that North thought keeping code to oneself helped competition. I’m not really sure I agree with him or not and I’d say I agree with RC when they say someone should at least attempt a replication before asking for code.
BTW, what code has Mann still not ponied up? Is it something from MBH98 still?
steven mosher (#28715):
Agreed.
Once the author(s) have the first publication with their data, there is little if any reason to hold it back. Actually, if they really wanted to milk the data for more publications it could be initially released with a license that allows replication and commentary only.
Boris:
Mann is a coauthor of Steig et al. (2008).
You feel that code is 100% released, or do you lack the expertise to make that judgement?
Boris,
I’m not aware that there is such an animal as a fully coupled model. There are models with various degrees of coupling, but AFAIK there are none that are fully coupled, i.e. fully integrated per time step calculation of both atmospheric and oceanic conditions. Linkie?
Boris (Comment#28722) December 19th, 2009 at 3:50 pm
“7. You’ll take our work and beat us out in bids for future work
A. Competition is a good thingâ€
Um, steven, stealing someone’s work is pretty much the opposite of competition, unless you are one of those anarcho-capitalists who don’t believe in any IP laws at all. I sure hope you aren’t.
Well lawrence Lessing is a friend if that gives you any clue. My point is this. If public funds have been used for the development of the science, then it’s not “your work.” When I worked for DOD the science we did was not “our work” Our patents all had to be assigned back to the funding agency. When a tax payer funds a university to do science, that science should be open. With regards to patents and IP, you can just count me in the group of people who work for the reform of these laws whenever I get the chance. ( psst ya I fought against the RIAA back in the day and made “illegal” copies of the CDs I bought “)
” I’d say I agree with RC when they say someone should at least attempt a replication before asking for code.”
That’s an impractical an uninforeable standard. Second, McIntyre did attempt to replicate Mann before he asked for the code. third Osborne himself had trouble ( did you in fact read the million words of the emails? ) Just point blank why do think someone SHOULD attempt a replication? because its good for their soul?
to “prove” they know how to follow instructions? Why? I fail to see what benefit follows from this. Santer is the perfect example of this Nonsense. he argues that he wont supply the code because anyone could do it in 2 hours. How exactly is that logical? gavin made the same argument in 2007. he argued that GISSTEMP could be written in 2 pages of matlab. false. Manifestly false. The algorithms as written DID NOT AND COULD NOT match the description in the text. because the text was silent on the particular question that I was asking. When the code was finally revealed I saw the real answer. The QA check I was specifically asking about was done BY HAND OFFLINE with no records whatsover.
The code answers questions better than the text.
Nobody has given me anything that even looks like an argument about why the code should not be given. The best you have is “generally we dont need the code” That’s your best position and you cant even back it up with actuals facts or tests or any argument of any kind. Sad.
Now the really CRAZY thing is the code that is of LITTLE USE, GCM code, is freely posted. In short, code that cant be used or tested by others is freely released. Talk about bonedheaded and backward.
Google search for “fully coupled climate model”.
JohnV
I think one thing people have missed, for example, was the way you and I could work together even though we had different viewpoints. You shared your code and I accepted it and used it. Partly because I knew that if I was going to make my BEST CASE to YOU using your code would take the method question off the table between us. I think the record shows that even though we’ve disagreed here and there we’ve managed to get along without any bad blood, name calling, suspicions etc. All the personal stuff just withered away.
Other folks:
In this whole affair I know its tempting for people to want to hold gavin and others to account for the past actions of others.
personally, I’m just going to praise rasmus for sharing code.
Boris (Comment#28774)
December 19th, 2009 at 10:35 pm
Interesting search results. Here’s what I found in just the first few:
“While proxy data give us a general idea of what early Paleogene SSTs should be, we cannot achieve those values in a fully coupled climate model experiment.”
http://web.ics.purdue.edu/~huberm/SEWALL.HUBER.SLOAN.PDF
“To remain at the leading edge of worldclass science it is essential that the UK community continues to develop our modelling capability, towards the objective of a fully coupled climate model…”
http://www.atm.ch.cam.ac.uk/acmsu/research_ukca.html
“…a feat that no fully-coupled climate model is currently capable of producing…”
http://esd.lbl.gov/research/projects/abrupt_climate_change/projects/
But just to show how interesting your idea is, I googled the exact phrase “leprechauns are real”
http://www.google.com/search?hl=en&rls=com.microsoft%3Aen-us%3AIE-SearchBox&rlz=1I7HPND_en&q=%22leprechauns+are+real%22&btnG=Search&aq=f&oq=&aqi=
John M.
boris is no hank roberts
JohnV,
“Have you ever tried to install and use SQL Server, Octave, or R? They are all complex and require months or years of experience to become a reliable user.”
There is a difference between highly complex and tuneable and Slop.
SQL Server has manuals and study courses available. What can you get from GISS even if you offer money??
JohnV,
“Lots of slow and steady progress towards understanding.
If you’re trying to make a point then just make it.”
I seem to remember, in my old fallible brain, that many times in history science has plodded along in a slow steady progress until a lightbulb moment (not a CFL) tosses it all into the trash bin.
In fact, I remember reading eminent scientists of the day stating that everything has been discovered there is to discover!!
Wonder where Climate Science, as exemplified by the IPCC, is going??