In today’s post discussing the recent error at GISS, which resulted in GISS pulling the obviously faulty data, Gavin decided to end with a paragraph which, evidently, communicates his final point about the role of models.
Which brings me to my last point, the role of models. It is clear that many of the temperature watchers are doing so in order to show that the IPCC-class models are wrong in their projections. However, the direct approach of downloading those models, running them and looking for flaws is clearly either too onerous or too boring. Even downloading the output (from here or here) is eschewed in favour of firing off Freedom of Information Act requests for data already publicly available – very odd. [additional kvetching snipped- lucia]
I had to blink a few times when I read this. I mean… huh? Gavin’s point about the role of models … in what… is… what?
- Who are these “many”? How does Gavin know their motives? Did he consult the oracle at Delphi?
- Why does Gavin imagine that testing the accuracy of IPCC projections, as communicated by the IPCC in their reports requires anyone to rerun any specific model? and though it’s not entirely necessary to testing IPCC projections?
- Why does Gavin think downloading data from The Climate Explorer is eschewed by these “many”? and
- What, pray tell, is wrong with “many” requesting results of post-processed data used in peer reviewed papers? (I assume here Gavin’s use of pronoun “many” means “one”. Steve McIntyre recently requested data from Benjamin Santer. When Ben refused to provide data, Steve filed a request under the FOI.)
I hate rhetorical questions, so I”m going to answer most. (I’ll leave the first as rhetorical though. I have no idea who “many” may be.) But now, here are some comments.
No one needs to run any AOGCMs to test IPCC projections.
The IPCC, when writing the AR4, came up with a specific methodology to create their projections for public consumption. These projections were based on specific models and specific model runs submitted to the IPCC who ultimately came up with projections based on those specific model runs.
The projections are available in the form of graphs and texts in IPCC documents. Some of the supposedly more robust projections are those applying to Global Mean Surface Tempearture (GMST) and are highlighted in section TS 5 of the Technical Summary of the AR4, available here. In the absence of significant volcanic activity or major changes in solar forcing, the “underlying climate trend for GMST” (i.e. expected value stripped of “weather noise”) is projected to rise monotonically, with the rise behaving nearly linearly during the first two decades of this century.
The IPCC communicated this information to the public using this graph:
Using these words (among others):
Figuring out that the IPCC AR4 comminicated a best estimate for the “underlying climate trend in GMST ” is “about 2C/century” for the first two decades does not require the reader to download and run an AOGCM. Determining whether the projected trend of “about 2C/century” materializes also does not require anyone, anywhere to download any of the AOGCMs used by the IPCC run them or test them.
In fact, it would be irrational to call any other trend the trend the IPCC projection: The actual, honest to goodness IPCC projections are based on the models and runs they selected using the process they ultimately chose. They IPCC authors communicated this trend.
Testing the IPCC best estimate for the projected trend merely requires comparing the projections the IPCC came up with and communicated to observations in the period where the projections applied.
Either these projections or others related to tropospheric temperature trends will match the real earth climate trends within the range of uncertainty of the observation or they will not.
As for Gavin’s ascribing motives: The reason the weather watcher are watching is the projections are agreeing with or diverging from the observations. Period.
Until the temperatures begin to or the IPCC revises its projections, weather watchers will continue to point out that it is difficult to reconcile trends of -1.5C/century with projections of 2C/century projected to apply right now.
Criticizing people for not downloading or running AOGCM’s is not going to distract anyone from the fact that the temperatures have not been rising this century.
Downloading “model data” is not eschewed by “many”.
First: I have no idea why Gavin imagines that “many” should be downloading the “model data”. After all, determining whether “about 2C/century” or any other IPCC projections specifically communicated by the IPCC falls in the range of climate trends consistent observations does not require downloading any model output from any source.
All that is required to test the projection for GMST is a) reading the IPCC technical summary and documents, b) identifying the projection is “about 2C/century” and c) comparing this to observations of real earth data.
People are downloading data anyway!.
Why does Gavin think anyone one is eschewing downloading the precious “model data”? Did he checked the logs at the services he links, run a script and fail to find the IP address of these “many”?
Does Gavin even read what’s going on at other blogs written by whomever the unnameable “many” might be?
If Gavin gout out a bit, he might shocked to discover that “some” bloggers have indeed downloaded data from “The Climate Explorer” and done a few comparisons between models and observations.
“Some” have even applied the tests in Santer17 to GMST. Few would suggest the results of this test would increase our confidence in projections based on multi-run averages used by the IPCC in the AR4, as the results indicate that if we accept the t-test described in Santer, would lead us to reject the hypothesis that an average based on 38 runs using the A1B SRES available at The Climate Explorer falls in the range of trends consistent with the observations of GMST since Jan. 2001. (To those wondering whether the method Santer et. al (2008) used to compare models to observations resembles the one Gavin described with such zest here: The answer is: No.)
Let me assure Gavin that Steve McIntyre also numbers among the “some” who downloaded climate data from archives in all sorts of places. With regard to requesting information from Santer: It appears Steve wants to figure out precisely what Santer did and whether certain pesky details affect the results. (Typical pesky detail associated with the choice of end point for the analysis discussed here.)
There is nothing wrong with asking authors for data!
So, evidently, Steve who is interested in Santer’s paper asked for data. Is this wrong? It is hardly unusual for people to contact authors of peer reviewed papers and request already processed data. The practice is routine. Many authors are willing to share data, particularly after it has been used in a major publication.
Evidently, Santer prefers not to grant this particular request. Maybe Ben Santer has a right to refuse. Maybe not. I have no idea. If Ben has such a right, he need not worry about any request under the Freedom of Information Act. No need for anyone to complain about the request at their blog. If he has no such right, then he will be obligated to hand over the data to SteveM.
Big whip. Does anyone anticipate any bad thing happening when SteveM gets the data?
So…What was the point of Gavin’s final paragraph?
Well… that was a long comment on a bizarre paragraph. But, as I said… huh?
Remember, that paragraph was supposed to make a point about the role of models. I know, it seemed to be nothing more than kvetching about the behavior of “many”, trying to assign pointless busy work to these unidentified “many”, and maybe some indirect attempt to defend Santer’s decision to refuse Steve’s request for data.
Yep, it’s just a garbled mis-mash!
So, here’s how I predict “many” will react to Gavin’s post:
“Many” will think Gavin is upset GISS has egg on its face. After all, they posted clearly bad data and pulled it after much laughter in the in what Gavin has been known to refer to as the “more ‘excitable'” parts of the blogosphere.
What many probably won’t discern from that post is a point Gavin probably would like to make: The fact that GISS screws up GISSTEmp from time to time, doesn’t mean there is no global warming.
GISS screwing up only means GISS screwed up– everyone screws up sometimes. This month was GISS’s turn.
But we know when they correct the blunder, the observational data will still show warming over the past few decades.
But as long as GISS did screw up, why not have some fun? Most readers here should feel free to guess the GISS’s future October anomaly (the one that will appear after Nov. 10.) I’ve got a brownie bet going on this!.
Yes GISS did screw up. The reason they screwed up, I believe, is because the bad data confirmed the NASA GISS cultural expectations. Can anyone imagine that GISS would have published a significantly lower number had the mistake gone the other way?
I wonder how many people noticed the error but were reluctant to say anything.
Mike–
The data were published on a Federal holiday. Who knows, maybe they have a script running and no humans are involved?
This was clearly embarrassing for them. I’m sure they will figure out how to write a script that creates flags and halt the process long enough for humans to double check things. These sorts of things can be done. (Are done in projects elsewhere.)
It’s pretty simple. When a person believes it is cooling and a warm data point shows up, they do some checking. When a person believes it is warming and a warm data point shows up they tend to be less diligent. And the same goes for the other side. some things that can help take the subjectivity out of this phenomena ( confirmation bias) are quality control processes, independent audit and reproducible research.
http://www.stanford.edu/~vcs/papers/Licensing08292008.pdf
Gavin Schmidt explains down under that any 10 year period is dominated by “weather noise”:
“Rajendra Pachauri’s assessment of the temperature record is in line with the assessment in the reports of the Intergovernmental Panel on Climate Change, so it is unsurprising that he should repeat what the organisation he heads has concluded. Pachauri is discussing longer time scales than a year or 10 – over such short periods simple weather “noise” is responsible for temperature fluctuations that do not reflect underlying climate change. Duffy’s suggestion that a temperature drop in the past year is significant is equivalent to assuming that because one or two spring days are cooler than a week before, summer won’t occur. ”
http://www.smh.com.au/news/opinion/letters/climate-doubts-based-on-shortterm-irrelevancies/2008/11/10/1226165474661.html?page=fullpage
Gavin likes to make claims like this providing no support for why a full year drop in temperature is equivalent to one or two spring days etc. Equivalent is a suitably vague word.
Obviously, there is more weather noise in two days and less weather noise 5 years. But if a 1 year trend is “equivalent” to two years of data, then it’s also “equivalent” to five years of data. (In fact, it’s closer to 5 years!)
There are methods to quantify the uncertainty due to noise in a signal. Gavin tends to, shall I say it, “eschew” quantitative assessments when making his puerile analogies at his blog or in his opinion pieces.
Though there is much sturm und drang surrounding Santer’s recent paper and it does have some, shall we say “issues”, at least that paper does try to quantify the uncertainty due to noise rather than just decreeing some aribitrary number of years is key.
Anyway, if you click some of the links: If I apply the method of Santer to individual model runs over spans of 30 years, models don’t look so terrific at predicting GMST.
“Who knows, maybe they have a script running and no humans are involved? ”
I think you are being very kind, How long have they had access to the data? No one checked it out?
The talk about how to apply quality control to the GISSTemp numbers seems to me to be way over the top.
Now GISS/NASA has estimated the costs for trying to apply a little quality control to the temperature calculations. What’s so expensive about actually thinking about the numbers for which you responsible? Looking at the fancy multi-colored plot and saying, Hmmm … that’s strange, it’s hotter than two Hells in Siberia in October; not to mention a couple of other places. That’s not expensive. And actually thinking about the numbers for which you are responsible is part of the job in the first place; the most fundamental part as far as I’m concerned. It’s a job requirement in Engineering.
Evidently, the estimate is based on this:
Who suggested they check everything manually? Or even compare to the weather underground?
Can’t Captain Obvious visit GISS and point out that it’s possible to add simple QA filters to a script? Why not just add a few lines to their script to check and flag these things:
1) Temperature anomaly for station more than 3 standard deviations from normal for that station or region. If yes, flag. List as “no-available” and wait until confirmation.
2) Absolute temperature matches last months to all significant figures. If yes, flag. Maybe add an if/then to supress the flag for tropical regions where this can actually happen. List as “no-available” and wait until confirmation.
3) Temperature anomaly for regions making up more than 2% of the worlds land mass more than 3 SD from normal? If ‘yes’, flag. List data as “not -avaialble” and wait until confirmation.
This would have caught the embarassing error. Gavin seems to be telling us the process of computing GISSTemp is already automated (as in possibly little or no human intervention before the data magically appear for the world to see. ) If so, the three QA checks described above could be implemented easily.
Presumably, the 0.25 FTE currently working on GISS temp could write this and shove it into the current code. Sheesh!
Meanwhile, since GISS has not yet done this, having the display held up until a human looks at it, thinks it’s ok and presses a “publish” buttin might be wise. That human presumably would have said:
“Northern Asia with positive anomalies of 14C? HUH? Geeh…. does that really seem right? Maybe I should wait a couple days rather than make GISSTemp the laughing stock of the blogosphere!” Then, he could email Hansen and ask what to do.
So, we have two possible solutions, neither of which would cost a half a million dollars to implement. DUH!
Speaking of which, I love
this comment:
Gavin – Your response to #24: No deal – and not a very constructive answer to start with “Rubbishâ€.
You are setting up a straw man by implying that the only alternative to accepting errors would be to wait for human checking.
You will probably be aware that figures are already (and logically) rejected for days where the recorded mid temperature falls outside the min and max temperatures (A not-uncommon phenomenon at sites where the figures are manually recorded and accidentally transposed).
It’s entirely possible, and reasonable, to ask for more automated error catching during the raw data processing, without requiring a wait for human scrutiny.
Such error catching *could* involve automatically rejecting the data if a “large†(tba) proportion of a sources figures were missing or identical to the previous set. Other obvious checks could include rejection if all values for a month were zero, or outside a “reasonable†(tba) range.
You appear to be resistant to a suggestion for technical improvement.
[Response: Huh? Did you even read the top post? I suggest exactly that. – gavin]
Uh… the top post suggests
So, he alludes to this being “tricky”, and describes something that would be cheap if done in a reasonable way. But does gavin suggest a method?
No. Later in comments, Gavin does describe in inefficient, manpower intensive method and suggests a 1/2 million dollar budget!
So… what was the point of Gavin then deciding to describe some half million dollar fix in detail? Are readers supposed to let their brains fall out of their skulls and not criticize Gavin’s elaborate explanation of an idiotic expensive process for fixing the process? Are we supposed to not notice this in comments just because he alluded how “tricky” automation might be and described possible simple fix?
Sheesh!
These guys really should hire a PR person. With the spinning and excuses, you’d almost think that they care less about the quality of the data and more about perception. Perhaps a better response would be more like:
“Some errors in our October GISS temperatures have been found, resulting in a gross overestimation of temperatures in some regions. The temperature record for October will be reviewed and corrected, and additional quality control measures will be added to avoid this error in the future.”
In my job, if something happens on a project I’m working on, there will be a post-mortem and steps must be outlined to avoid the same problem in the future. Management starts to get really upset if the same problem keeps happening. I’m only an engineer though, not a climate scientist, so maybe this doesn’t apply.
I knew that some scientists were so far from reality that they finished up living in a completely virtual model/world .
Schmidt belongs obviously to that sort .
So checking data like temperature is tricky ?
Sure , I admit that the process of looking at a thermometer and understanding the deep meaning of the temperature concept needs only those with brave hearts and crafty minds .
As for transcribing such a number and communicating it we enter the realm of intellectual giants to which , alas , we don’t belong .
However I happened to be advising a big company on a project consisting to model a presumed chaotic system .
This project involved downloading a large number of data , actually closing quotations of stock and raw ressources on Stock exchanges over the whole Europe . The pricing of this company was to be set by analysing those data .
Of course stock exchange data are a mere trivialities when compared to such lofty and difficult concepts like temperature but …. they also happen to be sometimes wrong .
That was the first question asked – “How are you sure the data is OK ?”
There was a trainee in the team (a young guy good at programming) who said “Only 3 things to do and already done .”
1) Check that the data was updated . Easy test : if X(n)=X(n-1) then wrong
2) Check that there is data : easy test if X(n) = 0 then wrong
With that we cover 90 % of the problems .
If you want to go to 99,9% you add if X(n) doesn’t belong Interval then wrong
The young man was paid 1500 € and it took him about 4 hours to code a routine that checked downloading tens of thousands data every day .
It took him farther 4 hours to code an automated mail to people on list for approving the data .
And it took him 8 hours to cover the last unprobable event and to inform the thousands of people making the price (with flashing red capitals :)) that the data X(n) was not approved and they could choose X(n) or X(n-1) according to their best judgement .
Of course in this last case I wouldn’t like to be in the skin of the guy who wouldn’t approve the data but the routine was written anyway .
So counting large , and even if it was much less tricky than temperatures , the cost of this BASIC control was about 160 € .
Add overheads , tests , documentation and whatnot and roundup to astronomical 5000 € .
I understand that Schmidt is afraid that in the case of temperature , it could be a half million ?
He should stop smoking his carpet .
TomVonk–
You forgot the cost of writing the peer reviewed article . . .
Hi Lucia – long time no commenting, I’ve been pretty busy, but I have been checking in once in a while…
Anyway, when I read Gavin’s post at realclimate it made a lot of sense to me, and I’m wondering at your (over-)reaction. The role of models (no he didn’t quite complete the thought, but it seemed obvious) is to find some way to calculate how the real world behaves, so you have a slightly better knowledge of the future than random guessing. I don’t know if you’re one of the “excitable” folks, but if you belong among them, you have certainly done the most of what could be called “modeling”, that I’m aware of – “lumpy” for example – and your primary collection of posts seem to center around testing the basics of comparing model predictions with reality. Fine, that’s worthy stuff.
But Gavin’s point, it seemed to me, was that with all the effort going on out there at critiquing – McIntyre, Watts, Pielke, etc. etc., nobody (other than your small efforts) seems to be actually taking the step of trying to create alternative models, or modify existing models, to gain real predictive power any better than what we have now. That seems a real lapse – why the neglect of modeling on the “skeptic” side?
Arthur, the answer to your question is very simple. From the point of at least some skeptics, myself included, the entire concept of trying to ‘gain real predictive power’ of a system as complicated as the Earth’s climate over decades is hopelessly flawed.
It’s not clear to me what Gavin’s point was, other than to misrepresent the motives of people like Lucia and he-who-must-not-be-named.
Arthur–
I don’t think I’m over reacting to Gavin’s point, whatever that might have been. I also have no idea who the “many” are. Gavin has the unfortunate habit of writing posts in which he criticizes unnamed collections of people for some sort of collective behavior that can be attributed to absolutely no one.
On the model front: There is no doubt that models have a role and you have correctly identified it. If Gavin wanted to explain the role of models he could.
But he tool a great leap from “the role of models” to criticizing people who compare the observations to projections IPCC projections based on those model. Among other things, he criticized them for not actually downloading models, gridded data or whatever. That criticism of “many” is unfair and irrational.
Moreover, if Gavin were to directly criticize the “many” for not downloading models and running, the rational response to his criticism is “Yawn.” or “So?”
I mean…. Why should people out in the blogosphere spend time downloading, running or creating AOGCMs? I note that if there are any points in favor of this idea, then Gavin has failed to reveal those reasons to the readers of his blog.
Had he revealed the reason you suggested, one could present a counter argument. I would ask this: why should the task of comparing projections to data to output of models be neglected in favor of running models?
And, with regard to the kevtch I skipped in Gavin’s final paragraph: Why should anyone volunteer to find bugs in Model E and report them to Gavin? Gavin and others at GISS are paid to do that. If they can’t get the world to do their job for free… well too bad for them!
If “many” in the blogosphere want to focus on comparing projections to observations, how is this a lapse? If Anthony wants budget his time and look at the physical installation of thermometers, how is this a lapse?
And if these things are a lapse, wouldn’t Gavin’s failure to drive out to stations and look at the thermometers be a lapse? (Of course it’s not. It’s not efficient for anyone to try to bite off the whole problem.)
The fact is, people in the blogosphere examine the temperature data products because it is something that can done with limited funds, using a lap top. Many statistical tests are direct applications of undergraduate level methods. Doing this doesn’t interfere with Gavin, or any modeler’s ability to develop ModelE or run Model E one bit. If Gavin’s complaint is that those in the blogosphere haven’t decided to become modelers, his complaint is groundless.
But, the fact is, we don’t know if that was Gavin’s complaint do we? It’s the complaint he did not state, but which you infer because you assume Gavin meant to communicate a complaint that could be understood by someone, somewhere.
As it happens, IPCC projections aren’t looking good when compared to data. Those who look at data notice this. The blogosphere exists, and they post. There is nothing wrong with this behavior. The look at GISSTemp data and notice GISS blundered and share information. There is nothign wrong with this.
Time to cash a reality check. How many programmers can I hire for a year for half a million quatloos? How many “keypunch” operators can I hire for that price? In the bad old days much data was keyed twice to eliminate simple errors such as transposition and the like. In the modern age I think a competent software engineer could design, implement and test a validation suite that would catch most errors of this form.
I would expect a system run by a US government agency for widespread consumption would have had these kinds of checks from the day dot. If that isn’t the case then someone should ensure that they are implemented post haste.
As Lucia pointed out, everyone screws up from time to time and this was their day. Perhaps there is a QA system in place and Steve at CA simply found a bug in it. Let’s hope so.
Arthur, I don’t know much about why McIntyre or Watts do not run models, maybe because they are not trained in that topic or not interested. Your claim that Pielke does not run models is to display some considerable ignorance on your part.
He is one of the founding fathers of atmospheric models and their application at weather and climate time scales, and one of the best texts on the topic that I read in grad school. Here is a link to his research group page and a long list of papers using models:
http://cires.colorado.edu/science/groups/pielke/
Dan–
I suspect Arthur meant Pielke Jr. doesn’t run climate models.
Hi Lucia-
I do not run climate models.
Though I am happy to evaluate predictions provided by those who do!! 😉
“Silence, whippersnapper! The Great and Powerful Oz has spoken!”
Really, it would be better for Gavin to say less and stop acting so prissy. He’d gain more respect by being business-like rather than acting like the cheerleader who’s football star boyfriend has been criticized. It may be frustrating to have little dogs nipping at your heels all the time, but vainly kicking at them isn’t going to stop them. Show some graciousness and class.
I think Arthur Smith is right, we simply do not have enough climate models yet. Everyone should create their own. Then when we run every single model, at least a few of them will agree with reality. Of course we won’t know which ones until after the fact.
I think that climate models as they are designed are flawed inherently, because they are trying to model a chaotic system using linear approximations of probable solutions of coupled differential equations applied over a grid of the globe.
It is mathematically inevitable that the results will diverge after a number of iterations in time, and there is no meaning in running such models with different parameters.
They have to be scrapped and a new paradigm used. I think the one by A.Tsonis et al, discussed at Climate Audit a while ago and now and then coming again into the recently quoted
http://www.climateaudit.org/?p=2223
is the way to go.
Using networks of harmonic oscillators seems to be the way to go for a new way of modeling. They have had some success in generating the current temperature stasis with just two of the ocean currents involved.
This sort of work is not something that can be done as a sideline or by retired people, like me, as it requires strong computing resources ( as all models). We can but sit on the sidelines and cheer or boo as the case may be.
I believe the September-is-also-October data is probably due to a massive temporal distortion arising from a horrible mishap at a secret Siberian laboratory. Either that or the Ð¼ÐµÑ‚ÐµÐ¾Ñ€Ð¾Ð»Ð¾Ð³Ð¸Ñ guys in Novosk hit the vodka a tad early. I am not sure which at the moment.
Gavin’s gratuitous slaps at the Unnamed One and his straw men allies were ill-advised. He should probably take his copy across the hall to the guys at Fenton Communications to avoid that kind of counterproductive output in the future so the RC sponsors get their money’s worth.
Speaking of temporal distortions, I just recently saw Björn Ulvaeus in a hideously weird caped outfit along with 3 other oddly dressed Scandinavians (Anni-Frid is not a Swede but Norwegian) the sight of which could have the effect of triggering drug flashbacks for many 50-somethings. Either that or raising temperatures across Russia by 9 degrees. I am not sure which at the moment. Who would post such a thing without a warning label?
Hi again – of course, besides Lucia I should probably have mentioned Roy Spencer, who has used simplified models for some analysis recently, Steven Schwartz’s simple ocean-atmosphere model, and Richard Lindzen who at one point claimed to be working on a sophisticated model including “cumulus convection” he was so concerned about. Yet none of these, or other efforts by “skeptics” seems to have materialized into any sort of realistic predictive model – why?
Or maybe I’m just not familiar with what has been done. I do note that the “NIPCC” report promoted by the Heartland Institute this past summer has a chapter on “Climate Models Are Not Reliable”, but as far as I can tell it throws its hands up in the air on any question of how they might be improved to be more reliable. No alternative model is quoted anywhere. The future is unpredictable! But I think we can do better than that. Hansen’s 1988 work compared with observations since then is pretty good proof of that.
The claim of lack of resources here is specious. Open source software has been developed for decades outside of regular corporate salaried channels; there’s no lack of free programming talent around just waiting for a challenge to prove itself. What’s more, there’s tons of free climate modeling software already available – download NCAR’s well-documented community climate model for instance. There’s EdGCM available for free, runs on a desktop PC or Mac. And if you need more power for your model, do like climateprediction.net and get millions of people across the internet to do your work for you…
So, really, why haven’t “skeptics” come up with realistic models that don’t show as much warming? Or have they, and I just don’t know about it? What’s up with that, really?
Gavin needs to remember the old adage “better to remain silent and be thought a fool than to speak and remove all doubt.”
Publishing this data was incompetence. This type mistake is SO obvious that it should have been caught. It has to place into doubt the quality measures they have in place.
Gavin’s explanation is worse.
Arthur–
So what if skeptics don’t build AOGCMs?
Is there some rule that says everyone on the planet or interested in the issue of global warming should be working on predictive models of any sort whatsoever? If so, then an awful lot of apparently good work in climate science fails the test. DOEs ARM program is focused on collecting data. They haven’t developed any AOGCMs. So, is that bad? Judy Curry has done hurricane work– and built no AOGCM ore predictive model. Is her work worth of he dust bin? Has Kerry Emmanuel built an AOGCM? Or developed a predictive model for climate?
As for lack of resources, why does your response address software and not more inportant resources like time and electricity? Even if all the software is free, why should the skeptics you list — or the “many” Gavin criticize why would anyone want to boost their electricity bills keeping their lap tops running day and night for climatepredictions.net? (That said, evidently, some of the many must, as I am aware the program exists.)
And why should the task you seem to be admire take precedence over figuring out if the models that currently exist are able to project future changes?
I’ll tell you why I spend time testing IPCC projections: I think testing whether the projections actually disseminated by the IPCC in the AR4 are consistent with data is more important than trying to develop a predictive model. I think it’s much useful than turning my mac into a drone for climateprediction.net. (But if someone else wants to help climateprediction.net out, bully for them!)
I suspect the main reason many don’t invest their time trying to develop predictive models is the think models are only useful if they have skill. For that reason, many notice the plethora of preditive models but believe there is insuffient attention paid to testing the predictive fidelity of the already existing models.
Under the circumstances, I think it’s irrational to develop more and more poorly tested models, and would prefer to spend my resources: i.e. time, internet connection and electricity, comparing the model projections to observations.
Also, I doubt if Steven Schwartz is a skeptic. But if you like, I’ll ask him.
Yes. Gavin’s explanation, and some of his replies in comments are worse than the article itself. The janitor in his office should bind Gavin’s hands and prevent typing until Gavin calms down.
My favorite comment was from John Phillips calling Gavin’s response ‘dignified and measured’. But touting models over data and whining about Freedom of Information requests seems more ‘outrageous and hysterical’ to me.
===========================================================
Kim– My favorite is the inline response to BD, which suggests the processing screens for unphysical outliers.
The outlier in Turhuanks was +13.53 C, a record for the entire observational history (since 1881). Since that time, the proportion of anomalies greater than 10 C is 0.4%.
The change in temperature was the suspicious 0C. The smallest magnitude Sept-Oct change ever is -2.8C. All changes are negative.
It’s true that no automated error traps can catch everything. But GISS processed data that contained numerous redundant unprecedented events. Is the QA limited to catching over flags like 999?
It’s also true the GISS system is hardly mission critical and mistakes don’t harm anyone’s life. But Gavin’s defense of the error is worse than the error itself.
In that comment, Gavin also complains about a tone of mock outrage. That’s not the tone I’m hearing!
Does anybody else remember Gavin Schmidt’s take on model vs. data last January, in reply to Tierney Lab’s questions such as “Are there any indicators in the next 1, 5 or 10 years that would be inconsistent with the consensus view on climate change?â€
Schmidt: “[the right question to ask] should be, are there analyses that will be made over the next few years that will improve the evaluation of climate models?”
http://www.realclimate.org/index.php/archives/2008/01/uncertainty-noise-and-the-art-of-model-data-comparison/
In the RealClimate world, observations are subordinated to the models. If you accept that, the rest of their blog starts making sense.
It seems like the hurricane path models are extremely useful out to four or six days. The weather models are also very useful out to maybe 5 to seven days. As it turns out that is sufficient for our needs. The reason these models perform so well is that they can be restarted every day to give us a continuing look ahead. Climate, on the other hand, is defined better by a thirty or fifty year period. The Global Climate Models are plagued by the same type of problems that hurricane and weather models have, however, we must wait many years to see if the predictive power was useful. As of now, they have NOT been useful. Climate changes in several decades to be wetter, drier, warmer or hotter. If we cannot predict these basic things more than a week into the future now, how can we be talking about skill in a computer program that is attempting to predict a range of these parameters on time scales of not only decades, but centuries. It seems to me that reliance on anything so obviously implausible, is ludicrous at best.
I remember when it was thought that computers could gain an intelligence equal to humans- artificial intelligence. A computer cannot even carry on an intelligent conversation after many years of programming. Building a skillful GCM is an admirable goal, however, it is only a goal. It has not been accomplished. After the GCMs have been continually updated and improved over a period of a century or so they might become useful, but I doubt it.
Arthur,
assuming I, or a team I was associated with, had the knowledge, skills, and computer resources to build an AOGCM, the current CONSENSUS would REQUIRE that it be VALIDATED!!
Now, you explain to us exactly HOW, and HOW LONG, it would take for this Consensus to accept a reasonable model.
Then tell me how this would be of any use to us years after all the changes in our Sociopolitical systems had already become entrenched!!!
Saying I told you so while gardening with wooden implements isn’t my cup of tea!!
TomVonk–
You forgot the cost of writing the peer reviewed article . . .
Lucia consider my post as an article and you have just peer reviewed it 🙂
I don’t charge my time here so the cost of writing the article is 0 .
Your peer review having been done – do you charge something ?
I am of course aware that the above costs bear no relevance to a peer reviewed article by NASA’s representatives especially when they are submitted to a rigorous quality control and necessitate the contribution of a large multidisciplinary team containing but not limited to cooks , plane pilots , waitresses and small green men .
But even then …
Re: (#6598)
“exactly HOW, and HOW LONG” would it take for a validation of the model? – (I have rather freely interpreted the question)
I think this question squarely raises a matter in which I have more than a passing interest.
The whole raison d’etre of a model is to avoid having to replicate the entire system which you are modelling.
In that respect it is similar to drawing a topographical map – essentially, the scale will be limited to what it is practical to reproduce. If you try to make the scale too large, you face a number of problems. The map will not fit on available media, the data files become too large to handle and so on. If you make the scale too small, the map becomes progressively more inaccurate. The ‘accuracy’ with which the map depicts the terrain is in proportion to the scale employed to produce it. But whilst theoretically, one can make the map as close to 100% ‘accurate’ as desired, simply by arranging for the scale to approach more closely to 1:1, you run into the problems mentioned.
At 1:1, the map/model becomes co-extensive with the terrain that is to be ‘represented’ and the map/model ceases to be of use – one might as well work on the real thing!
I make the following points –
1. A model is a simplification of a subject/system – it is a ‘scaled’ representation.
2. To the extent that it is a simplification, ANY model contains inherent inaccuracies which are not the same as errors arising as a consequence of incorrect inputs of parameters or equations and so on, but are a consequence of scaling.
3. The uses to which the model may be put are strictly limited by the extent of the ‘scaling’ – there is a scaling threshold.
It should be possible to quantify the scaling property for different purposes, but that is somewhat out of my field.
Without really knowing, I would guess that the uses to which current climate models are being put exceeds that scaling threshold.
lucia (Comment#6587) November 12th, 2008 at 4:28 pm
Kim– My favorite is the inline response to BD, which suggests the processing screens for unphysical outliers.
The outlier in Turhuanks was +13.53 C, a record for the entire observational history (since 1881). Since that time, the proportion of anomalies greater than 10 C is 0.4%.
The change in temperature was the suspicious 0C. The smallest magnitude Sept-Oct change ever is -2.8C. All changes are negative.
It’s true that no automated error traps can catch everything. But GISS processed data that contained numerous redundant unprecedented events. Is the QA limited to catching over flags like 999?
What surprises me lucia is that NOAA claims a fairly wide ranging system of QC tests which should have covered these eventualities, but they apparently didn’t work. NOAA has a much bigger investment in GHCN than GISS has in GISTEMP so I’m surprised at the error. Additionally I see no acknowledgment on the GHCN site that the error occurred at all!
Phil–
Yes. NOAA should have caught this too! SteveM wrote about NOAA, and has been complaining about GHCN for a long time.
It’s very odd neither one caught this.
In fact, one of the reasons things are caught in science is generally, everyone is very anal about catching these sorts of mistakes. So, ordinarily, both GHCN and GISS should be acting very embarrassed.
Aren’t acting defensive and attempting to shoot the messenger classic symptoms of severe embarrassment?
Dewitt–
Yes. Silence can also be a symptom of embarrassment, but it’s a more productive response than shoot the messenger.
I would say that acting defensive and attempting to shoot the messenger are really evidence of an inability to take responsibility and improve the situation.
Imagine this response instead, “Thanks for bringing this to my attention, I will make sure that the proper people are alerted and something is done as quickly as possible. I don’t know how this could have happened, but I will find out, and I’ll do everything in my power to make sure it doesn’t happen again. Please give me a day or two to get back to you and let you know what I found out.”
On reason non warmers would not produce a model is that is not clear how to do a model with useful predictive skill over the timescales which interest us short lived humans. It may not be possible because we lack the knowledge of the underlying processes. It may not be possible at all.
A project more on the scale of what individuals and small groups of amateurs could do is OpenTemp.org. With an open source implementation of GISSTEMP, we could patch to pickup the missing stations, and eventually directly collect all stations which are online.
However, this project seems to have gone in to hibernation, and it is in C# which is not a great choice if you want another open source champion to pick up the ball.
Sean,
I’ve visited OpenTemp and never been able to make heads or tails of the overall goals. He wants a tool to find trends in temperatures in the lower 48? Based on what?
Are there any processed sets available? I thought I read insinuations by “someone at another blog” that John’s program created a set but it was not different from GISS. But hunting around when I hunt around http://www.opentemp.org/main/ I find miscellaneous bits of code, a roadmap with no hyperlinks, no archives etc.
Lucia,
there are no archives for OpenTemp. You just download the program, list the stations you want to use and run the program. The result is a CVS file which you load into excel. If you ask johnV he’ll give you the list of stations he used. we should probably revisit it. It’s been a while. The goal was to build an emulation of GISTemp and then change assumptions ( stations) to see how well it tracked
Steve– No weighting by area? (Some regions have a zillions stations per unit area, some have few.) No UHI correction? No worrying about discontinuity due to moving?
Still, if someone wants to run it and explain how and why they picked certain stations, it might be interesting for them to run it and blog about it. I wonder why JohnV doesn’t run it and blog about what he found? It seems like writing it must have been a lot of work, and then it’s just sitting there doing nothing.
If open code can reproduce GISSTemp, and a reliable person or persons runs every month, when the real GISSTemp goes of the rails, we would know it instantly. The open source could have the Q/A anyone wanted to code.
Then as steven mosher says you could offer non compatable options like, include missing stations mode, correct obvious location errors mode.
Sadly the site has not been updated in 2008.
TomVonk (Comment#6557) “…they [stock price data] also happen to be sometimes wrong .”
This is where you go wrong, badly. Stock price data is never wrong – a price is a price is a price. $10.51 is $10.51 and never $10.51.5 or maybe on occasion $10.50.
What you describe in your comment is processing error, and the kid doing the programming was simply implementing some basic controls.
Measurement of physical quantities however is different – it is never correct and is always only an estimate (to some level of accuracy).
There is a world of difference between the data Gavin deals with and stock prices. Trust me on this one, I have dealt with both in my career – most recently with stock prices and interest rates – there is a difference.
Lucia, I believe there was no weighting by area. I’ll look at the source or ask JohnV. UHI. no correction, but JohnV focused on good sites in rural areas. This is one aspect that needs a double check. more later its getting late
Steve,
It would be interesting to do some comparisons. But in order to understand what any comparison might mean, the logic underlying choice of stations, weighting, no weight etc. needs to be translated into a human language, not C.
While various people disagree on conclusions, SteveM and Tamino share the desirable trait of doing analyses and presenting graphs, figures and narratives that explain what they think those analyses mean. OpenTemp appears to be missing that. (Or, JohnV provided the analyses, but what’s there is invisible because there are no accessible archives and the main contributor is silent, so we can’t ask or search.)