Roger Pielke Jrs’ letter: The modest proposal.
Nature Geoscience published a letter to the editor by Roger Pielke Jr.; the title is Climate predictions and observations. In his letter, Roger included a graphic illustrating IPCC temperature and sea level changes from the first through fourth assessement reports, and compared those to data.
Roger made some brief observations about the comparison, and closed his letter with these words:
Exercises such as these highlight the usefulness of projections of relevant climate variables that can be verified on timescales of a few decades or less. Such projections, no matter how accurate, can help to accelerate understanding about the human effects on the climate system as well as the limits to our ability to predict future consequences of those effects with accuracy and precision.
To facilitate such comparisons the IPCC should (1) clearly define the exact variables in its projections and the appropriate corresponding verification (observational) datasets, and (2) clearly explain in a quantitative fashion the exact reasons for changes to its projections from assessment to assessment, in even greater detail than found in the statement in 1995 regarding aerosols and the carbon cycle. Once published, projections should not be forgotten but should be rigorously compared with evolving observations.
I tendorse Roger’s suggestion to the IPCC. Careful attention to verification of past projections against later data would help the public and their chosen policymakers better understand the certainty of AGW while quantifying the predictive fidelity (or infidelity) of models and methods used to predict the magnitude of zero order metrics like Global Mean Surface Temperature and Sea Level Rise.
The IPCC should be aware that if they fail to clarify these things, others will nevertheless examine the data and draw their own conclusions. Climate scientists may or may not agree with those conclusions. However, absent clear verifications with full documentation of verification methods by the IPCC, those who wish to inform themselves have no alternative but to perform their own.
Readers who wish to thank Roger for his message can leave comments at Roger’s Blog.
Previous Post:
« Exploratory Statistics: Roger Cohen Looks at ENSO
Next Post:
Accounting For ENSO: Cochrane Orcutt »
16 Responses to “Roger Pielke Jrs’ letter: The modest proposal.”
You can leave a response, or trackback from your own site.
Peter Gallagher (4 comments.) April 2nd, 2008 at 1:22 pm
Lucia,
You may be aware that the Bulletin of the American Meteorological Society is about to publish this paper apparently confirming the accuracy of the IPCC climate models. The blurb for the article is as follows:
On my reading of the paper, however, the observations of the ‘current climate’ that they use to verify the accuracy of the models (Table 1 in the paper) appears to be from the period 1979-1999.
lucia April 2nd, 2008 at 2:11 pm
Thanks Peter!
Yes. often the comparisons are to earlier years. Oddly, that doesn’t bother me so much. (Though it may bother others.)
The IPCC does discuss AOGCMs quite a bit in their reports. But it’s not clear to me that’s sufficient if we wish to discover how well they predict zero order metrics like GMST.
The difficulty is that the IPCC doesn’t use output of coupled AOGCM models directly to make their projections. They use a hierarchy of models. Simpler models are created and tuned to the coupled models. Then the simple models are run using forcings. I assume initial conditions must be set for the simpler models as well. I discussed this here: http://rankexploits.com/musing.....ound-2000/
So, the difficulty is that even if the coupled AOGCMs models worked almost perfectly, uncertainty could be introduced when the much simpler tuned models are used to make projections into the futre. Of course, it’s also possible the tuned models work well. Still, clearly the predictive fidelity of the final product ought to be tested.
This can all get a bit confusing. This is especially so since whenever I read discussions at climate blogs (including those written by climate scientists) or articles explaining why the public should have confidence in the IPCC models, the people supporting the accuracy of models only discuss the more complex AOGCM’s. The simpler tuned models, which really provide the projections are rarely mentioned.
Vincent (7 comments.) April 2nd, 2008 at 8:05 pm
I thought there was criticism of you using 2001 to 2008 (or even 1998 to 2008) as not representing any climatic trend. But for them its AOK to use 1979-89? I’d prefer up to date data any day is,t that more relevant? I thought Christy’s recent paper showed that nearly all the models were out of line
Peter Gallagher (4 comments.) April 2nd, 2008 at 9:29 pm
@1495 (Lucia)
Thank you for the additional clarification, Lucia. I had misunderstood this cascade in the modeling, which your earlier post explains much more carefully. I’m not sure why this cascade exists unless it is because the full models are simply too clumsy or too big to set up and run regularly. In any case, as you say, the point is to know whether the cascade’s output fits the observed reality since they last set the ‘tuning’ parameters.
Patrick Hadley April 3rd, 2008 at 2:59 am
Following Peter’s comment about the new paper celebrating the success of past models, I am not sure that it is very impressive that the models between 1979 and 1999 projected the warming trend to stay much the same as it was in the recent past. It is surely relatively easy to predict that trends will continue to be basically as they are at present.
I wonder how many models they can find that predicted the flattening out of the warming trend since 2001, given that CO2 continued to rise as before?
terry April 3rd, 2008 at 5:00 am
I’m a member of AMS (in reference to Peter’s post.) As far as journals go I think their journals are fairly balanced. That said, I think the reaction to Reichler 2007 will be pretty interesting, and that’s a forecast you can take to the bank.
A quick read tells me they don’t seem to suggest anything about the projections of the future, and that the paper focuses entirely on the present.
Dave Andrews April 3rd, 2008 at 12:53 pm
Great site Lucia, even though I struggle to follow the maths.
Re models, I wonder how much has changed since Gerald North of Texas A&M was quoted in Science 292, p192 2001 as saying “There are so many adjustables in the models and there is a limited amount of obsevational data, so we can always bring the models into agreement with the data” Thus an evermoving playingfield.
lucia April 3rd, 2008 at 2:39 pm
Thanks Dave.
I don’t know now much the AOGCM’s have improved. Based on Rahmstorf’s justification of validating the 2001 projections using data from 1990, one might think some climatologists believe they haven’t been modified at all!
But other climatologists insist that there is constant evolution and effort to improve them. I tend to think the latter.
The AOGCM’s do contain many parameterizations; I have no idea what their final predictive skill is. From my point of view, the proof is in the pudding. If they can predict, then that will prove itself when their predictions are compared to data.
Dan Hughes (35 comments.) April 4th, 2008 at 6:49 am
Relative to Validation of predictions/projections/forecasts, was the present La Nina foretold by any organization? Several organizations are saying now, after the arrival, the recent coolness is a direct consequence of the phenomena.
Thanks for any info.
lucia April 4th, 2008 at 7:00 am
Dan,
The IPCC doesn’t predict weather. So, no, they didn’t predict La Nina. Recent coolness is likely the result of La Nina, just as late 2007 hotness was the result of El Nino.
With regard to the trend, these are supposed to cancel out over time, so that’s not a particularly important to Roger’s graph showing data since 1990. More importantly, Roger’s graph stops in 2007, so the recent coolness doesn’t show. Had he extrapolated the recent two months and added 2008 data, we’d see a big dip. ( That would also have been misleading, because only 2 months of 2008 data were available at the time Roger’s letter was published. So, I’m guessing Roger didn’t show extrapolated data because he prefers not to mislead reader.)
Scott Fraser (1 comments.) April 5th, 2008 at 11:07 am
Lucia,
I’m a real layman when it comes to the topics you and your colleagues are discussing, but reading Peter’s comment above about the recent paper that seems to validate the existing climate models caused me to think back on the process of building financial models (something I do know reasonably well) and it occured to me that modelers use data to construct and test their models. My guess is much many of the climate models in use today were developed using data sets similar to that used by the authors of the article Peter mentioned to test the validity of the models.
Seemed like a circular argument to me. I read the first few pages of the article. On page 5 the authors acknowledge the fact that:
“Present climate, however, is not an independent data set since it has already been used for the model development (Williamson 1995).”
Can you explain to a layman how the article Peter cited adds anything to the current debate. Seems to me that all it does is validate that the guys who built the models did their work well and that their models are an accurate expression of the data they used to build them. So what?
I’ve enjoyed reading your blog. Thanks for the efforts you are making.
Dave Andrews April 5th, 2008 at 12:04 pm
I agree with Scott when he says so what. The Reichler and Kim paper may validate existing climate models but how can policy decisions be based on these models when it points to so many problems with climate modelling - eg,
“However climate models are not perfect. Our theoretical understanding of climate is still incomplete and certain simplifying assumptions are unavoidable when building these models. This introduces biases into their simulations which sometimes are surprisingly difficult to correct.”
“Several important issues complicate the model validation process. First, identifying model errors is difficult because of the complex and sometimes poorly understood nature of climate itself, making it difficult to decide which of the many aspects of climate are important for a good simulation.”
“there is a lack of reliable and consistent observations for present climate, and some climate processes occur at temporal or spatial scales that are either unobservable or unresolvable.”
Dave Andrews April 5th, 2008 at 12:32 pm
OOps ! Posted before I’d finished.
They end with couple of “to hope”s ie
” it is to hope these advancements will enhance public credibility of model projections and help to justify the development of even better models”
“it is to hope that the present study will stimulate further research in the design of more robust metrics.”
Again I ask how can policy decisions be based upon these models?
Dave Andrews April 8th, 2008 at 5:57 am
Lucia,
Have you seen William Briggs post today on ‘Why multiple climate model agreement is not that exciting’?
Seems good to me, what say you?
lucia April 8th, 2008 at 6:55 am
Dave–
I just read it.
Briggs’s point seems to be that even if all models agree they could be biased. Yep.
Historically, computer models for transport of mass, momentum and energy have been known to agree with each other, and yet all give biased answers, when applied to simpler problems. There are reams of papers discussing the reason for biases in specific circumstances after each reason is discovered. The biases can often be shown to exist in simple problems, but propagate into more complex ones.
Why should climate science be the exception?
The models may be precise and accurate. The models may be insufficiently precise or accurate to make meaningful projections. (And I say “insufficiently precise or accurate because often in climate-blog-wars we read “good” or “bad”, with no quantification. When someone claims they are “good”, do they mean: “Can predict a trend to ±0.01C/century?” Or, “Look! We finally have the North Pole colder than the equator! And it’s finally raining in the tropics. Whooo hoooo!” Depending on the state of modeling in a particular field, “good” can mean anything; and what “good” means in a field is generally the sort of “tacit” knowledge Gavin discussed recently at RC)
In any case, plausible arguments that may or may not be sufficiently precise based on level of confidence in modeling choices or ability to hind-cast. But, the truth is, the only way to decide whether the nay-sayers or yay-sayers are correct is compare to data collected after projections are made.
That’s why I think these sorts of comparisons should be done. Ideally, I think the IPCC should propose the appropriate hypothesis test when they make their projection and they should apply these tests themselves and include the results in their assessment reports. Instead, they currently simply provide rather ambiguous, unquantified narratives explaining that the models did “well” in the past, and so we have some, generally unquantified, level of confidence in their projections.
Neil Fisher April 9th, 2008 at 7:42 pm
It is this very point that concerns me most - without independently verifiable projections with objective pass/fail tests, almost any climate change can be said to “generally agree with model output” and falsification is impossible. And they call it “science”…