The Blackboard

Lucia,

You may be aware that the Bulletin of the American Meteorological Society is about to publish this paper apparently confirming the accuracy of the IPCC climate models. The blurb for the article is as follows:

The study titled “How Well do Coupled Models Simulate Today’s Climate?” is due to be published this Friday in the Bulletin of the American Meteorological Society. In the study, co-authors Thomas Reichler and Junsu Kim from the Department of Meteorology at the University of Utah investigate how well climate models actually do their job in simulating climate. To this end, they compare the output of the models against observations for present climate. The authors apply this method to about 50 different national and international models that were developed over the past two decades at major climate research centers in China, Russia, Australia, Canada, France, Korea, Great Britain, Germany, and the United States. Of course, also included is the very latest model generation that was used for the very recent (2007) report of the Intergovernmental Panel on Climate Change (IPCC).

On my reading of the paper, however, the observations of the ‘current climate’ that they use to verify the accuracy of the models (Table 1 in the paper) appears to be from the period 1979-1999.

Thanks Peter!
Yes. often the comparisons are to earlier years. Oddly, that doesn’t bother me so much. (Though it may bother others.)

The IPCC does discuss AOGCMs quite a bit in their reports. But it’s not clear to me that’s sufficient if we wish to discover how well they predict zero order metrics like GMST.

The difficulty is that the IPCC doesn’t use output of coupled AOGCM models directly to make their projections. They use a hierarchy of models. Simpler models are created and tuned to the coupled models. Then the simple models are run using forcings. I assume initial conditions must be set for the simpler models as well. I discussed this here: http://rankexploits.com/musing.....ound-2000/

So, the difficulty is that even if the coupled AOGCMs models worked almost perfectly, uncertainty could be introduced when the much simpler tuned models are used to make projections into the futre. Of course, it’s also possible the tuned models work well. Still, clearly the predictive fidelity of the final product ought to be tested.

This can all get a bit confusing. This is especially so since whenever I read discussions at climate blogs (including those written by climate scientists) or articles explaining why the public should have confidence in the IPCC models, the people supporting the accuracy of models only discuss the more complex AOGCM’s. The simpler tuned models, which really provide the projections are rarely mentioned.

I thought there was criticism of you using 2001 to 2008 (or even 1998 to 2008) as not representing any climatic trend. But for them its AOK to use 1979-89? I’d prefer up to date data any day is,t that more relevant? I thought Christy’s recent paper showed that nearly all the models were out of line

@1495 (Lucia)

Thank you for the additional clarification, Lucia. I had misunderstood this cascade in the modeling, which your earlier post explains much more carefully. I’m not sure why this cascade exists unless it is because the full models are simply too clumsy or too big to set up and run regularly. In any case, as you say, the point is to know whether the cascade’s output fits the observed reality since they last set the ‘tuning’ parameters.

Following Peter’s comment about the new paper celebrating the success of past models, I am not sure that it is very impressive that the models between 1979 and 1999 projected the warming trend to stay much the same as it was in the recent past. It is surely relatively easy to predict that trends will continue to be basically as they are at present.

I wonder how many models they can find that predicted the flattening out of the warming trend since 2001, given that CO2 continued to rise as before?

I’m a member of AMS (in reference to Peter’s post.) As far as journals go I think their journals are fairly balanced. That said, I think the reaction to Reichler 2007 will be pretty interesting, and that’s a forecast you can take to the bank.

A quick read tells me they don’t seem to suggest anything about the projections of the future, and that the paper focuses entirely on the present.

Great site Lucia, even though I struggle to follow the maths.

Re models, I wonder how much has changed since Gerald North of Texas A&M was quoted in Science 292, p192 2001 as saying “There are so many adjustables in the models and there is a limited amount of obsevational data, so we can always bring the models into agreement with the data” Thus an evermoving playingfield.

Thanks Dave.

I don’t know now much the AOGCM’s have improved. Based on Rahmstorf’s justification of validating the 2001 projections using data from 1990, one might think some climatologists believe they haven’t been modified at all!

But other climatologists insist that there is constant evolution and effort to improve them. I tend to think the latter.

The AOGCM’s do contain many parameterizations; I have no idea what their final predictive skill is. From my point of view, the proof is in the pudding. If they can predict, then that will prove itself when their predictions are compared to data.

Relative to Validation of predictions/projections/forecasts, was the present La Nina foretold by any organization? Several organizations are saying now, after the arrival, the recent coolness is a direct consequence of the phenomena.

Thanks for any info.

Dan,
The IPCC doesn’t predict weather. So, no, they didn’t predict La Nina. Recent coolness is likely the result of La Nina, just as late 2007 hotness was the result of El Nino.

With regard to the trend, these are supposed to cancel out over time, so that’s not a particularly important to Roger’s graph showing data since 1990. More importantly, Roger’s graph stops in 2007, so the recent coolness doesn’t show. Had he extrapolated the recent two months and added 2008 data, we’d see a big dip. ( That would also have been misleading, because only 2 months of 2008 data were available at the time Roger’s letter was published. So, I’m guessing Roger didn’t show extrapolated data because he prefers not to mislead reader.)

Lucia,

I’m a real layman when it comes to the topics you and your colleagues are discussing, but reading Peter’s comment above about the recent paper that seems to validate the existing climate models caused me to think back on the process of building financial models (something I do know reasonably well) and it occured to me that modelers use data to construct and test their models. My guess is much many of the climate models in use today were developed using data sets similar to that used by the authors of the article Peter mentioned to test the validity of the models.

Seemed like a circular argument to me. I read the first few pages of the article. On page 5 the authors acknowledge the fact that:

“Present climate, however, is not an independent data set since it has already been used for the model development (Williamson 1995).”

Can you explain to a layman how the article Peter cited adds anything to the current debate. Seems to me that all it does is validate that the guys who built the models did their work well and that their models are an accurate expression of the data they used to build them. So what?

I’ve enjoyed reading your blog. Thanks for the efforts you are making.

I agree with Scott when he says so what. The Reichler and Kim paper may validate existing climate models but how can policy decisions be based on these models when it points to so many problems with climate modelling – eg,

“However climate models are not perfect. Our theoretical understanding of climate is still incomplete and certain simplifying assumptions are unavoidable when building these models. This introduces biases into their simulations which sometimes are surprisingly difficult to correct.”

“Several important issues complicate the model validation process. First, identifying model errors is difficult because of the complex and sometimes poorly understood nature of climate itself, making it difficult to decide which of the many aspects of climate are important for a good simulation.”

“there is a lack of reliable and consistent observations for present climate, and some climate processes occur at temporal or spatial scales that are either unobservable or unresolvable.”

OOps ! Posted before I’d finished.

They end with couple of “to hope”s ie

” it is to hope these advancements will enhance public credibility of model projections and help to justify the development of even better models”

“it is to hope that the present study will stimulate further research in the design of more robust metrics.”

Again I ask how can policy decisions be based upon these models?

Lucia,

Have you seen William Briggs post today on ‘Why multiple climate model agreement is not that exciting’?

Seems good to me, what say you?

Dave–
I just read it.
Briggs’s point seems to be that even if all models agree they could be biased. Yep.

Historically, computer models for transport of mass, momentum and energy have been known to agree with each other, and yet all give biased answers, when applied to simpler problems. There are reams of papers discussing the reason for biases in specific circumstances after each reason is discovered. The biases can often be shown to exist in simple problems, but propagate into more complex ones.

Why should climate science be the exception?

The models may be precise and accurate. The models may be insufficiently precise or accurate to make meaningful projections. (And I say “insufficiently precise or accurate because often in climate-blog-wars we read “good” or “bad”, with no quantification. When someone claims they are “good”, do they mean: “Can predict a trend to ±0.01C/century?” Or, “Look! We finally have the North Pole colder than the equator! And it’s finally raining in the tropics. Whooo hoooo!” Depending on the state of modeling in a particular field, “good” can mean anything; and what “good” means in a field is generally the sort of “tacit” knowledge Gavin discussed recently at RC)

In any case, plausible arguments that may or may not be sufficiently precise based on level of confidence in modeling choices or ability to hind-cast. But, the truth is, the only way to decide whether the nay-sayers or yay-sayers are correct is compare to data collected after projections are made.

That’s why I think these sorts of comparisons should be done. Ideally, I think the IPCC should propose the appropriate hypothesis test when they make their projection and they should apply these tests themselves and include the results in their assessment reports. Instead, they currently simply provide rather ambiguous, unquantified narratives explaining that the models did “well” in the past, and so we have some, generally unquantified, level of confidence in their projections.

Instead, they currently simply provide rather ambiguous, unquantified narratives explaining that the models did “well” in the past, and so we have some, generally unquantified, level of confidence in their projections.

It is this very point that concerns me most – without independently verifiable projections with objective pass/fail tests, almost any climate change can be said to “generally agree with model output” and falsification is impossible. And they call it “science”…

[...] above from Roger Pielke Jr. via Lucia at the Blackboard (please follow the gratuitous traffic driver link to her [...]

Why doesn’t the temperature graph go up to 2008?

Because this post is from April 2008, before annual data from 2008 was available.

Paul – check the date of Lucia’s post – 2 April, 2008.

PaulM– Long ago, I got in the habit of dating my own blog graphs precisely to make it obvious why data stops. I may need to learn to add these to other people’s graphs when I discuss their work.

[...] Originally I posted a graph from Roger Pielke Jr. via Lucia at the Blackboard because it was somewhat related and I wanted to give her some traffic. As luck would have it, few [...]