Real Climate Tries to Spin Pielke: A curious lesson 2.
Last week I mentioned Roger Pielke Jr’s interesting article comparing IPCC prediction/projections for Global Mean Surface Temperature observations of measurements taken on the true planet earth. Yesterday, Stefan Rahmstorf posted a rather odd commentary at Real Climate. I say odd, because, as far as I can tell, Stefan’s commentary seems to go off on tangents rather than discuss Roger’s major point which would seem to be:
It would be wise for
- the IPPC to formalize their methods to verify models,
- for communicate the results of those verifications clearly and quantitatively and
- communicate the reasons for changes in projections more clearly than found in recent documents.
I agree with Roger on these things. Standard verification methods, quantification and better communication would help the public better understand what is in store for the world’s climate.
So, now that I’ve described what appear to be Roger’s major points, let us examine what Stefan at RC says!
Stefan is concerned that Roger’s one page article doesn’t properly account for ‘noise’ in weather data.
Stefan Rahmstorf says:
If you are comparing noisy data to a model trend, make sure you have enough data for them to show a statistically significant trend.
Accounting for “noise” is important in many analyses, and to support certain types of conclusions. As it happens, it’s not quite clear that it’s necessary to support Roger’s argument in the specific paper. There is no point in logic that says Roger must do a full uncertainty analysis to suggest that the IPCC ought to be doing one!
But, I can think of an example where “noise” should have been accounted for formally. That would be Stefan Rahmstorf own Rahmstorf et al. 2007, published in Nature. Stefan’s article shows a graphic comparing observations of GMST to IPCC projections of GMST. That graphic includes uncertainties in predicted trends, but it includes absolutely no indication of any uncertainty associated with determining an underlying trend from noisy data.
Yet that graphic is the main evidence Stefan used to support his major points about the correspondence between projections of GMST and data. And that’s what Stefan is supposedly doing in his paper. David Stockwell’s recently discussed the uncertainty associated with Rahmstorf’s trend analysis here. Accounting for this uncertainty in determining trend from weather data would impact the conclusions in Rahmstorf et al. 2007.
What happens if we account for uncertainty in the noisy weather data?
It just so happens that using GMST data available since 2001, it is possible to show that we can exclude certain predicted trends as true and we can say this with a relatively high degree of statistical confidence. We find the 2C/century trend predicted/projected by the IPCC AR4 document is falsified to a confidence of 95%. Could 2C/century still be true for the underlying trend? Sure. Things that happen 5% of the time, happen 5% of the time! (Other caveats also apply as mentioned in my previous articles.)
The major results are shown graphically:
The red line is the trend projected by the IPCC. The fuzzy bands are the sorts of uncertainties they communicated to the public. The purple ones are trends consistent with data since 2001.1
Stefan seems to suggest there is a magic number of years required to evaluate any and all claims about weather trends.
Speaking of Pielke and John Tierney’s earlier discussions of 8 year trends (and forcing us to search for links), Stefan at Real Climate tells us:
We showed that this period is too short for a meaningful trend comparison.
And suggests,
“. . . 17 years is sufficient.”
Stefan provides no explanation why 17 years sufficient, and no explanation how the reader might determine whether 15 years, 12 years or 10 years might be enough. (Though, I should note that the criteria for sufficiency of data are somewhat well understood as a matter of statistical testing.)
Stefan’s “proof” that 8 years is too short is, evidently, a his own blog post containing a chart showing that eight year down-trends do occur during period with a long term uptrend.
What does Stefan’s previous blog post discuss? It discusses a period of time during which downtrends were caused by major volcanic eruptions. You can read more about why this sort of proof is totally absurd here . The argument can with these two graph which illustrates that, the longish dips in temperature are associated with volcanic eruptions:
It is no surprise that many aren’t entirely convinced by the Real Climate “proof” that 8 years is too short to test any and all claims of climate trends during any and all periods. The fact is that, when comparing data for GMST to predictions of GMST, everyone’s own tacit knowledge (aka common sense) tells them that they need to apply the first of Gavin and Stefan’s three rules from “lesson 1″:
Firstly, are the drivers changing as we expected?
In Real Climate’s example “proof”, the 8 year dips were associated with an unpredictable driver called “volcanic eruptions”. These are absent from the recent period and cannot explain the failure of the IPCC projections to track GMST since 2001. Additional statistical statements involving words like “standard deviation” and “uncertainty intervals” could be made, but need I make them here?
Stefan criticizes Roger Jr. for not properly distinguishing between forecasts and scenarios.
Stefan says,
3. One should not mix up a scenario with a forecast - I cannot easily compare a scenario for the effects of greenhouse gases alone with observed data, because I cannot easily isolate the effect of the greenhouse gases in these data, given that other forcings are also at play in the real world.
Indeed. When evaluating IPCC reports, modelers often wish to make the distinction between forecasts and scenarios. But I see no evidence Roger made this mistake.
But, since it’s Rahmstorf criticizing Roger Pielke Jr., one might ask: As a practical matter, with regard to emphasizing the distinction between forecasts and scenarios, how do Rahmstorf et al. 2007 and Pielke 2008 differ?
As lead author of Rahmstorf et al. 2007, Stefan compares projections to observations. He perceives differences between the two. He discusses why observations and projections may differ: This is a question of interest to people who develop climate models, and also of interest to the voting public. Yes. We would all like to know why the models and data are different.
In Roger’s article, Roger similarly compares the projections and observations. Like Stefan, he notices differences between the two. Roger suggests both the model projections and the comparisons to data are useful. But, rather than discussing why observations and projections differ, Roger suggests the information, as communicated to the public might be somewhat confusing. Roger then suggest the method of communicating the projections and their relationship to observations could be improved, saying:
Such projections, no matter how accurate, can help to accelerate understanding about the human effects on the climate system as well as the limits to our ability to predict future consequences of those effects with accuracy and precision.
To facilitate such comparisons the IPCC should (1) clearly define the exact variables in its projections and the appropriate corresponding verification (observational) datasets, and (2) clearly explain in a quantitative fashion the exact reasons for changes to its projections from assessment to assessment, in even greater detail than found in the statement in 1995 regarding aerosols and the carbon cycle. Once published, projections should not be forgotten but should be rigorously compared with evolving observations.
There is no evidence Roger has confused forecasts and projections. Roger treats projections as projections, calls them projections and discusses how the information in projections could be made more useful to people (like voters and policy makers) who wish to better understand climate.
And though Roger doesn’t say this, it may be that he would suggest that the forecast/projection confusion might be confusing to the public, and maybe clearer explanations on the part of the IPCC could clear up this confusion.
Yet, Roger’s suggestion that the IPCC could improve the utility of their projections by making verification clearer and more quantitative, seem to bother Stefan who complains:
1. IPCC already showed a very similar comparison as Pielke does, but including uncertainty ranges.
But I would say this: The IPCC’s comparisons are similar to Roger to the extent the two share deficiencies.
Roger’s comparison uses no standard statistical techniques to account for uncertainty in noisy data; neither do the IPCC or Rahmstorf’s. Roger’s comparison does not quantify the degree of agreement of disagrement; neither does the IPCC (or Rahmstrof). Roger’s method is purely visual; so are Rahmstorf and the IPCC’s.
These features are flaws in when they appear in the IPCC documents in Rahmstorf, but not Roger Pielke’s paper. The reason is subtle: Roger’s point is better, more quantitative comparisons are required. He does not appear to be holding his method up as the gold standard, or even claiming it is adequate!
Conclusion
I concur with Roger the IPCC documents require clearer explanations discussing why the model projections change over time. The documents should be expanded by one or two pages to include clear discussions of verification of past IPCC projections. The results of these verifications should be quantified and communicated clearly and unambiguously. The exact reasons for changes in projections should be explained, and when possible related to the quantative discussions of the verifications done on past projections.
As it stands, the narratives in the IPCC amount to qualitative feel-good ruminations that provide little guidance to those who wish to better understand how we are to interpret the current projections. Better comparisons are required. I applaud Roger for saying so, and Nature for publishing the article where he said this.
End notes:
1. Note: My use of the word “projection” is in the sense of “extended line”. As in: The tree limb projects over the fence. Given the size of the IPCC chart, and the crudeness of my graphics tools, it’s difdicult for me to draw short lines and check that I inclined to match the trend I wish to show. In future, I plan to be careful with the use of the term “projection” when I mean, “I extended the line due to the limitations of my crude graphics tools.”
2. Roger Pielke blog is having difficulties. The best link to his response to Real Climate post seems to be: Rogers’s response:
Previous Post:
« Slow Down Boris: Wordpress plugin!
Next Post:
“If the conclusion on CO2 is desperately self-evident …”: Congress should act! »
39 Responses to “Real Climate Tries to Spin Pielke: A curious lesson 2.”
You can leave a response, or trackback from your own site.
BarryW April 11th, 2008 at 11:12 am
Their examples expose another problem with their position by using volcanic eruption related down turns. They keep saying the models account for the major drivers. If previous downturns were due to volcanism and since there is a dip that they have no verifiable explanation for and is not volcanic, then there must be drivers they have not accounted for, so how can they claim robustness for the models?
lucia April 11th, 2008 at 11:22 am
BarryW–
And moreover, we know they believe those dips are due to volcanic eruptions because the fact that they did hurry up, code up the model after pinatubo and predicted the temperature dip before it happened is part of the proof the models have some predictive ability.
Clearly, the models have some predictive ability. The real questions are: how much? So the pertain to quantifying that ability. Also, the issue with respect to the full IPCC modeling process, does the full modeling process result in something people can understand? Does it help them make useful decisions?
(It’s also necessary to make sure we don’t solely go off onto the “How goo are AOGCM’s” question. As, they represent only a portion of the models and methods. The full methodology start with creating scenarios, using one sort of model, estimating forcings– using models– running AOGCMS, creating and tuning “simple models” to match AOGCM’s, running the simple modles and then doig some statsitical treatment.)
BarryW April 12th, 2008 at 7:08 am
I’m not sure I’d count what is almost retrofitting a model to what should be a known driver as showing predictive capabilities. I’m coming to the position (admittedly one tempered by a lot of ignorance) that the models are useful for measuring sensitivities because you can hold other variables static. Because of the number of apparent (and unquantified) feedbacks in the real world I doubt their predictive capabilities. Atmoz has a post on the Gray hurricane predictions that says that the average from previous years is a better predictor than Gray’s predictions. Admittedly that is weather and not climate. I wonder how the models are doing (as opposed to what the IPCC has derived from them) relative to the time frame you’ve been looking at? Are they any better than a short term average or tend.
lucia April 12th, 2008 at 7:14 am
BarryW:
I don’t know the answer to that. The reason I limit myself to what the IPCC document communicate to the public is that document is supposed to be, in some sense, a consensus of what climatologist thought most likely at the time the document was written.
I have no doubt that someone, somewhere has models, and individual model runs, that would match the current behavior. I also have no doubt that someone, somewhere could dredge up models runs that would be so hilariously incorrect as to leave us gasping.
But given that the IPCC– and climatologists– are relying on some sort of averaging to project into the future, it seems reasonable to discuss only those results that were communicated as plausible.
bender April 12th, 2008 at 7:39 am
Nice analysis and a fair parsing of the exchange. Maybe there is a cure after all for the muddy-headed thinking about “weather noise”, “internal climate variability”, and model & data and uncertainty that passes in the literature as “science”. A higher level of scrutiny is clearly warranted. A reply from Dr. Rahmstorf would be most welcome.
lucia April 12th, 2008 at 7:46 am
Thanks bender! (I don’t think I’m on Real Climate’s radar screen. And, for some reason, Wordpress doesn’t seem to ping them the way it pings other people I link. I don’t know quite how that happens, since I thought that pings just go “out”, but can be ignored.)
Trivia for the minute: When the email alerting me to your comment arived,I clicked to the screen, faced the window and said “S..t! It’ snowing!” I went to the window, opened it, there are flakes– they are wet and not accumulating. But…
I may need to look up when we last had mid-April snow. I’m sure it’s happened, but, darn! I set up my spring house last week to get my seedlings started, and I don’t want it covered in snow!
Ian Castles April 12th, 2008 at 8:28 pm
I share Bender’s view that a reply by Dr. Rahmstorf to the serious criticisms that have been made of Rahmstorf et al (2007) would be most welcome. This should have been a higher priority for him than that of posting his ‘Model-data-comparison, Lesson 2’ [for Roger Pielke, Jr] on RealClimate.
But all seven of the Rahmstorf et al co-authors have some responsibility in this matter, as do the six prestigious research institutions in five countries (Germany, France, Australia, the UK and the US) with which they are affiliated. And so too does the journal in which the article was published (which, incidentally, was the US-based ‘Science’, not the UK-based ‘Nature’).
Stefan Rahmstorf was a Lead Author of Chapter 6 of the 2007 WGI report. Other Rahmstorf7 co-authors were Coordinating Lead Authors, Lead Authors or expert reviewers of Chapters 1 (Somerville), 3 (Parker), and 5 (Cazenave and Church) . The article was a product of members of the IPCC milieu, and was published online in the same week as the release in Paris of the IPCC WGI report (or, strictly speaking, its Summary for Policymakers).
Yet another co-author of the article in ‘Science’, James Hansen, has recently written to Australia’s Prime Minister (and in similar terms to all Australian State Premiers) calling on him to exercise his leadership ‘on a matter concerning coal-fired power plants and carbon dioxide emission rates in your country.’ Hansen explains that, because of the urgency of the matter, he has not collected signatures – but he offers the names of seven Australian scientists whom Mr. Rudd could consult.
Hansen also says that he had ‘read and commend[s] the Interim Report of Professor Ross Garnaut’, which has been submitted to all Australian Governments. According to this Report, ‘Comparisons between observed data and model predictions suggest that the climate system may be responding more quickly than climate models indicate’ (Rahmstorf et al, 2007) and, specifically, that ‘Global mean surface temperature increase since 1990 has been measured at 0.33 C, which is in the upper end of the range predicted (sic) by the IPCC in the Third Assessment Report in 2001, as shown in Figure 5 (Rahmstorf et al 2007).’ As Lucia says in the above post, the paper on which the Australian report has relied ‘includes absolutely no indication of any uncertainty associated with determining an underlying trend from noisy data.’
Francois O April 13th, 2008 at 8:09 am
There is a very interesting paper about how to deal with uncertainties in models. Although the paper is not about GCM’s, it is about carbon cycle models, where they also have to deal with the same two types of uncertainties: there is uncertainty in the observational data, and there is “structural” uncertainty, that is uncertainty about the physical processes themselves. The abstract reads:
In their conclusion, the authors also add:
This is more or less a candid admission that you can always retrofit the model to the data if your data or model uncertainty is large enough. But that doesn’t help your predictive ability, because if there are processes that are not included in your model, you can miss the boat entirely.
On another topic, am I the only one who has problems with what is called “noise” in climate data? From a physicist’s point of view, noise comes from the measuring instrument. Or else, it is an intrinsic, and probabilistic, feature of the phenomenon, like quantum noise in light measurements. But climate, albeit chaotic, is deterministic to a large degree. What is called noise is deterministic variability. Furthemore, this variability is not independent from any underlying trends, because it’s a nonlinear system with many feedbacks involved. A major volcanic eruption could, in principle, change the course of climate by triggering albedo feedbacks. Major El Ninos do not occur randomly either. The heat that they dissipate must come from somewhere. So even if we can’t predict exactly what next year’s average temperature will be, we could, in principle, have an understanding of the variability that constraints our predictions, to a narrower and narrower range as this understanding gets better and better. That’s what wheather (not climate) models do, in fact. My point is that it’s just to easy to dismiss the variability as “noise”, as if it were entirely unpredictable. In fact, it just points to our ignorance of what’s really going on.
Francois O April 13th, 2008 at 8:13 am
P.S. It’s also snowing (again!) in Montreal this morning…