As some readers know, there seems to be some debate over whether hypothesis tests of the IPCC projections should use earth data or gcm ‘model data’ to the variability of 8 temperature year (or any number of year) trends observed on the earth’s surface. I’m of the opinion that it is best to use real earth weather variability when applying hypothesis test, but I’m open to using “model weather” if and only if that model weather is not inconsistent with “real earth weather”.
A first check of an AR(1) process suggested by Gavin suggested as “closer” to models than the one that fits real earth weather indicates that Gavin’s “closer” process falsifies when compared to earth weather. So, that comparison did not inspire me to switch to “model weather noise” when testing hypotheses about the current trend in GMST.
However, the test is also not the death knell for the idea of using “model data” to describe the “weather noise”. Gavin didn’t actually say the AR(1) process he suggested describes the model weather well. (Also, I have reason to believe he doesn’t think earth weather noise is AR(1).)
So, even though that AR(1) process falsifies relative to real earth weather, this important more general question remains unanswered:
I don’t know the answer to this, so I’m going to be fishing out IPCC projections for the 21st century, and comparing the characteristics “weather noise” to recent weather noise”.
I’m going to find the comparison rather interesting. We’ve recently experienced a “quiet” time with respect to volcanic eruptions. I think this permits a more stringent comparison of model’s ability to simulate “weather noise” than other periods. Why? Well, when volcanos are erupting, one might expect the month-to-month and year-to-year variability is largely the result of the system responding to the rapidly varying external forcing. ( Major volcanic eruptions can result in changes in forcing comparable in magnitude to the entire amount experienced since the industrial revolution, but these changes occur over the course of only a few years.) So, the observed variability of GMST might be large even if the internal variability for a slowly forced atmosphere is rather small.
Let’s look at some data!
So far, I’ve looked at 2 sets of projections for the 21st century. I’m currently grabbing data and I’ve looked at two sets of projections for the A1B SRES. EchoG, is particularly interesting in that it appears to have particularly violent “weather noise” and consequently, does result in extremely large variations in 8 year temperature trends.
The difficulty is that the “weather noise” in EchoG is amazingly variable. If you examine the figure below, you’ll see how violently EchoG weather see-saws up and down compared to recent weather:
Can this violent see-saw behavior be quantified?
I always like to quantify a bit– as people can sometimes agree on numbers even if they don’t agree on what those numbers mean. So, here are a few numbers:
- EchoG weather noise is big! If we fit a ordinary least squares fit trend to monthly data the individual members of the ECHOG ensemble for SRES A1B, using projections from Jan 2001-Jun 2008, the standard error for residuals are 0.23C, 0.18C and 0.22C. In contrast the standard error for residuals to the trend for real earth data based on a merge of NOAA, HadCrut and GISS Land/Ocean is 0.10C. So, on a month to month basis, EchoG “weather noise” is twice as variable as real earth weather. (If we corrected the earth data for measurement noise, we’d conclude earth weather is even less variable. So, EchoG is quite variable!)
- EchoG lag 1 correlations look big. The lag 1 autocorrelation for monthly GMST based on the three runs of EchoG weather are ρ=0.78, 0.80 and 0.77. In contrast, the autocorrelation for real earth data based on the “merge (3)” set is ρ=0.44. (Recall that the autocorrelation always falls bewteen 1 and -1, so this is quite a sizeable difference. That said, estimates of autocorrelations based on small amounts of data are imprecise.)
- EchoG’s estimated 8 year variability in trend looks big. The estimate standard error of the 8 year trends using Red-Corrected OLS for individual members of the ensemble are: s= ±8C/century, ±8C/century, and 12 C/century. In contrast, the estimate of the variability of earth trends is ±0.5 C/century. The estimate of the variability basedon ECHOG is are high both because a) the month-to-month variability of ECHOG is high and b) the lag-1 autocorrelation for EchoG weather is very high.
- Anomaly differences persist a long time. I compared the three ensemble members (aka trajectories or realizations of weather) to each other. I expected that each member’s temperature projection would periodically rise and fall, crossing the other member’s temperature projection.
Oddly, this is not so. The temperature of member ‘0’ is consistently low relative to the other two. On average, member ‘0’ is 0.4C cooler than the other two trajectories and this persist for two centuries, as can be seen by comparing the blue (trajectory 0) to the red and yellow below:
(I set the baselines to 2001-2008. )
Have we learned anything?
Well, at least one of the models in the IPCC projection has some whopping weather noise. It’s both much more variable than our recent real earth weather and also exhibits stronger lag1 autocorrelations.
I can’t really say much more for now. Tentatively, I’d say this model appears to have “weather noise” that is not characteristic of the “weather noise” the earth has exhibited lately. I plan to be looking at more “model data” and comparing to past non-volcanic periods to see whether the model exhibited unrealistically high weather variability back in the 30s when we had another period with minimal amounts of volcanic activity. That period should give us a nice long run to better determine whether the disagreement over the variability of “weather” is due to the short sample of earth weather or model inaccuracies.
very interesting. A basic question: are the three model runs using different initial conditions? (I assume that aspect doesn’t change with the realizations, but from the graphic you present..?)
Steve– Funny you should ask that. I was just looking that up at LLNL!
The sort of use different initial conditions– but sort of the same. 🙂
Evidently, EchoG ran 5 cases to hindcast 1850-2000. The 3 individual cases above are each initialized with one of the hindcasts. So, they should differ by an amount one might expect to represent the spread of “weather noise” for ‘planet EchoG’.
Why one stays below the others is a mystery to me. I’m trying to find out if there is some convention whereby run “o” has a different baseline, because the only other set I looked at also has a cold run ‘o’. In that specific case, the -0.4C difference for run “0” persists for 350 years! I find that very surprising.
The excess “noise” looks like it’s the result of inadequate damping in the model, which makes me think of the positive feedback that these models indicate.
Perhaps a model that was better able to reproduce weather fluctuations would come up with a lower climate sensitivity.
Is there any way to try to figure out how much damping is missing from the model and what effect this had on the climate sensitivity? Would it be possible to modify your simplified model to produce various levels of weather noise and then see how this effects the sensitivity?
Isn’t the modeled noise above characteristic of models which run violently out of control, and have to be corrected spuriously? Doesn’t this support what Gerald Browning and Tom Vonk say about the state of Climate Modeling. Following the tracks is a bit like sitting with a learner driver trying to stear in a straight line.
Just published: D. KOUTSOYIANNIS, A. EFSTRATIADIS, N. MAMASSIS & A. CHRISTOFIDES “On the credibility of climate predictions†Hydrological Sciences–Journal–des Sciences Hydrologiques, 53 (2008).
Again another peer-reviewed assertion that the models are useless or have NO predictive value
http://www.atypon-link.com/IAHS/doi/abs/10.1623/hysj.53.4.671
Vincent- I saw that paper a little while ago. I didn’t like that the study’s coverage only spanned eight stations. Of course it wasn’t funded, but it would be very interesting what results one would come up with if the coverage were much larger.
Lucia, check out the paper vincent cites ( hey i’m in the acknowledgements) have a look at the hurst coef
of the model runs for the data you have
Steve– They didn’t seem to look at EchoG.
I’m looking at the “no volcano/no bucket-transition” period for this data. What a hoot!
As long as this model is in the mix, IPCC projections can NEVER falsify using Gavin’s method. This looks like a planet with weather created by Greek gods in some ’50s movie like Jason and the argonauts!
As I understand the situation, climate modelers have insisted that their models are not designed for, and are not capable of modeling short term weather variability. They insist, however, that they can successfully model long term trends, since the shorter term weather variabilities tend to “average out”.
On the other hand, they are apparently representing the variation between their model simulations as a measure of weather variability. There seems to be an implication that the models are capable of simulating something they weren’t designed to do in the first place. This seems illogical.
What am I missing?
Re: 4539
Duane,
You’re not missing anything. The dogmatists will hold up GCMs as successful and reliable no matter what, since by definition the model must right. Real world data [or a our gracious host calls it: real world weather] does not appear to matter much. As a result, even when a particular model is blatantly wrong it’s nevertheless still right: it’s what’s verifiably happening in the real world, we are told, that’s somehow out of sync. Think of Trenberth, Schmidt, et. al. reasoning in this matter as a scary re-run of Monthy Python’s “Dead Parrot”.
Ya lucia, it looks like a ping pong ball bouncing back and forth between some boundary conditions.
I’d side with Chris on this one and suspect underdamping. looks like a bang bang system??? does that make sense?
Tetris–
In this instance, the use of “they” may be inappropriate. It’s specifically Gavin at RC who suggests that I need to use the spread for the models because that supposedly tells us something about the variability of real earth weather. Or, at least that’s what I understand him to be saying. I think it should be fine to use earth data to hypothesis test models.
However, the fact that my hypothesis tests involve only 90 months data does mean there are limitations to my ability to test models. So, some aspects of Gavin’s criticism are legitimate. This puts me in a bit of a difficult spot with regard to proving something in a manner that is closer to definitive– in the sense that nearly everyone has to agree something is “proven”.
So, right now, we can have someone like JohnV suggest that my statistical model (AR(1) “weather”) doesn’t account for the 11 year solar cycle: That’s right, it doesn’t. Moreover, 90 months isn’t enough data to test whether or not the solar cycle matters. So, while some here are convinced by my analysis, other’s aren’t. Those doubts are legitimate– but based things we don’t know. (For example, loads of climate modelers insist the effect of the solar cycle is so negligible that they ignore it when running climate models! )
Anyway, I’m trying to do some analyses looking at “model data” to resolve some issue. But, I can’t entirely explain until I do them. So, in the meantime, y’all are going to be seeing some posts whose main purpose will be rather mysterious until we get to the point where I roll all the bits together! (Some may remain mysterious.)
The main things is: I will be comparing “model weather” to “earth weather” and doing some tests of their characteristics. I don’t know how the test will come out. But, the thing about blogs is, whatever I get, you’ll most likely see as I arrive at steps that I can explain at all. That means this will definitely be a “let the chips fall where they may” analysis!
In the meantime, until such time as I become convinced “model weather” is a reasonable facsimile of “earth weather”, I’ll post hypothesis tests every month until such time as I become convinced I can’t. I’ll also be widening uncertainty intervals as I identify features that are quantifiable. So, for example, next month, I’ll overlay the effects of 0.04C uncertainty in measurements on the hypothesis test. (I may do that sooner even– to show the effect on this month’s results. )
Chad,
I didn’t like that the study’s coverage only spanned eight stations.
The paper has been updated following comments since it was presented at EGU, (including comments by Gavin!).
The eight station coverage is a problem but at least all stations had over 100 years record. But as you say the study wasn’t funded, yet surely it is this type of study that climate scientists should be doing so that they can verify their models? Why are they not doing these studies? Why have papers like Stainforth et al, Phil Trans. R. Soc A (2007) 365, 24145 2161 ,which was eloquently critical of climate models, been largely ignored?
I think I know the answer- do you?
here
http://www.fourmilab.ch/hackdiet/www/subsection1_2_3_0_5.html
Dave-
What is the title of that paper? “Confidence, uncertainty and decision-support relevance in climate predictions” by Stainforth, et al?
I have reason to believe he doesn’t think earth weather noise is AR(1)
I’m also inclined to think weather noise does not approximate white noise. I’ve detrended temperatures series in various ways. The distribution of slopes of detrended temperatures appears different to the distribution you’d expect from white noise in a AR(1) process. Just to give you an example, if you detrend the GISS series with a third-order polynomial, you get basically a flat series, right? But then if you get 10-year slopes from the detrended series for every year from 1881 to 1998 (looking ahead) there’s a wide distribution that goes from about -3.0 to 3.0 degrees / century (where the average should be zero for a detrended series, of course). 1998 is actually not unusual at -0.3. I’m not sure it becomes unusual even if you consider volcanic activity.
Here’s the part of the Stainforth et al paper that people seem to be ignoring:
But, yeah, there’s a little something for everybody in there.
Boris. Your quote, “range of possibilities highlighted for future climate”, doesn’t claim the models actually predict reality, just a range of possibilities.
I have always been puzzled that the IPCC never factored in some random “volcanic” activity into the models. They explicitly state in AR4 and previous reports that they specifically ignore aerosol injections from volcanoes. It doesn’t seem right to assume that a typical century of volcanoes has no overall effect on the projected temperature. Each volcano (we tend to get 1 to 2 per decade of noticeable size) depresses temperatures a degree or so for years. One has to assume that over a period of a century that this is a substantial hit on the overall temperature increase, yet the IPCC ignores this forcing. Why? Is there some reason to believe that this volcanic forcing is somehow cancelled out? Do they expect the whole century to be volcano free? It seems to me this puts a bias on higher temperatures.
Saturn-
I think it may be because the aerosol forcing is not as persistent as, say, CO2 forcing. The aerosols in the troposphere get washed out fairly quickly. So an eruption would have cause a noticeable temperature drop, but it will rebound fairly quickly. Also, it may also simplify defining scenarios. Imagine each scenario they laid out times 3. For example you’d have say A1 with no volcanic activity, with little volcanic activity, with moderate volcanic activity, or with large volcanic activity. It would create more work for the modelers to do. Just a guess.
It worries me to think that decisions are made on “life or death” of the planet earth because of too many models. They could have just put in an “average year” from the last 500 or so years. Volcanoes are pretty well documented so I’m pretty sure they could get data that was reliable on the number and severity.
I understand the aerosols are washed out over a year or 2, maybe 3 in a big one but these things happen 10 times a century or more. So, the effect must be significant on the overall temperature increase. I would like to know for the 20th century if you remove volcanoes from the record what would have happened to temperature? I know they have a way for accounting for volcanoes in the models. So, why not assume an average mix just so we don’t believe the results were jimmied to be high.
Consider that everytime a volcano happens now they have to explain why temperatures will remain low for years and what impact it will have on their long range predictions. Wouldn’t it be better if they could say we already counted that in our models. The predictions are unchanged.
Boris,
That’s the standard bit they have to put in in order to continue to receive funding. The rest of the paper is a damning critique of climate models, eg,
Here, our focus is solely on complex climate models as predictive tools on decadal and longer time scales. We argue for a reassessment of the role of such models when used for this purpose
There is no compulsion to hold that the most comprehensive models available will yield decision-relevant probabilities, even if those models are based upon ‘fundamental physics’
Statements about future climate relate to a never before experienced state of the system: thus, it is impossible to either calibrate the model for the forecast regime of interest or confirm the usefulness of the forecasting process
Finally, model inadequacy captures the fact that we know a priori, there is no combination of parameterizations, parameter values and ICs which would accurately mimic all relevant aspects of the climate system. We know that, if nothing else, computational constraints prevent our models from any claim of isomorphism with reality, whatever that phrase may mean
In cases where the reliability of the forecasting system cannot be confirmed, the ability of our models to reproduce past observations in detail gives us some hope that the model forecast may provide valuable guidance for the real world. Climate models fail this test
They then list the areas where models need improvement :-
Examples are the inclusion of a stratosphere, a carbon cycle, atmospheric/oceanic chemistry, ice sheet dynamics, and realistic (ie. statistically plausible equivalents of real world behaviour) ENSO structures, land surface schemes (critical for the exploration of regional feedbacks),diurnal cycles, ocean eddies and many others
And note that Models of such complexity, at high resolution and with suitable exploration of uncertainty are not going to be available soon
Chad,
Yes that’s the one.
Nice logic…you trust a scientist who you also claim is lying to get more money.
That the models need to be improved does not mean they are not good at doing stuff already. So take the quote mining to Uncommon Descent.
Hey Lucia,
In your calculation for the standard error for residuals using the merges NOAA, GISS, and HadCrut, did you change the baseline? If so, how did you do it? I’m still looking for monthly temperatures for the GISS and HadCrut baseline.
Chad–
The baseline doesn’t matter when calculating the standard error for residuals or the slope. But, I do rebaseline. Every month, I find the average for the entire period of interest, and rebaseline each set individually to make the average temperature “zero”. This causes the trend lines to all intersect in the “middle” year, and is useful for graphing the results.
Boris,
Do you realize that part of the sentence you quoted, “The range of possibilities highlighted for future climate at all
scales clearly demonstrates the urgency for climate change mitigation measures… ” was fluff?
Further analysis might demostrate a threat and an urgency, but the range alone demostrates nothing of the sort. Further analysis might rule out all possible threat responses other than mitigation measures, but the range alone demostrates nothing of the sort. A range of possible climates clearly demostrates a range of possible climates. The rest is clear as mud.
Re: 4545 [Lucia] and 4558 [Boris]
Lucia: it is not just simply Schmidt. None other than Trenberth -in an original letter to Nature- last year argued that models do not need real world data to run. Interestingly enough, he also argued that GMCs are useless as forecasting tools and that if anything, we should start focusing more on regional models using real world regional data to see if we can them work – in fact not unlike what Pielke Sr has been arguing. His comments caused so much horse feathers to hit the fan in the dogmatists camp that he subsequently penned a climb down of sorts, fudging the issue.
Boris: do you trust Mann, Briffa, Thompson, Schmidt, Hansen and their ilk? If you do, keep in mind that there is very substantial evidence that they all have taken mind boggling liberties with the truth in order to secure ongoing funding. Fact is, it is the alarmist/dogmatist camp [not the skeptics] who over the past 20 years have been the beneficiary of over US$2 billion in government funding by keeping the AGW/ACC circus alive and going.
tetris says:
Please keep your fantasies and conspiracy theories to yourself. Or don’t. Whatever. the “very” and “mind boggling” were a nice touch, though.
Raphael sez:
And the range is mostly bad for us. And some of the range is very very bad.
‘This research has been completed. No further funding of any kind will be considered or accepted for continued study in this area.’
(edit: oops, that came too late, I was trying to respond under comment 4558)
Boris,
The range may be bad to very very bad for us, but the range itself says nothing to the validity of that assessment. Because the range says nothing on the subject, it does not clearly demostrate “the urgency for climate change mitigation measures.” The statement is fluff.
If I allow that lapse of logical consistancy, we would still have a problem with urgency of measures to mitigate climate change, rather than the urgency of measures to mitigate (common usage of the word) the threats of climate change. Proactive and reactive adaptation are legitimate responses to the threats of climate change. The claim of a clear demostration of “the urgency for climate change mitigation measures” disregards all but one response to the threat of climate change. Ergo there is no clear demostration and the statement is still fluff.
Re: 4563
Boris,
In the real world [as in Enron or WorldCom]] as opposed to the “freedom of legal consequences” lahlahland the Hansens, Schmidts, Manns and the others are free to operate in, these esteemed gentlemen would either be in front of a judge or behind bars by now for manipulating or withholding data, just like the Enron guys.
Gore raised US$320 million on the back of a what the UK High Court ruled was a pack of lies. But no, says Boris, that is not lying to get the money, that’s just an example of tetris having one of his conspiracy theory bouts.
Keep up the good work Boris: ignorance is bliss but delusion is Nirvana…
Boris,
So take the quote mining to Uncommon Descent.
Its funny how pro warmers always resort to this attempt to slur people when they don’t have any other answer.
Used to happen to me regularly at Open Mind (Closed Mind?)no matter how many times I told them that I believed in evolution and the geological record of Earth’s history.
OOPS,
I guess “believed” was the wrong word to use in the context:-). ‘Accepted’ was what I meant.
Boris is (probably) well meaning. In his eyes, that excuses everything.
OMG. A CO2 bomb dropped in my back garden. Boris,.. Anyone, what should I do ??? What’s the number for the IPCCCCCCCCC……………..Arthur, is that you? Can you help? Can Copyright Law help? No…Honestly, I’m not going to infringe the Copyright God. Arthur? Don’t you hang up on me!!!!….