When I look at quantitative issues, I’m going to look at a hindcast period and a forecast period. Both are “no-volcano” periods for must tests.

Below, I have compared the 12 month lagging average of FGOALs temperature hindcast/prediction from 1980-2030 (SRES A1B after 2000):

There is a hidden problem in this approach, which is that many of the GCMs use forcings in their hindcasts that they do not use in their forecasts. I don’t have the numbers here at the moment, but for example, many of them use things like black carbon and aerosols and solar and volcanoes in their hindcasts but not in the forecasts.

Since you are using both hindcasts and forecasts, I’d have to assume that the variance would be much greater in the former than in the latter …

All the best,

w.

I’ve been trying different tests. On the one hand, the models do poorly generally. But on the other hand, the only test that can’t be shot full of bullets for some statistical “issue” are the non-parametric ones. And for those, the limited number of runs is making things difficult. ( If you want to show something happens less than 1 in 20 times, for many tests it’s nice to have at least 20 samples. For some non-parametric tests, you only need 5 samples. More is better– but 5 is a key value because that’s the number where (1/2)^n < 0.05, meaning at least a result of “I got heads five out of five times” does happen less than 5% of the time. But for the types of tests where I need 20, I often have only 17 independent “samples” and for the type where I need 5 I generally only have 1-4 indepedendent samples.)

Lucia, I appreciate your sense of humor! BTW, that was NOT a bald attempt at getting you to do my bidding, but simply a question.

It appears CCSM3 did not upload any runs for the SresA1B. So… nope. They seem to be the only group that did not run an SresA1B case. So, I’m not looking at CCSM3– at least not yet.

But, of course, if you are interested in that model, you c ould download the data and look at it to learn whatever you hope to learn from looking at it.

http://nsidc.org/news/press/20080610_Slater.html

“Choosing the optimum combination of factors is an intuitive art that trades off completeness and( the modellers hope) accuracy for tractability and economy……..such a trade -off between accuracy and economy is not ’scientific’ per se, but rather is a value judgement, based on the weighting of many factors”

Here’s what Bader et al said in the recent ‘Climate Models: an assessment of strength and limitations’

Climate modeling has been steadily improving
over the past several decades, but the pace has
been uneven because several important aspects
of the climate system present especially severe
challenges to the goal of simulation.

Almost 20 years on from Shneider’s comments it is clear that whilst improvements may have been made the models are not ‘fit for purpose’ to drive policy decisions and the climate science community really needs to open up and acknowledge this fact.

Thank you for the link to the Rind 2008 paper. It deserves to be widely read. At the moment only CO2 Science seem to have picked up on it.

Here’s the press release you missed:

http://downloads.climatescienc.....elease.pdf
http://www.doe.gov/6442.htm

I don’t remember it getting a lot of attention, but 100+ page reports rarely do.

Thanks for your patience. Believe it or not I did actually take a few classes of this mathy stuff. Unfortunately, I now realize that remembering enough for an A and understanding are not mutually inclusive. Of course, I no longer remember enough to even get an A, though I certainly had hoped the understanding of the principles remained. But, alas, I discovered a small (huge) gap in my understanding because of one visualization used by a professor which failed to serve it’s purpose.

Ah the curses of pre-programmed responses to certain visualizations, and the desire to get a degree rather than a desire to learn.

Wise man say, when the teacher draws a bulleye demostrating low precision and high accuracy, you shouldn’t wonder about weapon and ammo specifications, range to target nor skill of shooter.

Wise man say, when learning about accuracy and precision, and the teacher draws a bullseye with a shot pattern demostrating high precision and low accuracy, your thoughts should not wonder why the shooter did not adjust between shots.

Sorry, it was rather an “eureka!” moment. And allow me to say, things just make alot more sense now.

“why has the modeling effort progressed so little in a quarter century”?

I would note that <a href=”David Rind has published a paper asking the same question.
It is doubtful that
averaging different formulations together will end
up giving the “right” result, especially because we
have no way of knowing whether the various choices
that have been made even circumscribe the proper
sensitivity.

The 2005 article was tres RC in that (a) both the title and tone drip with more condescension than average and (b) it ducks the substance of the criticism entirely. The Wegman report (done by statisticians who did a Tamino on Mann et al.) lays out very substantive set of methodological criticisms beyond what McIntyre and McItrick did. The report also says that Mann does not appear to understand ‘principle component analysis which is precisely what is recited to us Dummies in the 2005 RC post. The RC ‘defense’ simply repeats the same problematic conclusions with an eye roll as if that were forever sufficient.

I happen to think that paleoclimate work is fascinating and I have great respect for the difficulties involved. It’s a shame that a valuable but infant discipline was called upon to do so much politicized heavy lifting. Instead of being hammered for overselling the conclusions of their work Mann et al should have been in a position to be respected for moving the discipline as far as they have instead of correctly being seen as agents for an agenda.

Roger’s question is hypothetical. I don’t necessarily mean to ask “Has weather that falsifies predictions occurred?” Though, if one finds a method to test the hypothetical question, one naturally then applies it to weather we’ve had.

In your example on hurricanes, the answer is “that depends”. The problem is, in your hypothetical, you didn’t describe what the ensembles project and you don’t describe what we know about real hurricane tracks.

Also, what makes sense in testing depends on what is studied and what the model projections looked like.

If a hurricane projection model always said the the 67% confidence intervals for the hurricane landing location was anywhere between NovaScottia and Panama, and the average location of a “hit” was Maryland, the model would be useless.

And yes, there are questions one can ask to falsify based on the average “hit” location!

Lucia, the situation is also much worse when one considers the way the media report climate change. They may never refer to the fact that results are based on models, just ‘new research’ or ’scientists say’ . If they do refer to the models they will rarely qualify it with a statement of limitations. Thus the wider general public never gets close to the matters that are being discussed in certain parts of the blogosphere.

Francois O,

I couldn’t agree more with your comments in #5013 and #5024. They were what I was driving at by my simple question posited above.

“This, FO, is why one-dimensional EBMs will not do. You need a 3D hydrosphere to intiate the kinds of powerful negative cloud feedbacks that could cap warming.”

My point was that it may be that you can never get there. That it is just too complex a system to be modeled accurately, therefore you will always get unsatisfactory, if not unfalsifiable results. This is pretty pessimistic, I agree. But maybe we need one or more major breakthroughs before that problem can be tackled. So in the mean time, a simple model may just be good enough, at least for policy purposes.

When I did my work on the carbon cycle, I was amazed that I could get such good fits of CO2 vs time with a very simple model using a few coupled differential equations. I was also amazed that nobody had tackled that problem before. Of course the problem was, in the end, that I could get good fits with an infinite number of parameter sets, so that was a strong limitation. But I really just needed more, and more precise, data. And that is a big problem with GCM’s too: we lack the real world data. If there had been more money poured at gathering as much data as possible since the past 20 years, maybe there could be an intensive effort to get a single model that matches those data accurately enough. All easy to say, of course…

Obviously, if there were, “types of weather,” which, “would falsify consensus predictions,” there would be some question about accuracy of the models. But, I would ask, “Are the models good enough for their purpose?”

I would draw a parallel between climate model ensembles and ensembles used for hurricane track predictions. Because of uncertainties and lack of information, no single model for hurricane tracking is adequate to determine its final destination. When utilized as an ensemble, the likely track is much clearer.

If a hurricane rounds Cuba and is predicted to hit Texas, but makes a sudden turn and hits Florida instead. Does that falsify the model? Or does it just show that the model isn’t 100% accurate.

I don’t know much about either modelling or validation of model ensembles. But I wonder if considering this parallel might help. Could you show a hurricane tracking ensemble falsified using the same techniques? Would it be an fair test of that ensemble? I am not suggesting you look at data for hurricanes. I am simply using it to look at the question from a different perspective.