Does the IPCC describe uncertainties in a quantitative way that is accessible to readers? I expressed my opinion in comments: the IPCC generally resorts to qualitative discussions, and in particular, they become quite qualitative when discussing individual models.

I think this vagueness is unfortunate as it results in various people arguing about whether the models are “good”, “bad”, “useful”, “useless” without ever figuring out whether they disagree about a) the quantitative measures of accuracy and precision or whether they b) agree about the level of accuracy and precision, but disagree about whether that level of accuracy is sufficient for a particular use.

For example: Are we arguing whether or not the models predict the temperature of the planets surface within ±5C? Or do we agree on they can do that, but we are arguing about whether ±5C is sufficiently accurate for some particular use? (And in that case: what use?)

So, my contention is this: The IPCC, at least in the AR4, tended to avoid describing the level of accuracy easily digestible form.

This was not always the case. Atmoz provided a link showing that a clearly labled Taylor diagram was included in Chapter 8 of the IPCC’s Third Assessment Report (TAR.) These sorts of diagrams permit readers to quickly assess how closely each model matches observed data overall.

Diagrams weren’t provided for everything in the TAR, but some where. So, clearly the IPCC did at one time provide some accessible quantitative assessments of the individual models on which they based their projections.

However, as far as I can see, the authors of the Fourth Assessment Report (AR4) tended to omit these sorts of clear synopses from Chapter 8. In the AR4, narratives discussing individual models start out seeming quantitative. Alas, like a mirage in the desert, this “seeming” vanishes as the reader takes the time to examine the figures while reading the text.

Let’s look at an example: The comparison of simulation results to observations of mean surface temperature.

Comparison of Simulation Results of IPCC Models to Observations of Mean Surface Temperature

Section 8.3 of the WG1 report to the AR4 is called “Evaluation of Contemporary Climate as Simulated by Coupled Climate”, and so is the section where one might seek comparisons. Section 8.3.1.1.1 discusses temperature and we find discussion and images of:
a) observations of annual average temperature for the period 1980-1999,
b) the the difference between predictions of annual average temperature averaged over all models and observations and
c) the difference between observations and predictions of annual average surface temperature by individual models.

Instead, I’ll skip the discussion of (a) and (b) , and focus on (c)

With regard to individual models, the text says, in words:

“Individual models typically have larger errors, but in most cases still less than 3C except at high latitudes (see Figure 8.2 b and Supplementary Material, Figre S8.1) Some of the larger errors occur in regions of sharp elevation changes ….

Ok.. so I read “less than 3C”. That seems quantitative. But what does the qualifier “in most cases” mean? And what of the caveats: a) outside the high latitudes, in regions without sharp elevation changes, c) not including the eastern portions of tropical basins. How much does that leave?

Unfortunately, the figures aren’t conveniently located near the text. So, we must find Figure S8.1 in the supplemental materials. Here is the first individual model shown, BCC-CM1:

Using my eyeballs finely tuned to detect colors, I would say errors are greater than ± 3C in many regions of the globe. All of Central America and the Gulf of Mexico are dark blue (-5C). There is quite a bit orange (+3C-+5C) over Asia– at the latitude of Chicago, which I think is not considered “high” latitude. What of the dark blue (-5C) over south east Asia?

So, would you say this model has errors less than less than ±3C except at high latitudes? Or that outside the high latitudes errors are less than 3C except in places with rapidly varying terrain?” (Like, for example, Florida and the Gulf of Mexico? )

Admittedly, BCC-CM1 appears to be the worst vis-a-vis this particular metric. So, what about other models? Not to pick on models by bloggers… but….

What about GISS Model E?

On this metric, GISS Model E looks better than BCC-CM1. But, I think it also displays large regions with errors noticeably greater than ±3 C. — and mean regions well away from the poles.

But maybe you’ll see something different when you look at this:

What are we to make of those large blobs of dark blue (-3C to -4C) off the east coast of the mid-atlantic states? Or the fire engine red off the coast of Equador, Peru and Chile?

I guess off the coast of Equador and Peru is in the eastern part of a tropical ocean basin. But… isn’t there some red next to Chile?

In any case, at least in my opinion, the wording of the IPCC document tends instill the idea that errors are bounded by roughly ±3C. That’s the only number mentioned. Certainly, the words never mention the magnitude of the maximum errors on the graphs: 5C. In many individual models, errors this large appear in fairly large regions, both inside and outside the high latitudes.

What about the rest of the GCMs?

Scanning over the two pages of graphics, overall, the rest of the models look as good (or as bad) as Model E. Model GFDL-CM2.0 seems to be off by more than ±3C over nearly the whole northern hemisphere. Or… maybe it just seems that way to me. Being off by 5C is certainly never called out in the narrative.

So do the words and the figures match?

Let’s read these again:

“Individual models typically have larger errors, but in most cases, less than 3C except at high latitudes (see Figure 8.2 b and Supplementary Material, Figre S8.1) Some of the larger errors occur in regions of sharp elevation changes ….

Well, clearly this is spin. The commonly occuring 5C is never mentioned. Even given the caveats, seems rather misleading. At first glance, reading the words without looking at the graphs tucked away in a separate section, I thought the text suggested

“More than 50% of models have maximum errors less than ±3C (except outside high latitudes.”

Now, I’ve looked at the graphs. I’m not going to try to count specks, the text can’t possibly mean that. Maybe it means:

“Individual models typically have larger errors than the average of all models, but in most cases, if we exclude the high latitudes, (where the errors are typically as high as 5C for nearly all models) then the error is less than 3C in 50% of the remaining area. In the other 50% of the area outside the high latitude regions, the errors for the poorer models are frequently as high as 4C-5C– but the errors for the better models are usually no larger than 4C.

Or, it could mean something else. I can’t really integrate all those color pictures by eyeballing them.

Frankly, I think it might have been better if the IPCC had quantified what they believed about individual models using descriptive statistics.

How could the IPPC have quantified the data comparison?

There are many options; here are three:

1. The IPCC could have provided a Taylor Diagram, calculating the rms of the errors indicated in the full collections of these figures. The result for each model could have been indicated and labeled in the legend.
2. The IPCC could have produced figures similar to Figure 8.11 which shows the rms for climatological seasonal patterns, but used it to describe this metric. (They also could have included for all models used in predictions instead of only 14. )
3. The IPCC could have provided a table showing the rms error for each model.

The data were available to turn into pretty color graphics. That means the data were available to post-process and create Taylor diagrams, tables or other charts.

This would have clarified the uncertainties for readers.

Alas… no…. in the AR4, the IPPC gave readers ambiguous prose accompanied by pretty colored pictures buried in the supplementary materials. And this is why, I periodically say the IPCC resorted to vague qualitative discussions of the accuracy of the models they used to make projections.

If any one in comments or at another climate blog wishes to defend the virtue of these models by citing literature that makes up for these lapses in the AR4, I would welcome that information. I’ll happily put up with rhetorical eyerolls. 🙂