The Blackboard

After Steve McIntyre established that equation (12) Santer17 (pdf ) does not contain a typo, I decided to apply the paired t-test in Santer17 section 4.2 to test two hypotheses; both are similar in form to the one Santer17 refers to as “H2″. My tests however, relate to Global Mean Surface Temperature (GMST) and are based on slightly different ensembles; to distinguish between Santer’s H1 and H2, I’ll number my hypotheses H3 and H4.

The hypotheses I tested are

1. H3: Averaging 38 trends in GMST computed from 38 runs, is the “multi-run” trend consistent with observations?
2. H4: Averaging over the 19 best estimate of the trend from each of 19 models, is the “multi-model” trend consistent with observations?

Both hypotheses will be tested using equations (12) & (13) in Santer17, but the test quantities will be modified so as to test the slightly modified hypotheses.

Since Santer17 relies on the Nychka method to correct for serial autocorrelation, and this method is know to result in some false positives when noise really is AR(1), I also repeated the test using the correction suggested by Lee & Lund 2008. Monte Carlo test indicate this method slightly over compensates and results in uncertainty intervals that are too large when the noise really is AR(1).

Observations

For this test, I use data from HadCrut, GISS, NOAA/NCDC and I also averaged the three to obtain a “Merge 3″ set.

Models and Runs

For this test, I use the 38 runs downloaded from The Climate Explorer are discussed here.

Nomenclature

As much as possible, I will be using the notation in Santer17 (pdf ). Note that the subscript “o” is used for observations, “m” for model values.

Calculation of Model Statistics

All quantities are computed as described in equations (12) and (13) in Santer17, using supporting equations as detailed in Santer17 section 4.2. However, because I define my “ensemble” slightly differently, I modify computation slightly.

Under hypothesis H3, each run is treated as one realization in an ensemble of all possible runs of models that could pass the IPCC AR4 criterion. To calculate an average, I simply summed all 38 trends and divided by 38. All other averages over the ensemble were then computed in the normal way. This test is interesting because Gavin appears to have previously discussed averages and standard deviations using averages over individual runs.

Under hypothesis H4 the average the average trend from all runs of a particular model is treated as one realization of all possible models that could pass the IPCC AR4 criterion. After computing the 38 trends based on individual runs, I computed the average trend for each model. So, if a particular model was associated with 5 runs, I summed the five trends and divided by 5. This created an ensemble of 19 “models”. The average over this ensemble was then calculated in the normal way. This test is interesting because the IPCC appears to base their projections on the multi-model average over all models.

In contrast, in section 5.1.2, Santer17 applies their test to each model individually. So, for their tests, the runs for a particular model constitute the ensemble for that model.

The numerical results for key quantities are provided in the table below. The top of the table provides quantities that are computed for either the models or the observations. The lower portion provides results for the t-test comparing the multi-model best estimate of the trend to each observational set.

 

The final row in the table indicates the conclusions of hypothesis test: H3 is rejected at a significance level of 5%.

That is to say: The best estimate of the trend based on the multi-run average is found inconsistent with the observations during the period from Jan 2001-Sept. 2008. This particular test says we should treat the hypothesis that the average over model runs matches observations as false.

Are you wondering which values in the table tell us H3 is rejected? Ultimately, it’s the comparison of “tinv(5%,DOF)” and “d₁*” for a particular observational set. The quantity d₁* is the difference between the best estimate of a trend and the observed trend normalized by the standard error for the difference in these two quantities.

If d₁ is larger than tinv(5%, DOF), then we reject the hypothesis the best estimate of the trend based on the multi-run average is consistent with that particular data set. Note that tinv(5%) is approximately equal to 2, but may be higher or lower. When the number of degrees of freedom is very large, tinv(5%, DOF) approaches 1.96; otherwise, it is larger. The precise value depends on both the significance level and the number of degrees of freedom.

Multi-Model Results

The computations to test H4 are similar to the multi-run test performed above. However the average, standard deviation and number of samples for the “model” is modified to match the values for the ensemble of 19 models rather than 38 runs. The best estimate of the multi-model trend based on averaging over the best estimate from 19 models was 2.59 C/century; the standard deviation was 2.25 C/century. The values for the observations are unaffected.

The main result is the best estimate based on the multi-model ensemble is found inconsistent with observations at a significance of 5%. So, this test says we should treat the hypothesis that the best estimate of the trend based on averaging over models matches observations as false.

Lee and Lund Results

I repeated both tests using the modification described in Lee and Lund (2008). This involves modifying equation (6) in Santer to account for a finite number of observations. The final conclusions were unchanged: The best estimate of the trend based on both the multi-run average multi-model average is inconsistent with observations. That is: The result of the tests guides us to treat the both the best estimate of the trend based on the average over runs or the best estimate of trend based on the average over models as false.

Caveats

This result is interesting because it demonstrates that if we account for model uncertainty as discussed in Santer17, the best estimates based on either multi-model or multi-run averages are inconsistent with observations. This result is for a forecast rather than a hindcast, and so represents a test of models ability to predict events before they occur.

As in my previous article discussing Santer17, I note that there are caveats associated with the application of the method of Santer17. I advise readers to read the paper; I plan to discuss these sometime in the future.

Future plans

There are some caveats associated with application of Santer17 to data since Jan 2001 or any data.

The first caveat relates to Santer17’s use of AR(1) noise. Their tests is based on the assumption that all deviations from linear trends in the observations is weather noise and that it is AR(1). As I see it, the use of AR(1) tends to under estimate the confidence intervals, resulting in too many false positives. That is, we say the models are wrong when the are correct.

Santer17 discusses this, and it’s a fair point. We know from previous tests that treating the weather noise as “ARMA(1,1) or “AR(1)+White Noise” widens the uncertainty intervals. I’ll be posting the this month’s test using ARMA(1,1) at some point. (They will not have changed drastically since last month.)

Despite that caveat, I think it is interesting to examine the results if we use the method in Santer17, but we should bear in mind it would be wiser to use ARMA(1,1).

The second caveat is a bit more complicated. Santer17 treat all deviation from a linear trend as noise and assume this noise is “iid”. The assumption that this “noise” is independent across models certainly untrue for models runs that include volcanic forcing in as external forcings. I’ll be discussing this in detail later, so I will not belabor it now.

For now: The GMST since 2001 fails the Santer17 test for the “multi-model ensemble-mean trend” based on the 19 models and 38 runs I have been able to download.

Handy back links

New visitors to the blog frequently ask the same questions. In anticipation of this, I am providing some links to previosu articles.

1. Previous article on Santer17.
2. The choice of 2001 as “start date” and its effect on results of hypothesis tests is discussed here.
3. Effects of ENSO is discussed many places. Most recently, here.
4. The choice of statistical model matters. Last month’s test using AR(1)+white noise is here. An updated analysis for GISS using data through September 2007 is here.
6. The 38 runs downloaded from The Climate Explorer are discussed here. Checking for updates and downloading additional runs is on my “to do” list.