The problem:
Testing whether IPCC simulations faithfully reproduce the climate trends exhibited by earths temperatures is complicated by stochastic variations (i.e. noise) overlying the climate (i.e. deterministic) response. These stochastic variations appear in both the simulations and observations and arise from the earth’s own internal variability and the stochastic external forcings. Because these exist, evaluation of the fidelity of simulations must be statistical in nature.

Strong eruptions of strato-volcanoes number among the strongest of stochastic forcings and elevate the amount of noise contained in temperature time series. However, unlike stochastic variations arising from the non-linear nature of the earth’s climate system itself, it is possible to observe when volcanic eruptions occur, estimate the anomalous radiative forcing due to the volcanic eruption and account for the deterministic portion of the response of the climate to these eruptions. One method climate modelers use is to run repeat cases of AOGCMs with differing initial conditions and averaging.

A graph showing the average of 55 runs from AOGCM’s forced using modelers estimates of 20th century forcings and the projection into the 21st century is shown below:

(Note, for blog purposes, the multi-run average includes some runs that did not simulate volcanic eruptions.)

The estimated effect of the eruption due to Pinatubo stands out clearly in this graph.

So, far, this is of course not a problem: Models attempt to simulate the effect of volcanic aerosols on the response of the climate. They appear to have some success.

However, we now turn to the issue of testing whether the model simulations correctly reproduce the long term trend in temperature. Suppose we take the approach of first determining the linear trend in the surface temperatures– and estimate for the uncertainty including the deviation due to the volcanic eruptions. We then determine the linear trend in surface temperatures– and estimate for the uncertainty including the deviation due to the volcanic eruptions. These using these two trends, we pools the uncertainties and test whether the two trends are similar.

In principle, this method sounds fine.

In this case, we have a problem. Because the assumptions underlying any trend analysis of this sort include this one: The deviations in repeat samples should be uncorrelated from each other. That is: the fact that all or most model runs share the deviation due to Pinatubo when it occurred and all hypothetical repeat samples of the earth’s surface trend would include Pinatubo eruptions violates this assumption.

I discussed this issue way back in October 2008, and explained that when the deviations from lienar behavior from repeat realizations of “weather” are correlated accross samples treated as independent, the effect is to make the statistical test insensitivity. That is: it will fail to reject models that are wrong too often.

As analysts, scientists or simply rational people, our goal should be both to make as few mistakes as possible. This means we wish both to reject models that are wrong and while accepting those that are right. Both errors have negative consquences. In frequentist statistics, standard practice is to select the rate at at which we reject models that are correct. Typically one might chose a type I error of α=5%. Once this is chosen, we should see the method that rejects models that are wrong at the highest possible rate. Unfortunately, if we use the method of of comparing trends oulined above, when we can trick ourselves into thinking our type I error is α=5%, when it actually is something lower. In the process, we also increase the type II error.

This is sub-optimal.

So, what to do? Well, the answer is actually easy: Analyze the difference between the observed surface temperatures in simulations. These series of differences is not expected to exhibit any shared response to Pinatubo because the shared response in each counter act each other. (This is, by the way, a very standard way to deal with this issue. )

However, as we have seen, a climate blog warrior elsewhere decided to do something more complicated: Apply a “correction” to the earth’s surface temperature based on a lagged linear regression on volcanic aerols. To understand why that method is inferior to the more straightforward, and mathematically trivial method of analysizing the difference between observations and simulations, we need to discuss the sort of relation we expect bewteen volcanic aerorols, forcing and the earth’s temperature. Let’s dive into that.

What sort of relationship do we expect bewteen volcanic aerosols loadings and forcing?
Stratospheric aerosols shade the surface and we expect aerosols to cool the earth. However, owing to the heat capacity of the earth’s climate system, the effect is not instantaneous. This notion is reflected somewhat by multiple linear regression that incorporate a time lag. But how well would this work?

The answer to that is, “It depends.” Specifically, it will depend on the way the earth’s climate system responds to externally applied forcing and on how the forcing is applied.

Let’s look at two cartoon examples for ways forcing are applied; both will use a single lump heat capacity for the earth with a specified time constant. The choice of an earth with one time constant for simplicity– not realism. What is important is that the earths’ climate does exhibit a time constant, and that time constant is not extremely short.

Example 1: If a we apply some sort of perfectly sinusouidal forcing, (and nothing else) we can create time varying temperature v. aerosol loadings curves that resemble the yellow (forcing) and green (temperature) tracings below:

Note the response of the earth’s temperature anomaly adopt the sinusoidal shape of of the forcing, but exhibits a lag. If we identify that lag, we find an almost linear relationship between the earth’s temperature and the applied forcing:

Note the correlation between temperature and forcing in this cartoon example is almost perfect. There is a detectible hysterisis which arises from the discretization into 20 bins in my Excel spreadsheet. (If there is little noise from other processes, any decent analysist could probably tease ‘noise’ out.)

This is the sort of qualitative behavior we expect for the annual cycle of the earth. In this case, using a lagged linear regression to correct for the forcing would work almost perfectly: the best correction I can obtain using my excel spreadsheet for the cartoon problem i shown in blue. The only residuals in the corrections would arise due to slight uncertainties due to discretization. (In my cartoon, this arises becasue I apply use 20 bins for the sin wave and can only get lags in even bin increments.)

Example 2: Now, let’s apply an irregular forcing. In this cartoon example, I will make a pseudo-volcano erupt for 2 time periods out of every 17 with the amount of ‘aerosol’ in unjected selected by the “rand()” function in excel. The aerosols present in at time “t” will drop out with some exponential rate of decase. The forcing will be proportional to this aerosol loading. Fiddling with parameters, I can rather easily concoct a ‘temperature’ & ‘forcing’ v. ‘time’ graphs that looks like this:

Note that even though the temperature in this cartoon example is absolutely, totally and completely the result of the forcing function– and some sort of strong causal relationship would be evident to anyone looking at the graph, the correlation between forcing and lagged temperature is poor for any an all lags. At the optimum temporal lag the correlation between temperature and forcing looks like this:

In this case, a correction based on a linear regression at an optimum lag will do a poor job accounting for the instantaneous effect of volcanic aerosols on the temperature anomaly. Compare the blue “corrected” temperature to the yellow original temperature in the graph above; you will see the correction does almost nothing.

Could we do better?
Yes. If we have a phenomenologically based correction that faithfully represented the response of the earth’s climate, we could correct the effect of “volcanoes” just as successfully as correcting the effect of periodic forcings like the annual cycle. What is even more interesting is that if one had a phenomenologically based correction for the perfectly sinusoidal forcing, that would also result in a “corrected temperature” that is also zero at all times.

So, when available phenomelologically based corrections are always preferable to corrections based on lagged linear regression.

We could try lots of cartoon examples. The general rule will be: To the extent that the forcing is regular and periodic, the relationship you will be able to find a fairly decent correlation bewteen forcing and lagged temperature: In this case both physics based and simple lagged fits will ‘correct’ for the effect of a parameter fairly well. In contrast, to the extent that the forcing is irregular in magnitude a-periodic or episodic, you will find the correlation between forcing and or temperature will tend to be poor. I suspect many readers will notice that volcanic erruptions occur at irregular intervals and at different levels of forcing. This suggests a correcting for volcanic forcin by applying a lagged correlation will be a poor choice.

Does this mean that an analysist can’t come up with a good way to deal with episodic forcings?
Of course an analysist who knows what they are doing can deal with episodic forcing– at least sometimes they can. In the case of comparing results of AOGCM runs to observations, they can do so easily!

Let’s consider what we claim to be trying to do. Suppose an analyst does wants to compare an observatioin of surface temperatures affected by a volcanic forcing and a simulation of the exact same time period. In particular, she would like to avoid ignoring the known treating the climate signal due Pinatubo in both series as uncorrelated across the two series. The reason is that if she treates these Pintubo dip in the observation and the exact same dip in the simulation as uncorrelated, that correlation across the two series will distort the analysis.

The classic solution to this problem is to recognize that the models have been driven by volcanic forcings, and the method of eliminating the common signal in the two series is to subtract one from the other and analyze the difference in the two series. Moreover, what is intersting is that the method of subtraction will work both when the applied forcing was sinusoidal and might have been corrected successfully using a linear correlation to lagged temperature and will work when the applied forcing is irregular — as volcanoes are.

So, if one wishes to devise a test that is senstivite and minimizes the rate of both false positive and false negatives in outcomes of hypothesis testing there is no reason not to analyze the difference between models and simulations. I can think of only a few reasons one would not rely on analyzing differences. These are a) they didn’t think of it, b) they want to devise a test that is insentive, but appears to compensate for effects like volcanic eruptions. In the second case, you might see someone applying a linear regression to lagged temperatures as a function of volcanic aerosols an using this to “correct” for volcanoes.

This gets us back to where we started: If we are trying to create a senstivite test to detect differences in the climate response and obsevations, we might just as well start out by, well, subtracting and comparing the differences! Once this is done, no further correction for response to volcanic aerosols is required. The “correction” for the response of the earth’s climate to volcanoes in implicit in the analysis method.

Is analyzing differences a fancy novel thing? Nope. It’s done all the time. Whenever we are interested in whether a population of paired items are similar or different, taking differences is the most sensitive way to test.

What do the differences between multi-model runs and observations look like?

Below I show the difference between a multi-model run average of 20th century runs extended into the 21st century using the A1B scenario and observations is shown below:

Note that the excursions due to volcanic eruptions vanish. After subtraction, what remains is the sum of 1) the difference between the multi-model mean climate response in the model runs and the earths climate response, 2) the stochastic variability of the earth’s climate system and 3) the residual stochastic variability in the models remaining after averagin gover the runs.
If model simulations correctly capture the earth’s response to applied forcings, then the first listed item will be identically zero, and should exhibit no trend. Note also: The best fit trend for the difference between Hadley observations and the multi-run mean is negative. That is, we could formulate the hypothesis that the trend in the graph shown above is zero, and test that hypothesis that the trend show in the graph is really zero.

On Friday, I revealed results of my test– but I am not explaining my analytical choices. I think I have now explained why I test the difference. In upcoming posts, I will explain how I deal with temporal autocorrelation in the residuals.