The David Rose of the Daily Mail wrote a post with the title Global warming stopped 16 years ago, reveals Met Office […] has set off quite a snit. In turn, the Met Office posted a their answers to questions David Rose asked the Met office. Naturally, this spawned quite a kerfuffle, Anthony has posted twice!

I was in the middle of a “script-reorganization” so I couldn’t comment quickly. I’m at a stopping point where I can discuss things so I’ll pause to comment a bit claims in Rose’s article and statements in the Met Office’s response to David’s question. I’ll probably post again next week to engage the issue “cherry picking”. (That discussion requires the planned script functionality extension which motivated the “organization” activity.)

First: I want to state claims a reader coming across Rose’s article might infer he is making. Three of these claims appear to be:

1. Global warming has occurred. He writes: “… global warming is real, and some of it at least has been caused by the CO2 emitted by fossil fuels”.
2. Warming may be slower than the level catatrophists (whoever these people might be) have claimed. He writes, “the evidence is beginning to suggest that it may be happening much slower than the catastrophists have claimed”
3. Global warming stopped 16 years ago. This claim is in the title presented in bold to anyone who comes across the article.

Given the kerfuffle, it’s worth nothing that the three claims taken collectively are entirely in agreement with this “clarifying” graphic the Met Office shows:

Unfortunately, if presented as a rebuttal or response to any question Rose actually asked, that graph is an utterly irrelevant red herring.

Because the Met office graph is irrelevant as any sort of “rebuttal” from the Met Office, I’ve pondered a bit on organize my comments on these claims. I’ve decided the best thing to do might be to simply show the answers I would have given to Rose’s first two question. The third is such a strangely worded leading question that I don’t know how it really can be given a good answer. (I think the Met Office’s answer is rather silly– but I’ve developed some sympathy for that after I tried to think of a good answer to it. Possibly, the best answer would be “I don’t understand what you are trying to ask. Could you clarify?”)

Now, on to my answer to Rose’s two questions:

Q.1 “First, please confirm that they do indeed reveal no warming trend since 1997.”

I would have understood this to mean no real warming trend since Jan 1997. So my answer would have been: No, I cannot confirm “no warming trend since 1997”. Using HadCrut4, the trend from January 1997 – Aug 2012 is positive. That is: There has been a slight amount of warming during that period.

I might have supplemented that with a graph. The one below shows the trends based on HadCrut4 and two other agencies:

(Note: circles are expansion to include measurement uncertainty. I haven’t updated HadCrut4 — it uses HadCrut3 info.)

The HadCrut4 trend is shown in purple. At +0.05C/dec, it hovers above the solid black line indicating “no trend”. So, it shows warming. Slight warming– but still, warming.

No more blathering required for this question.

Q.2 “Second, tell me what this says about the models used by the IPCC and others which have predicted a rise of 0.2 degrees celsius per decade for the 21st century. I accept that there will always be periods when a rising gradient may be interrupted. But this flat period has now gone on for about the same time as the 1980 – 1996 warming.”

My answer to this would be that if Jan 1997 was not a cherry pick, and if HadCrut4 contains no error, then the trend since Jan. 1997 suggests temperatures are rising more slowly suggested by the multi-model mean in the AR4. That is: The mean warming in the AR4 models is faster than consistent with the earth weather since Jan 1997.

I diagnose this using the trace circled in red below:

The trace outlined in red shows the multi-model mean over models that used A1B to project into the 21st century. The blue error bars indicate the ±95% uncertainty in trends estimated using a “Model centric” method. That is: Examining models with more than 1 run projecting into the 21st century, I estimated the variance in 188 month runs for that model. I then found the average variance over the 11 models that permitted that operation, and took the standard deviation and used that as the “multi-model” estimate of the standard deviation of 188 month trends in a particular model. The ±95% spread in weather is illustrated by the span of the dark blue uncertainty intervals inside the red oval trace. Notice the bottom of that span lies above the HadCrut4 188-month trend.

This means that based on that particular choice of start year, when compared to HadCrut4, on average the AR4 models are running warm. This is true even though the HadCrut4 shows a warming trend.

At this point, I might have discussed cherry picking a bit. A case can be made that in this particular case, Jan 1997 is a rather inapt start point. But… if I were the Met Office suggesting that, I know perfectly well, the next question is:

I’m asking about 15 years because that’s a claim in a peer reviewed paper. What about that old claim that 0C/dec could last up to 15 years?

These papers use a method that permits the Met office to write misleading things like this:

So in that sense, such a period is not unexpected. It is not uncommon in the simulations for these periods to last up to 15 years, but longer periods are unlikely.

The analysis method used to justify claims like that involved taking standard deviation of trends over all models runs ignoring the fact that different models show different trends. Using that standard deviation around the multi-model mean shows the HadCrut4 trend falls inside that spread of runs; it also shows 0C/dec very near the lower boundary of the ±95% confidence intervals. It is on this basis that (some) people justify stating that we should “expect” to see 1 in 40 “earth” trends as low as 0C/dec even if the underlying trend matches the multi-model mean.

As I observed at the time: If you are trying to detect whether the ensemble is biased that method is wrong. Moreover, if you are trying to estimate the variability of earth trends (or even the spread of due to weather in a typical model) that method is just wrong.

It’s wrong because with respect to deciding whether the difference in the multi-model mean and the earth realization is due to weather we need as estimate of variability due to “weather” or “internal variability”. And real difference in mean warming from model to model have nothing to do with “weather”, or “internal variability” or anything of the sort. That is the structural difference in mean warming in different models. Not. Weather.

An estimate of the ±95% spread of trends inside “weather” enhanced by the structural uncertainty in the model mean trends is shown to the right of the uncertainty intervals I circled above. That is: this is the spread of all “weather in all models”. Note that spread is larger than the spread of “weather” in a typical model and the bottom of that trace just grazes 0C/century and HadCrut4 lies inside this trace.

It is that spread grazing zero that justified the claim of “15 years” in the papers that made that claim.

So what does the earth trend falling outside the spread of “weather” but inside the spread of “weather + structural uncertainty” mean? It means

a) if we estimate weather based on models, that HadCrut4 trend is outside the range consistent with the multi-model mean trend but
b) the earth trend is inside the full range of weather enhanced by the spread due to structural uncertainty.

How can this happen? Some models show less warming than the multi-model mean; others show more warming than the multi-model mean. Given this situation, it is possible for an earth trend to fall inside the range of weather consistent with “all weather in all models” because they fall inside the spread of weather associated with models that are– on average– warming more slowly than the multi-model mean.

Now, before I end this, I’d like to observe a few things:

1. In this particular case, 15 years is a cherry pick. We know that 1998 was one of the largest El Ninos seen in a long, long, long time. So, in some sense, we know that’s not a “good” year to really start. I’m not going to show results over a range of start years– if I did, I won’t reject the multi-model mean if I pick a large variety of years. (It’s close– but no cigar.)

   Rather than show a bunch of different start years, I am extending my script to create a method that is more “robust” against cherry picking. That is: It will compare the deviation in absolute anomalies along with the deviation in trends. That metric is harder to cherry pick because if you pick the start date in a “warm” year, the trend is low, but the mean difference is high. In contrast, if you pick the start date in a “cool” year, the trend is high, but the mean is low. I’d show that, but it requires more coding. I’ll be showing this a bit later this week. I don’t know if using the “new” method will result in a reject or accept since 1997. But since cherry picking is rampant (on all sides), I want to budget time on creating a “harder to cherry pick method”.

2. My focusing on Rose’s questions means I’ve failed to slam him for “global warming stopped”. There is no evidence global warming stopped in any meaningful sense.

   First: The 15 year trend is positive. That’s “slow warming” but not “stopped”

   Second: we know the trend over the past 50 years or century are both positive. Rose says so himself. And we know temperatures are variable. So, there is ample evidence that we have been in a warming trend; it should take quite a bit of cooling to decree we have evidence it actually stopped. Specifically, if we are estimating weather using the “model centric” method I use here, if we took the uncertainty intervals from models and placed them around ‘0’, we need to see a 188-month trend of approximately -0.13 C/decade before we can reject a hypothesis that warming is positive. That level is the ±95% window (one sided) that would correspond to a forced trend of 0C/decade. A +0.04C/dec is certainly well above -0.13C/dec.

   To be sure, it may be that when he wrote “stopped” Rose meant “paused” or “slowed”. But generally, “X stopped” gives the impression X is not about to resume at any moment. When applied to random variables, it gives the impression one believes the expected value for X has “stopped” being what it was before. The fact is: there is zero evidence that we should expect the trend in the upcoming 15 years to not be positive. Even without any consideration of models — merely resorting to extrapolation– there is every reason to expect a positive trend over the upcoming 15 years.

This was pretty rambling. But the highlights:

1) If picking 1997 was “fair”, the HadCrut4 trend falls outside the ±95% confidence window defined by the multi-model mean and “typical model weather.
2) This suggests the multi-model mean is warmer than consistent with earth weather.
3) The earth trend falls inside the ±95% spread of “weather + structural uncertainty” of models. This suggests some individual models are not too warm.
4) The Met office conflates (2) and (3) in a misleading way. To be fair, I think they likely aren’t doing it on purpose. They are collectively a bit confused and they have mislead themselves. Nevertheless. both (2) and (3) are true.
5) There is no evidence “global warming stopped”– or at least there isn’t any evidence if “stopped” is defined as “stopped and we should not expect it to resume”.
6) There’s a good chance Met office is going to continue to have difficulty defending the model mean. After all, if the anticipated El Nino is either weak or non-existent, next year Rose may well announce that we’ve had 16 years 0C/dec!
7) It really would help if they “discover” that the multi-model trend is biased high and stop trying to suggest the discrepancy can be explained by “weather” (or worse- volcanoes!)

Currently, I’m just adding tests to my graphs. For that reason, the discussion will be brief merely explaining the additional informaiton added to one of the graphs from Observations V. Models: “Model Weather”. where I discussed a method to see whether the earth weather trend falls inside the spread of “weather trends” using the estimate of variability of model trends based on repeat runs for matched periods from that model.

I told readers I’d add the multi-model mean. I’ve actually added 3 versions of the multi-model mean shown in bold below. My “preferred” version is labeled “MT”.

(Careful readers will note that for convenience I am drawing data from a 2nd hand source, and these trends end in June or July.)

The three versions of the comparison to the multi-model mean are:

1. M1: This trace represents the 95% range of weather for a forecast data point assuming the 11 models used to create the forecast all replicate the probability distribution of “earth weather”. I computed the multi-model mean for the test period over the 11 models with more than 1 run. I estimated the spread due to the “weather” in each model by taking the square root of the mean variance. (I’ll call this the pooled mean s.d.) That weather is shown in slate blue. I estimated the uncertainty in the model mean using the ratio of the standard deviation of the model mean over the 11 models to the square root of the 11; this is shown in grey. I then extended the “weather” uncertainty intervals by pooling the 95% spread of weather and the standard error in the sample mean. These are shown in slightly lighter blue. If earth weather was measured perfectly, and that range was a forecast we would expect trends in the forecast period to have a 95% probability of falling in that forecast.

   Carrick correctl observed that a full comparison requires inclusion of measurement uncertainty in the earth trend. I estimated that and added it to the uncertainty intervals. The lower bounds of the trend including measurement uncertainty are shown with open circles. (See illustration below for more details.)

2. MT: This trace represents the 95% range of weather for an as yet unseen realization under the assumption that all models used to compute the A1B SRES replicate the probability distribution of “earth weather” but we were only able to estimate the standard deviation of trends due to “weather” using the 11 models with repeat runs. The only difference between this and the previous test is the multi-model mean is a bit higher and “22” was used to compute the standard error in the mean. This is the multi-model mean actually used in the AR4, and for that reason, I consider it the “preferred” mean to test.

   When the earth trend falls below the open circles, this indicates that the earth trend as “weather” falls outside the range that is consistent with the multimodel mean– assuming we accept the pooled variance of weather in models as the best estimate.

3. ME is called “M enhanced”. This is the result one would get if one estimated the uncertainty using the spread of trends in each model (i.e. what the models actually estimate as uncertainty in trends due to weather”) and then added the spread in the mean trends (i.e. scatter in the location of open circles) into the estimate in the “weather trend”.) This could be justified if the difference in the model trends was due entirely to statistical error and not due to any real difference in trends. In that case, the uncertainty intervals computed under ME enhanced would on average be equal in size to those computed under M1. But we would have a larger number of samples and so get a more precise results. However, in this case, it’s actually pretty easy to show that trends in the forecast region differ from model to model. (I have not shown this test, but I’ve done it.) So, this method results in a spread that is not an estimate of the variability in 138 month trends due to “earth weather”. Rather, it is simply an estimate of the ±95% range of runs including both the effects of “weather” and the fact that some models show more warming and some show slower warming. (This is, btw, the set of assumptions that published papers used to insist the current trends are consistent with the models. They may be consistent in the sense that some models show less warming than the mean and/or are very variable. But this method does not contradict any diagnosis that the ensemble is biased warm.

Note: All graphs assume probability distributions for trends are Gaussian (i.e. Normal).

Guide
There were questions on reading the graph in prior posts. This might help:

In the previous post, SteveF asked

There are big differences in the uncertainty range for different models. Is this mainly because some models have fewer runs (which adds to uncertainty) or because some models have more run-to-run variability?

I answered, but I also thought I would create a graphic that shows the standard deviation of 150-month trends as computed over repeat runs of models forced using the A1B SRES and also compares those to the spread estimated using ARMA11. Below, the standard deviation of 150-month trends computed based on repeated samples of models with more than 1 run are shown in ‘slate blue’ as “95% confidence intervals” centered on zero rather than their respective multi-model means. The arima estimates for GISTemp (red), HadCrut3(purple) and NOAA(green) are shown by horizontal lines. The 95% confidence interval based on the “pooled standard deviation” for all models is shown to the far right in black. (A pooled standard deviation is based on taking the square root of the average of the variance.)

Note that generally speaking the variability over repeat samples of model runs is larger than estimated using ARMA applied to the model data. If one thought all models agreed and created the correct spectral properties of “weather”, then one might explain the discrepancy by saying that “weather noise” is not ARMA11. (And this might even be so.) But — oddly enough– if we treat noise in models as ARMA11 gives more or less correct sized uncertainty intervals for 150 month means in the aggregate, though not for individual models. For some models, it would be too high; others too low. All this means is that the weather noise in models is not “ARMA11” (which would be evident if we examined their residuals in any case.) It is also the case that weather noise in individual models do not have identical spectral properties. (No. I’m not going to show them for every model. But I’ll simply point out that the slope-o-grams for models differ dramatically see slopeograms.)

Nevertheless, it is useful to look at a feature that strongly affects the estimate of the variability of 150 month trends. This is the standard error or residuals to a linear trend. When computed during periods with no fast variations in the forced trend (i.e. no Pinatubo-like eruption) this quantity gives some measure of the short termvariability due to weather in models relative to that seen on planet earth. The pooled mean of for the ‘se’ of residuals over repeat periods during the forecast period are show with slate blue circle, the pooled mean over all models are shown with black circles and the ordinary mean over all models with grey circles. The associated standard deviation in the ‘se’ over the periods in the forecast period are also shown. The ‘ses’ associated with the single 150 month trend for observations from 2000-now are showin with red, purple and green lines for GISTemp, HadCrut and NOAA/NCDC respectively:

Using the “method of the eyeball”, it seems that on the whole, the models tend to have larger short term variability than seen on the real earth. The model variability tends to be higher than the observed variability, and this is true despite the fact that measurement uncertainty means that the standard error residuals for the earth includes contributions from ‘earth weather” and measurement uncertainty.

This doesn’t necessarily tell us whether the models over or under estimate the longer term variability or even the variability of 150 month trends. But, I think it’s worth nothing that despite the lack of “ENSO”, the some models contain a surfeit of variability at frequencies below 150 months– and I would say that some of that variability is both (a) wild and (b) clearly wrong. (Honestly, I’m not sure weather is sufficiently predictable to permit agriculture on planets ECHAM or FGOAL also known as “the wild-crazy weather models used to claim the recent earth trends falls inside the range consistent with models”.)

Hmm… I guess I should also compare the average lag 1 auto-correlation. Those are measurable statistics. I haven’t done that recently and if I’ve done it in the past, I don’t recall the result I get. I’ll add that.

Previously, I showed that if we the observations of of Global Surface Temperature using “linear forced trend + ARMA11-noise” , we would reject the null hypothesis that the trend in the multi-model mean temperature for AR4 models correspoding to the A1B SRES matches the trend corresponding to three observational data sets (GISTemp, NOAA/NCDC and HadCrut3) using a number of choices of start years– though not all possible choices. (See Arima11 Test: Reject AR4 Multi-Model Mean since 1980, 1995, 2001,2001,2003.) I leave it to readers to decide what the result means–and merely report that the rejections are occurring.

Of course, when interpreting what the rejections mean, one might observe that for some periods analyzed, the forced trend may not be linear and even when it is, the noise may not be ARIMA. So, it is useful to look at the result a different way. Today, I’ll show what we get if we limit our tests comparing least squares trends from simulations to those from observations as “test statistics” and use estimate the variability trends over independent realizations of “weather” from the spread of trends in an individual model. This test can be applied to individual models– which I will do. (It can also be applied to the multi-model mean if we make additional assumption. When the assumptions apply, I strongly suspect that test is has greater statistical power. I will defer that comparison.) There are 11 models with more than 1 run; results will be shown for each of these 11 models.

I’ll provide the main result first, and explain the test afterwards. The main results are encapsulated in the graphs below:

The main findings are:

1. If we apply test whether the observation the trend associated with Global Mean Temperature computed starting in Jan 2000, we reject the ‘null’ hypothesis that the observation falls within the spread of “weather” characteristic of a particular model in (2,5,4) cases based on comparisons to (GISS, HadCRUT3 and NOAA) respectively. If we test the same null hypothesis using trends that begin in Jan 2001, we reject the null in (4, 6, 5) cases when observations are based on (GISS, HadCRUT3 and NOAA) respectively.
2. Using monte-carlo analysis simulating the above test 10⁴ times to account for the fact model results are independent of each other but all cases are compared to the same observation, I find if this test was applied to a system of models which reproduced both the mean trend and the variability of mean trends correctly, we would expect the false positive rate of rejection seen for trends starting in 2000 to be (10.0%, 4.3%, 5.7%) for GISS, HadCRUT3 and NOAA) respectively. For the number of rejections seen when the test is applied to trend beginning in 2001, false positive rate would be (6.0%, 3.3%, 4.6%) for the number of rejections seen with (GISS, HadCRUT3 and NOAA) respectively.

   If we use the significance level of 5% (i.e. 1-95%) as our criteria for significance, this test would guide us to reject the hypothesis that HadCrut or NOAA trends are consistent with weather in all models and that it is unlikely the result the mis-match in trends since 2001 occurred by chance. However, comparison to GISS suggests that even if models correctly replicate the earth trend and variance in trends, the number of rejections observed could have arisen due to chance. Using the same significance level but applying the test using trends since Jan 2000, the number of rejections for both NOAA and GISTemp are not sufficiently large to reject the null hypothesis that model means match the observations, but the number of rejections is sufficient to state the HadCrut3 model does not match the ‘weather’ in at least some models.

   (Note: I am assuming I would reject models as biased if we had too many rejections on either extreme with zero rejections in the other direction. So, the rate at which we would see rejections with model all “too warm” is one half that indicated.)

Those are the main results. Once again: What these statistically significant mismatches for some start years and some observational data sets mean could be a matter of some debate. For example: Among other things, I will need to change the script to use HadCrut4 when it is reporting reliably. (The report through Aug 2012 is available now, but last month data were only available through Dec. 2010. I have not yet updated my script to read it.)

But, no matter what they mean, I think it’s useful to show where the relative agreement or disagreement between models and data. (I will later be adding the multi-model mean. I just haven’t done so yet.)

For those interested in how things were computed, I’m going to give a very cursory description.

Trend for period of interest
The observed trends were computed over the 150 (138) months beginning in Jan 2000 (2001). These are indicated with horizontal lines above; (Red=GISTemp, Green=NOAA/NCDC, Purple=HadCrut3). The mean trend for models were computed for the same period of time by first computing the trends over N available runs, and then averaging over all N runs. Each trend is indicated with a brown open circle.

Estimate of standard deviation of “model trends”
For each model, the variance model trends by computing the average of the variance over “N” runs 12 non-overlapping 150 (138) month periods of simulations for models with more than 1 run. This corresponds to the estimate of the variability in trends would would expect to arise from “weather” if a particular model is correct. The ±95% range for spread of weather around the model mean is indicated by the vertical blue lines with the inner most horizontal tick.

Uncertainty in model mean trend.
The sample mean trend for each model is an estimate of the mean that would be obtained in the limit that the number of runs was infinite. However, this quantity is uncertain. The standard error for this estimate was computed as the ratio of the standard deviation to the square root of the number of runs. This quantity was pooled with the standard deviation for model trends to create the ±95% range for “weather trends” we would expect given the additional uncertainty in our estimate of the mean trend. That range is indicated by the outer blue tick.

Note that in principle, if the model is correct in the sense that the model mean over an infinite number of runs would reproduce the expected value of the earth’s mean trend and the expected value of the variance in the earth’s trends if– hypothetically– we could keep rerunning the earth weather, an individual realization of earth’s weather should fall inside the outer-tick of the blue line with probability 95%. However, the observation of earth weather also contain measurement noise while models do not. When we compare an individual realization of the earths trend to “model weather”, we must include this measurement error. (Note: We don’t need include it when we estimate uncertainty in trends based on the residuals to a linear fit in a time series.)

Estimate of measurement uncertainty for observations
To estimate this quantity, I first obtained estimates of the uncertainty in annual average temperature data for each observational data set. I then created annual average temperature series for each data set, and added “red noise” with a specified value lag 1 correlation coefficients to the published data set with residuals that reproduced the estimate for the errors in annual average temperatures and computed the least squares trend. For each choice of lag 1 correlation coefficient, repeated this 100 times and computed the standard deviation of the resulting trends.

This process was repeated for lag 1 correlation coefficients from r1=0 to 0.9 at intervals of 0.1. I then selected the the largest estimate of uncertainty due to measurement errors corresponding to estimated values of uncertainty in individual annual average temperatures. (The maximum occurred with lag 1 coefficients of about r1=0.7 or 0.8. It’s likely the r1 coefficient for measurement errors is a lower value; my estimate likely represents an upper bound on the possible magnitude of the uncertainty due to measurement error.)

Estimates of the observational error were computed for each observational data set. These were then pooled with the uncertainty due to “model weather”. The extended lower bounds for observations consistent with the model data are shown with open circles (red=GISS, purple=HadCrut, blue=NOAA).

Diagnosis for individual models.
In this method, we diagnose that an observation falls outside the range consistent with “model weather” if the trend associated with the observation (shown with a horizontal line) falls below the open circle corresponding to the pooled uncertainty resulting from: 1) “model weather”, 2) uncertainty and the determination of the model mean and 3) uncertainty due to measurement uncertainty.

Diagnosis for models collectively.
Because we are testing 11 models against observations, we would expect to get 1 case “false positive” rejection at p=95% to occur more than 1 in 20 times. However, determining how often we might see several at a time is complicated by the fact that the tests are not independent. Every model is compared to the same earth trend.

To determine the rate at which “m” models reject, I replicated the methodology used to create the full graph above under a “null-set” assumption that a) all models and the earth share the same mean trend b) all models and earth share the same variability of “N month trends”. That is: I find the false positive rate (i.e. rate at which we would reject a “correct null” ) that we would reject m out of 11 models under the assumption all models are “right”. If this false positive rate false below 5%, we conclude it is unlikely that an event of this magnitude would happen through random chance. That is: we can reject the notion the that earth weather is consistent with weather for that particular model.