If you love climate blog wars you’ll want to see this graph 
Notice that the trend in surface temperatures from January 2001-December is positive; the trend from Jan 2008 to March 2009 is positive. Yet…. “mysteriously”… the trend from January 2001 to March 2009 is negative.
Go figure!?
So, what does amazing event mean? Not much. No law of mathematics says the longer term trend should fall between the two subtrends. The fact that it doesn’t tell us very little, particularly since monthly surface temperature anomaly exhibit temporal auto-correlation.
Still… I thought a few of you would like to see this amazing relationship. 🙂
Update
Two people have suggesting the “mystery” behavior is due to the gap between Dec. 2007 and Jan 2008. I adjusted the graph so the first short trend ends and the second short trend begins in Jan 2008. The “amazing” behavior persists:
Truly “astounding”. 😉
It is kind of interesting. It shows the power of cherry picking.
Of course this is just a slick way of eliminating the data you don’t want to include. We’ve seen this piecemeal segmented linear fitting before, including more elaborate versions in Mannian reconstructions. Lies, all lies.
Lucia,
But surely, using GISS data, this is like building a house of cards on a tidal mud bank…
Tetris–
The same sort of “amazing” behavior could be shown with any time series with a noticable temporal autocorrelation. It could be synthetic made up data: We’d still see this!
But the December 2007 to Jan 2008 trend is very NEGATIVE
Anne–
True. But, believe it or not, I could get the same “amazing” result by including Jan 2008 in both trends!
I disagree. It tells us a lot.
It tells us to be very careful and circumspect when tempted to make inference about trends – for example, is 1950 to date a short term positive trend against a much longer zero trend? We don’t have the data to answer that.
It gives the impression that December 2008 was one hell of a month!
When I pull out the ocean cycle influences (ENSO, AMO and the south Atlantic) out the temperature series over this time period, I get:
– warming of 0.02C per decade with GISS,
– cooling of 0.077C per decade with Hadcrut3,
– cooling of 0.078C per decade with RSS, and,
– cooling of 0.076C per decade with UAH.
So even though there is a La Nina at the end of the series and a downturn in the AMO, temps are still cooling – maybe the solar influence is showing up (and the GHG influence is not or it is very small).
Bill
Are there enough cyles to pull out the influence of the AMO? Particularly considering the uncertainty in ocean measurements during WWII and general uncertainties in earlier eras.
Do you have uncertainties associated with your estimate for the corrections due to these cycles?
Lucia:
There is a third trend that connects the interior end-points of your two trends.
For the AMO, there are three up cycles and two down which is not much to rely on but the AMO is not as regular as the recent past make it looks like.
Some of the reconstructions going back farther in time show wider variation and less predictability so I’m not looking at the AMO as a regular “cycle” just an influence like the ENSO is an influence which, as well, doesn’t really have predictable cycles.
There are 1,650 data points (monthly) in these reconstructions and the average error is mostly within +/-0.2C (with a few outliers getting up over +/- 0.3C) These errors are mostly random white noise however. If they continued for several years at a time either positive or negative I wouldn’t use those time periods as I have done here. There are a few periods where it is over or under for up to 3 years at a time or so which is not so good of course. But it is what it is.
F-stats are getting up close to 400. R^2 is 0.772 for the Hadcrut3 reconstruction.
Here is what the Hadcrut3 reconstruction looks like to February 2009 (don’t have access to the newest one on this computer.)
Lucia,
How would one go about deciding the minimal number of points needed to say that there are statistically-defined (reliable?) trends in a series?
Presuming that it even is proper to look for trends over various parts of a series, I image that it would be related to the series length, the amplitude of the the data, the number of embedded trends that is “reasonable”, etc. I think I’ve seen this sort of thing in regime change analysis, but that is looking for discontinuities, not changes in slope.
Any thoughts?
Question: If you convert a time series, AMO or PDO say, to the frequency domain, are there statistical tests to show that a peak at some frequency may be significant?
Jon,
I updated adding a graph to show the “mystery” is not solved by overlapping the end points of the two subtrends.
Bill,
I behind here. Where do you learn the magnitude of the AMO, ENSO and PDO indices in 1871?
Gary,
The term “reliable” is a value judgment. Statistically, you can report uncertainty intervals given some set of assumptions about the probability distributions of errors/noise etc. in the data. Then, someone can decide if a ±1C/century error is “reliable” or if they think you need a ±.001 C/century error to call the trend “reliable”.
For a given set of data, the uncertainty intervals will get smaller and smaller as you collect more and more data.
What we sometimes see is people want to define “reliable” as “now gives me the answers I prefer to be true.”
AMO index from 1856 to 2009 is here. Anybody know an easy way to convert year by row and month by column data to a single column in Excel?
Lucia,
The reliable AMO index numbers go back to 1856, the reliable Nino 3.4 index goes back to 1871. The southern AMO numbers come from the Smith and Reynolds ocean reconstruction dataset which goes back to 1854. There is a new version now ERSSTV3 and I haven’t started using it but the numbers are slightly different so I’ll have to see what it shows.
I don’t use the PDO (which goes back to 1900) because it is just an extension of the ENSO (the accumulated impact of the last two or three ENSO events or the last year or two of ENSO events). The PDO’s impact is not as definitive as the ENSO and it doesn’t provide as good of correlations so I just stick with the ENSO.
Dewitt Payne,
I use the formula from this page to covert matrix arrays into a single column. You have to play around with it a little before you can get it to work. Then just save the spreadsheet when you get it working – Converting.xls – and continue to use the same setup.
Without this formula, I wouldn’t be doing any of the work I have done.
http://www.cpearson.com/excel/MatrixToVector.aspx
There is probably a simpler method.
DeWitt Payne (Comment#13085)
Don’t know how to convert in excel, but I can do it in R. Here it is in a single column with dates.
amo series
Let’s see, you have a relative long term trend with little positive displacement. You then have a comparatively short term trend BUT it has a LARGE displacement AND that displacement is all negative to the first trend. What’s the big deal???
kuhnkat–
Reality: This sort of behavior appears frequently in trend analysis.
Climate blog talking points: Deep climate has noticed a subcategory of this type of behavior comparing long term trends in temperature to two trend embedded in the long term trend and had decided these sorts of things have some sort of meaning. Rather than initially suggesting what he thought it meant, he set explaning this sort of thing to me as a quiz. My answer is: this sort of thing means nothing.
As long as we’re on the subject of weird trends, here’s my offering for a funny plot.
This plots the difference of GISS and HadCRUT3 monthly global anomalies since Jan 2000.
As you know, if you plot the GISS and HadCRUT3 datasets individually, the OLS trend of the two differ by about 0.1 deg/decade, with GISS being about 0.1 deg/decade and HadCRUT3 being essentially zero.
As it turns out, the OLS trend using the differences for the entire period is about the same as the difference between the individual trends, but…one can treat the data as consisting of two discrete time periods; pre July 2004 and post July 2004. Those two segments are essentially flat. The graph I linked to also shows the average for the two periods (solid lines) along with +/- 1 sigma (dotted lines). If you do an OLS treatment of the two segments independently, the two slopes are virtually zero.
Since the two averages differ by about one sigma, using the IPCC vernacular, can we say it’s “likely” that something strange happened July 2004?
(The sharp downward peak is July 2004, and it is not included in either period for this data treatment. For the record, if one does an OLS fit for the pre July 2004 period it is -0.008 deg/decade and for post July 2004 it’s 0.02 deg/decade. Again, for the entire period, the OLS slope is about 0.1 deg/decade.)
Maybe we should have a Ministry of Silly Graphs.
http://www.youtube.com/watch?v=wippooDL6WE&NR=1
JohnM– That appears to be a very silly graph. But before I stick my neck out and decree that it must be silly…. we’ll have to wait for someone to investigate whether GISS and Hadley did something in 2004. (After all, we need to cover our a.. here!)
Be careful when converting an n x 13 matrix like that amon.us.long.data file. When representing the decimal year, it should be:
year + 0.5/12 + (month-1)/12
Some data sets that you find whose dates are in decimal use this representation as opposed to
year + month/12;
DeWitt Payne-
I had wondered the exact same thing a while back and I think the answer is anything above tinv(1-0.05/2,n-1)/sqrt(n) is significant. In Matlab, I corrupted a sinusoid with some zero-mean white noise and took the Fourier transform. I know exactly where the peak frequency is and I found virtually everything around that frequency was below that figure above. Be sure to window your data before you take the transform or you’ll get spectral leakage and you’ll end up finding spuriously significant frequency components.
Lots of pathologies can occur when you forget sampled data is not necessarily from an analytic function.
f(x) = x mod 1 – x/10 is increasing everywhere the derivative is defined but has an overall negative trend.
Here is another wierd trend – GISS Model E Aerosols forcing.
[In watts /metre^2 – to convert to temperature impact – multiply by approximately 0.32C.]
Talk about your contrived chart. (And I realise now I did not include the Indirect effect of Aerosols in my GISS Model E components deconstruction. I thought I had but apparently not.)
The total temperature impact from Aerosols (volcanic are separate) is nearly -0.6C!!
When Hansen said a few weeks ago that they didn’t have good Aerosols data and more-or-less just pulled the numbers out of hat – he was NOT kidding.
http://data.giss.nasa.gov/modelforce/RadF.txt
“When Hansen said a few weeks ago that they didn’t have good Aerosols data and more-or-less just pulled the numbers out of hat – he was NOT kidding.”
Which is pretty damn bad when you think about it! They pretty much need that forcing to get any kind of realistic 20th century response. And yet it is a SWAG if there ever was one.
Hmm, you need to take intercepts into account when you average
trends, we have something like
ya=0.009689425x-18.88162 % positive trend, ia=intercept sa=slope
yb=0.1514964x-303.855 % positive trend, ib=intercept sb=slope
y=-0.0026x+5.7573 % combined trend
and now, to combine slopes you need to do a little math with the design matrices to obtain
(-0.0934957*ia + 0.0934957*ib -186.8013*sa + 187.8013*sb ) =
-0.0026
The key word being defined. An index like the AMO may not be an analytic function, but it is, in principle at least, continuous and continuously differentiable.
Lucia, I believe this effect is called Simpson’s Paradox Have a look at the sample graph in the top right hand corner.
Apparently it’s well known and sometimes treated in introductory stats books, but I only stumbled across it a few months ago.
From the wiki page “… [it] also occurs with continuous data:[20] for example, if one fits separated regression lines through two sets of data, the two regression lines may show a positive trend, while a regression line fitted through all data together will show a negative trend, as shown on the picture above.”
Perhaps it might explain why CO consistently appears to get lower (or negative) trends than OLS – something to do with ensemble averages perhaps?
CO doesn’t give consistently lower trends as a method. It just gave lower trends for a period of time. I haven’t checked that method in a while. If climate scientists claim they prefer OLS using the (1+r)/(1-r), I’m content to use that.
But somehow, my showing we get the rejections either way didn’t sway the more alarmist bloggers!
DeWitt Payne (Comment#13125) April 27th, 2009 at 11:39 am
The key word being defined. An index like the AMO may not be an analytic function, but it is, in principle at least, continuous and continuously differentiable.
I don’t know what model is preferred for these data but a typical statistical assumption for doing linear regression is that all the noise is independent no matter what the sampling (and, yes, I know that assumption is not necessary). This makes the noise white in the limit. That would make the underlying function continuous but no where differentiable.
In any case, the example I gave captures the behavior of the plot. Increases everywhere you can measure it locally and decreases in trend.
I’ve done some monte carlo experiments to compare the different regression methods on 1) a linear trend + white noise and 2) a linear trend + red noise. I found CO produced a probability distribution identical to the distribution for OLS on the whitened series. Whereas OLS on the reddened time series had a much wider spread. CO appears to be the best method, as far as the trend line is concerned. I don’t remember my results exactly for the standard error calculation, so I won’t comment as to how the methods differ in that respect.
Joe Triscari (Comment#13141) April 27th, 2009 at 11:46 pm ,
Isn’t that confusing the sample with the population? The sample always consists of a finite number of discrete points with associated measurement error and so is never a continuous function. But the population, the AMO index e.g., may exist at all time scales and resolutions short of the quantum level. So for practical purposes it is continuous and continuously differentiable.