http://landshape.org/enm/

He sources the smoothing technique to a paper by Michael Mann, “On smoothing potentially non- stationary climate series”, GRL, 2004. Mann advocates the use of a “minimum roughness” constraint for the end of a time series.

http://holocene.meteo.psu.edu/.....nGRL04.pdf

Steve McIntyre has some things to say about this technique, in another context.

“As I noted in the earlier post, Mann’s “minimum roughness” constraint, when translated from inflated Mannian language, boils down to a reflection of the series both horizontally and vertically around the final value”.
“When I wrote a little routine to implement Mannomatic smoothing, I noticed something really funny. I know that it seems bizarre that there can be humor in smoothing algorithms, but hey, this is the Team. Think about what happens with the Mannomatic smooth: you reflect the series around the final value both horizontally and vertically. Accordingly with a symmetric filter (as these things tend to be), everything cancels out except the final value. The Mannomatic pins the series on the end-point exactly the same as Emanuel’s “incorrect” smoothing”.

http://www.climateaudit.org/?p=1681

Looking at the series in Rahmstorf et al, which ends in 2006, this appears to be the case. I don’t think this solves our problem re the relationship between the series trend & the IPCC chart, however if 2008 should turn out to be quite cool it would be interesting to see this chart updated at the end of the year, using this technique.

….that year to year temperature fluctuations are not completely random, but dependent on previous years’ temperatures. You yourself argued in a previous thread that Schwartz underestimated his time constant, so you were obviously giving a simple statistical test which you do not truly belive in.

On the first part of your comment: Yes. I’ve tried to comment on the fact that I’ve neglected serial correlation in the residuals in the current (and most recent) blog post. Serial correlation in the residuals matters.

I’m actually looking at fixing that up right now, but the reason I’m comfortable neglecting it with annual data is the correlation isn’t very big for annual values. (It should be looked at, but it’s not all huge and doesn’t affect the estimate of how long it should take to falsify or validate. The residuals for monthly data are highly correlated.)

I will be writing more about this though.

On the second bit: It’s not the serial correlation in the residuals I criticized Schwartz for. (Although that does matter.) What criticized him for was looking at the data in an odd way that made it difficult to separate out actually measurement error from real weather variability. His instrument measurement error does not obey conservation of energy. (Tamino also neglected to recognize this distinction, and that’s why Tamino’s simulated temperatures don’t look like temperature measurements. )

Funny you should mention Kreysig. I’m getting beta and alpha error from… “Advanced Engineering Mathematics” by…. Erwin Kreyszig

I know what correlation is. Before I bowed out of research to be a slug, I wrote things like The influence of a mean fluid velocity gradient on the particle-fluid velocity covariance. However, I’m not used to dealing with these sort of time series. In turbulence experiments, you either design to sample so fast you can pick up the full shape of the autocorrelation function or you sample so slow, you make sure your data point are uncorrelated.

In between makes data analysis difficult and error prone. It’s best to avoid it.

http://rankexploits.com/musing.....-suggests/

Hans von Storch addressed this problem here:

http://coast.gkss.de/staff/storch/pdf/misuses.pdf

Storch gives examples of “detrending” date, but that was for AR(1) models. I’m not sure how it would work
against more complex models.

From what I understand of his article, you’ve got to run it against truly random data: Schwartz would say that every 5th year constitutes random data, I suppose you would say every 8th year.

2. As I said before, I was blissfully ignorant of autocorrelation until reading Climate Audit. I get the impression that if a process is truly random, the residuals should follow a random distribution.
I get this idea for an autocorrelation test from an old textbook: “Introductory Mathematical Statistics”, by Ervin Kreyszig. It gives a test for “runs”- checking if a run of heads or tails, (or in this case temperatrue fluctuations above or below the trend line). For 2m events, m heads and m tails, the mean number of runs should approach m + 1 as a limit. The variance in number of runs should approach (m(m+1))/(2m-1). This approximation is supposed to work well for 2m ove 20. I presume that with strong autocorrelation, the number of ‘runs’ of temperature above or below the trend line would be signicantly different than m+1.

The number of runs of 1, 2, 3 etc from flipping a coin should be 1/(2^(n+2)), so in a sequence of 1024,
you should get about 1024/8 = 128 runs of 1 head, 128 runs of 1 tail, 64 runs of 2 heads, 64 runs of 2 tails, etc. As a second possibility, maybe a Chi test would work to test for autocorrelation in this case.

I’m sure there are better test already, I’d just like to know if my two off the top of my head ideas make
sense, or these off the top of my head ideas are like dandruff-small and flaky- Alan McIntire

Wikipedia:

Type I error, also known as an “error of the first kind”, an α error, or a “false positive”: the error of rejecting a null hypothesis when it is actually true.

Me

The value “α” (alpha) just listed is the probability that one might decree that a null hypothesis is false when it is true.

Wikipedia’s word <==> my words.

“Reject the null hypothesis” <==> “Decree the null hypothesis false”.
“When it is true” <==> “When it is true”.

By both Wikipedia and my choice of words, alpha error is a false positive. Commonly, the null hypothesis is the equivalent is “nothing happened”.

So, “Rejecting nothing happened” is finding “Something happened”. If nothing actually happened (that null hypothesis is true) then you got a false positive.

You say “The value “α” (alpha) just listed is the probability that one might decree that a null hypothesis is false when it is true. ” This sounds like a false negative, type II error.

Type I error (measured by alpha)is:
Type I errors (the “false positive”): the error of rejecting the null hypothesis given that it is actually true; e.g., reject hyposthesis of no difference (That is you conclude there is a difference) when there is, in fact, no difference.

Type II errors (the “false negative”): the error of failing to reject the null hypothesis given that the alternative hypothesis is actually true; e.g., Failing to reject the hypothesis of no difference, (that is you conclude that there is no difference) when there is in fact there is a difference.
Don

BTW– since many people argue by rhetorical question, I want to make sure new visitors know that mine aren’t.

Most applied statistics I’ve done have been for laboratory experiments. When designing a lab experiment, we usually try to take enough data to avoid having to tease out results using the sorts of techniques econometricians or paleo-people have to do. We aren’t stuck with waiting for the earth to go around the sun and weather to happen.

I do know that in this area, which is politically contentious, I want to explain what the answer is if someone insists that “no warming” must always be the null hypothesis because they think that’s some sort of inviolable rule. I also want to explain what the answer is if someone else insists that “x C/century” must be the null hypothesis because that’s what some governing body says. There are repeated arguments that amount to arguing over what one must accept as the null hypothesis at various blogs.

Bayesian gets used to mean that we assume some sort of probability to some original hypothesis, right?

E.T. Jaynes
“Probability Theory: the logic of science”

One of the reasons hypothesis testing isn’t math itself is that you need to say what the hypothesis is.

When doing tests on past data, this permits soooooo much cherry picking. It often doesn’t even seem like cherry picking to the people doing the analysis. But the people you are trying to convince always recognize the cherry picking. (Why start hurricane counts in 1970? Or assume ‘X’? Or look at temperatures since precisely 1998?)

In some ways, stating what the test hypothesis and describing how to do the test before collecting data, encourages clarity. Doing this before data comes in means we don’t have each person suggesting a hypothesis they actually cherry picked. We also don’t end up with people insisting on precision for the steps that they know in advance help their point and going all loosie-goosie on steps that, if implemented fully, hurt their point.

I am totally imprecise on red noise. I also do no tests for normal data or normally distributed residuals etc. I’m doing this with the precision one would generally do to estimate how much data you will need before figuring out the budget for doing an experiment.

These sorts of scoping calculations are actually rather routine done for lab or field work! You might not do this formally, or document it, but you will do it on the back of an envelop. Who wants to start an experiment only to discover that the uncertainty is so large that even if an effect is real, you probably can’t prove it in less than 1,000 years?

“The analysis of data inevitably involves some trafficking with the field of statistics, that gray area which is as surely not a branch of mathematics as it is neither a branch of science.”

But we need people who will traffic with stats. I’m just glad that I’ve not had to be one of them.