Oh dear! Yes, I suppose so. Then of course, if you compare the graphs from both versions you could say “see, the length of smoothing makes no difference”.

From what I have been reading the 11 and 15 are bogus anyway, the actual filter uses 31 point smoothing. So it is really data spanning 31 years that is smoothed not 11 or 15.

Lucia,

I must be a bit gullible. Despite believing that I knew the meaning of causal, I now realise that I have allowed myself to be duped into thinking that the transformed symmetric filter is causal.

You were right all along. It is not a question of whether you actually have data at t=n+1, the point is that you are not allowed to use it in the computation of Yn if the filter is to be called causal.

One can make a philosophical case that using Xn+1 when it is actually known is not the same as inventing it when it is still in the future. That does not change the fact that if you use Xn+1 when computing Yn, it ceases to be a causal filter.

I am a slow learner but mostly catch on in the end.

“My question for you is whether you think the Copenhagen report should be changed to show the dashed lines when they also update the typo showing the wrong period for the smooth.”

The first sentence of the caption to Figure 3 in the Copenhagen report has already been amended, as follows:

“Changes in global average surface air temperature (smoothed over 15 years) (corrected from 11 in the first version of this report) relative to 1990.”

Isn’t this open to the (mistaken) interpretation that the Figure itself has been amended?

“I agree that it would be desirable to indicate this on the plot somehow. It’s a continuous taper, so a sudden transition to dots doesn’t quite work either.”

Strangely enough the original paper (Moore et al 2005) outlining the method used by Rahmstorf said “A new approach makes use of singular spectrum analysis (SSA) [Ghil et al., 2002] to extract a nonlinear trend and, in addition, to find the confidence interval of the nonlinear trend.”

Maybe this shows how to compute the error bars that Rahmstorf choose not to place on his graph.

Clearly the weight of the first estimated point when first used is quite low and so the potential for revision is small. Even so, I am unclear how you can put an error band on it without a weather noise model.

By the time you are using all 15 estimates, they are about half the total weight for the last point. It then matters which years you get right as each is weighted differently. It seems you need an accurate weather noise plus climate trend uncertainty model before you can compute the error bands.

Isn’t this where we came in?

The UKCP09 project is going to be a good test to see how well it can be done.

http://www.ukcip.org.uk/index......Itemid=345

ou can insist, if you wish, that the filter has to be causal at each point, making no use of future points even though they are known.

That’s the definition of a causal filter. Yes. It rules out symmetric filters which are acausal.

IfF you wish to explain describe advantages of symmertric filters relative to assymetric filters, that’s fine. We can discuss this. But your seeing advantages to useing acausal filters doesn’t turn them into causal filters any more than my explaining that red is pretty than blue can transform red into blue.

Almost everyone agrees that filtering end points is hard and the results are both unstable and error prone. This is precisely why Rahmstorfs conclusions which are based entirely interpreting what the end points of his filtered results mean is cuckoo. This, and statements he has made suggesting he did not pad, indicate that he did not really grasp the problem with his ‘analysis’ which interpreted the meaning of the endpoint region inappropriately.

Jorge, as I’ve said, when you approach the endpoints, something about the fit is necessarily lost. In MRC etc, it is the strength of smoothing, or if you prefer the effective filter width, which is tapered. Your distinction that these points are subject to revision is also a good way of putting it. I agree that it would be desirable to indicate this on the plot somehow. It’s a continuous taper, so a sudden transition to dots doesn’t quite work either.

However, as I’ve also said, it’s a universal problem in smoothing time series on finite intervals, and I haven’t seen anyone else come up with a good presentation of the situation.

That’s one of the virtues of using the time-invariant filter with linear extrapolation

Which is a-causal and sues linear extrapolation to guess the future values.

I don’t know why you think that the fact that the posts which specifically motivated this post appeared at this blog suggests this post is unrelated to them and not motivated by the comments in those posts. But, if you do; you do.

The distinction between causal and acausal becomes very blurred when time varying coefficients are used in the N/2 endpoint region. It is a purely technical definition as the outcome is identical.

This is why I prefer the “subject to revision” terminology. The fact is that whether you use an explicit or implicit extrapolation, the older end points will only stay the same when new data arrives if you have made the correct extrapolation.

No amount of huffing and puffing about whether you used padding or not, whether the filter is called causal or acausal can change the provisional nature of the N/2 region at the end of the series.

The only honest thing you can do is what Lucia suggested and show that part of the plot with dashed lines. If you want to be scrupulously honest you will add a footnote warning that the dashed area is “subject to revision”.

If I have it right, this would cover the last 11 or 15 years of the Rahmstorf graphs.

My question for you is whether you think the Copenhagen report should be changed to show the dashed lines when they also update the typo showing the wrong period for the smooth.

If you say yes, you would get the seal of approval from Richard Feynmann for scientific integrity, if not, you are just another spinmeister. I have never had any reason to doubt your intelligence or integrity so I am sure you will support this change to the graph.

And Jeff Id, it sounds as if what you are recommending is padding with zeroes. That is still padding, and has the big downside that it varies with any offset applying to the data. Add a constant (eg for temp, change from anomaly to C or to K) and you change the shape of the smooth. But it isn’t true that these “fancier methods” have no advantage. They avoid introducing artificial lag.

And Jorge, yes, near the end point, you have to depart from time invariance, of course. And if it was just a matter of choosing any filter you like, then yes, you could get any answer you like. That’s one of the virtues of using the time-invariant filter with linear extrapolation – it’s a pre-defined constructor which departs continuously (as you approach the endpoint) from the time-invariant form, and preserves zero lag, and is causal in the sense that it does not use future values beyond the data range (except for the extrapolation device used to construct the time-varying asymmetric filters).

And Paul P, no, I still don’t accept that the main point of this post was to discuss the use of the smoothed values. That has, as Lucia says, been dealt with at length in previous posts. What she said in this one was “Today, I am going to mostly discuss a method of guessing the future data used with some smoothing methods: Mannian Minimum Roughness Criteria (MRC) for short.”. That is what generated all the talk of “bogus guesses”. And my contention is that there is nothing wrong with that method. The criterion itself does not involve extrapolation. And it has some merits.

Reflecting on the X axis only places the mean as the most important and will curl a trend back on itself relative to the actual data.

For my own work I usually just let the sliding window empty out at the end which is nearly the same thing but places less weight on non existent data than reflection and is difficult to manipulate a result. I see no advantage to these fancier methods in most cases.

People might argue that the method I describe is uneven at the end of the graph but that’s a result of the available information, making up information doesn’t typically improve the result unless the physics behind the curve is known.

While odd filtering methods are common in science, I don’t find Lucia’s description of bogus out of line. It really is the fabrication of data to do any reflecting, padding or whatever in cases when the endpoint is critical to the analysis.

If Rahmstorf decided that he was smoothing specifically to create a projection based on the empirical data, and he said any particular smoothing method happened to be his favorite method, I would be not call those his guesses “bogus”. After all, in that event, he would simply be explaining that smoothing was his way to predict the future.

In that case, I’d ask him to clarify what he predicted or projected as the temperature for 2010 using his method and test that against data when it arrived. I suspect it wouldn’t take long to show his smoothing method has very little skill as a projection. But predictions are hard. As long as someone admits that the data points involved in the smoothing are, themselves, predicted, I have no problems. But using them and making conclusions that treat those data as real is bogus.

I don’t understand why you can’t see that it’s the application of data smoothing and the way the results are being interpreted that Lucia is objecting to. I admit that her language is a bit more colorful than I would use, “bogus guess” for example. But with a smoothing filter such as this the endpoints are, no matter how reasonable, necessarily extrapolations. That’s just a fancy name for “guess”. As I said, I wouldn’t use the qualifier “bogus”, but for all we know they may be; most guesses are wrong to some degree or another. Until we get more data we won’t know for sure. Now there’s nothing wrong with extrapolations per se; it depends on how they are used. That’s the whole point.

I think we are slightly going astray with regard to terminology.

As far as I understand, there are three important classifications applied to these filters. The first is whether the filter is recursive, meaning that the calculated values Yn, with their own coefficients, enter into the computation as well as the observed values of Xn with their coefficents.

The second is whether the filter is causal, meaning that no future values are used. As practical matter I believe most recursive filters are causal, and also, most causal filters are recursive. This is because of the computational burden involved with a recursive acausal filter and the generally poor performance of a non-recursive causal version.

The third term that is important is called time invariant. This means that you are always working with fixed coefficents.

Now time varying filters can also come in various flavours involving causal/recursive but they are in fact a different class. What you have been trying to convince us is that the acausal time invariant filter, that needed to guess the future, can be morphed into a causal one by algebraic means. What is clear though, is that in the process, we have moved into the class of time varying filters.

This has very important implications, because in principle, you are now free to use completely different coefficients anywhere you like in the whole time series. A heavy smooth where you don´t like the look of a bump or a lighter smooth if you want to encourage a trend to develop during the up swing of a bump, at the same time as flattening out the down swing.

In practice, I can´t imagine actually doing anything so blatant but when it comes to the causal filter that was derived from the original acausal one you are now forced to make conscious choices about the weightings for the endpoints of the time series that are not the same as those in the rest of the time series.

The fact that these choices are dictated by the endpoint infills of the acausal filter really just shows that being able to call it causal is a hollow victory.

A time varying causal filter, in this case, turns out to be identical to an acausal time invariant one.

The bogus guesses have not disappeared even though it is technically a causal filter.

I am no great expert on digital filters and have only designed one, a third order Butterworth, in my life. That may be because it was 100% 6809 assembly language.

bove I worked through your Mann extrapolation example to show that it’s application does just yield, as a matter of algebra, a regular asymmetric filter near the end. Do you see anything wrong with that filter?

Everywhere except the final point, the computation includes data collected after the point you have filtered. So, it’s acausal everywhere except the final point. At the final point, the result is the original filtered point; so it’s not any sort of filtered at all.

Are you asking if something is “wrong” with this? Or are you continuuing to suggest the “filter” is causal. In reality, the only place where it is “causal”, it is not a filter.

As for the idea of a filter being “wrong”, I think I’ve already made it pretty clear that it’s the interpretation of the results and application to testing projections based on this filter– particularly when the whole test relies on the region near the end point– that is objectionable.

Lucia, above I worked through your Mann extrapolation example to show that it’s application does just yield, as a matter of algebra, a regular asymmetric filter near the end. Do you see anything wrong with that filter?

You responded to Steven, who said he can’t see any point in using any causal filter.

So, with respect to steven’s comment about causual filters, what point were you trying to make?

How can constructing points which preserve past moments in the absence of future moments not be “extrapolation”?