The best way to track recent data is to not smooth it at all.

If tracking the end points was one’s goal when one writes a papers when the end points happened to point in the direction of one’s favorite theory, is it any wonder the practice has lead to accusation that one specifically designed previously unknown end point treatments to emphasize the behavior end-points he assumed meaningful?

I guess in the case of Rahmstorf, he hunted down a little know end point treatment and then tweaked his value of M when the end points no longer did what he liked. This is pretty pathetic as his entire paper is nothing more than a comparison of what amounts to comparison of observations to this bizarrely smoothed end point.

The whole episode stinks badly of people (particularly Rahmstorf and co-authors) acting under confirmation bias.

Of course, you can do all the arithmetic to make the projection implicit rather than explicit.

I don’t know what advantage shifting the line might have. It should result in greater oscillations when new data come in.

Is you impression they do one of the following:

A) Fit OLS to last M data points. Use that trend but slide up to the end data point, and assume data along that line.

B) Fit OLS through the last M data points, forcing it through the final data point. Assume future data long that line.

Both methods would have odd effects. But… I’m under the impression they do one of the other. Do the do either? Or something else.

I did pick it because it shows something interesting, as opposed to highlighting specifically what happened in say Rahmstorf’s analysis.

BTW– I picked the end filter because that extension would either be, or be close, to the one that would cause the minimum revisions if the data were trend+white noise. It obviously doesn’t work well for the sine wave.

Now you’re using a 29 year filter to identify 20 year cycle “noise”. That’s never going to work very well. As you show, significant periodic behaviour remains, and inevitably contributes to the estimate of trend. That’s not really an error. It’s an ambiguity about whether the sine, with period close to the filter length, is to be regarded as “signal” or “noise”.

The same is true of attempting to filter the last 30-years of global mean temperature data. There is roughly a 11-year oscillation in global weather associated with the solar cycle sitting on top of any long-term trends from human activity.

In other words, Lucia’s example is completely apropos to the real-world problem here. Given that, remember who you are also criticizing when you comment on her example!

However, in the event that one does’t go the ARMA route ignores autocorrelation or moving averages, as far as I can determine one would use the end point smoother I discuss in this article, which, as Nick points out is neither the Mann or the Grinsted fit.

But, even then, the econometricians writing one of the papers Nick linked would be advising coming up with some methodology to figure out the decent fit to use when extrapolating. Maybe if one went to that trouble, you could come up with a smoothing that did not go all wobbly at endpoints. But if one is going to try to use ARMA to figure out the extrapolation required to fit the end points… why not just show the ARMA projection in the “end point” region? Or show the both the ARMA projection, the smooth obtained using the ARMA and the extension. (The reason is obvious: If people understood you were using the ARMA to create the smooth, they’d realize they should suspect the smoothed trend in that region. All those traces become clutter.)

It may be time to drop the idea of smoothing to the latest data.

I look at a number of different data data sets where the smooth is stopped before the end of the actual data. I do not experience any frustration or sense of loss at this. Of course, these charts are simply presenting data and not trying to “sell” a particular interpretation of the data!!!

Thanks. i needed a good laugh.

Though I think that if you extrapolate all the different issues and variations that NIck has brought up, there is only one solution:

2M+1 = total length of the data, including the data R did not bother to show;

then dashed line after the actual data ends.

We all agree something has to give. People keep explaining that smoothing is imperfect at endpoints. You keep popping in and “explaining” to us that we need to understand that smoothing is imperfect at end points.

But you seem to be missing the points people are making which are that because smoothing is imperfect at end points:
a) People should not interpret the curve at end points as somehow ‘real’ and then diagnose the progression of climate change based on this.
b) People should avoid showing these regions without highlighting the fact that the information in the smoothed region near the end point can be very, very deceptive.

The MRC way is to try to avoid bias, and accept the loss of noise attenuation there.

MRC maybe a way to try to avoid bias, but trying is not the same as succeeding.

MRC does not succeed in avoiding bias. It substitutes one sort of potential biase for another sort of potential bias. If you get lucky then MRC works out fine. Otherwise, it does not.

Then a comment – as you say, the analyst isn’t supposed to know the structure of the example. The aim is to apply a lowpass filter to identify and eliminate “noise”; then the “trend” is the slope of the remaining curve. That’s all you can do.

Now you’re using a 29 year filter to identify 20 year cycle “noise”. That’s never going to work very well. As you show, significant periodic behaviour remains, and inevitably contributes to the estimate of trend. That’s not really an error. It’s an ambiguity about whether the sine, with period close to the filter length, is to be regarded as “signal” or “noise”.

So while I’m not sure that your MRC is one of the known versions, I’d agree that any endpoint rule is going to locally enhance this residual oscillation in the estimated trend. As I’ve said elsewhere, you’re running out of data and something has to give. The MRC way is to try to avoid bias, and accept the loss of noise attenuation there.

There is something about both smoothing and fitting to higher order polynomials that sometimes cause people disengage their grey cells and believe the nice smooth graphs. It’s also a bit more difficult to find discussions of potential problems placed in the context of smoothing.

If someone is going to actually try to smooth to end points, it might be nice if someone knew of a reference that discussed a series of tests to ensure ones smoothing filter did not become pathological in some specific “cartoon” problems.

For example, things a “halfway decent” end point condition should not do.

* Ends should not “turn down” when the ‘true’ underlying trend is positive an the noise is AR(1). This would disqualify padding using the average over the full range or padding using the current value.

* If the ‘true’ trend is a sine wave + a linear trend, the predicted value of the “smooth” curve should never flail out as it does in the figures in the post. (I could try to explain that in mathematical terms, but would need to make another sketch.)

It might be that padding using a linear extrapolation based on the final (2M+1) points would greatly reduce the problem of the smoothed curve in the end points display deviate from the fictional “underlying trend” just as much as unsmoothed noise did in the first place. (Only, now the impact is worse because the analyst things he reduced these deviations, so he things what he sees in the end points is real.)

The example of the sine waves reminded me in a way of Yule (1926), the original article on spurious regression, but which spends quite a bit of time discussing the interaction between correlations and sine waves (online at http://www.climateaudit.org/pd.....6.jrss.pdf )