The Blackboard

But Rahmstorf et al do not use the Mannian minimum roughness.
They use ‘a’ minimum roughness criterion due to Moore Grinsted and Jevrejeva, which is rather different, see code posted by Jean S on your other thread (sorry for duplicating!):

What the code does is tack on M extra points at each end that lie on straight lines. These straight lines are found from a fit of the first (or last) mp points of the original data x. (As explained by Grinsted in the passage quoted by Jonathan).

But this is not ‘minimum roughness’ as originally proposed by the great Mann himself, which reflects the data horizontally and vertically about the final point, forcing the smoothed version to go through the last grid point (although Mann did not seem to realise this until it was pointed out by McIntyre). At first glance it seems more sensible than minimum roughness, although I am sure you can find examples where it gives bizarre results.

Lucia, I think your fig 1 is not quite right – with MRC the smoothed curve MUST go through the last point, with any symmetric filter, [as is clear from your example with y's, eg (y2 + y3 + y3 + y3-y2)/3 = y3] but yours doesnt.

Lucia –

The Fallacy of Rahmstorf´s Entrails.

This is my name for this particular form of self deception. It deserves a name and to become part of the scientific folklore handed down to future generations.

In essence it goes like this. We have a theory and we have made some predictions about some variable. A few years go by and we get hold of some observed values of this variable. Unfortunately, the data are rather noisy and hard to compare with the predictions so we decide to smooth the observations. So we look in the cupboard for a cookbook method and we apply it with some parameters that give a good match to the original predictions.

What we don´t fully comprehend is that the cookbook code is actually inventing future observed values to produce the smoothed curve for the observations. The end result is that we have shown that the actual matches the predicted, as we hoped but we have lost sight of the entrails that were consulted to get the future values required to smooth the past so that it matched the original predictions.

Who knows whether any of this was intentional. It really does not matter because it is so clear that, up to now, we have not proved anything at all. I feel rather sorry for Rahmstorf but he will just have to takes his lumps like the rest of us when we make a mistake.

Unless climate scientists wish to be the laughing stock of the entire world, they will get this nonsense pulled from the Copenhagen synthesis report.

I am really impressed with the clarity of your explanation of the technicalities here. It has completely exposed the daft nature of trying to verify predictions by making even more predictions.

Paul–
Yes. My smooths were smoothed by JeanS to using the Grinstead method. I’ll fix that.

At first glance it seems more sensible than minimum roughness, although I am sure you can find examples where it gives bizarre results.

The Grinstead method has to be better than the Mann method. But, in my opinion, you should never “test” a projection against data that has been smoothed using fictional future values. It doesn’t make any sense.

There are other problems with Rahmstorf. For example, the smoothed data points should carry along with them the uncertainty in figuring out where the origin is. (That is: Everything is pinned to some “known” temperature in 1990. This is supposed to be some sort of expected value for the temperature in 1990. But how do you “know” that value? Under any sensible notion involving noisy data, there is uncertainty in your knowledge of that temperature. Rahmstorf doesn’t add that. Then there is no uncertainty shown at the end points. It’s all just wretched.)

So you pick the even extension if you think the data will go back in the future and the odd extension if you think it will keep going. Easy as 1-2-3! (Granted there is some ambiguity, depending on whether you like 1-2-3-2-1 or 1-2-3-4-5 better…)

And much better than convolving with a signal with a linear extension derived from the points being smoothed, that’s more convoluted than you can possibly imagine!

oliver– The “beauty” of all of this is that in addition to the complication of all possible smoothing methods, the
a) analyst gets to sit in his office and chose which ever entire method he currently likes, plus the magnitude of the parameter after the data are observed and
b) two years later after publishing a paper, he can “tweak” the method if the results no longer look as good as they did two years earlier.

This is all so much more scientific than, say, using a standard method like a “t-test”.

When I do smoothing, and ’tis rare when I do it, I start with the most recent data and work backwards so that the so-called edge effect is seen in the oldest data. Am I missing something?

Jorge: “I feel rather sorry for Rahmstorf” Two points.

The reviewers must assume some responsibility for errors – from the Preface.

This report has been critically reviewed by representatives of the Earth System Science Partnership (ESSP)ii, by the parallel session chairs and co-chairs, and by up to four independent researchers from each IARU university. This extensive review process has been implemented to ensure that the messages contained in the report are solidly and accurately based on the new research produced since the last IPCC Report, and that they faithfully reflect the most recent work of the international climate change research community.

“Unless climate scientists wish to be the laughing stock of the entire world, they will get this nonsense pulled from the Copenhagen synthesis report.”

Although Stefan has said he will inform the editors of the error, we should not assume he will do so, or even if he does, that the editors act on that advice. Ian Castles relates a similar situation with the IPCC where promises to fix an error by Australian Hennessy have gone unaddressed by the higher-ups. They should be informed separately and independently.

Because 2008 is lower than 2006, the guess for 2006 is also lower than 2008

\
Should that be “Because 2008 is lower than 2006, the guess for 2010 is also lower than 2008″

It looks like Mann’s statistical errors is what are being confirmed in multiple studies.

Lucia,
I really cannot see where you are going with this post. It’s well known that if you smooth data, you lose information at the ends. If L is the smoothing interval, then the loss begins at end-((L-1)/2) and is progressively worse until you reach the end. So you must show a full curve, it has to be interpreted with appropriate caution. All the ways in which it could have been derived have faults. They include:
1. Guessing the future. The problem is obvious, but it is often done, by various ways. The guessed values become more influential as you approach the endpoint.
2. Use a one-sided smoothing formula near the end. This sounds more satisfactory, but isn’t. If you use only past data, then any trend estimate you make is a lagged value – it should be attributed not to the present, but to a midpoint of the data period you used. And then when you look at what you have done, it turns out that you have used a rather lopsided smoothing window to get that lagged value. So this is rarely done.
3. Use future data predicted from the past. The problem here is that it is really a version of 2. In fact, every one-sided formula can be seen numerically as just a two step process where you use some of the past data to extrapolate future points, then use a two-sided formula for smoothing. So the tradeoff comes with how far you go back to find data for your prediction. Variants are:
3a – just guess that future values will be the last value used. This has the least lagging effect, but is obviously a minimal information predictor. It has the bad effect of artificially amplifying the effect of that value.
3b MRC, a variant which uses that single past value with a fitted slope. This uses more past data (to fit the slope), and increases the lag effect, but is a more realistic predictor. It still has undue dependence on that last value.
3c The Grinstead method, which uses more past data to give a better extrapolation. OK, fine, but again the tradeoff. By using still more past data, you have a further lagged estimate. Your trend estimate is more reliable, but is centred on a point further in the past.

And so on. There’s no good answer. MRC is no better or worse than others.

Smoothing loses information throughout the time period, not just the end-points.

The climate is noisy and chaotic. Perhaps the chaos element does not contain information (certain chaos modelers would disagree with that) but parts of the “noise” certainly do. For example, the climate modelers certainly appreciate the “noise” created by a La Nina a little better now after the headaches it has caused them recently.

Smoothing just reduces the information which is available – information which we can use to better understand what is going on with the climate.

Most people like nice smooth lines, like some people prefer lines going up. But the climate does not operate on the smooth lines that most people prefer.

The smoothing period should really be limited to the timescales of the force that you are interested in, operates under.

Monthly temperatures are already smoothed from an infinite number of individual temperature readings down to an average over 30 days. Why would one smooth that even more. The greenhouse effect and the energy from the Sun operates at the speed of light, modulated by the ability of atmosphere, land, ocean and ice-sheet molecules to store up that speed of light energy. It does not operate at a timeperiod of 14 years in any case.

Perhaps this is a naive question, but why are moving averages preferred over exponential smoothing? The latter doesn’t require any guesses for the future and, to me, seems a better choice because it gives priority to the data for year N when computing the smoothed value at year N.

(I’m sure I’ve asked this before elsewhere, but don’t recall having received a good answer.)

I really cannot see where you are going with this post.

Going? I’m not “going” anywhere. Various people are asking what is involved in these various sorts of endpoint assumptions which are described briefly in articles by Mann and Rahmstorf. I am providing an answer, describing by example. I happen to think examples with illustrate the method help the math averse gain a better intuitive feel for just what is involved, so I picked a concrete example.

there’s no good answer. MRC is no better or worse than others.

I agree there is no good way to smooth endpoints. My point is not to suggest that MRC is worse it is to describe what sorts of things are done. Why would you think my point is to say MRC is worse?

Of course I’m sure you are aware this fits in with the other posts. Although my main point is simply to describe what is involved in some particular types of smoothing methods, I will also point out that some climatologists, people are using results of smoothed data, including points that are computed based on guessed future data to “test” models. Rahmstorf did this.

Believing one can “test” projections by comparing to “data” that has been smoothed using guessed values is utterly, totally and completely foolish. Using MRC is no more foolish than other methods. However, explaining any individual method of smoothing in concrete terms illustrates why Rahmstorf’s idea one can test this way is error prone relative to standard statistical methods like t-tests, which many were taught in school.

Jorge: let them present the work… as is… unchanged. Long term…can only help the cause of AGW skeptics LOL

why are moving averages preferred over exponential smoothing?

The aren’t necessarily. However, it is easier to explain their properties when giving simple examples.

No matter what smoothing you use, you end up with a problem smoothing endpoints.

Ralph, the problem with exponential smoothing is that it doesn’t give an estimate of the current value – the estimate is of a value some time in the past (lagging). To see this, try smoothing a straight line. That is smooth already, and you’d expect no change. But the line is shifted, to the right. A centred moving average, say, won’t do this.

Nick–
Oddly,I think the straight line wouldn’t shift if we rebaseline after smoothing (as Rahmstorf does and most climatology papers do). The baseline is set to force the smoothed value through zero in Jan 1990. The rebaselining would shift things back.

But exponential smoothing will distort more complex curves in ways that are different from moving averages. If the true temperature is a sinusoid, the moving average at least puts the minima and maxima in the correct calendar years. Exponential smoothing won’t.

Lucia,
My query as to where you were taking this related to your using terms like “bogus guesses”, with lots of embroidering with terms like bog-o-city. That’s pretty loaded language for what is just extrapolation using varying amounts of recent information. Every centered method does this at the ends, whether explicitly or implicitly.

I read Mann’s original 2004 paper. It actually makes good sense. He says basically that since you can’t compute centred smoothing estimates for near-end points, you should replace them with estimates based on some desired property for the curve. He chooses minimum second derivative (roughness). That sounds pretty reasonable. It’s mathematically equivalent to the use of extrapolation. And as I said, it’s no panacea – it’s just another point in a trade-off spectrum of currency of information versus variance.

It’s very reasonable to criticise the error in the Figure caption mis-identifying the smoothing interval. The original 2007 complaint about using the end-point of a smoothed curve to suggest that a coming breakout from the error range is also reasonable. But this talk of bogus guesses seems to be just piling on.

David, Your #15514, last para.

Just to clarify. I wrote to Kevin Hennessy in March 2006 to point out the error in the Whetton et al (2005) CSIRO Research Paper, of which Kevin was a co-author. He replied that “I agree with you. The error will be corrected”, copying his email to eight colleagues at CMAR Aspendale (Greg Ayers, Chris Mitchell, Penny Whetton, Ian Watterson, Roger Jones, Kathleen McInnes, Paul Holper and Simon Torok).

When the error had not been corrected after about six months, I raised the issue with another co-author of the paper, Roger Jones, with whom I was having an exchange on John Quiggin’s blog. Roger’s reply indicated that there’d been a stuff-up of some kind so I assumed that, the matter having again been brought to notice, a correction would finally be issued. But it hasn’t been.

The IPCC didn’t come into my correspondence with the CSIRO people, because it never occurred to me that a non-peer-reviewed work would be cited in an Assessment Report – but in fact Whetton et al (2005) WAS cited in WGII Chapter 11, of which Kevin Hennessy was Coordinating Lead Author. I would have thought that the fact that the report had been cited by the IPCC strengthened the case for the CSIRO issuing a correction, but I can only assume that the powers-that-be in the CSIRO thought the opposite.

I agree that the editor(s) of the Copenhagen Synthesis Report should be advised of the error in the caption to Figure 3.

Nick #15227: In Mann 2004 the procedure of double reflecting is actually a bit of a throw-away – just one way of doing it. And the issues are laid out quite well in it. Whatever you think of him, and I am sure he has made mistakes, I have found his papers are always at the coal-face, which engenders my respect.

Ian thanks for that. One of the main editors is Will Steffen, Executive Director ANU Climate Change Institute.

Another important revelation from Jean S on the previous thread:
The caption on Fig 3 should not be changed from “smoothed over 11 years” to “smoothed over 15 years”. It should be “smoothed over 29 years”! (or maybe 31, I’m not quite sure).
Rahmstorf et al 2007’s ‘Embedding period’ is the extension required, half the size of the smoothing window!
Perhaps this is another error that Stefan Rahmstorf “hadn’t noticed”?

This means that the only part of the smooth red curve in Fig 3 that does not involved guessed temperatures is the short section from 1985-1993!

Lucia,

I far prefer the title “Disco Stu” criteria to MRC. It has stornger prvenance, predating Mannian MRC and is essentially the same. Can be best summarised as:

“look, it’s going up.”

Paul, it is 2M-1 (I rechecked), i.e., 29 years for M=15, which of course means that only M-1 values are actually needed for padding.

A small clarification: it seems that guessed values are only used in the end. That is, they first filtered the whole temperature series, and then cut it to 1970. At least this is the way I’m able to replicate the figures.

Nick –

Surely you can see that any method that relies on future projections to verify past projections is not science. It is reading entrails, tea leaves or other occult practice.

The only time you can safely extrapolate is when you have a detailed knowledge of the forces acting on the variable. In this case, the whole point of the exercise is to find out whether we do have that knowledge embedded in the computer projections.

Frankly I do not care at all about the actual smoothing method that is employed. If it explicitly or implicitly uses future values it cannot be used to make a judgement now. It may be very disappointing that we have to wait for future information to make a correct statement about the past, but that is inherent in a smoothing process.

Any proof that is obtained by the smoothed observations is in fact conditional on the future matching your guesses. Until that future occurs, your apparent proof is just another guess. It is not science and has no place in any publication that purports to be scientific.

The fact that this nonsense can be passed off in properly reviewed literature simply shows that self deception or ignorance is widespread. Certainly, I would plead ignorance as I had never fully realised the implications of what was going on under the hood.

Once your eyes are opened you can see that the Rahmstorf graphs have no scientific value at all.

No, Jorge, there is nothing wrong with extrapolation here. It’s just a device to get what you eventually want – a set of weights to make the weighted sum used for smoothing. Extrapolation is just one linear combination; smoothing is another, and as a matter of algebra, you can exchange between expressing the smoother in internal values or extrapolated values.

Take Lucia’s last example above, with two reflected points added, and use with a five-point centred moving average. Then the two last smoothed points are .2*(y0+y1)+.6*y3 and y3 (the rest are just the true moving averages). These are perfectly sensible smoothing averages, although the final point is not smoothed at all. The fact that extrapolations were used to construct them does not matter; the advantage of the method, as I explained on another thread, is that it ensures that both constant and linear functions smooth into themselves. Put that another way, smoothing constructed this way is guaranteed to preserve zero and first moments. That’s an important property.

This is actually a well-known technique in numerically imposing boundary conditions in ode and pde. You create ghost nodes or particles with extrapolated values, and use a derivative approximation, substitute, and then apply the resulting operator expressed just in interior node values. It just simplifies the algebra.

Having said all that, I should now say it another way. When MRC etc use future values, they are not asserting that they are “true”. They are asserting that those values impose the end condition (min 2nd deriv) that they want.

Jorge, again (and Lucia)
I’ve now seen a lot about what Mannian smoothing is doing and why. Firstly this idea of lag-free smoothing. You’ll be familiar with the effect in smoothing electrical signals – because you can only use present and past values, there’s always a phase shift to the past. With time series analysis, for past points you have access to values on both sides, and can make a lag-free filter. Symmetry is enough; past and future are equally represented.

But as your smoothing point moves towards the present, you start running out of future. To keep a balance, you have to upweight the future you have, which is primarily the latest point. So in the example of my previous post, the penultimate point has a weighting of 0.6 on the current point (y3), which balances the more spread out past points (y0,y1). And when you try to smooth the latest point, there is no future, so you can’t use the past (and avoid lag). So the final point has to be unsmoothed.

So these apparent eccentricities of MRC etc are actually the requirements for lag-free smoothing up to the boundary.

Nick–

I read Mann’s original 2004 paper. It actually makes good sense. He says basically that since you can’t compute centred smoothing estimates for near-end points, you should replace them with estimates based on some desired property for the curve.

It makes a heck of a lot more sense to simply admit you can’t smooth the data near the end points and avoid trying to decieve yourself into thinking you can give an endpoint treatment a fancy name and get decent smoothed information around the endpoints.

Thinking that Mann (or someone else) has come up with a solution for the issue is exactly what is bogus.

It would have been much better if Mann (and others) were to simply write a paper decreeing that their smooth lines should just stop n/2 prior to the end point.

It’s fine for you to suggest that someone should mask what’s wrong with results that depend on guessed future data by vaguely discussing ‘error bars’ as if the errors in that region are at all comparable to this inside the range where the smooth is computed based on real data. But the issue is qualitatively different and requires a different adjective.

Jean S, thank you again for correcting me. So it is ‘only’ the period 1993-2008 where the smoothed curve is corrupted by guessed data. Your ‘forensic climate science’ on this is quite remarkable.

Nick, this is all very well, but Rahmstorf is not using MRC. If he did he would get pictures like Lucia’s, which are not what he wants. So he is picking and choosing both the endpoint extrapolation algorithm and the length of the smoothing interval to get the answer he wants. As Jorge says, this is not science. And yes, Lucia’s latest comment is spot-on.

It must be axiomatic that using extrapolated values to estimate trends at end points is

The alternative to extrapolation is shortening the smoothing period, and shifting the weighting to an assymetrical distribution. This is done in many economic statistical circles (surprised Ian Castles didn’t refer to it)

This gives different challenges for interpretation (the closer you get to the end point the less robust and subject to subsequent change the calculation is). But must be superior to a result that effectively says:

“we estimate the historical trend to be X because we have assumed that the future trend will be X (or Y or whatever)”

Lucia,
I’ve agreed that smoothing becomes increasingly problematic as you get towards the end, as is well known. However, climate scientists certainly aren’t the first to try to smooth graphs to the endpoint.
Here for example, is the Australian Bureau of Statistics advising on how to do it.
And as I’ve argued above, Mann’s method does about the best possible; it tapers the degree of smoothing as required rather than allowing lags or other distortion.

Geckko,
No, you’re missing the point. Using extrapolation with a symmetric smoothing filter at the endpoint is exactly equivalent, mathematically, to using an asymmetric filter. It’s just a way of creating an asymmetric filter with certain desirable properties.

Just to drive home that point, here is the key quote from that Australian Bureau link:

For this reason, symmetric filters cannot be used at either end of a series. This is known as the end point problem. Time series analysts can use asymmetric filters to produce smoothed estimates in these regions. In this case, the smoothed value is calculated ‘off centre’, with the average being determined using more data from one side of the point than the other according to what is available. Alternatively, modelling techniques may be used to extrapolate the time series and then apply symmetric filters to the extended series.

That’s saying both that extrapolation is OK (last sentence), and that it is an alternative to an asymmetric filter.

Nick –

There are lots of ways to smooth data but you cannot use the resulting curve up to the end points without inventing data or accepting the lag. I don´t care whether it is computed padding of the window or using some implicit derivative of the past data.

The only honest smooth will be a lagging one.

No matter how fancy you make the extrapolation procedure your smooth at the end point is conditional on having made the right choices about the future. The whole point is that a global temperature is a very difficult thing to extrapolate into the future – if it was easy we would not need complex models to try to do it.

We have seen very clearly that Rahmstorf clearly failed to extrapolate correctly in his first paper because the addition of two more years completely changes the slope of his smoothed curve. The question of changing the parameters is a side issue for me. The simple truth is that he has tried to validate some earlier computer projections using a smoothed curve that itself cannot be validated until some other future extrapolation turns out to come true.

I do not always have a problem with extrapolating. For example, if I have an unknown RC electronic filter I will happily predict future values of the exponential after a step change in input once I have enough observed values to calculate the time constant.

This is the exact opposite to the global temperature situation because we have no validated way to extrapolate. It is precisely this validation for the models that we are trying do. To suggest that the unchecked validity of a different extrapolation can be used for this purpose is nothing less than insanity.

It is a widespread insanity for sure and until these last few threads I had thought that it was simply a technical argument over the rival merits of varying filter constuctions. Indeed, I even joined in on a few occasions but I can now see clearly that that was all a side issue.

You just cannot call on a curve as evidence of another truth now when it will take time to show the curve was true in the first place.

It is plainly a bloody nuisance to have to wait for the end of the window to be able to get evidence for what is happening now but the alternative is straightforward self delusion.

I have joined the ranks of the undeluded.

Nick

This is actually a well-known technique in numerically imposing boundary conditions in ode and pde

Sure. But that application is totally irrelevant to what Rahmstorf and Mann are doing. When you write a code to solve odes (ordinary differential equations) and pdes (partial differential equations like conservation of mass, momentum and energy inside AOGCMS) the system of equations includes a statement of what the boundary conditions are supposed to be.

So, you apply the ones dictated by the actual problem at hand. (For those not well grounded in physics, if computing heat transfer from inside your house, you might set the boundary condition for temperature at the outside wall of the house to a constant. For heat transfer from a heating element, you might set the heat flux to a known value, which means you will set the trend to a known value. All these assumptions will be reported when you describe the results.)

When you write a code, you don’t just set your boundary conditions based on your intuition about what the answer should look like!

What Rahmstorf and Mann is more like the second: With data already in hand, the analyst picks the end point treatment. They tend to select the one that makes the end points look the way they “like”. Then, they give the end point treatment a name, and with no further ado, proceed to make conclusions about what is happening in the world.

But in the case of Rahmstorf, it turns out his chosen end point treatment with m=15 affects the final 28 years of data. (Jean S clarified that Rahmstorf’s m is not the size of the filtering window but the extension describing the number of guessed years required to compute the final point!)

When MRC etc use future values, they are not asserting that they are “true”.

Oh? Who are “they”? It is certainly true that some people neither say, nor imply that the future values are true. They say nothing about them at all.

But the point is, apparently smoothed past observations, that is those within values within N/2 of the end point are also not “true”. These were computed based on the “not true” values from the future.

Papers like Rahmstorf’s and Mann’s present these “as true” and advance conclusions based on comparison of those “not true” values to things like model projections, theories etc.

So, even if no one looks at, presents or discusses the actually fictional future values guessed using a method they think gives the “right” future, people do use those values to compute “partually guessed data” that is then used to “tests” their own theories.

This is a method positively designed enhanced confirmation bias.

If the climate community, including Rahmstorf, were to get in the habit of putting a vertical line at the point where the “smoothed data computed based on fake future data” begins, they would figure out their methods of “testing” are cukoo.

Lucia,
Have you read that section from the Australian Bureau of Statistics site? It seems to endorse exactly what Mann is doing.

Jorge–

It is a widespread insanity for sure and until these last few threads I had thought that it was simply a technical argument over the rival merits of varying filter constuctions. Indeed, I even joined in on a few occasions but I can now see clearly that that was all a side issue.

You have captured the reason I did not wish to post a technical discussion of the rival merits of varying filter constructions in the abstract.

No matter which we use, the end points have different behavior than the interior. When presented as one smooth continuous curve, the user is tempted to believe this nice smooth curve is real and climate scientists are using those smoothed curves and making conclusions about agreement between smoothed data and projections.

Nick seems to agree the end points are a problem, but seems to want to limit the discussion to the smoothing itself and forget about the larger issue which is: How are climate scientists using these smoothed results?

It would be bad enough if they were relying on comparison in the central region where better methods of comparison exist, but at least centered smoothing doesn’t distort the heck out of everything.

But the fact is, people like Rahmstorf are specifically making conclusions about the fidelity of model projections based on the endpoints of smoothed graphs. Nick can post all sorts of defenses that, when properly used, MRC is no worse than any other method, but my points (which many seem to get) is that any and all smoothing methods would be inappropriate for Stefan’s purpose.

The fact that anyone (including Stefan) can change the appearance of the endpoint by cherrypicking whatever end point treatment they like after the data come in, select their filter length (m=11 now m=15) etc. only makes this application worse.

Nick–

Have you read that section from the Australian Bureau of Statistics site? It seems to endorse exactly what Mann is doing.

So? Does the econometric discussion have anything to do with later using the smooth data to test projections?

You really aren’t getting my point, are you?

Does this have anything to do with later using the smooth data to test projections?

But your post, with all its scorn for “bogus guesses”, is about Mann’s method. He didn’t use it to test projections.
And Rahmstorf didn’t use Mann’s method either.

Nick, I dont really think it does. It points out that there are several different ways of doing it. It also says that the asymmetric filtering near the endpoints dampens different frequencies (obvious since its a shorter filter) and introduces a phase shift (obvious since it is effectively lagged) – hardly an endorsement.
But this is all irrelevant for Lucia’s main point about retrospective adjustment of the method to obtain the required result – but I see from above that you agree largely with this, so it seems we are pretty much in agreement.

Nick–
You seem to want to think each blog post must be isolated from the context of the conversation between my readers and me. There is no such rule, there is no rational reason for such a rule and even if you wish to create one, it’s not going to work.

Here’s the context: The previous two posts are on the topic of using smoothing to test projections. The second sentence in my post alludes that this s an answer to questions that arose in the specific context of using smoothing to test projections. You will note that I said:

In comments, on articles in the “Fishy” series, people have been asking about the whole “end point condition” issue and how it relates to Rahmstorf’s foolish “method” of testing projections.

So, it should be clear to anyone reading that the comments in this post are written to address the questions posted in the context of the previous two posts.

I consider it perfectly acceptable to write a post addressing questions my readers have and providing answers that makes sense in context.

If you want to suggest that smoothing makes sense in entirely different contexts, like writing code that solves the navier stokes equations, or economists trying to figure out if Christmas sales really are stronger than sales in July, fine. But you know perfectly well that people developing cfd codes don’t cite Mann to explain how they applied their boundary conditions. There are valid reasons to smooth some things, but it must be done cautiously. Neither of the specific examples you give is relevant to the context of my discussion and equally importantly, they have nothing to do with how Mann, Rahmstorf or climatologists are actually applying smoothing to test theories or communicate information about what’s occurring with the worlds climate.

I wish Tamino would do posts like this, instead of just cheerleading with an appearance of explaining science.

Lucia I see no justification for applying acausal filters to data that presumably result from a causal process, except that it makes for pretty graphs.

The necessary and sufficient boundary conditions for fluid flow equations used in CFD are set by the eigenvectors for the equation system. Because the supplied boundary information determines the solution interior to the solution domain, that boundary information must be correct. Sloppy boundary conditions can easily destroy numerical solution methods. Sloppy boundary information can also easily defeat the best efforts to attain accurate solutions in the domain.

Don’t mess with Mother Nature.

The other thing the smoothing allowed was to bump the Hadcrut3 and GISS temp lines up from where they really are (versus the 1990 baseline). The Tar projections start at Zero but, effectively, Hadcrut3 and GISS do not.

If 1990 was the baseline year, then 2008 temps are only about 0.1C higher than 1990 (Hadcrut3 is 0.14C and GISS is only 0.06C higher).

The chart looks like temps should have increased about 0.3C under the TAR climate projections from 1990 to 2008. So, once again, the models are off by more than half.

This smoothing is just another manipulation of the data.

Steven,
No-one’s using an acausal filter. Like all linear smoothers, Mann’s is just a linear combination of values within the data range. The numerical weights can be seen as derived by applying a symmetric operator to an extrapolated range. Or, equivalently, as being chosen to minimise the second derivative (and preserve zero and first moments), which does not involve the idea of extrapolation at all.

Nick

which does not involve the idea of extrapolation at all.

Oh common NIck! Here’s the definition of causal and acausal filters from Wikipedia:

The word causal indicates that the filter output depends only on past and present inputs. A filter whose output also depends on future inputs is acausal.

Mann’s end point treatment only applies to acausal filters. That is: filters that use future data to compute the value of the current smoothed data point. MRC has no practical application in a causal filter because these have no end point problem.

The fact that the method of dealing with endpoints ends up being computed in a way that can be expressed by applying various weights to past data does not transform the method into a “causal” filter. Of course the method of computing the end points relies only on past data. The future hasn’t arrived.

Mann’s own discussion of the method relies on extrapolation by mirroring. The weights are determined based on this form of extrapolation.

Re: Nick Stokes (Comment#15569) July 1st, 2009 at 4:23 pm
“…equivalently, as being chosen to minimise the second derivative (and preserve zero and first moments), which does not involve the idea of extrapolation at all.”

How can constructing points which preserve past moments in the absence of future moments not be “extrapolation”?

Lucia,
I’m referring to the causal treatments of endpoint values – ie not using “bogus guesses”. The use of acausal (eg symmetric) filters for smoothing interior regions of a time series is not controversial. A centered moving average, for example, is acausal on that interpretation.

Nick–
What oliver said.

Oliver and Lucia,
Again, preserving zero and first moments is equivalent to the proposition that constant and linear functions are unchanged on smoothing. At each point, the moments are calculated on a window of known points. In Mann’s method, this window gets narrower as you approach the endpoint.

Nick–
Mann’s method of treating endpoints is only used with acausal filters. Of course the method of dealing with endpoints can’t actaully use future (unknown) data.

You responded to Steven, who said he can’t see any point in using any causal filter.

So, with respect to steven’s comment about causual filters, what point were you trying to make?

Well, the primary question is, what point was Steve trying to make? If he’s saying that you should use only one-sided filters even when you know “future” values, then that is an extreme view that I think he should justify. And my response would have been off target.
I assumed that he was taking up the earlier claims that Mann is using “bogus guesses” (acausality) at the endpoints. And my point is that he is not.

Lucia, above I worked through your Mann extrapolation example to show that it’s application does just yield, as a matter of algebra, a regular asymmetric filter near the end. Do you see anything wrong with that filter?

Nick–

bove I worked through your Mann extrapolation example to show that it’s application does just yield, as a matter of algebra, a regular asymmetric filter near the end. Do you see anything wrong with that filter?

Everywhere except the final point, the computation includes data collected after the point you have filtered. So, it’s acausal everywhere except the final point. At the final point, the result is the original filtered point; so it’s not any sort of filtered at all.

Are you asking if something is “wrong” with this? Or are you continuuing to suggest the “filter” is causal. In reality, the only place where it is “causal”, it is not a filter.

As for the idea of a filter being “wrong”, I think I’ve already made it pretty clear that it’s the interpretation of the results and application to testing projections based on this filter– particularly when the whole test relies on the region near the end point– that is objectionable.

Nick –

I think we are slightly going astray with regard to terminology.

As far as I understand, there are three important classifications applied to these filters. The first is whether the filter is recursive, meaning that the calculated values Yn, with their own coefficients, enter into the computation as well as the observed values of Xn with their coefficents.

The second is whether the filter is causal, meaning that no future values are used. As practical matter I believe most recursive filters are causal, and also, most causal filters are recursive. This is because of the computational burden involved with a recursive acausal filter and the generally poor performance of a non-recursive causal version.

The third term that is important is called time invariant. This means that you are always working with fixed coefficents.

Now time varying filters can also come in various flavours involving causal/recursive but they are in fact a different class. What you have been trying to convince us is that the acausal time invariant filter, that needed to guess the future, can be morphed into a causal one by algebraic means. What is clear though, is that in the process, we have moved into the class of time varying filters.

This has very important implications, because in principle, you are now free to use completely different coefficients anywhere you like in the whole time series. A heavy smooth where you don´t like the look of a bump or a lighter smooth if you want to encourage a trend to develop during the up swing of a bump, at the same time as flattening out the down swing.

In practice, I can´t imagine actually doing anything so blatant but when it comes to the causal filter that was derived from the original acausal one you are now forced to make conscious choices about the weightings for the endpoints of the time series that are not the same as those in the rest of the time series.

The fact that these choices are dictated by the endpoint infills of the acausal filter really just shows that being able to call it causal is a hollow victory.

A time varying causal filter, in this case, turns out to be identical to an acausal time invariant one.

The bogus guesses have not disappeared even though it is technically a causal filter.

I am no great expert on digital filters and have only designed one, a third order Butterworth, in my life. That may be because it was 100% 6809 assembly language.

Nick,
For what it’s worth, I’ve understood since the beginning of this topic that Lucia’s main point was the use of the smoothed endpoint in comparisons with model projections. In reality they are both models of different types so the comparison makes no scientific sense. Then changing one of the model parameters, in this case the smoothing window size, and presenting it as an update to a previous analysis where only new data had been added is dishonest at the least.

I don’t understand why you can’t see that it’s the application of data smoothing and the way the results are being interpreted that Lucia is objecting to. I admit that her language is a bit more colorful than I would use, “bogus guess” for example. But with a smoothing filter such as this the endpoints are, no matter how reasonable, necessarily extrapolations. That’s just a fancy name for “guess”. As I said, I wouldn’t use the qualifier “bogus”, but for all we know they may be; most guesses are wrong to some degree or another. Until we get more data we won’t know for sure. Now there’s nothing wrong with extrapolations per se; it depends on how they are used. That’s the whole point.

Paul–
It is the application that bothers me.

If Rahmstorf decided that he was smoothing specifically to create a projection based on the empirical data, and he said any particular smoothing method happened to be his favorite method, I would be not call those his guesses “bogus”. After all, in that event, he would simply be explaining that smoothing was his way to predict the future.

In that case, I’d ask him to clarify what he predicted or projected as the temperature for 2010 using his method and test that against data when it arrived. I suspect it wouldn’t take long to show his smoothing method has very little skill as a projection. But predictions are hard. As long as someone admits that the data points involved in the smoothing are, themselves, predicted, I have no problems. But using them and making conclusions that treat those data as real is bogus.

The whole problem with this smoothing is that in mirroring by two directions you’ve assumed trend is the important information you want to extend. While not a priori wrong, it leaves the door open for manipulation of endpoints by the externally motivated individual — and that’s exactly how it’s being used. Without a blink from the team..

Reflecting on the X axis only places the mean as the most important and will curl a trend back on itself relative to the actual data.

For my own work I usually just let the sliding window empty out at the end which is nearly the same thing but places less weight on non existent data than reflection and is difficult to manipulate a result. I see no advantage to these fancier methods in most cases.

People might argue that the method I describe is uneven at the end of the graph but that’s a result of the available information, making up information doesn’t typically improve the result unless the physics behind the curve is known.

While odd filtering methods are common in science, I don’t find Lucia’s description of bogus out of line. It really is the fabrication of data to do any reflecting, padding or whatever in cases when the endpoint is critical to the analysis.

Steve McIntyre has <a href=”http://www.climateaudit.org/?p=6473posted today on the actual code used by Rahmstorf. The claim that padding wasn’t used has been falsified beyond a reasonable doubt.

Dewitt,
It depends on what “used” means. I’ve added a comment at Steve’s site. Ultimately, as I’ve contended above, they are using an asymmetric filter near the ends. The neatest way to construct this is using extrapolates (padding). But you can construct and justify the same filter using arguments interior to the data interval.

And Jeff Id, it sounds as if what you are recommending is padding with zeroes. That is still padding, and has the big downside that it varies with any offset applying to the data. Add a constant (eg for temp, change from anomaly to C or to K) and you change the shape of the smooth. But it isn’t true that these “fancier methods” have no advantage. They avoid introducing artificial lag.

And Jorge, yes, near the end point, you have to depart from time invariance, of course. And if it was just a matter of choosing any filter you like, then yes, you could get any answer you like. That’s one of the virtues of using the time-invariant filter with linear extrapolation – it’s a pre-defined constructor which departs continuously (as you approach the endpoint) from the time-invariant form, and preserves zero lag, and is causal in the sense that it does not use future values beyond the data range (except for the extrapolation device used to construct the time-varying asymmetric filters).

And Paul P, no, I still don’t accept that the main point of this post was to discuss the use of the smoothed values. That has, as Lucia says, been dealt with at length in previous posts. What she said in this one was “Today, I am going to mostly discuss a method of guessing the future data used with some smoothing methods: Mannian Minimum Roughness Criteria (MRC) for short.”. That is what generated all the talk of “bogus guesses”. And my contention is that there is nothing wrong with that method. The criterion itself does not involve extrapolation. And it has some merits.

Nick –

The distinction between causal and acausal becomes very blurred when time varying coefficients are used in the N/2 endpoint region. It is a purely technical definition as the outcome is identical.

This is why I prefer the “subject to revision” terminology. The fact is that whether you use an explicit or implicit extrapolation, the older end points will only stay the same when new data arrives if you have made the correct extrapolation.

No amount of huffing and puffing about whether you used padding or not, whether the filter is called causal or acausal can change the provisional nature of the N/2 region at the end of the series.

The only honest thing you can do is what Lucia suggested and show that part of the plot with dashed lines. If you want to be scrupulously honest you will add a footnote warning that the dashed area is “subject to revision”.

If I have it right, this would cover the last 11 or 15 years of the Rahmstorf graphs.

My question for you is whether you think the Copenhagen report should be changed to show the dashed lines when they also update the typo showing the wrong period for the smooth.

If you say yes, you would get the seal of approval from Richard Feynmann for scientific integrity, if not, you are just another spinmeister. I have never had any reason to doubt your intelligence or integrity so I am sure you will support this change to the graph.

Nick–
The only point when the Mannian filter becomes causal is the final one which is that’s not filtered at all. If you examine the method use to compute the smoothed value at the second to the last point, it uses data from the final one making computation of that point “causal”. I don’t know how you can even begin to argue there is anything causal about the use of this filter.

That’s one of the virtues of using the time-invariant filter with linear extrapolation

Which is a-causal and sues linear extrapolation to guess the future values.

I don’t know why you think that the fact that the posts which specifically motivated this post appeared at this blog suggests this post is unrelated to them and not motivated by the comments in those posts. But, if you do; you do.

Lucia,
On causality, as Jorge said, the commonly understood meaning becomes blurry on a time series interval. You can insist, if you wish, that the filter has to be causal at each point, making no use of future points even though they are known. As I said above, that is extreme, and rules out virtually all the symmetric filters that are actually used in the interior regions. For that reason, I thought that Steven was using a more relaxed version, in which the filter used only points from within the known data range for a given interval.

Jorge, as I’ve said, when you approach the endpoints, something about the fit is necessarily lost. In MRC etc, it is the strength of smoothing, or if you prefer the effective filter width, which is tapered. Your distinction that these points are subject to revision is also a good way of putting it. I agree that it would be desirable to indicate this on the plot somehow. It’s a continuous taper, so a sudden transition to dots doesn’t quite work either.

However, as I’ve also said, it’s a universal problem in smoothing time series on finite intervals, and I haven’t seen anyone else come up with a good presentation of the situation.

ou can insist, if you wish, that the filter has to be causal at each point, making no use of future points even though they are known.

That’s the definition of a causal filter. Yes. It rules out symmetric filters which are acausal.

IfF you wish to explain describe advantages of symmertric filters relative to assymetric filters, that’s fine. We can discuss this. But your seeing advantages to useing acausal filters doesn’t turn them into causal filters any more than my explaining that red is pretty than blue can transform red into blue.

Almost everyone agrees that filtering end points is hard and the results are both unstable and error prone. This is precisely why Rahmstorfs conclusions which are based entirely interpreting what the end points of his filtered results mean is cuckoo. This, and statements he has made suggesting he did not pad, indicate that he did not really grasp the problem with his ‘analysis’ which interpreted the meaning of the endpoint region inappropriately.

Thanks Nick –

“I agree that it would be desirable to indicate this on the plot somehow. It’s a continuous taper, so a sudden transition to dots doesn’t quite work either.”

Strangely enough the original paper (Moore et al 2005) outlining the method used by Rahmstorf said “A new approach makes use of singular spectrum analysis (SSA) [Ghil et al., 2002] to extract a nonlinear trend and, in addition, to find the confidence interval of the nonlinear trend.”

Maybe this shows how to compute the error bars that Rahmstorf choose not to place on his graph.

Clearly the weight of the first estimated point when first used is quite low and so the potential for revision is small. Even so, I am unclear how you can put an error band on it without a weather noise model.

By the time you are using all 15 estimates, they are about half the total weight for the last point. It then matters which years you get right as each is weighted differently. It seems you need an accurate weather noise plus climate trend uncertainty model before you can compute the error bands.

Isn’t this where we came in?

The UKCP09 project is going to be a good test to see how well it can be done.

http://www.ukcip.org.uk/index......Itemid=345

Re #15661, Jorge

“My question for you is whether you think the Copenhagen report should be changed to show the dashed lines when they also update the typo showing the wrong period for the smooth.”

The first sentence of the caption to Figure 3 in the Copenhagen report has already been amended, as follows:

“Changes in global average surface air temperature (smoothed over 15 years) (corrected from 11 in the first version of this report) relative to 1990.”

Isn’t this open to the (mistaken) interpretation that the Figure itself has been amended?

Ian –

Oh dear! Yes, I suppose so. Then of course, if you compare the graphs from both versions you could say “see, the length of smoothing makes no difference”.

From what I have been reading the 11 and 15 are bogus anyway, the actual filter uses 31 point smoothing. So it is really data spanning 31 years that is smoothed not 11 or 15.

Lucia,

I must be a bit gullible. Despite believing that I knew the meaning of causal, I now realise that I have allowed myself to be duped into thinking that the transformed symmetric filter is causal.

You were right all along. It is not a question of whether you actually have data at t=n+1, the point is that you are not allowed to use it in the computation of Yn if the filter is to be called causal.

One can make a philosophical case that using Xn+1 when it is actually known is not the same as inventing it when it is still in the future. That does not change the fact that if you use Xn+1 when computing Yn, it ceases to be a causal filter.

I am a slow learner but mostly catch on in the end.