Like the wake behind a jet ski, the discussion of fishy smoothing operations just doesn’t want to dissipate.
Today, I’m going to show another specific example of surprising (to some) things that can happen with smoothing. I’ll be using synthetic data generated using a function that illustrates the odd feature, so the discussion in some sense is a “cartoon”. That is: it captures the essential features I wish to communicate; in fact, it exaggerates them. The reader should be aware of the exaggerations inherent in cartooning- – even when these cartoons involve some amount of math.
The organization of the post is as follows:
- Describe the synthetic ‘data’ that will be used to show the odd smoothing effect.
- Show what happens when we smooth the data highlighting the different effects of smoothing on interior points and end points.
- Show what happens to the updated graph when 2 years of additional data are added.
- Suggest (once again) that it is foolish to try to test a theory or hypothesis to any smoothed data in the end region.
The synthetic ‘data’ .
To illustrate some odd things that can happen, I’m will generate ‘data’ based on the sum of a linear trend and a sine wave. Data are shown below.
Any reader looking at that graph could have probably figured out the functional form of that data even if I hadn’t described it. I did select it to specifically demonstrate what smoothing reveals, fails to reveal or makes even more confusing that simply looking at raw, unsmoothed data.
As we proceed, we will assume that
- the analyst wants to detect something he considers to be the “underlying trend”, associated with the straight line.
- If the analyst saw this function repeat over and over and over, he would consider the sine wave “natural variation”, which he is not trying to predict. His goal in smoothing is to filter out the “natural variation” leaving what he consider to be “the underlying trend”.
- The analysis doesn’t know (or even believe) the “underlying trend” is linear. In fact, part of his goal when filtering is to try to learn if “the underlying trend” changed. So, instead of just fitting a straight line, he is smoothing in the hope of discovering the non-linearity in “the underlying trend”. When he looks at the smooth curve, he will ask himself “Has the trend changed”?
- The analyst also plans to test a theory about the recent behavior of the system by eyeballing the agreement between the theory and the smoothed curve.
In the spirit of a cartoon analysis, we will also assume the analyst somehow never notices this is obviously the sum of a sine wave and a straight line.
Application of M=15 smoothing filter with MRC end points.
We’ll now examine what happens if we decide to apply a triangular filter with a half-width of M=15 to data ending in 2006. In this case, I will apply the “Minimum Roughness Condition” (MRC) by padding computed by extrapolating using the least squares fit to the final “M” data points. This method of guessing future endpoints makes sense if one believes that trend over the final M data points reflects the “underlying trend”, and not the “noise”. (The relationship between the triangular filter and Rahmstorfs M=15 filter was discussed by SteveM.).
Following the Copenhagen/Rahmstorf tradition, I will not compute any uncertainty intervals for the smooth trend. Violating that tradition, I will highlight the data used to compute the least squares line (in blue), show the least squares trend (with a green dashed line), show a few of the ‘guessed’ future data points (blue triangles) and use different colors to indicate region where the smoothed curve is computed using the triangular filter applied to obsered data only (purple), and where the curve was computed using the same triangular filter, but including guessed not-yet observed data (dark blue). The boundary is marked by a vertical dashed blue line.
Here’s how the graph looks:
Assuming the analyst had the goals highlighted previously, he might focus on the following things:
- When the smoothed curve was computed using observed data only, the filter dramatically reduced the oscillation due to the sine wave. Using the “eyeball” technique, the analyst might conclude the smoothed data “speaks for itself” giving us insight into the “underling trend”. If the analyst considered the sine wave “noise” and the line “signal”, he might conclude the filter successfully removed the noise, leaving the “signal”. Success!
- In the “smoothed using extrapolated data” region , the smooth curve is smooth. The second derivative is small: There are no wiggles. This curve somehow seems “less noisy” than the previous one. However, it’s interesting to note that the magnitude of the ‘smoothed’ temperature in 2008 actually exceed the maximum value in the observational period. So, while the smoothed line is “smooth”, it’s not necessarily closer to the “underlying trend” the analysit is hoping to uncover using this technique.
Now, assuming the analyst didn’t have the benefit of the vertical blue line, and the green line showing the extrapolation, let’s ask ourselves: Might the a naive analyst conclude that smoothing in the end region tells him something about the behavior of “the underlying trend”?
It would be foolish to do so.
To emphasize how foolish, I added the continuation of the smoothed trend that will appear when the next 17 data points are collected. Notice how the low amplitude sinusoid will re-appear when more data are observed? In reality, the dramatic change in appearance of the smoothed line in the en points tells us nothing about the underlying trend. Nothing. Nada. Rien.
The reason the smoothed curve shows dramatic changes is the properties of the smoothing function changed. Specifically, the functional form of filter was held constant, but it was applied to guessed future data and the guess was inadequate. (Or, if you want to look at this Nick Stokes way, the filter weights were modified. The method of recomputing the weights can be discovered by a) assuming the future data falls on the least squares line shown and b) asking a bright 14 year old to do some algebra to recompute weights.)
Now that I’ve explained the problem with diagnosing data trends by examining the smooth graph, I want to show the same graph again. Remember that in this discussion, the analyst is specifically interested in comparing the predictions beginning in 1990 to the smoothed data. Here’s the part of the relevant part of the graph:
Notice that nearly the entire comparison period is in the problematic end point region. I haven’t shown what the “theory” might be, but I think most would suspect it would be foolish to make any conclusions about the relative agreement between a theory and the smooth data in this region.
Addition of 2 more years data
It’s hardly surprising that addition of 2 more years of synthetic data to this particualr example dramatically changes the behavior of the smoothed curve. In the following graph, I added data for 2007 ad 2008, and recomputed the smoothed curve. The new extension is shown in aqua:
It’s easy to see that the trend in the “end region” was changed dramatically; the new end-trend looks like a downward facing fishhook.
Is the new end-trend better than the previous trend? Not really. Because we happen to know the functional form used to generate the synthetic data, the new trend looks better. But if we didn’t happen to know that, we would have no way of knowing whether the updated extension was “better” or “worse” than the previous one.
Is the new trend worse than the pervious one? should we “correct” our proceedure by changing M? Changing M slightly isn’t really going to help. We could make a new series of graphs using M=17, and we’d see the problem will still exist. If the problem really was the magnitude of M, we’d need to at least double M to see any real improvement in the method. Small changes might let us change the answers current appearance such that we “like” it better; that’s all.
What we should know is that we should be very, very careful interpreting the meaning of the trend in the end region.
Advise to those considering smoothing.
First, I admit that this example exaggerates the dangers of smoothing. I picked a functional form that would specifically highlight a particular danger. This specific problem would not arise if the data was better described by a trend plus white noise. Nevertheless, it’s worth recognizing that we usually don’t know the underlying form of the trend or the noise. So, being aware of all possible smoothing artifacts can alert us to dangers.
Given that, here are some points worth considering:
- Smoothing has its uses. It’s less problematic when applied to interior data points and used mostly to make pretty pictures. It’s most problematic when applied to end points and used to test scientific hypothesis.
- Even when making pretty pictures, avoid showing the smooth trends computed in end regions. People have a difficult time disambiguating and may focus on the dramatic changes in the curve as it moves from the interior regions to the end regions.
- If at all possible, show uncertainty intervals all along the smoothed graph. Explain the assumptions (including the statistical model for errors) you used to compute the uncertainty intervals.
- If you don’t know how to compute the uncertainty intervals, at least highlight the difference between the smoothed trends in the end region and the interior. Change the color of the curve, make it dashed, or show a vertical line as I did. Some indication to alert the reader (and the possibly naive analyst) is necessary to prevent people from making unwarranted conclusions.
- If you don’t know how to compute uncertainty intervals, scrupulously avoid discussing the agreement between smoothed data in end regions to any theories, projections or prediction.
- If you must make comparisons between observations and theories, seek out other methods to test hypotheses. Smoothing may be your favorite hammer, but you’ll that doesn’t mean you should treat every data analysis task as a nail.
1) But what if you just know your understanding of the trend is right? Wouldn’t be OK to apply smoothing to end points in that instance–especially if only bad people disagree with you?
2) Pielke Sr. today posted a radical thought that any search for temperature trends may be a waste of time. What is Weather Noise
He argues that linearized graphs of temps in search of trends is less meaningful than looking for the amount of heat gained or lost because climate is more likely to shift in jumps rather than gradual trends. Don’t differentiate–integrate.
George–
If you just know the trend, there is no need to do any analysis. 🙂
I agree with Roger Sr. that we don’t need to create linearized graphs of trends. But I think, like other techniques, they can be useful. I think we need to be very careful saying one method is flat out better than another method. Instead, we can point out the problems with application of “method A” in the context in which someone specifically used the method.
So, for example, Rahmstorf application of smoothing to test the TAR is peculiar and error prone. That doesn’t mean careful practitioners who are familiar with the dangers of can’t apply it to other problems, taking pains to avoid the conceptual and applications errors R made.
Similarly, applying linear trends doesn’t always work, but sometimes assuming a trend makes a test more sensitive than just checking whether the average from period A is different from the average in period B.
Very nice. Can I also recommend a soundbite version of this argument by JS at Climate Audit (comment 83 on the Rahmstorf Rejects IPCC Procedure thread).
What you’re showing is also true of OLS if your data is sinusoidal and you pick the wrong start or end points, which is my contention on any range that doesn’t cover the 60 peak pattern in the temp data.
Nicely put.
The example of the sine waves reminded me in a way of Yule (1926), the original article on spurious regression, but which spends quite a bit of time discussing the interaction between correlations and sine waves (online at http://www.climateaudit.org/pdf/others/yule.1926.jrss.pdf )
BarryW–
Yes. It’s possible to apply OLS incorrectly too. But at least almost everyone applying OLS computes uncertainty intervals, and it’s easy find material discussing some tests that should be applied after fitting data to a trend.
There is something about both smoothing and fitting to higher order polynomials that sometimes cause people disengage their grey cells and believe the nice smooth graphs. It’s also a bit more difficult to find discussions of potential problems placed in the context of smoothing.
If someone is going to actually try to smooth to end points, it might be nice if someone knew of a reference that discussed a series of tests to ensure ones smoothing filter did not become pathological in some specific “cartoon” problems.
For example, things a “halfway decent” end point condition should not do.
* Ends should not “turn down” when the ‘true’ underlying trend is positive an the noise is AR(1). This would disqualify padding using the average over the full range or padding using the current value.
* If the ‘true’ trend is a sine wave + a linear trend, the predicted value of the “smooth” curve should never flail out as it does in the figures in the post. (I could try to explain that in mathematical terms, but would need to make another sketch.)
It might be that padding using a linear extrapolation based on the final (2M+1) points would greatly reduce the problem of the smoothed curve in the end points display deviate from the fictional “underlying trend” just as much as unsmoothed noise did in the first place. (Only, now the impact is worse because the analyst things he reduced these deviations, so he things what he sees in the end points is real.)
Brilliant. Perfectly clear. Lucia, you are wasted doing whatever you are doing. You should be teaching. Well, you should be doing research as well. Mind like a steel trap. Cannot compliment you enough on this one.
Lucia, first a query. What version of MRC are you implementing? It isn’t Mann’s. It looks more like Grinsted, but my understanding, based on David Stockwell’s version of the code, is that they project with the slope you’ve used, but from the last data point.
Then a comment – as you say, the analyst isn’t supposed to know the structure of the example. The aim is to apply a lowpass filter to identify and eliminate “noise”; then the “trend” is the slope of the remaining curve. That’s all you can do.
Now you’re using a 29 year filter to identify 20 year cycle “noise”. That’s never going to work very well. As you show, significant periodic behaviour remains, and inevitably contributes to the estimate of trend. That’s not really an error. It’s an ambiguity about whether the sine, with period close to the filter length, is to be regarded as “signal” or “noise”.
So while I’m not sure that your MRC is one of the known versions, I’d agree that any endpoint rule is going to locally enhance this residual oscillation in the estimated trend. As I’ve said elsewhere, you’re running out of data and something has to give. The MRC way is to try to avoid bias, and accept the loss of noise attenuation there.
Nick–
I’m using the one I described, making no particular claims that it matches anyone’s.
We all agree something has to give. People keep explaining that smoothing is imperfect at endpoints. You keep popping in and “explaining” to us that we need to understand that smoothing is imperfect at end points.
But you seem to be missing the points people are making which are that because smoothing is imperfect at end points:
a) People should not interpret the curve at end points as somehow ‘real’ and then diagnose the progression of climate change based on this.
b) People should avoid showing these regions without highlighting the fact that the information in the smoothed region near the end point can be very, very deceptive.
MRC maybe a way to try to avoid bias, but trying is not the same as succeeding.
MRC does not succeed in avoiding bias. It substitutes one sort of potential biase for another sort of potential bias. If you get lucky then MRC works out fine. Otherwise, it does not.
lucia (Comment#16119)
July 10th, 2009 at 5:05 pm
Thanks. i needed a good laugh.
Though I think that if you extrapolate all the different issues and variations that NIck has brought up, there is only one solution:
2M+1 = total length of the data, including the data R did not bother to show;
then dashed line after the actual data ends.
Lucia, have you read the Moore, Grinsted sea level articles? Or the Rahmstorf sea level articles? 🙂
“MRC does not succeed in avoiding bias. It substitutes one sort of potential biase for another sort of potential bias. If you get lucky then MRC works out fine. Otherwise, it does not.”
It may be time to drop the idea of smoothing to the latest data.
I look at a number of different data data sets where the smooth is stopped before the end of the actual data. I do not experience any frustration or sense of loss at this. Of course, these charts are simply presenting data and not trying to “sell” a particular interpretation of the data!!!
KuhnKat–
The other difficulty is that Nick has cited a paper that discusses creating endpoint smoothers that are unbiased at the endpoints. The paper points out that for smoothers that are based on either explicit or implicit prediction of the future, the unbaised smoother with that will result in minimum change in the future is one based on an the best fit curve to the data. The discuss ARMA fits.
However, in the event that one does’t go the ARMA route ignores autocorrelation or moving averages, as far as I can determine one would use the end point smoother I discuss in this article, which, as Nick points out is neither the Mann or the Grinsted fit.
But, even then, the econometricians writing one of the papers Nick linked would be advising coming up with some methodology to figure out the decent fit to use when extrapolating. Maybe if one went to that trouble, you could come up with a smoothing that did not go all wobbly at endpoints. But if one is going to try to use ARMA to figure out the extrapolation required to fit the end points… why not just show the ARMA projection in the “end point” region? Or show the both the ARMA projection, the smooth obtained using the ARMA and the extension. (The reason is obvious: If people understood you were using the ARMA to create the smooth, they’d realize they should suspect the smoothed trend in that region. All those traces become clutter.)
NIck Stokes:
The same is true of attempting to filter the last 30-years of global mean temperature data. There is roughly a 11-year oscillation in global weather associated with the solar cycle sitting on top of any long-term trends from human activity.
In other words, Lucia’s example is completely apropos to the real-world problem here. Given that, remember who you are also criticizing when you comment on her example!
Carrick,
I wasn’t criticizing, just commenting. For any reasonable filter, there are inevitably frequency bands (noise) suppressed, bands passed with little attenuation, and an inbetween region. Lucia has chosen an example from the latter. Nothing wrong with that, as long as it is understood. In fact it’s more interesting.
Nick– I did intentionally pick a “problem” signal to illustrate. I also picked end points that highlight the problem for this signal/filter combination. That’s why I said this exaggerates to show the problem.
I did pick it because it shows something interesting, as opposed to highlighting specifically what happened in say Rahmstorf’s analysis.
BTW– I picked the end filter because that extension would either be, or be close, to the one that would cause the minimum revisions if the data were trend+white noise. It obviously doesn’t work well for the sine wave.
Nick–
Grinsted and Moore’s paper isn’t very explicit about precisely how they apply their bc and I dislike backing information out of either matlab or R code (because I don’t use either.)
Is you impression they do one of the following:
A) Fit OLS to last M data points. Use that trend but slide up to the end data point, and assume data along that line.
B) Fit OLS through the last M data points, forcing it through the final data point. Assume future data long that line.
Both methods would have odd effects. But… I’m under the impression they do one of the other. Do the do either? Or something else.
for grins apply various smooths to the historical record. see how often its misleading.
Lucia,
Sorry I missed this query. Yes, I believe it is A). Compared with your method, there is just an offset, same slope. And as I said in a later thread, I think your method has some advantages, overall, tho’ G’s may appear to track better at the end.
Nick– In one of those papers you cited when we were discussing Mann’s MRC, they mentioned that using OLS to project gives you the minimum for future revisions provided that the data really does follow a linear trend and the noise is white. They also mention the possibility of using an ARMA fit to project, then smooth.
Of course, you can do all the arithmetic to make the projection implicit rather than explicit.
I don’t know what advantage shifting the line might have. It should result in greater oscillations when new data come in.
Lucia, I agree about the OLS minimum revisions – I think it’s the same as I was saying here #16219. Mann and Grinsted (less) accept greater oscillations in order to try to best track the most recent data. They have a case, but OLS looks good to me.
Nick–
In the context of filtering, I don’t see how any positive case can be made for doing things that “try to best track the most recent data”. The purpose of filtering is to smooth out the higher frequency oscillations. What is the point of specifically designing a filter to smooth out the oscillations in the interior, while maximizing the impact of high frequency “noise” at the end points?
The best way to track recent data is to not smooth it at all.
If tracking the end points was one’s goal when one writes a papers when the end points happened to point in the direction of one’s favorite theory, is it any wonder the practice has lead to accusation that one specifically designed previously unknown end point treatments to emphasize the behavior end-points he assumed meaningful?
I guess in the case of Rahmstorf, he hunted down a little know end point treatment and then tweaked his value of M when the end points no longer did what he liked. This is pretty pathetic as his entire paper is nothing more than a comparison of what amounts to comparison of observations to this bizarrely smoothed end point.
The whole episode stinks badly of people (particularly Rahmstorf and co-authors) acting under confirmation bias.