The blogger Steven Goddard has been on a tear recently, castigating NCDC for making up “97% of warming since 1990” by infilling missing data with “fake data”. The reality is much more mundane, and the dramatic findings are nothing other than an artifact of Goddard’s flawed methodology. Lets look at what’s actually going on in more detail.
Whats up with Infilling?
The U.S. Historical Climatological Network (USHCN) was put together in the late 1980s, with 1218 stations chosen from a larger population of 7000-odd cooperative network stations based on their long continuous records and geographical distribution. The group’s composition has been left largely unchanged, though since the late 1980s a number of stations have closed or stopped reporting. Much of this is due to the nature of the instruments; many USHCN stations are manned by volunteers (and are not automated instruments), and these volunteers may quit or pass away over the decades. Since the 1980s the number of reporting USHCN stations has slowly declined from 1218 in the 1980s to closer to 900 today, as shown in the figure below. As an aside, this is quite similar to what happened with GHCN, which birthed the frustratingly-persistent “march of the thermometers” meme. Unsurprisingly, the flaw in Goddard’s analysis mirrors that of E.M. Smith’s similar claims regarding GHCN.
As part of its adjustment process, USHCN infills missing stations based on a spatially-weighted average of surrounding station anomalies (plus the long-term climatology of that location) to generate absolute temperatures. This is done as a final step after TOBs adjustments and pairwise homogenization, and results in 1218 records every month. The process is really somewhat unnecessary, as it simply mirrors the effect of spatial interpolation (e.g. gridding or something more complex), but I’m told that its a bit of an artifact to help folks more easily calculate absolute temperatures without having to do something fancy like add in a long-term climatology field to a spatially-interpolated anomaly field. Regardless, it has relatively little effect on the results [note that 2014 shows only the first four months]:
You can’t really tell the difference between the two visually. I’ve plotted the difference below, with a greatly truncated scale:
Where did Goddard go wrong?
Goddard made two major errors in his analysis, which produced results showing a large bias due to infilling that doesn’t really exist. First, he is simply averaging absolute temperatures rather than using anomalies. Absolute temperatures work fine if and only if the composition of the station network remains unchanged over time. If the composition does change, you will often find that stations dropping out will result in climatological biases in the network due to differences in elevation and average temperatures that don’t necessarily reflect any real information on month-to-month or year-to-year variability. Lucia covered this well a few years back with a toy model, so I’d suggest people who are still confused about the subject to consult her spherical cow.
His second error is to not use any form of spatial weighting (e.g. gridding) when combining station records. While the USHCN network is fairly well distributed across the U.S., its not perfectly so, and some areas of the country have considerably more stations than others. Not gridding also can exacerbate the effect of station drop-out when the stations that drop out are not randomly distributed.
The way that NCDC, GISS, Hadley, myself, Nick Stokes, Chad, Tamino, Jeff Id/Roman M, and even Anthony Watts (in Fall et al) all calculate temperatures is by taking station data, translating it into anomalies by subtracting the long-term average for each month from each station (e.g. the 1961-1990 mean), assigning each station to a grid cell, averaging the anomalies of all stations in each gridcell for each month, and averaging all gridcells each month weighted by their respective land area. The details differ a bit between each group/person, but they produce largely the same results.
Lets take a quick look at USHCN raw data to see how big a difference gridding and anomalies make. I went ahead and wrote up some quick code that easily allows me to toggle which dataset is used (raw or adjusted), whether anomalies or absolutes are used, whether the data is gridded or not, and whether infilled data is used or not (in the case of adjusted; raw USHCN data has no infilling). Its available here, for anyone who is interested (though note that its in STATA).
Here is what we get if we take USHCN raw data and compare the standard approach (gridded anomalies) to Goddard’s approach (averaged absolutes) Â [note that 2014 is omitted from all absolute vs anomaly comparisons for obvious reasons]:
This compares absolutes to anomalies by re-baselining both to 1961-1990 after the annual CONUS series have been calculated. The differences stand out much more starkly in the difference series below:
This difference is largely due to the changing composition of  stations in the network over time. Interestingly, simply spatially gridding absolute temperatures eliminates much of the difference, presumably because the other stations within the grid cell have similar climatologies and thus avoid skewing national reconstructions.
The difference series is correspondingly much smaller:
 Here the differences are pretty minimal. If Goddard is adverse to anomalies, a simple spatial gridding would eliminate most of the problem (I’m using USHCN’s standard of 2.5×3.5 lat/lon grid cells, though the 5×5 that Hadley uses would work as well).
So what is the impact of infilling?
Lets do a quick exercise to look at the impact of infilling using four different approaches: the standard gridded anomaly approach, an averaged (non-gridded) anomaly approach, a gridded absolute approach, and Goddard’s averaged absolute approach:
Goddard’s approach is the only one that shows a large warming bias in recent years, though all absolute approaches unsurprisingly show a larger effect of infilling due to the changing station composition of the non-infilled data. We also have a very good reason to think that there has not been a large warming bias in USHCN in the last decade or so. The new ideally-sited U.S. Climate Reference Network (USCRN) agrees with USHCN almost perfectly since it achieved nationwide spatial coverage in 2005 (if anything, USCRN is running slightly warmer in recent months):
Update
Another line of evidence suggesting that the changing composition is not biasing the record comes from Berkeley Earth. Their U.S. temperature record has more stations in the current year than in any prior year (e.g. close to 10,000) and agrees quite well with NCDC’s USHCN record.
Update #2
The commenter geezer117 suggested a simple test of the effects of infilling: compare each infilled station’s value to its non-infilled neighbors. My code can be easily tweaked to allow this by comparing a reconstruction based on only infilled stations to a reconstruction based on no infilled stations, only using grid cells that have both infilled and non-infilled stations (to ensure we are comparing areas of similar spatial coverage.
The results are unsurprising: the two sets are very similar (infilled stations are actually warming slightly less quickly than non-infilled stations since 1990, though they are not significantly different within the uncertainties due to methodological choices). This is because infilling is done by using a spatial weighted average of nearby station anomalies (plus the average climatology of the missing station to get absolute temperatures).
Its also worth pointing out that these problems plague some of Goddard’s other recent claims, like his assertion that the first four months of 2014 were the coldest on record: http://stevengoddard.wordpress.com/2014/05/14/coldest-year-on-record-in-the-us-through-may-13/
Here is what the results should actually look like: http://i81.photobucket.com/albums/j237/hausfath/USHCNgriddedanomaliesaveragedabsolutesjan-april_zps397d3fbb.png
Nick has a couple of links up worth reading:
http://moyhu.blogspot.com/2014/05/nonsense-plots-of-ushcn-adjustments.html
http://moyhu.blogspot.com/2014/05/the-necessity-of-tobs.html
http://moyhu.blogspot.com/2014/05/ushcn-adjustments-averages-anomalies.html
The last post has a mathematical demonstration of the flaw in Goddard’s analysis.
I don’t mind people making mistakes, but it bothers me when they don’t correct them after they’ve made them, especially when the error appears in their post, rather than in a comment on a post.
Yep, Nick’s posts are quite good. I meant to link to them, but it slipped my mind. Goddard has moved on from homogenization to focus on infilling, though the same methodological issues plague his analysis of both.
Thanks, Carrick. I find it useful to remember, about spatial infilling, homogenisation etc, that when you replace station readings with some average of local readings and then weight average the lot, the end result is just a reweighting. And if you have done the interpolation right, it’s an area-based reweighting.
I do wish USHCN would give up on the absolute temperature stuff; they do it as well as it can be done, but to no good purpose, and it encourages many to do worse. Suppose you tell someone that the CONUS average was 56°F last year. Anyone who wants to make sense of that will ask – well, what is it normally? In other words, what’s the anomaly?
Nick,
The absolute temperature may be of interest to some people. For example, farmers, landscapers, people designing streets or buildings and such may want absolute temperatures and would wish to consult them when making decisions about new products. I’m not sure if the care what CONUS absolute temperatures are, but they could easily want temperature in a county.
I have no idea if this is why absolutes might have been preferred historically. But it’s worth nothing that absolute T does matter for some issues– they just aren’t very useful for assessing whether temperature are increasing or decreasing over time.
Indeed. You can also use deterministic factors like elevation to create absolute temperature fields much more granular than what you get by interpolating between individual observations. The easiest way to get good up-to-date absolute temps is to create a simple spatially-interpolated anomaly field for the current month and add it to the climatology field for the baseline period.
Lucia is on to an important distinction irt to anomalies vs. absolute temps: Life happens in actual temps.
Anomalies are a derivative product that impact most people indirectly at best.
Long after the popular madness of climate obsession passes people will still care about what actual temperatures are.
It sure was hot last week: +1C on the old anomaly scale.
The only way that this data tampering could be legitimate is if there has been progressively increasing selective loss of warmer data from the temperature record. That isn’t very credible.
http://stevengoddard.wordpress.com/2014/06/06/illinois-manufactured-temperatures-warming-at-23-degrees-per-century/
Hmm, you say regarding the Raw Gridded Anomalies minus Averaged Absolutes that “This difference is largely due to the changing composition of stations in the network over time.”
This should be true if there is no methodic error, yes.
But what pure station composition mechanism exists that over 100 years would cause the difference to increase monotonically, when both adding stations (early years 1900-1940) and removing stations (later years 1980-2013)?
Adding stations in the early years increases the difference.
Removing stations in the later years ALSO increases the difference.
I’m not sure that simply adding and then removing stations can produce this effect, at least not if the addition or removal was random.
Or am I missing something here?
Do you have a plot of the Raw ungridded anomalies versus the ungridded absolutes?
Goddard just seems very confused. Zeke’s explanation is clear. Goddard should read it.
“Where did Goddard go wrong?” and
“the frustratingly-persistent “march of the thermometers†meme. Unsurprisingly, the flaw in Goddard’s analysis mirrors that of E.M. Smith’s similar claims regarding GHCN.”
The political nature of this “attack “article only illustrates the deep held conviction that something is badly wrong in climate world.
It also shows that Goddard and Smith are giving vent to the deeply held suspicions that data is being manipulated and manufactured by the custodians of the data.
The fact that an attack has to be made shows that some of the these suspicions may in fact be real.
A cross pollination of backup , similar to Cook and Lewindowsky and also the 97% meme is no surprise. First we have Zeke who seems straight and convincing, then Nick Stokes, confirmation. followed by Eli Rabett May 20, 2014 at 7:36 pm at Nick’s Moyu blog.
With a bit of luck Carrot Eater will be along as well.
Absolute temperatures will get another flogging,
” If you use absolute temperatures and your station network composition is changing over time, you will tend to get climatology-related artifacts that are comparable in magnitude to any trend effects you are looking for.” NS
Along with ” But who needs to know the US average temperature?”
Well guess what, NS you might also get a real true trend showing artifice rather than artifact.
Steven Mosher in to defend his mates [I almost thought he was Carrot Eater reading “march of the thermometersâ€} but the vocabulary, though perfect, was not refined enough for Steven.
I hope Goddard posts here as well. Real Data ie absolute temp observations, sure beats mathematical shell games.
Who is this Goddard person and why does he matter? Does he have a space flight center named after him?
FACTS
Zeke March 2nd, 2010
There are 782 stations in GHCN with a start date prior to 1900, an end date after 1990, and at least 100 years of data
Carrot Eater March 2nd, 2010
If you look at Hansen’s 1987 paper, there is a huge station drop-off around ~1970. all that missing data got filled in later, so that the dropoff moved to ~1990
BillyBob March 2nd, 2010
” Almost all (if not all) GISS stations are now airports and non-rural.”
Zeke March 2nd, 2010
“GHCN was more rural (as a % of stations) post-1992 than pre-1992.”
BillyBob March 2nd, 2010
†all of the US temperatures – including those for Alaska and Hawaii – were collected from either an airport (the bulk of the data) or an urban location.â€
WQiki 7,280 fixed temperature stations in the GHCN catalog color-coded by the length of the available record. Sites that are actively updated in the database (2,277)
“(USHCN) was put together in the late 1980s, with 1218 stations chosen from a larger population of 7000-odd cooperative network stations based on their long continuous record.”
“slowly declined [???] from 1218 in the 1980s to closer to 900 today,” [??? some added as well]
My point would be that over a quarter of the so called stations are now virtual infilled stations. They do not exist but you keep using the data as if it was real so as to keep a stable chart presentation.
Worse than this some stations may have been added [correct if wrong] in airports and cities while others are dropped off so the Number of true stations with strong historical records ie ones that existed when the 1218 were chosen are now only about 609. Thus only 50% of the original record only is being used, yet this is being claimed as a true historical record and graph.
Who gets to choose which stations are dropped off. Are all the new American stations in Airports and cities. How were they Kriged to fit into the old result. Why do all redos of data sets keep going up in temperatures
Zeke, you are a gate keeper, you are linked to the data collectors and computers I think. If all of what I have said is true and you do not get the idea that people are a bit sus”
[ Then you must live on another planet Way out in space surrounded by Cowtans emitting a positive warm rising temperature. Lucia, feel free to cut this bit of sarcasm out if it is too rough ]
Let me rephrase:
.
Who is this Goddard person and why does he matter? Emperor Lrrr of Omicron Persei 8 demands to know! Is he a Goddard of the Earthlings space flight centers?
Thanks Zeke (and Nick)
I wish I had more time to spend on the math. Translating your post into RobertSpeak, I understand:
1. Of the continuously reporting stations, the impacts of adjustments is nill. I assume Goddard has never claimed otherwise.
2. Of the missing stations, the temperatures are estimates.
3. Applying gridding and reporting as anomalies to the continuously reporting and estimated stations shows no drift.
4. Applying gridding and reporting as anomalies to the continuously reporting stations shows no drift.
5. By implication, the estimates for the no longer reporting stations may be open to challenge.
As always, it is a great pleasure when you stop by.
Thank you
Steve Goddard has a post, “Temperature Vs. Temperature Anomaly”
It starts:
“A few mathematically illiterate people believe that somehow the use of anomalies would make NCDC data tampering go away. Below are the anomaly and absolute temperature graphs. As you can see, it makes absolutely no difference. All that the use of anomalies does is shift the Y-axis.”
So for all you mathematical illiterates who didn’t know that “anomalies” means just subtracting the mean from the final result, here is SG’s challenge:
“I am happy to debate and humiliate anyone who disagrees with this basic mathematics”
Steve Goddard,
Yes, infilled data in IL might be warming at 23 degrees per century.
Or your method could be wrong.
http://i81.photobucket.com/albums/j237/hausfath/USHCNILinfillednoninfilled_zpsa1854dbc.png
http://i81.photobucket.com/albums/j237/hausfath/USHCNILinfillingeffectabsolute_zps1ed76c27.png
Not sure I need to invoke Occam’s Razor on this one.
RobertInAz,
On continuously reporting stations the effect of infilling is nil (by definition). Adjustments do have an effect, but thats another discussion.
The main take-away is that there is a right way that everyone in the literature (and on the blogs, e.g. Jeff Id’s approach or Anthony Watt’s paper) that uses spatial weighting and anomalies. Steve Goddard’s way is not correct, and is introducing spurious artifacts.
.
BobD,
There could be different effects in the early or late years if there are different reasons why stations started back then and ended today. If stations were started or stopped reporting at random, it wouldn’t matter at all. If stations in, say, New England are more likely to be run by volunteer observers and stations in the south are more likely to be automated airport-based stations, then a dropout of mostly New England stations (because the observers passed away or otherwise stopped reporting) would lead to a warming bias in recent years if you just averaged absolutes. Anomalies, or gridded absolutes, would mostly avoid this issue, as discussed above.
Here is raw non-gridded anomalies vs. non-gridded absolutes:
http://i81.photobucket.com/albums/j237/hausfath/USHCNaveragedanomaliesaveragedabsolutes_zpsf820a413.png
http://i81.photobucket.com/albums/j237/hausfath/USHCNaveragedanomaliesminusaveragedabsolutes_zpsca31b8c8.png
Anomalies make a pretty big difference during periods (early and late) where the composition of the network is changing. Anomalies don’t introduce any bias, so if you are trying to look at change over time in a series with changing composition there is no reason not to use them (and many reasons to!).
angech,
You mentioned that “you are linked to the data collectors and computers I think.” While my fiancee often claims that I’m linked to my computer, I’m pretty sure she is speaking metaphorically.
Zeke,
I have no problems with the use of anomalies, and you’re quite right, one of the main reasons to use them is to ensure consistency when comparing incomplete records.
Thanks for the graphs of non-gridded anomalies vs non-gridded absolutes. They show that the absolutes have cooled over time relative to the anomalies.
This tells me that in the early years there was a net gain of cooler stations with time, and later on there was a net loss of warmer stations. Is that likely? I would have thought (as per your New England example) that as stations moved to airports and cities that the opposite would have been true in the later period.
It’s also interesting that the gridding process on its own effectively doubles this difference rate. It looks a bit suspicious to me (ie: there may be an inherent error somewhere), especially as you don’t appear to have a ready explanation for why the difference is so beautifully monotonic over time, while the station count looks like an inverted U-shape. Is this something that was not noticed and analysed previously? I find that difficult to believe.
It seems to me that each step of this process must be fully and exhaustively understood and communicated if the adjustments are to be believed, because right now there appears to be a lack of clear understanding of what is actually happening in the metadata. I’m not saying it’s wrong, but it may well be.
Step 1 would be to explain why the difference after gridding is so linear (and why gridding doubles the anomaly/absolute difference rate), and step 2 would be to explain why gridding the absolutes produces almost the same result as gridding the anomalies.
Not everyone appears to agree with Zeke that there are no problems.
http://www.drroyspencer.com/2014/01/u-s-temperatures-1973-2013-a-alternative-view/
Zeke, Fortunately married before they had personal computers up and running, Suggest more going out and less computers as is my wife’s advice to me.
So to be clear
there were “ 1218 stations (USHCN) in the late 1980s
So to be clear
there were “ 1218 real stations (USHCN) in the late 1980s
There are now [???] original real stations left-my guess half 609
There are [???] total real stations – my guess eyeballing 870
There are 161 new real stations , all in airports or cities added to the graph
There are 348 made up stations and 161 selected new stations.
The number of the original 1218 has to be kept like the stock exchange to have a mythical representative temperature or temperature anomaly over this number of sites.
Nobody has put up a new thermometer in rural USA in the last 30 years and none has considered using any of the rural thermometers of which possibly 3000 of the discarded 5782 cooperative network stations.
And all this is Steve Goddards fault.
Can someone confirm these figures are accurate and if so why any trust should be put in this Michael Mann like ensemble of real, adjusted real and computer infilling models.
Nick Stokes. Absolute temperatures matter a lot. when you get a reading in Celsius it is what it is. An anomaly is purely a mathematical digression from this which is slightly easier to use in mathematical models.
There is a large amount of difference and significance in a 2 degree spread at – 263 degrees, at 15 degrees, and at 30,000 degrees say on the sun.
Even on the earth at sea surface temperature a couple of degrees change in the Arctic in winter is quite a massive different heat input to 40 degrees in summer at the equator.
Personally I don’t mind which of the two readings you use, as Steve pointed out there should not be any differences in the change per individual weather station.
What I do mind is your quibbling about it purely to obfuscate the argument. 15 degrees average sure sounds fine to me and 16 degrees if it ever arrives would be even better.
Assuming that the derivative temperature anomaly product is correct, what does it show of any importance?
The term “difference without distinction” comes to mind.
As angech points out, a 0.5 C anomaly in a system that has a built in dynamic range of 30.0 C is possibly less than a crisis. And many of the derived anomaly trends we see promoted as a crisis are much smaller.
I’m an old school sceptic. I work as an engineer. I am not familiar with the in’s and out’s of temperature measurement and the complex series of adjustments that it undergoes. Nor do I intend to go down that road. So what do you do? Well, sceptics have some rules of thumb. They are fallible. Sometimes the rules of thumb guide but at other times they mislead. But this is still better than trying to pin a ribbon blindfolded on a spinning wheel.
So is Steve Goddard credible? He has come up with some interesting ideas. He doesn’t always get things right. He is partisan. He often doesn’t admit to mistakes or tries to defend his position in a way that’s intellectually not credible. I know this, because on a fairly black and white issue (if there is such a thing in this field) he didn’t admit to his mistake with me. In fact, he tried to conflate issues and behave in a way that frankly wasn’t honest. So do I trust Steve? No. But he still finds interesting stuff that other more credible sources occasionally pick up on. Do I trust people who endorse Steve has being credible? No I don’t either.
Now on the other side of the fence, there is a guy like Nick Stokes. Same story. Issue was black and white. I picked him up on a mistake. Did he concede? No. He danced around and tried to conflate the issue. So do I trust people now who endorse people like Stokes? No I don’t.
So what am I left with? A whole bunch of people, partisan, heavily invested in positions, who want to be ‘right’ not in the intellectually vigorous sense, but in the sense a lawyer wants to ‘win’ a case.
That doesn’t mean Stokes, Zeke, Goddard or other partisans are wrong on everything. In fact, they might be right on most things most of the time. But as a famous sceptic once observed, with this lot, you really do have to pay attention to the pea under the thimble, and that’s not easy because those hands are moving fast.
“Absolute temperature” is well defined by physics, and it is not what is measured in °C or °F.
Will Nitschke:
Great post. I feel the same way.
Zeke (Comment #129987)
“Steve Goddard,
Yes, infilled data in IL might be warming at 23 degrees per century.
Or your method could be wrong.”
I think the latter. I’ve posted my Illinois calc here.
It is very clear that NCDC uses every trick in the book to manipulate the global and US temperature trends.
Large datasets like this are easy to manipulate with algorithm selection/site selection/selective adjustments that all sound perfectly reasonable to those that “like” their data manipulated.
I see Tom Karl leaning over someone’s desk two or three times per month saying “What happens if we change this X? Oh, well that’s no good. Let’s change that Y. Oh darn, that’s no good either. Well, we have to drop in that new list of stations that Tom Pederson came up with.”
I always wonder if the people who read and/or contribute to The Blackboard ever wonder if the graphs they produce actually mean something in the world somewhere or if they are simply exercises in squiggology. I wonder where the line between the two things lies and how a scientist would recognize it.
Andrew
I took Goddard to task over this as well in a private email, saying he was very wrong and needed to do better. I also pointed out to him that his initial claim was wronger than wrong, as he was claiming that 40% of USCHN STATIONS were missing.
Predictably, he swept that under the rug, and then proceeded to tell me in email that I don’t know what I’m talking about. Fortunately I saved screen caps from his original post and the edit he made afterwards.
See:
Before: http://wattsupwiththat.files.wordpress.com/2014/06/goddard_before.png
After: http://wattsupwiththat.files.wordpress.com/2014/06/goddard_after.png
Note the change in wording in the highlighted last sentence.
In case you didn’t know, “Steve Goddard” is a made up name. Supposedly at Heartland ICCC9 he’s going to “out” himself and start using his real name. That should be interesting to watch, I won’t be anywhere near that moment of his.
This, combined with his inability to openly admit to and correct mistakes, is why I booted him from WUWT some years ago, after he refused to admit that his claim about CO2 freezing on the surface of Antarctica couldn’t be possible due to partial pressure of CO2.
http://wattsupwiththat.com/2009/06/09/co2-condensation-in-antarctica-at-113f/
And then when we had an experiment done, he still wouldn’t admit to it.
http://wattsupwiththat.com/2009/06/13/results-lab-experiment-regarding-co2-snow-in-antarctica-at-113%C2%B0f-80-5%C2%B0c-not-possible/
And when I pointed out his recent stubborness over the USHCN issues was just like that…he posts this:
http://stevengoddard.wordpress.com/2014/06/03/antarctica-gets-cold-enough-to-freeze-co2/
He’s hopelessly stubborn, worse than Mann at being able to admit mistakes IMHO.
So, I’m off on vacation for a couple of weeks starting today, posting at WUWT will be light. Maybe I’ll pick up this story again when I return.
Engineers who work with data realize:
Empirical measurements trump synthetic data (infilled data, interpolated data, extrapolated data, etc.) Synthetic data is only used as a last resort. You always use empirical measurements when you have them. The only reason anyone uses synthetic data rather than empirical data is to cheat.
Empirical measurements are sacrosanct. You don’t adjust them or tweak them; that’s cheating. Adjusted data is synthetic data.
In regards to this, Anthony Watts asked Zeke these simple questions:
What is the CONUS average temperature for July 1936 today?
What was it a year ago?
What was it ten years ago? Twenty years ago?
What was it in late 1936, when all the data had been first compiled?
We already know the answers to questions 1 and 2…, they are 76.43°F and 77.4°F respectively, so Zeke really only needs to answer questions 3 and 4.
Zeke?
Taking all you say at face value, I observe your “Raw Gridded Anomolies and Absolutes” chart, and think about the Warmists’ claims I keep reading. That last year was one of the warmest on record; that there was no real cooling ’40s thru ’70s, while CO2 was rising; that the warming rate ’70s thru the present is unprecedented; that there really is no pause in warming since mid ’90s; that most of the warmest years on record occurred since 1990. Your chart does not seem to support those claims.
I have also been reading a lot of the popular press articles Goddard finds, about events in the late nineteenth and early twentieth century, and I find that Arctic melting and glacier loss was far greater back then; there were more major hurricanes, more and worse droughts and fires; more and worse summer heat and winter cold; and much more.
So I have to ask, why do you spend so much time focusing on Goddard, when there seems to be little to support the notion of catastrophic warming and extreme weather even in your own facts, and I don’t see any support for the idea that any of that will stop if we only stop burning fossil fuels.
On this temperature stuff, having looked at some real records and paper originals and a discontinued reporting site, I am profoundly happy that I am not responsible for making good science out of the mess. Further, most people I talk to do not know about weather record methodology nor would they care. They do care about weather. Therefore, consider . . .
On his Real Science blog, “Steven Goddard†(SG) has several interesting headings at the top, including Bad Weather. The items therein represent many hours of searching and reading and often he made a separate post of these. One of my complaints of the CAGW crowd is their apparent lack of historical awareness. On WUWT the comments of “tonyb†[an actual historian] provide very detailed discussions of historical climate. There, likewise, “Jimbo†provides lists of documents reporting on past weather and climate (and a lot more). These folks make a valuable contribution, including SG.
When high profile politicians stand before TV cameras and make statements about current weather being unprecedented it is good to have documentation of the truth.
Re: Louis Hooffstetter (Jun 6 08:09),
Say you were measuring the boiling point of water daily and you found that it varied. You wouldn’t adjust your data to compensate for barometric pressure because that would be cheating. Right.
“You wouldn’t adjust your data to compensate”
DeWitt: Non rhetorical question:
Why would you change your data to reflect a measurement that didn’t happen? In other words, you are preferring a made up number to one that is produced by a measuring device.
Andrew
“I see Tom Karl leaning over someone’s desk two or three times per month saying “What happens if we change this X? Oh, well that’s no good. Let’s change that Y. Oh darn, that’s no good either. Well, we have to drop in that new list of stations that Tom Pederson came up with.—
Can I assume this is hyperbole? While I fully agree summarized data can be slanted be selection and summary technique, I hardly think the process would play out in full view of the entire staff.
P.S. Also, justifying adjusting numbers in case A doesn’t necessarily justify adjusting numbers in case B.
Andrew
DeWittPayne,
You might want to reconsider what you wrote after reflecting on what Louis said. Empirical measures are what the properly adjusted device measures, variances and all.
It would be those depending on models over empirical measures who would steadfastly refuse to record the empirical data results.
“I see Tom Karl leaning over someone’s desk two or three times per month saying “What happens if we change this X? Oh, well that’s no good. Let’s change that Y. Oh darn, that’s no good either. Well, we have to drop in that new list of stations that Tom Pederson came up with.â€
This is the kind of conspiratorial ideation that skeptics need to avoid to be taken seriously.
Steve,
Your own book documents that at least some of the climate catstrophe guys are not above playing fast and loose.
I propose that we “freeze” the historical temperature record prior to 2011.
No more adjusting. There is no reasonable explanation that 1901 temperatures need to be changed every 14 days.
Bill:
That would create real problems for the catastrophists…
Re: hunter (Jun 6 11:13),
TOB, for example, is a proper adjustment, just like barometric pressure is a proper adjustment for boiling point. All empirical measurements are based on models of one sort or another, temperature especially.
Hmmmmmm.
One has a difficult time detecting a consensus of skeptics.
You would think that the folks dishing out the big oil money could herd these kittens more efficiently!
angech (Comment #129995)
“Nick Stokes. Absolute temperatures matter a lot. when you get a reading in Celsius it is what it is.”
Really? Can you tell me the absolute temperature of your state last year? The US? Last month? Does it really matter to you?
>sigh<. For most people I know, the 'empirical measure' of something is whatever an accurate sensor measures. I see a whole lotta word parsing going on, which makes things so tedious. An empirical measure of the bp of water would show the fluctuations due to baromtric pressure. it would the application of the gas law to put the variations into a meaningful context. What skeptics are noticing is that years after the measurements were made changes are done post hoc that have the coincidental result of supportiing the claims of people who have shown themselves to all too often be less than reputable.
As to Nick's comment: Obtuseness fits him like a glove.
Louis Hooffstetter (Comment #130004)
“Empirical measurements are sacrosanct. You don’t adjust them or tweak them; that’s cheating. Adjusted data is synthetic data.”
The adjustments that you are complaining about are actually not to empirical measurements. There is no instrument that spits out a monthly average. They are adjustments to calculated results.
This is a significant distinction with TOBS, say. No-one is saying that the instrument is wrong. They are saying that the naive calculation of monthly average doesn’t give what is required for the subsequent calculation of a state or national index.
If you are an engineer designing a jetty, you need to know the sea level. You can go out with some IR device and get a reading to the nearest mm. But of course, that is not the information you need. There is a lot more calculation to do. Your reading is not sacrosanct.
“TOB, for example, is a proper adjustment”
Looks like DeWitt made a declaration.
Let’s see if he has any evidence to back up his claim.
Andrew
Nick,
The problem is that the climate obsession industry is deriving a product whose only use is to support the climate obsession industry. And we have already seen plenty of well documented noble cause (at best) corruption and rent seeking. The product seems to run counter to historical records, since it makes past heat waves, Arctic ice fluctuations, MWP, etc. disappear into the memory hole. As to your absolute temp gambit: growing seasong are made with temperatures, not daily anomalies. Pointing out that a place that has been typically -25.0o is now, based on some derived infilled average, -25.75o as if that is a world crisis only emphasizes the hysterical nature of climate obsession.
Re: Andrew_KY (Jun 6 15:13),
Sorry, not going to hold your hand on that one. The literature is out there, do your own research.
TOB, for example, is a proper adjustment, just like barometric pressure is a proper adjustment for boiling point. All empirical measurements are based on models of one sort or another, temperature especially.
Sure TOB is an adjustment needed when using a measurement for climatology. The statement still stands that you do not change the measurement. You use the same recorded measurement and fix your TOB calculation. I’m on the old school engineers side on this. Anomalies cannot be translated back to temperatures. You cannot determine the average temperature in 1936 using anomalies.
All empirical measurements are based on models
This is a pointless statement. Math itself is a model, so what?
Th USHCN corrections are not credible
http://stevengoddard.wordpress.com/2014/06/06/ushcn-temperature-adjustments-are-not-credible/
Andrew_KY (Comment #130022)
‘“TOB, for example, is a proper adjustmentâ€
Looks like DeWitt made a declaration.
Let’s see if he has any evidence to back up his claim.’
Here is my offerring.
Steve Goddard,
You are still doing it wrong.
.
Infilling has no real effect on temperatures: http://rankexploits.com/musings/wp-content/uploads/2014/06/USHCN-infilled-noninfilled.png
.
The new USCRN network doesn’t show any of this massive 0.4 F warming bias you seem to find after 2004: http://rankexploits.com/musings/wp-content/uploads/2014/06/Screen-Shot-2014-06-05-at-1.25.23-PM.png
.
Satellite records over the U.S. agree well with homogenized USHCN temperature data: http://rankexploits.com/musings/wp-content/uploads/2013/01/uah-lt-versus-ushcn-copy.png
.
There is a reason why folks like Anthony who agree with you on a lot of the issues don’t support your work. You can’t make up your own temperature reconstruction method that is prone to bias, and point to those biases as evidence of nefarious adjustments. Revise your code to use anomalies and some sort of spatial weighting if you want your results to be taken seriously. There is a reason why the RSS record you cite reports in anomalies and not absolutes…
Re: Andrew Krause (Jun 6 15:26),
Of course they can. You can’t calculate an anomaly without knowing an average to subtract from the actual temperature. The actual temperature is then the average plus the anomaly. This is really basic. Of course you can’t translate a gridded average anomaly back to an individual station temperature for a particular day, but you always lose information when you average over space and time.
Louis Hooffstetter,
Actually, averaging absolute temperature readings from stations to figure out the “true” U.S. absolute temperature is a pretty poor approach. Stations tend to be biased toward lower elevation (there aren’t too many on top of the Rocky Mountains, for example).
A better approach would be to use a climatology field that uses elevation and other factors to calculate more accurate long-term climatologies, and add in spatially-interpolated anomalies from stations. Anomalies have a convenient property of being strongly spatially correlated; absolute temperatures not so much.
Here is an accurate comparison of USHCN Raw and Adjusted vs. UAH and RSS over the lower 48: http://i81.photobucket.com/albums/j237/hausfath/USHCNrawhomogenizedRSSUAH_zps0b5d1c5c.png
Trends are:
USHCN Homogenized – 0.26 C per decade
UAH – 0.23 C per decade
RSS – 0.17 C per decade
USHCN Raw – 0.16 C per decade
UAH agrees pretty well with Homogenized data; RSS with raw data. I’m not sure we can conclude anything either way, except the uncertainty in satellite measurements is of the same magnitude as uncertainty in surface measurements over this period.
Oh, and before anyone complains about it, tropospheric amplification over land is actually right around one: http://climateaudit.org/2011/11/07/un-muddying-the-waters/
Screencap image of the changes in GISS global temperatures (which comes from the NCDC GHCN-v3 and ERSST) made on May 15, 2014. Yellow and strike-out are monthly temperatures which have changed from the previous version on April 12, 2014.
Hard to see but “ChangeDetection” does not allow a direct link. This just shows all the changes.
http://s29.postimg.org/em4ea0xtj/GISS_Glb_Changes_May_15_2014.png
This is crazy. About one-third of the historical temperature record is changed by more than 0.01C once or twice per month. The older records are down of course. They are even changing 1998 which they used to leave alone.
There is no logical reason why February 1883 needs to change to -0.37C from -0.36C as of May 15 2014. They already lowered it 34 times previously so all the “TOBs” changes should have been made already.
There needs to be a new law “freezing” the old temperature record so that we can know what is actually happening. Write your Congressman.
I would suppose that Steve Goddard does not realize that by pushing his flawed calculations that he takes away from the discussion of the more basic unanswered questions concerning temperature adjustment algorithms used by groups such NCDC (USHCN and GHCN). I get somewhat frustrated by the drift in blog exchanges which appear to me to be getting away from these basic questions and issues.
Concerning the temperature adjustments using breakpoint methodologies, I would pose the question again about constructing benchmarking simulations with non climate station changes introduced that we either know or think the adjustment algorithms might have problems handling. Setting up proper simulations requires knowing all the non climate factors that might affect temperatures. While we are all familiar with the more obvious ones and the ones that are more readily handled by breakpoint algorithms and other methods, the question arises whether there can be changes that algorithms/methods will have difficulty finding and providing proper adjustments. If we find such holes in the algorithms/methods we could at least conjecture whether those conditions could exist in the real world of temperature stations.
I have worked with various conditions of non climate affects on simulated station temperature series where station differences are used with breakpoint methods. Those conditions that are difficult to impossible to find are where the non climate effect(s) on a given station are causing the temperature to change slowly over extended periods of time. While the temperature changes are incrementally small, over extended periods the change can be significant.
Part of the problem in finding these slowly developing changes on temperature is that, while near neighbor stations can have excellent correlations in their temperature series, the trends produced by these stations can be significantly different. My analyses indicate that most of the trend variations among near neighbor stations can be accounted for by making some reasonable assumptions about the red and white noise using an ARMA model. Some of the difference (when using adjusted temperature data) could be attributed to unresolved non climate changes. It is not these unresolved changes that are the major unknown here but rather that the red/white noise makes finding non climate trends difficult to impossible.
“There is no logical reason why February 1883 needs to change to -0.37C from -0.36C as of May 15 2014. They already lowered it 34 times previously so all the “TOBs†changes should have been made already. ”
Zeke? Steve? Anybody?
George Orwell
“He who controls the past controls the future. He who controls the present controls the past.â€
I became a skeptic when the medieval warming period got “disappeared”.
Bill Illis (Comment #130032)
“There needs to be a new law “freezing†the old temperature record so that we can know what is actually happening. Write your Congressman.”
RobertInAz (Comment #130035)
“He who controls the past controls the future. He who controls the present controls the past.”
You can save the stamp, Bill. No-one is controlling the past. NCDC provides GHCN unadjusted daily and monthly here. They don’t change. You can use them to calculate your own index if you want.
“Anomalies cannot be translated back to temperatures. You cannot determine the average temperature in 1936 using anomalies.”
Huh.
somebody needs to do a simple example before he posts again.
Let me make it simple.
You start out with absolute. Lets just make up some january data
1936 jan 1C
To calculate an anomaly you pick a refernece period.
say 1951-80.
you sum up the 30 jans in those years. Say the average is 2C
2C is thus the “normal” or average for the reference or “base” period.
now you subtract 2C from every jan.
Jan 1936 becomes ……… 1C-2C or -1C
waala!! magic..
Higher math I know.
So jan 1936 is -1C anomaly..
anomaly from what?
from the 1950-1981 jan average of 2C
How do we get back the temp in 1936
oh good god its a differential equation.. no wait its addition
-1 + 2 = 1
“http://s29.postimg.org/em4ea0x…..5_2014.png
This is crazy. About one-third of the historical temperature record is changed by more than 0.01C once or twice per month. The older records are down of course. They are even changing 1998 which they used to leave alone.”
You and goddard have never taken the time to study the RSM method. When you do you will see why it constantly revises the past.
Its one of the artifacts of the method. It results from constantly re assessing the Reference station as more and more data comes in.
That is one of the reasons for not using a reference station.
See steve mcintyres posts on RSM..
once we got to the bottom of RSM then RomanM and Tamino saw how you could get rid of the concept all together.
In short, every time you add data you are re figuring what station should be the reference. Then other stations are stitched to that station to create a long record.. then anomalies can be taken.
But from month to month there is not ensured stability in which station will be selected by the algorithm as the reference.
As Mcintyre showed the order in which stations are combined has an effect. Change the order and the answers wiggle around.
Solution. DONT USE RSM
Why? it has this weird effect of looking like you are rewriting the past.
Steven Mosher – there is a perception that the past is always being revised down which would reflect some kind of bias. If it was just due to the reference changing then it should yo-yo up and down. Is that just a perception that it always goes down or a valid observation?
http://climateaudit.org/2007/08/08/a-new-leaderboard-at-the-us-open/
I find it quite ridiculous that people who want to teach us about the climate do not know what an “absolute temperature” is. I was going to suggest that they call it “Celsius temperature” instead.
But then a red-blodded American also refuses to know what a °C (which Americans write “deg C” or just “C”) is.
I guess that climatology can be defined as climate physics without the physics.
Steven Mosher (Comment #130038)
June 6th, 2014 at 9:24 pm
You and goddard have never taken the time to study the RSM method. Its one of the artifacts of the method. It results from constantly re assessing the Reference station as more and more data comes in.
—————————–
Let’s examine the math on how the Reference station method could change February 1883 to -0.37C from -0.36C.
Option 1: They found 500 new station records last month for February 1883 that have an average anomaly of -0.38C instead of the average of the other 500 stations at -0.36C. Now 1000 stations are at -0.37C.
Option 2: They found 5000 new stations for the reference period of 1981-2010, which were 0.02C warmer than the existing 5000 stations which dropped the anomaly in February 1883.
Option 3: not math but messing around.
Sorry, your reference explanation is simply a “diversion” instead.
“you will see why it constantly revises the past.”
“it has this weird effect of looking like you are rewriting the past.”
I love Mosher’s word games in the morning.
Andrew
Steven Mosher (Comment #130037)
Okay I am wrong, I get your point. I should have stated ..
You cannot get an actual temperature from a gridded, infilled anomaly. Is that an accurate statement?
I also had a humorous moment when I thought, what if you found out your starting temperature was subsequently adjusted.
It would be interesting to review the editing of the past records with the weather histories of the areas involved.
My take on this is that what we are really seeing is more climate obsessed hand wringing about what is actually nothing of any significance at all.
“Mountains out of molehills” comes to mind.
IOW: Whatever derivative products are developed, the climate is still not generating a crisis.
Re: Alexej Buergin (Jun 7 05:04),
I sympathize. It would be better to use the term measured temperature or actual temperature in C or F rather than the absolute temperature. Absolute temperature is already defined as degrees Kelvin.
I have given more thought to this matter. Back at grad school I had occasion to run ag trials using statistical designs, latin squares and fractional factorials and the like. Being agriculture, we always had missing data that had to be made up in order for the designs to be analyzed. There are statistical procedures for doing so. The key constraints are that the made up data may not, not, not change the statistical parameters of the population or subpopulations; second, the more data is made up, the less confidence one has in the results. Made up data cannot add information to a design; only the observed data contributes information.
First off, accepting Goddard’s assertion that 30% of the NOAA data is “constructed” statistically, that is way beyond any number that is statistically defensible. That fact degrades our confidence in any conclusions made.
More importantly, Lucia’s post implies that the “constructed” data is taken from the actual observations of nearby stations, and thus is likely to match what the missing station would have produced. I surmise Lucia offers this as a defense of the process and a defense of using the whole resulting dataset to reach conclusions. I am cynical of the motives of the people running the process, especially after Pelosi and Obama have announced their belief that the agencies have been purged of all “flat earthers”.
But the claim can be tested. Each “constructed” datapoint can be compared to the observations of the stations most adjacent, forming a subset. If the process has been statistically legitimate, the values firstly should match closely. The constructed datapoint must not materially change the mean or the standard deviation of the subset. Secondly, the deviations of the constructed datapoint from the nearby observations, besides being small, should add up to zero for the entire dataset. Constructed deviations above or below its neighbors should be equal in a set of 21,000 observations per month, as Goddard calculates. Goddard claims they do not add up to zero, but introduce a huge and growing bias. Lucia claims (I deduce) that the deviations will add up to zero because the process is legitimate.
This is eminently testable. Have at it, boys.
These remarks aren’t about USHCN although I suspect they may also broadly apply to that dataset.
If you think no one is manipulating the data you’re too trusting. Most of the people publishing these datasets are true acolytes of AGW.
Let me tell you the story of Isla De Pascua(Easter Island).
It is a lone island with absolutely no other stations within 1200 km. Probably close to twice that distance. Up to 2011 Ghcn made six raw datasets available to Giss in v2. Giss combined them into one dataset as was normal procedure at that time. That dataset became the homogenized dataset. No changes were made since the pop. was under 10,000 and lights=0. The trend-line for that period of record (1942-2011) was a negative -0.25C.
Now Ghcn combines the datasets into one, and then adjusts it before Giss gets it. The 2014 Ghcn adjusted data for that station has an average of about 0.61C subtracted from the raw data over the entire period 1942-1985. Then magically over the course of the next five years the adjustments are reduced to zero, and from 1991 to 2011 the average difference is about 0.02C. The trend-line 1942-2011 for this adjusted dataset is a marvelous positive 0.81C. With no other stations for comparison I don’t see how they justify reducing the temperatures in the first 40+ years of the record.
Giss is now using a second dataset for this station. Appears to be SCAR data. It runs from 1942-1996 with scattered values after 1996 up to 2006. It has a negative trend-line even greater than Giss v2.
Further information:
Giss is now using two datasets for about 50 stations confined to the S. Hemisphere and a few of those stations actually have three datasets. Back in 2011 Giss combined all the raw records and then created one homogenized record per station. I don’t remember ever hearing about it when they started using multiple homogenized records from a single station. They have been doing it for at least a year. I think using multiple datasets for one location results in undue and erroneous weight being given to that location when combining with other stations with one record within 1200 km of any particular sub-box. The 2011 Giss trend indicates such a problem. Mmmm, cherries!
Here is a graphic of the Isla De Pascua which includes both current datasets along with the single combined dataset Giss used in 2011.
http://i59.tinypic.com/2hpimpf.gif
Another suspect case is Hawaii. Two station were still being used by Giss until February 2013. They are Honolulu and Lihue. No more recent data has been provided in the Ghcn adjusted file since then. I was worried I might have missed the news of the big volcanic eruption destroying all the weather stations there. But then I discovered the data is up to date in the raw dataset. Without that data the entire state has been left blank for more than a year. What can be the justification for not putting that data in the adjusted file?
Not that I’ve become suspicious or cynical, but I noticed for DJF of the 1998 El Nino year the Hawaii land temperature anomaly was about 1.4c lower than the global land anomaly. For DJF of the 2010 El Nino year the land anomaly was about 1.2C lower than the global land anomaly. If they are lucky enough to keep that 4.5 million km2 area blank through the end of the year, who knows? Might be a chance to set a new record if a big El Nino develops.
Here’s a land data map of Hawaii DJF 1998. For those who are slow, it’s the isolated light blue area in the upper left-hand quadrant which has now been blank for more than a year.
http://data.giss.nasa.gov/cgi-bin/gistemp/nmaps.cgi?sat=4&sst=0&type=anoms&mean_gen=1203&year1=1998&year2=1998&base1=1951&base2=1980&radius=1200&pol=reg
Nick Stokes re the absolute temp and can I tell and do I care?
Well yes I can tell and I do care. I listen to the news most nights and get a weather report which gives a minimum and a maximum for the next day. This can be very important as to what I choose to wear and where I might consider going in that day. Try doing that from an anomaly. As for the absolute temp in ones state. You seem to have left out a few concepts such as time of day , elevation, location in the state etc .Where I live is an average of 8.19 at this time of year but if I lived at Mildura it would be 13.15 ,On top of Mt Hotham it would be an average of 3.67 and in Melbourne it would be 6.7.
The US last month seat of pants calculation taking in Alaska and Hawaii, and those funny bits like Guantanamo, Solomon Islands lots of Summer say 17.3 degrees but correct me if you wish. Nearly froze on the Grand canyon rim in the three mile island year ? 1979 April? Gee it was cold that year.
You really are hung up on not having absolute temps, aren’t you.Because then you have to face the fact that adjustments are being made when perhaps they shouldn’t be and, worst of all , it isn’t really much of a bother half a degree warmer or colder in absolute terms over a hundred years , is it.
Steven Mosher, much and all as the RSM is constantly readjusting temperatures based on the addition of new data surely if Texas for example, had a temperature, real, at the one and only location at the very start of your record ,say in 1878, or 1910, or whenever and that temp was say 6 degrees. Then it was 6 degrees. You cannot say oh no our data shows that it should have been 2.5 degrees, the next week 2.7 etc. It was measured at 6 degrees. The temperature data cannot be changed to suit your model. It was, is and will be 6 degrees there no matter how much your mathematically ept model massages the data.
You can say it should have been 2.7 or our model won’t work but it still is 6 degrees. Particularly and especially at the start.
I don’t mind about the present because if you tell me it should be 9 degrees in Texas at that spot today based on your model and I lived there I and I could see on the unadjusted weather reports that the average was 6 I would know that your model was just modelling. At least I presume the temps we get given are real ranges?!!! Don’t tell me they are adjusted as well.
PS no comment from Zeke, Nick etc on how many original real stations are left from the 1218 started with, how many are models only (infilled in other words) and what percentage of original models will they reduce to in order to keep the global warming anomalies going up as the real data refuses to cooperate?
DeWitt Payne (Comment #130049)
If you want to be really, really correct: It actually is °C and °F, but only K (since 1967).
Because in 1967 the ° from what was then °K was abolished; so we cannot say “a ° is a ° but point 0 is different for Celsius and Kelvin” anymore.
angech (Comment #130054)
Did anybody ever got an explanation from the BEST minds in climatology for the adjustments to Reykjavik in 1940?
http://stevengoddard.wordpress.com/2012/05/10/hansen-cheating-in-iceland/
geezer117,
Good suggestion, I’ve updated the post with an analysis that compares infilled-only stations to non-infilled stations in their same 2.5×3.5 lat/lon gridcell. This provides a rough station pairing approach, without requiring me to spend a few hours writing a proper pairing code. Unsurprisingly, infilling has no real effect. The first graph in my post also shows this rather well, comparing a US-wide reconstruction with and without infilling and showing that there is no difference.
Mosh,
Actually, your explanation of adjusting distant past temperatures as a result of using reference stations is not correct. NCDC uses a common anomaly method, not RFM.
The reason why station values in the distant past end up getting adjusted is due to a choice by NCDC to assume that current values are the “true” values. Each month, as new station data come in, NCDC runs their pairwise homogenization algorithm which looks for non-climatic breakpoints by comparing each station to its surrounding stations. When these breakpoints are detected, they are removed. If a small step change is detected in a 100-year station record in the year 2006, for example, removing that step change will move all the values for that station prior to 2006 up or down by the amount of the breakpoint removed. As long as new data leads to new breakpoint detection, the past station temperatures will be raised or lowered by the size of the breakpoint.
An alternative approach would be to assume that the initial temperature reported by a station when it joins the network is “true”, and remove breakpoints relative to the start of the network rather than the end. It would have no effect at all on the trends over the period, of course, but it would lead to less complaining about distant past temperatures changing at the expense of more present temperatures changing.
.
angech,
As I mentioned in the original post, about 300 of the 1218 stations originally assigned to the USHCN in the late 1980s have closed, mostly due to volunteer observers dying or otherwise stopping reporting. No stations have been added to the network to make up for this loss, so there are closer to 900 stations reporting on the monthly basis today.
.
To folks in general,
If you don’t like infilling, don’t use infilled values and create a temperature record only from the 900 stations that are still reporting, or from all the non-infilled stations in each month. As the first graph in the post shows, infilling has no effect on CONUS-average temperatures.
I would suggest that this discussion would proceed better if those who are presenting what they think are adjustment problems would put those questions (and answers) within the context of the methodology used by GHCN to make temperature adjustments. The people at GHCN have attempted to put the adjustments on a more objective footing by using the breakpoint methodology on paired station series differences. Some adjustments can be made based on meta data concerning non climate changes to the stations whereby it is my understanding that the breakpoint method is applied but with reduced significance requirements. An appropriate question than might be whether the adjustment in question was the result of an objectively found breakpoint or one prompted by the more subjective meta data. I have asked the GHCN people to provide that information, but the last time I looked it was not available. That information would also help me in my analyses of the adjustment methods.
I would be most interested in seeing a post by Zeke or other investigators of temperature adjustments go over the elements that are required or should be required in putting together a simulated temperature series with known non climate change effects on temperature that could best test the performance of the available adjustment algorithms. In addition the investigator could also show and explain any recent benchmarking test results. I would like also to see a discussion of the needs for benchmarking test that might clearly show the limitations of the adjustment methods and whether those limiting conditions could exist in the real station world.
I have little interest in motivation of investigators, but any that I have would go more to what is not said or done than what is.
Kenneth,
We have a paper in open review at the moment that touches on precisely that question: http://www.geosci-instrum-method-data-syst-discuss.net/4/235/2014/gid-4-235-2014.html
I’d encourage you to submit a peer review if interested.
Zeke – still not clear about your statement of the methodology not effecting trends.
I am sure that there was major battle back in 2007 about the relative position of 1998 behind 1934 as the hottest year in the USA temperature records. I linked above to a post by Steve McIntyre which defines the relative positions of various years.
My understanding is that GISS which feeds from the NOAA dataset now has 1934 well below 1998. Any explanation as to whether this is down to a more recent adjustment or is data or process driven?
clivere,
Many reasonably well educated people find it non-credible that so many allegedly neutral process adjustments all result in a derived product that reduces the well documented temps of the past. The flip side of this for me is what if it is true? That somehow the past was systematically over reported on temps, and out heroic climatologists of today just happen to have the analytical cure. And if we are in *reality* dealing with high temps, then were is the beef? If we are so much warmer, and still not facing the promised climate apocalypse, why not? But the evidence temp of tampering is strong in NZ and Australia. Not just the US. So as much as I respect what Zeke and Steve are saying, I don’t think we are seeing the complete analysis yet.
Bob Koss
“Let me tell you the story of Isla De Pascua”
I’m not sure what the point is. As I understand from your plots, in 2014 an adjustment was made that changed the trend, then later it was set back to where it had been. That at least runs counter to the claim that adjustments always cool the past.
I think I can see what has happened. In 1977 there was a sharp dip, and a subsequent rise. The change detector thought the dip was illegitimate, but passed the subsequent rise. That can happen; change detection is never perfect, it’s used when it does more good than harm. Anyway, this was fixed. That could have been by better algorithm, or just that the new data did not dip so suddenly.
The Hawaii story seems to be a sceptics favorite lately. I can’t see why; Hawaii still has two stations with adjusted data, and it isn’t a very big area. Honolulu and Lihue, for some reason, have recent data gaps that stall the adjustment algorithm, even though they are still reporting.
angech (Comment #130052)
“Nick Stokes re the absolute temp and can I tell and do I care?
Well yes I can tell and I do care. I listen to the news most nights and get a weather report which gives a minimum and a maximum for the next day. This can be very important as to what I choose to wear and where I might consider going in that day. Try doing that from an anomaly.”
That’s for your town. In fact, no-one will try to tell you an anomaly for that. Anomalies are used for combining data to produce a regional average, usually over a month or more. Not umbrella information.
If you live in Moyhu, say, the BoM won’t tell you a monthly average, and it doesn’t help much to know that Victoria had an average of 20°C. But if you know that Vic had a month anomaly of +3°C, that does tell you it was probably warm in Moyhu.
Zeke, thanks for the paper link. I’ll have a look at it to see whether it addresses the issues that are the most interest to me. It would be good to have you to do another thread on that paper.
How do you calculate an “anomaly” without having the real absolute temperature?
Obviously, one starts with the actual temperature first, monthly average temperatures etc.
There is so much “diversion” explanations from Zeke and Mosher in this thread that an objective person should understand what is going on here.
They appear to have taken on the role of NCDC defenders/apologists because the believers need to continue to defend the faith.
There is nothing wrong with Goddard using absolute temperatures because that is how an anomaly is calculated in the first place. I’m tired of seeing this type of obfuscation.
No, Bill. It’s not obscuration, it’s math. Goddard is provably wrong here.
If you aren’t following the explanations, I suggest you don’t immediately assume the problem is with the people trying to explain it to you.
I think in this case, it’s you not making any effort to learn something new.
Carrick (Comment #130067)
June 7th, 2014 at 7:08 pm
—————————
I’m a math geek with a degree in math.
I understand what is going on here.
Bill, supposing you did understand what’s going on, I don’t think that would result in you defending Goddard’s work.
The mathematical explanation of the flaw is very clearly written out here:
http://moyhu.blogspot.com/2014/05/ushcn-adjustments-averages-anomalies.html
If you think there are flaws in Nick Stokes’ arguments, please feel free to share those criticisms. It looks pretty bullet-proof to me.
Bill,
Here’s the essence of it. Any time series T of observations can be expressed as the sum of two parts – the expected values T0 and the anomalies T1. T0 might just be a longterm average. T1, the difference from expected, is the main information content.
If you want to turn the data into a regional average, and make inferences from the pattern, you’d like that to be based on the information content. Working with anomalies ensures that.
Otherwise the average can be expressed as the sum of a part dependent on the station T1’s and a part dependent on the T0’s. If the latter contributes a pattern, it is spurious. It is not derived from the information content of your data. Instead, in SG style averages, it comes from month to month changes in the selection of T0 series.
You can substitute those expected values (constant long term station averages T0) into a SG method and derive something quite like the results he displays.
Steve Goddard, Mosher, Zeke and Tamino are like economists. No matter how many you put end to end they still won’t reach agreement.
That is why I defer to cooler heads such as DeWitt Payne and Leonard Weinstein.
I prefer to access the raw data and make up my own mind so I downloaded the GHCN v2 data to prepare these posts:
http://diggingintheclay.wordpress.com/2010/12/28/dorothy-behind-the-curtain-part-1/
http://diggingintheclay.wordpress.com/2010/12/30/dorothy-behind-the-curtain-part-2/
I can’t find the GHCN v2 on line any more but when I compare it to v3 the warming slope is obvious even to an old fogey who learned his statistics back in 1959 from J.C.P.Miller.
http://en.wikipedia.org/wiki/J._C._P._Miller
In “Climate Science” you can’t rely on the past remaining unchanged as Richard Lindzen pointed out to the UK Parliament in February 2012.
gallopingcamel (Comment #130071)
“In “Climate Science†you can’t rely on the past remaining unchanged as Richard Lindzen pointed out to the UK Parliament in February 2012.”
Yes, he did. This was his evidence.
You can’t rely on the past if you lift stuff from someone who has looked up the wrong thing.
Hmmmm…..people with math degrees and professors of climatology are not clever enough to see the strong maths in this. And still no reasonable explanation as to why the adjustments are consistently reducing the past temps and increasing the present.
And still no perception that even if the adjustments are correct, what these derived trends are showing is still that nothing dangerous or unusual is happening in the climate.
A bit tangential, but for years Dr. Pielke Sr. was berated for pointing out that surface temps were not really that important. The opinion leaders, when surface temps were still perceived as supporting the apocalyptic consensus, dismissed him. And when he pointed out that carbon black could be a major influence, we had the CO2 control knob as a way of rejecting his argument.
Now Trenberth is chasing the missing heat into the depths of the ocean and carbon black is emerging as a huge influence. And no apology to Dr. Pielke, Sr. I wonder if the anomaly trends are going to turn out the same way much ado about the wrong things. It seems that no matter how dramatic you can dress up the anomalies, the reality is that not much is happening.
Carrack, your link to Moyhu showed Nick Stokes attempting to discredit SG with 6 diagrams talking about a spike in 2014 but all 6 graphs only went to 2000 why the heck is that.
Zeke has a post at SG where he admits that there are only 650 real stations out of 1218 . This is a lot less than only 918 that he alludes to above. Why would he say 650 to SG ( May 12th 3.00 pm) and instead #130058 at the Blackboard about 300 of the 1218 stations have closed down.
Can Zeke give clarity on the number of real stations (raw data) and the number of unreal stations using filled in Data in the 1218 stations.
Well, this is, as they say, something that everyone who deals with these data data sets knows, but it being so basic, it just about never is discussed. Back almost ten years ago McKitrick and Essex published a paper full of Goddard class misleadings. Amongst them was the truism that in a thermodynamic sense the earth does not have a single temperature, and if you want to really quibble, and they did, no point does either because no part of the system is at a true thermodynamic equilibrium.
Having freed temperature of its physical constraints they then went on to compute monthly averages of various powers of absolute (K) temperatures from a few selected stations across latitudes and discovered the annual cycle with a large “standard deviation” for any trends thus, in their view negating the claim that there has been any trend.
Some comments on this in early Rabett, the basic take home of which is, of course, what Zeke and Nick are pointing out, that anomalies allow comparisons of temperature trends at different locations across time and location. Absolute temperatures are worthless for that purpose.
Nick Stokes (Comment #130063)
Wrong interpretation, Nick. You mis-understand. Ghcn has not changed their adjusted data back from their marvelous 0.81 positive trend. They still use it as if it were correct in their current database.
Perhaps I should have been more explicit when I said ‘Giss is now using a second dataset for this station.’ That dataset is not a replacement for Ghcn. Giss has instead added a second dataset which I labeled (2014b), probably from SCAR, which has a large negative trend which is similar to the Giss 2011 combined dataset, and are using (2014b) in addition to Ghcn dataset(2014a). Giss is using both of those datasets. It makes no logical or physical sense for Giss to be using two such disparate datasets for the same location simultaneously. Neither does make sense to ignore the close alignment Ghcn raw has with those two down-trending datasets(Giss2011, 2014b). The Ghcn data has been extensively adjusted, and when it meet expectations no one bothered to check any further. Not Ghcn, nor Giss. Shoddy work.
As for Hawaii. Those two stations with current data, Hilo and Kahului, have not been used by Giss for a few years. The empty ocean attests to that. In the raw Ghcn, Honolulu and Lihue are up to date. That data all carries the designation ‘C’. Here I quote Ghcn. “C = Monthly Climatic Data of the World (MCDW) QC completed but value is not yet published”. Ghcn adjusted has published thousands of data points with that designation during the whole year 2013, I can’t think of a legitimate reason for a US entity to be more than a year behind reporting state data which is readily available in one of their own databases and leaving it unavailable for use.
angech (Comment #130074)
“6 diagrams talking about a spike in 2014 but all 6 graphs only went to 2000 why the heck is that.”
You’re not very good at reading graphs. The x axis is marked (by R) in years multiple of 20. The data shown is up to date.
“Zeke has a post at SG where he admits that there are only 650 real stations out of 1218 . This is a lot less than only 918 that he alludes to above.”
When I last looked a few weeks ago, in 2014 numbers reporting were Jan 891, Feb 883, Mar 883, and 645 for April. Many are staffed by volunteers and some reports are late. So 918 sounds right.
Nick, I cannot understand your post It seems that you split your data into real and infilled sub groups,
There appear to be a large number of these infilled stations 1218 -650 = 668 according to Zeke at SG and here.
there are claims that the real data is not located in the right areas to be useful for graphing the areas due to differences in latitude and elevation.
The artificial sites at the best locations give a “true grid” for the 1218 “stations”.
One knows what the true readings for these artificial sites “should be” , put them in and the adjust the real sites to what the artificial sites say the temperature should be..Zeke says each month one takes the infilled data from the unreal stations. I guess it ” comes in” from the computer programme primed with a need to rise up as CO goes up otherwise known as Steven’s Climate Sensitivity factor which is being adjust downwards from 3.0 to 1.4 currently due to the satellite pause.
One then has to look for non climatic break points, AKA real data,behaving badly which has to be removed.
Fortunately when you do this the difference between the raw R1 data and the final F1 data is almost eliminated as Nick so elegantly shows. Bravo for the shell trick.
Nick, thank you for your reply with the actual numbers of real stations. I think I understand about the low number in April not being the final number and how this can manufacture a spike if the later readings come in lower.
I do not understand why SG would not wait a month to have all the data in.
I am worried about the graphs on your website. They are in 20 year intervals and you state they are up to date.
Yet there is no 2000 listed at the bottom of the graphs or 2014 or 2020. I trust you are not doing a Tamino on me.
Your article reads as if all the graphs apply to SG and that you are both talking about graphs going up to 2014.
I think it would be the height of hypocrisy to post a bunch of graphs that appear to me to end in 2000 , I defy others to look and disagree with me, and then attack SG with comments that purport to apply to his work in 2014 , with graphs that one would assume to go to 2014 but actually go nowhere near this period.
Perhaps you should re look at your own graphs.
I state again, they only go to 2000.
Bob Koss (Comment #130076)
Bob, sorry I misunderstood your plots. I agree that GISS has two datasets for Easter Island differently adjusted, and that should be resolved. FWIW, my station trend calculator gives -0.31°C for 1942-now for unadjusted GHCN.
People like to focus on cases where adjustment greatly increased the trend. It isn’t all one-way, though there is an upward bias. I’ve looked at the distribution of trend effects here.
Bob Koss (Comment #130076)
Here is the NOAA GHCN story for Easter Island. The adjustment that you complain of is theirs. GISS now uses their adjusted data. They give some details.
Here is Honolulu.
USHCN from December 2002 versus what it is now.
http://s29.postimg.org/473ylf2c7/Conus_USHCN_vs_2002_Version_Apr14.png
That trend calculator is a nifty tool, Nick.
Anthony says: ” I booted him from WUWT some years ago, after he refused to admit that his claim about CO2 freezing on the surface of Antarctica couldn’t be possible due to partial pressure of CO2.”
.
After that ridiculous post by Goddard at WUWT, I stopped reading anything he writes. I even emailed Anthony and suggested he needed some kind of screening to cull that kind of nonsensical post from WUWT. Goddard is utterly uninformed about the basics of science; really, he has not a clue about how things work, and that is what allows him to constantly write the delusional rubbish he does. He is just a Skydragon slayer who happens to write about things other than the impossibility of warming caused by infrared absorbing gases in the atmosphere.
.
Message to Steve Goddard: Please stop helping so much. You are a discredit to all the people who have legitimate and reasoned skepticism about the extent and rate of future warming. Your nonsensical arguments are meaningless, simple to disprove, and discounted by anyone who actually knows a bit of science and math. They are just distractions form the questions that actually should be asked and the arguments that actually should be made.
Bill Illis,
The correct comparison is the 2002 version versus the current version, but with the slope for the current version calculated only through 2002 (apples to apples).
.
But whatever the difference in slope is, the question is if the adjustments are legitimate/correct, not if there have been adjustments; everyone agrees there have been adjustments.
I doubt you will publish this but:
A significant portion of current USHCN Monthly data is “Estimated”
For example: “For just California December 2013, 18 out of 43 are Estimated. The Estimated stations average 8.12C and the “Real†stations average 7.02.”
http://sunshinehours.wordpress.com/2014/06/04/estimated/
http://sunshinehours.wordpress.com/2014/06/05/ushcn-2-5-estimated-data-is-warming-data-arizona/
http://sunshinehours.wordpress.com/2014/06/07/ushcn-2-5-how-much-of-the-data-is-estimated/
SteveF:
Moreover, most people would I think admit that TOBS and station change corrections are necessary.
I think the only real question just whether the adjustments improve the accurate enough to be for the application that the data are being used for.
On that note, it’s important to understand that an adjustment that may be adequate for global mean temperature, may be insufficient for studying regional scale temperature changes.
Brandon and I had a discussion on this thread about “regional smearing” by e.g. the BEST algorithm.
As I pointed out on that thread (and the following is adapted from a comment on that thread), you can use the Climate Explorer to produce gridded average temperature series (which you can then use to compute trends).
http://climexp.knmi.nl/selectfield_obs2.cgi?id=someone@somewhere
Here are the trends (°C/decade) for 1900-2010, for the US SouthEast region (longitude 82.5-100W, latitude 30-35N):
berkeley 0.045
giss (1200km) 0.004
giss (250km) -0.013
hadcrut4 -0.016
ncdc -0.007
Berkeley looks to be a real outlier.
My theory is this is because Berkeley assumes an isotropic kriging function:
That is the interpolating function is assumed to be independent of the orientation, whereas, in practice, the correlation in temperature variations can be strongly zonally (latitudinally) stratified.
angech:
2000 just the last labeled tic mark of the graph (I’ve picked one for illustration, the others follow the same pattern.) The graph line clearly goes well to the right of that.
As Nick states, the graphs in that post up-to-date.
Zeke (Comment #130060)
Zeke, I have read the paper and agree with the importance of what the authors are attempting achieve with benchmarking evaluations and appreciated how comprehensively it was covered. I suppose the details of that benchmarking cannot be reavled at this time since that would defeat the purpose of a blind test. I liked the paper’s intentions of a doing a complete analysis of the how an algorithm performs on benchmark testing. The European benchmarking coauthored by, I believe, Venema did a better job in that respect than the one devised by the GHCN people in evaluating their own product.
I cannot determine whether the process will cover specifically introducing non climate temperature changes that would put the algorithms to the extreme test of finding gradual changing non homogeneities and then discussing whether those conditions could reasonably exist (mainly unnoticed) in the real world.
Zeke, I see that a number of the authors for this paper have direct or at least close interests in various temperature data sets. Who will be providing benchmarking and evaluations and how will the interests of the performance testing be kept separate from those of temperature data sets?
Also it was not clear on my first reading of the paper how the “real” world non climate changes to be added into the homogeneous temperature series will be determined and documented.
My next step will be to view the discussion page.
Carrick,
Yes, most people would agree that station move and TOB adjustments are legitimate. But it seems like some people can’t (or won’t try to) understand those types of adjustments and why they are needed to get a more accurate historical representation. I find it weird there is so much time spent arguing about this sort of thing; it seems just too obvious to be generating the arguments it does.
.
More interesting I think are differences between lower tropospheric warming and ground station warning, which is quite striking: http://woodfortrees.org/plot/rss-land/to:2009/trend/plot/uah-land/to:2009/trend/offset:0.138/plot/best/from:1979/to:2009/trend/offset:-0.45
.
This kind of discrepancy points to boundary layer effects being very important for surface temperature trends (with warring papers on this subject some years back… IIRC, Ben Santer and friends versus the Pielkes and friends). As we once discussed, there are plenty of transmission towers of sufficient height that a reasonable boundary layer temperature profile could be developed by placing temperature sensors at different altitudes on the same towers. That would be interesting data.
.
That regional discrepancy between Berkley and all the others is very interesting. I wonder if the Berkley algorithm could be modified to take latitudinal trends into account.
I posted this on WUWT. It is a companion to my posts in comment 130087.
Using USHCN Monthly v2.5.0.20140509
This is the percentage of monthly records with the E flag (the data has been estimated from neighboring stations) for 2013.
Year / Month / Estimated Records / Non-Estimated / Pct
2013 Jan 162 802 17
2013 Feb 157 807 16
2013 Mar 178 786 18
2013 Apr 186 778 19
2013 May 177 787 18
2013 Jun 194 770 20
2013 Jul 186 778 19
2013 Aug 205 759 21
2013 Sep 208 756 22
2013 Oct 222 742 23
2013 Nov 211 753 22
2013 Dec 218 746 23
Same data by state showing the ones over 35%.
2013 Jun AR 5 9 36
2013 Jul AR 5 9 36
2013 Sep AZ 5 8 38
2013 Oct AZ 5 8 38
2013 Nov AZ 5 8 38
2013 Mar CA 15 28 35
2013 Jun CA 15 28 35
2013 Jul CA 15 28 35
2013 Aug CA 17 26 40
2013 Sep CA 16 27 37
2013 Oct CA 21 22 49
2013 Nov CA 19 24 44
2013 Dec CA 18 25 42
2013 Feb CT 2 2 50
2013 Mar CT 2 2 50
2013 Apr CT 2 2 50
2013 Jun CT 2 2 50
2013 Jul CT 2 2 50
2013 Apr DE 1 1 50
2013 Jan FL 7 13 35
2013 Feb FL 7 13 35
2013 Mar FL 8 12 40
2013 Apr FL 8 12 40
2013 Jul FL 7 13 35
2013 Aug FL 7 13 35
2013 Sep FL 8 12 40
2013 Oct FL 8 12 40
2013 Nov FL 8 12 40
2013 Dec FL 8 12 40
2013 Aug GA 7 11 39
2013 Sep GA 7 11 39
2013 Oct GA 8 10 44
2013 Nov GA 9 9 50
2013 Dec GA 9 9 50
2013 Dec KY 3 5 38
2013 Jun LA 6 9 40
2013 Dec LA 6 9 40
2013 Oct MD 3 4 43
2013 Mar MS 11 18 38
2013 Jun MS 11 18 38
2013 Jul MS 12 17 41
2013 Aug MS 11 18 38
2013 Sep MS 13 16 45
2013 Oct MS 12 17 41
2013 Nov MS 11 18 38
2013 Dec MS 17 12 59
2013 Jan ND 6 11 35
2013 Feb ND 6 11 35
2013 Jun ND 7 10 41
2013 Aug NH 2 3 40
2013 Oct NH 2 3 40
2013 Oct NM 8 13 38
2013 Jul OR 12 21 36
2013 Jun TX 15 24 38
2013 Aug TX 16 23 41
2013 Sep TX 14 25 36
2013 Nov TX 15 24 38
2013 Dec TX 15 24 38
2013 Dec VT 3 4 43
Bruce,
A good chunk of that is likely late reporting (the old co-op system isn’t particularly automated). I’d expect the percent of missing stations to drop down closer 17% after a few more months.
Also, as I’ve said before, the infilling is simply a weighted average of surrounding station anomalies plus the long-term climatology for the station that hasn’t reported. Thats why it has no effect on the trend: http://rankexploits.com/musings/wp-content/uploads/2014/06/USHCN-infilled-noninfilled.png
The numbers are 10-11% from 1895. I doubt it is late reporting.
Jan 14467 123664 10
Feb 14344 123787 10
Mar 14447 123684 10
Apr 14735 123396 11
May 14608 123523 11
Jun 14758 123373 11
Jul 14927 123204 11
Aug 14931 123200 11
Sep 14658 123473 11
Oct 14517 123614 11
Nov 14471 123660 10
Dec 14164 123967 10
Or 2012 is still trickling in …
2012 Jan 95 912 9
2012 Feb 86 921 9
2012 Mar 94 913 9
2012 Apr 115 892 11
2012 May 119 888 12
2012 Jun 125 882 12
2012 Jul 133 874 13
2012 Aug 159 848 16
2012 Sep 171 836 17
2012 Oct 181 826 18
2012 Nov 190 817 19
2012 Dec 203 804 20
1998?
1998 Jan 161 1023 14
1998 Feb 152 1032 13
1998 Mar 140 1044 12
1998 Apr 146 1038 12
1998 May 170 1014 14
1998 Jun 169 1015 14
1998 Jul 170 1014 14
1998 Aug 171 1013 14
1998 Sep 158 1026 13
1998 Oct 162 1022 14
1998 Nov 145 1039 12
1998 Dec 145 1039 12
Indeed, in 1895 the stations in question hadn’t been built yet :-p
No one is questioning the fact that some of the USHCN network has atrophied over time. Stations have closed, volunteer observers have quit or passed away, etc. The network was originally created in the late 1980s, and NCDC made a conscientious choice not to add in new stations to replace ones that stopped reporting.
However, the actual percent of stations that have closed is closer to 20% than 40%. Much of the apparent increase in missing stations in 2013/2014 is due to reporting delays, rather than an uptick in the rate of station closures.
Zeke, if E flag data has no effect, maybe you could post Arizona and we can compare.
http://sunshinehours.wordpress.com/2014/06/05/ushcn-2-5-estimated-data-is-warming-data-arizona/
I could be wrong and I’d be happy to stop commenting on this if I was.
https://sunshinehours.files.wordpress.com/2014/06/az-uschcn-final-v2-5-0-20140509-from-1895-oct.png
Well, it depends on the month and state.
1998 is still missing for some states/months.
These are just the 35% and over:
1998 Jan AZ 9 13 41
1998 Jun GA 8 15 35
1998 Mar MD 5 9 36
1998 Aug MD 5 9 36
1998 Nov MD 5 9 36
1998 Dec MD 5 9 36
1998 Jul NV 5 8 38
1998 Sep NV 5 8 38
1998 Oct NV 5 8 38
Ok, here is all stations (including infilled) compared to non-infilled stations for AZ. See a difference: http://i81.photobucket.com/albums/j237/hausfath/USHCNAZinfillednoninfilled_zps0767679b.png
Here is another way to look at it:
This shows infilled stations only (only E flags) compared to non-infilled stations in Arizona grid cells (2.5×3.5 lat/lon) that contain both infilled and non-infilled stations. Again, little difference.
http://i81.photobucket.com/albums/j237/hausfath/AZinfilledvsnoninfilled2_zpsda365cc2.png
Zeke, you are using anomalies. The baseline would have used both infill and non-infill data.
I broke it down by month so anomalies are not really necessary.
(And your baseline changes for different graphs).
Bruce,
Anomalies aren’t just for removing the annual cycle… If your station network has a changing composition over time, not using anomalies will introduce bias due to climatological factors (like the average elevation of the station network) changing over time.
Here is a good simple explanation by Lucia of why you need to use anomalies for something like this: http://rankexploits.com/musings/2010/the-pure-anomaly-method-aka-a-spherical-cow/
Does you baseline include both infill and non-infill data to calculate the baseline?
Does that not skew the anomalies?
In a perfect world with perfect non-infilled data I will consider that anomalies should work.
But Arizona isn’t that big (it pretty much fits in a 5×5 grid) and using monthly averages gives one an idea what is going on with infilling.
Hi Bruce,
You don’t really even need to grid anomalies for AZ. Just look at the recent period with missing data (say, 1990 to 2013). Generate two datasets: one with all non-E-flagged values and the other will only E-flagged values. Convert both into anomalies relative to 1990 to 2013 and compare the results.
Of course, really short record E-flagged stations won’t really work that well; a better analysis would be to determine the effects of infilling by comparing all station anomalies (regardless of the flags) to anomalies from all non-infilled stations (excluding E flags). The difference between the two will be the effect of infilling on AZ temps.
Thats what I did here: http://i81.photobucket.com/albums/j237/hausfath/USHCNAZinfillednoninfilled_zps0767679b.png
Arizona – Oct 1895 to 2013
https://sunshinehours.files.wordpress.com/2014/06/az-uschcn-final-v2-5-0-20140509-from-1895-oct.png
Real Data = 0C/Decade
Estimated Data = .39C/Decade
All Data = .05C/Decade
411/2661 rows were Estimated.
(No anomalies were used)
Try using anomalies :-p
So you want to compare Infill anomalies using an all data baseline to non-Infill anomalies using an all-data baseline?
Incorrect. Not useful.
No, I suggested calculating the Arizona temperatures using anomalies for all stations (including E-flagged ones) and for only non-infilled stations (excluding E-flags) and comparing the difference. Anomalies are calculated for each individual station calendar month. I’d suggest using a period of 1960-1980 or so, as almost all stations have non-infilled data and infilling won’t affect baselines.
1960 to 1980 inclusive. Month / E / Not E
About 15% or so
1 84 441
2 78 447
3 84 441
4 75 450
5 84 441
6 72 453
7 72 453
8 85 440
9 74 451
10 76 449
11 86 439
12 85 440
Bruce,
Thats fine. It won’t mess up anomaly calculations. Whereas not using anomalies will introduce artifacts due to differing underlying climatologies.
Lets make a deal. You graph using my method (no anomalies) and post the results.
And then I’ll calculate using anomalies.
Bruce,
Here is infilled (all stations) vs non-infilled (excluding E flags) for Arizona using averaged absolutes (no gridding or anomalies). You can see pretty big climatological biases in the non-infilled data near the beginning and the end of the record when the station composition is changing (the middle is pretty similar since there are few missing stations).
http://i81.photobucket.com/albums/j237/hausfath/USHCNAZinfillednoninfilled_zpsa21ef8f0.png
Zeke showed us this difference between the new high quality USCRN network and USHCN.
http://rankexploits.com/musings/wp-content/uploads/2014/06/Screen-Shot-2014-06-05-at-1.25.23-PM.png
But this is the actual difference in the trends between the two.
http://s21.postimg.org/54hlteysn/USCRN_difference_USHCN_Apr14.png
Temperature trend is around 0.17F higher over just 9 years.
Yes Bill, the new pristine USCRN station network has a slightly higher trend than the old USHCN network. But its a short time period, so trends will be a bit more variable.
Zeke: If I remember correctly, the new USCRN has three thermometers inside an actively ventilated shelter. What happens if the shelters deteriorate with time? Say the outside gets dirty or the paint degrades, changing the albedo of the shelter? Maintenance that restored initial conditions could produce an abrupt breakpoint, which your usual algorithms might want to correct by reducing earlier temperatures. Is anyone doing breakpoint analysis on the USCRN or studying the effect of shelter mainenance?
Frank,
As far as I know no homogenization is being done (or is planned) for USCRN stations. They are not in old painted wooden boxes (like some of the older co-op station), and are designed to last for decades with minimal maintenance.
Here is what the new instruments look like (you can see the three sensors): http://cdn2-b.examiner.com/sites/default/files/styles/image_content_width/hash/42/df/42df45159afa2d0c0fdb90984029ebfd.jpg?itok=53ZXR4CL
Zeke, you need to each month to get a better idea how the distortion occurs.
Even your data though shows that the infilling technique just tries to continue existing trends along after the trend ends.
1995 to 2013 is all exaggerated because all the infilling does is keep the 1979-1995 trend going up up up.
Infilling might work when there is little trend or trends go one direction forever.
Bruce,
For the unteenth time, infilling is fine. Averaging absolute temperatures when the station network composition is changing is not fine. Infilling doesn’t “continue the existing trend”.
Here is a quick experiment for you: plot the absolute temperature of all stations in AZ that continue unbroken from 1950 till today. Plot the absolute temperature of all stations (without infilling) even if they don’t continue up to today. Compare the results. Ask yourself why you don’t see a big drop after 2000 in stations that don’t have any infilling.
There is a very good reason why everyone uses anomalies for these things.
There are no AZ stations from 1950 on that have zero E monthly records.
October
NAME E Count
AJO 12
BUCKEYE 5
CANYON DE CHELLY 7
CHANDLER HEIGHTS 6
CHILDS 12
FT VALLEY 13
GRAND CANYON NP 2 7
HOLBROOK 11
KINGMAN #2 9
LEES FERRY 9
MIAMI 1
PARKER 14
PEARCE SUNSITES 5
PRESCOTT 6
ROOSEVELT 1 WNW 9
SACATON 28
SAFFORD AGRICULTRL CTR 3
SAINT JOHNS 8
SELIGMAN 7
TOMBSTONE 13
TUCSON WFO 13
WHITERIVER 1 SW 13
WICKENBURG 14
WILLIAMS 4
YUMA CITRUS STN 11
Ok, use ones with the minimal amount of missing data. The point is that you will see that the apparent decline in recent years is an artifact of absolute temperature differences in the network from the climatology of the stations reporting. No station with complete reporting over that period will actually show a pattern of temperature change that resembles what you get when you average all the absolutes. This is why we use anomalies.
Zeke: Thanks for the reply.
My question is: What do we know about the possibility that station maintenance produces breakpoints in records of conventional stations and/or the new USCRN stations? “Maintenance” is meant to refer to changes that restore initial observing conditions. Gradual deterioration of a station will introduce a bias in the trend. For example the decrease in the albedo or ventilation of a shelter could gradually introduce a warm bias into the record of a station. If maintenance produces a breakpoint, that breakpoint should not be corrected. Whenever a breakpoint is caused by restoration of initial observing conditions, it shouldn’t be corrected.
I think in this case anomalies are used to hide the large amount of fabricated data.
What can I say Bruce, I just can’t stop hiding the decline :-p
NOAA has some very informative visualizations of individual station data, how it has been adjusted, and what is the effect on trend. I showed upthread the data for Honolulu.
There has been a conspiracy theory going around about how Honolulu has recently been dropped from the adjusted file because it was showing signs of downtrend, and the senior NOAA execs who watch this like a hawk nobbled it. But the larger story is that in the unadjusted data Honolulu has had a long and steep uptrend, which was much reduced after adjustment. It’s not all one way. It’s a similar story with Lihue, Kauai.
Anyway, I found the NOAA plots very helpful but hard to access. So I posted a portal so you can find and click on stations by name.
Carrick (Comment #129970)June 5th, 2014 at 3:01 pm
Nick has a couple of links up http://moyhu.blogspot.com/2014…..alies.html
Carrick (Comment #130089)June 8th, 2014 at 9:33 am
angech: 2000 just the last labeled tic mark of the graph (I’ve picked one for illustration, the others follow the same pattern.) The graph line clearly goes well to the right of that. As Nick states, the graphs in that post up-to-date.
I have gone back to Nick’s Moyu post and the graphs have been redacted/changed/altered to now show the graphs going to 2014.
As I commented before the graphs did not even have 2000 at the base and only went up to 2000.
I do not know why this happened, either Nick read my comments and changed them to the correct graphs or my computer was not getting the full graphs.
Not sure why you and Nick are playing tag on this. I would have appreciated a comment like “yes, there was a mistake and the full graphs were not posted, thanks for alerting me . Now changed. But I can cope.
SteveF (Comment #130091)June 8th, 2014 at 10:09 am
“Carrick, Yes, most people would agree that station move and TOB adjustments are legitimate. But it seems like some people can’t understand those types of adjustments and why they are needed to get a more accurate historical representation. I find it weird there is so much time spent arguing about this sort of thing; it seems just too obvious to be generating the arguments it does”.
I find it kind of weird that when you spend so much time arguing about this sort of thing that you do not sit back, do one of those alpha pauses [Null -A] and think about the reason. Other people have brains as well and the fact that a large number of people do not come to your obvious conclusion might mean that the conclusion is not that obvious, duh. Or even that you might be wrong but let’s not go there. Just do the first step OK.
Bruce,
give it up.
Zeke has made it quite clear.
1. We use anomalies rather than actual temperatures because we do not have enough real temperatures to use.
2. real temperatures do not occur in real places, like the top of a mountain or in the middle of a mountain range ,these altitudinal and location places can only be reached by a computer program.
3. real temperatures are not easily able to be measured in the right locations to give a true [also known as kriged] gridded data set for organizing graphs properly. A spherical cow needs to be created to put virtual infilled stations into these spots.
4. having got the true computerized plots in place every correct 100 K with some correctly located to the top of hills and in the middle of forests We can now correct the remaining pesky real data which shows climatic breaks, otherwise known as real weather and which causes TOB problems in a seasonal sense ie Hawaii and Alaska where the changes do not occur at the same time as the rest of the USA.
Also real temperature is staying flat whereas the infilling must add an up going trend for expected climate sensitivity.
Would Zeke for one like to declare this comment nonsense”
eg “There is no modification made to the infilling which incorporates CO2 sensitivity to continue an up-growing trend in the calculations”.
I’d like to see that!
Robert Way? Anyone?
anech:
The figures went to 2014 before you commented. You’re just nuts.
angech,
I have read and thought about the many arguments for why the warming trend is being ‘fabricated’ via adjustments, loss of stations, infilling missing data, etc, including the long exchange between Bruce and Zeke on this thread. I find arguments like Bruce’s consistently unpersuasive. That sort of argument indicates to me the person making it doesn’t really understand the issues. Is any historical record based on incomplete data from 100+ years ago going to be without problems? Heck no! And that is why adjustments are often needed. Lots of very skeptical people (like Jeff Id, among others) have reached the same basic conclusion: the adjustments to the temperature record are needed and are mostly legitimate. BTW, your 130126 comment is beyond ridiculous. You ought to consider commenting only at Goddard’s blog. A Deus.
Angech,“I have gone back to Nick’s Moyu post and the graphs have been redacted/changed/altered to now show the graphs going to 2014.”
I haven’t changed the plots.
Personally, I would like to see the average temperature of the surface of the world at this moment -not the average of the maximums over the past 24 hour or the minimums over the past 24 hours or the like. Then I could ompare it to the actual average 1 year ago, and not worry about anomalies, and also get a real idea bout what is happening to a number that I would understand
Nick. OK funny old things computers. others may comment.
Steve F Wouldn’t comment at SG. AW has made it clear he is an agent provocateur.
Is any historical record based on incomplete data from 100+ years ago going to be without problems? no! And that is why adjustments are often needed.
Let’s try some logic
It is not going to be without problems
. You can make adjustments. No, No, No, No.
You can make interpretations, theories , guesses.
You do not change the original real data ever That is not science and not legitimate.
No response to my earlier question which I know bothers a lot of people. Why has 1934 been reduced incrementally relative to 1998 over the last 15 years.
See here
http://burtonsys.com/climate/Re_CSRRT_Enquiry–Burton–US_Surface_Temperature_USHCN_chron.html
and here
http://sealevel.info/GISS_FigD.html
angech: “You do not change the original real data ever”
As far as I can tell, nobody’s doing that. What people are saying is that a temperature measurement of X, taken a while ago under certain conditions (e.g. equipment, measurement location, time of observation) is not directly comparable to a measurement Y taken today with current equipment &c. Instead, one should compare X’ with Y, where X’ = X plus adjustments. Or, you could compare X to Y’ (adjust current measurements); doesn’t matter if all you’re interested in is the change.
It would be much simpler if all measurements were taken with perfect instruments at consistent locations and with consistent methods. But not true.
HaroldW (Comment #130133)June 9th, 2014 at 7:24 am
angech: “You do not change the original real data everâ€
As far as I can tell, nobody’s doing that.
Try reading Zeke earlier in thread(Comment #130058)
June 7th, 2014 at 11:45 am. It is being done all the time. He says
“Mosh,Actually, your explanation of adjusting distant past temperatures as a result of using reference stations is not correct. NCDC uses a common anomaly method, not RFM.The reason why station values in the distant past end up getting adjusted is due to a choice by NCDC to assume that current values are the “true†values. Each month, as new station data come in, NCDC runs their pairwise homogenization algorithm which looks for non-climatic breakpoints by comparing each station to its surrounding stations. When these breakpoints are detected, they are removed. If a small step change is detected in a 100-year station record in the year 2006, for example, removing that step change will move all the values for that station prior to 2006 up or down by the amount of the breakpoint removed. As long as new data leads to new breakpoint detection, the past station temperatures will be raised or lowered by the size of the breakpoint.”
Congratulations to our hostess on her 30th Wedding Anniversary today which I just discovered. A real achievement to complete 30 years of marriage. Hope you have many more happy years.
JD
Zeke (Comment #130058)
…If a small step change is detected in a 100-year station record in the year 2006, for example, removing that step change will move all the values for that station prior to 2006 up or down by the amount of the breakpoint removed. As long as new data leads to new breakpoint detection, the past station temperatures will be raised or lowered by the size of the breakpoint.â€
——
Think about that for awhile.
If there is a “cooling” breakpoint detected, all the historical records are then cooled by the same amount. Let’s say trying to simulate the current station environment in the past. If the old station was as screwed up as the current station, it would have recorded a lower temperature. Semi-logical but not really.
So, if there is a cooling trend in the recent record (as in the US and the global record in the last several years), there is more chance some “break-point algorithm” will detect a cooling breakpoint and then adjust all the historic temperatures downwards.
And then let’s then say the algorithm has a bias toward detecting “cooling” breakpoints versus “warming” breakpoints”.
Viola. Simple mathematical procedure to continue adjusting the temperatures downward. Every darn month that is and sometimes twice per month.
Even worse, let’s say your breakpoint algorithm has a bias that changes through time, so that warming breakpoints are identified early in the record and cooling breakpoints are identified later in the record.
Double adjustment potential.
I note that I have repeatedly asked for a histogram of the breakpoint changes over time by how many were warming breakpoints versus cooling breakpoints in both the NCDC algorithms and the BEST algorithms (this would be about the 8th request now).
Given the silence from my request(s), I assume it is not a “good” story or that this has never even been calculated.
It is a “required” calculation in my opinion but you have never seen it.
Thanks! Believe it or not, it’s also my mother-in-law’s birthday. There were two ‘open’ weekends permitting us to take a honeymoon during June the year we wanted to get married. One would have been my brother-in-laws birthday; the other my mother in laws birthday. We picked the first of the two.
Bill Illis (Comment #130174)
“I note that I have repeatedly asked for a histogram of the breakpoint changes over time by how many were warming breakpoints versus cooling breakpoints in both the NCDC algorithms and the BEST algorithms”
For USHCN I think this is what you want.
For the effect on trends in GHCNV3, histograms are here.
Nick Stokes
June 9th, 2014 at 9:15 pm
————–
Okay, there are more warming breakpoints in the earlier part of the record than in the later periods in USHCN. Certianly seems to be more cooling breakpoints in the later period. Is this what I was saying?
And in the second link, GHCN V3, there are just more breakpoints that are cooling, especially around -0.05C for example.
Breakpoints in GHCN vs time?
angech (#130140)
You quoted Zeke as saying: “As long as new data leads to new breakpoint detection, the past station temperatures will be raised or lowered by the size of the breakpoint.”
Would it help if “past station temperatures” were more precisely stated as “adjusted past station temperatures”?
I can understand disputing whether the adjustments are correct, but not that what’s changing is the adjusted temperatures rather than the recorded ones. Perhaps I’m misreading your comments.
Nick Stokes (Comment #130072)
June 7th, 2014 at 11:40 pm
You have no shame! Creating “Straw Men” does nothing useful.
Remember I have a copy of the GHCN v2 data set. If you can explain why today’s GHCN shows a dramatic warming trend compared to v2 I will take you seriously. Did you read my posts on Greenland?
Enough with your BS already.
Harold W (Comment #130179) June 9th, 2014 at 9:37 pm
angech (#130140)
You quoted Zeke as saying: “As long as new data leads to new breakpoint detection, the past station temperatures will be raised or lowered by the size of the breakpoint.â€
Would it help if “past station temperatures†were more precisely stated as “adjusted past station temperatures�
I can understand disputing whether the adjustments are correct, but not that what’s changing is the adjusted temperatures rather than the recorded ones. Perhaps I’m misreading your comments
Misreading Zeke’s comment above and mine
“angech: “You do not change the original real data everâ€
when you say
“As far as I can tell, nobody’s doing that.”
Zeke said quote ” the past station temperatures will be raised or lowered by the size of the breakpoint.â€
furthermore if you had looked at his posting he confirmed it by staing the alternative
“An alternative approach would be to assume that the initial temperature reported by a station when it joins the network is “trueâ€, and remove breakpoints relative to the start of the network rather than the end”
This means they all are changing the recorded temperatures, all the time , every month and only in the past.
The latest temperature models have to be correct.
When a pause occurs past real temperatures have to be adjusted downwards by this method.
Ask Lucia.
This is a bit like hitting rabbits on the head when they pop up from holes. Where is Eli when you want him.
Nick (Comment #130177)
My reading of the histograms is that there is a consistent trend increase of about 0.3°C/century due to the adjustments. I presume we should ignore the large spikes at zero in each case, as they reflect zero adjustments.
Is that correct?
If so, it means that a significant proportion of the warming seen over the past century is from adjustments.
BobD (Comment #130182)
“I presume we should ignore the large spikes at zero in each case, as they reflect zero adjustments.”
No, you shouldn’t ignore them. But the mean adjustment differential is written underneath, and it is about 0.3C/cen.
That is a reasonable estimate of the difference to land temperature. It would make a difference of less than 0.1°C/century to land+ocean.
Nick Stokes (Comment #130183)
Thanks, I was reading it correctly then.
What is your theory as to why there is such a large net effect after adjustments?
I’ve got another question, while we’re on the topic of GHCN – what inputs do the national bodies have to the adjustments? Do they submit metadata in the form of station histories?
Nick Stokes (Comment #130183)
Well, it’s debatable I suppose. If we’re interested in how many of the adjustments that were made, were made in either direction, then they should be ignored, as “no adjustment” doesn’t feature in that discussion. I would argue for excluding them from the mean calculations too, for the purposes of determining the balance between positive and negative trend effects.
Also, why is the zero case shown using the -0.02 to 0.00 grouping? Considering how many there are going to be in each histogram, surely choosing groupings -0.01 to 0.01, etc. would make more sense?
BobD (Comment #130184)
“What is your theory as to why there is such a large net effect after adjustments?”
MMTS introduced a differential of about 0.15°C. That would be about half. Otherwise, probably a bias in the discontinuities they pick up. Stations moving out of town etc.
“what inputs do the national bodies have to the adjustments? Do they submit metadata in the form of station histories?”
No input. The algorithm is designed to avoid metadata, which is patchy and inconsistent. It is more reproducible to use a consistent algorithm on observations.
Nick Stokes (Comment #130187) Thanks Nick.
One last question – what post-processing (sanity) checks are done on individual stations, to see whether adjustments that were made were reasonable within the context of the station histories?
For example, it must be a simple matter to find some adjusted stations with known good histories and check them.
Nick Stokes (Comment #130187)
I lied, I have another question. You mention “stations moving out of town and so on.”
I agree this will produce a cooling breakpoint. However, it is well known that the reason for this cooling effect is purely artificial, since the town temperatures were only elevated in the first place because of UHI and/or sheltering. Hansen (2001) deals with this problem.
What steps are undertaken in the breakpoint analysis to ensure that an adjustment is NOT made in these cases?
BobD (Comment #130186)
“Also, why is the zero case shown using the -0.02 to 0.00 grouping?”
That’s just the way R does it, by default. You tell it how many columns you want, and it places them. The zeroes are significant; obviously if 99% were unadjusted, then a quite large adjustment on the others would not have much effect.
Sanity check?
Here is extensive testing; it’s mainly using synthetic data, since with real data you may know there is a station move, but not what the difference should be. They do test with the real MMTS transition.
For GHCN there may be a TOBS issue with MMTS. ROW doesn’t usually have volunteers, so TOB was stabler than USHCN. But with MMTS the TOB is set to the standard midnight, which is cool biased wrt any daytime TOB.
BobD (Comment #130189)
” However, it is well known that the reason for this cooling effect is purely artificial”
No, you don’t know that. UHI is only an issue if it changes. It may have been stable prior to the change, in which case the downchange is real.
Nick Stokes (Comment #130191)
Most cities do not have stable populations, and most cities are growing around the world.
What checks are done then to work out if the UHI has remained even, and if the downchange is real?
BobD (Comment #130192)
I think you’ll need to read some of the papers.
Nick Stokes (Comment #130193)
In Williams (2012) it says this:
In other words, they acknowledge exactly what I’m saying, but no checks are carried out, am I right?
I don’t care how you torture the data, if the estimated stations show a higher trend than the actual measured data, then you are defending fraud.
I skimmed over the comments, and I didn´t notice any comments about kriging. I´ve used oil and gas reservoir 3D dynamic models, which required a detailed grid describing the conditions at the beginning of the model run. As you can imagine, when we try to describe rocks and fluids located a few Km under the surface we lack the required coverage, so we have to interpolate the data to fill the grid.
Evidently some of you are quite knowledgeable about this topic, and I understand kriging is used to infill data quite often. I´m also wondering if the infilling used to fill a grid my not be subtly different than one used to report a regional or global temperature anomaly? When we fill in the grid we have to be quite focused on trying to arrive at the best guess for the specific grid cell…but quite often the cell is upscaled, and in some cases in the oil industry we do a refined grid over “tricky spots” (which I´m not sure is worth the effort in most cases).
Anyway, from an intuitive point of view I would do kriging of some sort, consider altitude, distance to large heat sinks, and possibly run a fine grid over mountain chains?
I wouldn´t get too bent out of shape trying to estimate the global average anomaly unless you want to impress a politician. However, having smarter data to fill in a grid sure makes sense.
Which reminds me, I assume you have other grid data sets, for values such as precipitation, humidity, pressures, heat capacities, and the way the different light frequencies reflect off the surface? That data can probably be squeezed to give you improved temperature kriging.
I checked adjustments for Barrow Alaska used in Ghcn v3. Umiat is nearest station at a distance of 276 km. Its record consists of 1945-1954 and 1991-2000. On top of that it has some holes. I don’t think they could have used that to compare with Barrow.
Barter Island is the only other station within 500 km(I think that’s their comparison limit) at a distance of 499 km. But that station has also been adjusted. Perhaps by a few different stations within 500 km of its location. But most importantly, it is missing a decade of data starting at the point in time when Barrow starts to be adjusted backwards in time.
What reference did they use to adjust Barrow? Did they adjust Barrow from Barter raw and then go back and say Barter raw isn’t correct, so we will also adjust it? Or did they first adjust Barter and use that adjusted data to then adjust Barrow raw? Either way it sure seems unfounded. Their adjustments warmed the past, but if it isn’t right it shouldn’t be done. Here is a graphic just to show just how schizophrenic the adjustments are for those two stations.
http://i61.tinypic.com/2cmlkrl.gif
Bob Koss (Comment #130197)
Bob, I don’t have any way of finding out excatly how the algorithm worked there. But it’s a good case for using the NOAA station info. Here’sBarter Is and Barrow W
I note that they are both cases were a strong uptrend was much reduced after adjustment.
Nick Stokes (Comment #130190) :
The specific sequence of breakpoints for the counting bins can be specified as a vector of values in the hist function. I would have thought that the large number of zeroes in your plots would have made it obvious that the “zero interval” be centered over that value when checking for symmetry.
Nick Stokes (Comment #130198)
I mentioned they had warmed the past, so the reduced trend is unsurprising.
Those NOAA charts showing the anomaly difference are sort of misleading. Those adjusted anomalies are based on the now adjusted data which has raised the temperature during the base period 51-80 compared to the raw data. They are subtracting apples from oranges. While it leaves the impression small changes were made, the actual adjustments are surprisingly large and rather erratic. At least that is how I view it when looking at my graphic of actual temperature adjustments above.
The question remains. What was the procedure used which justifies such large and erratic changes? I’m not going to locate and dig through numerous papers to try and figure it out. Was just hoping someone here had a reasonable explanation.
Bob,
I can’t speak to what exactly NCDC does to Barrow, AK, but here is Berkeley’s homogenization for that station: http://berkeleyearth.lbl.gov/stations/167759
There are 8 documented station moves in the metadata. The one in 1998 in particular introduces a pretty large step change vis-a-vis neighboring stations.
Also, for reference, here is the paper that describes NCDC’s homogenization approach: ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/papers/menne-williams2009.pdf
.
Fernando,
While the Berkeley approach uses Kriging, most other temperature series uses much simpler spatial interpolations, either simple distance-weighted averaging or averaging stations within large grid cells (e.g. 5 lat by 5 lon). Historically there hasn’t been much focus on identifying fine-grained local features, as most folks are looking at global or regional averages. Now, however, both Berkeley Earth and Cowtan and Way use kriging in their reconstructions.
Zeke:
I addressed this a bit in a comment upstream.
I would consider the temperature variations on a 5°x5° cell to be the “minimum resolution of interest”. However, the typical radius used by GISTEMP is 1200-km, but you can choose 250-km instead.
I was unable to find the equivalent “effective radius” for BEST in their most recent publication. As far as I can tell, they don’t publish their currently used kriging function… Zeke—can you point us to a document that shows the kriging function they are using now and what the effective averaging radius of that kriging function is?
I focused on the SE US because it is a fairly unique region meteorologically. It also happens to be one of the few regions on the globe that have experienced a net cooling over the last century.
As I noted above,
For calibration, the distance from New Orleans to the Canadian border is 2000-km (so a 1200-km radius spans north-south the entire US) and East-West it’s roughly 2400-km from Albuquerque to the US East Coast.
So what you and I might consider “regional scale” is a very different thing that what e.g. BEST seems to consider “regional scale”.
Carrick,
Berkeley’s standard product is a 1×1 lat/lon gridded field. We have quarter degree fields for CONUS and Europe where station density allows as a separate product.
Berkeley does tend to result in smoother trend fields than other products, and this has a notable effect in the U.S. southeast. We covered this field smoothness question a bit in our AGU poster this year: http://wattsupwiththat.files.wordpress.com/2013/12/agu-2013-poster-zh.pdf
We focused on the period from 1979-present, which isn’t really a period that shows much Southeast cooling. Its a challenge to figure out which is the “correct” degree of smoothness of trend fields, and if divergences in the Southeast are due to uncorrected localized bias or a real temperature signal that Berkeley is removing by oversmoothing.
Zeke, thanks for the reference.
The differences between GISTEMP 250 vs 1200 km trend does suggest to me that the effect is due to smoothing (smearing of higher latitude regions with larger trends into regions with smaller trend).
Do you have an estimate for the effective smoothing radius for the kriging function used by BEST?
It would be interesting to use an EOF based formulation, such as used by NCDC, and experiment with different truncations. Retaining higher order terms should result in more spatial resolution, but probably at the expense of increased noise levels.
Carrick,
I don’t know the effective smoothing radius for Berkeley’s kriging offhand; I’ll ask Robert at our group meeting today.
Argghhh! What a screw-up!
That last graph of mine in comment #130197 is crap. Had a one second power interruption last night while I was working on Barrow and Barter. Computer didn’t even shut down. Didn’t realize it at the time, but my spreadsheet corrupted several columns of temperature data and interpreted them as text. So all the calculations are wrong. After a total redo the Barrow adjustments don’t look so bad, but I still wonder what they compared it to.
Zeke (Comment #130205)
Thanks for the link to the station moves. I’ll look at the pdf later when I’m not so irritated.
It doesn’t appear Ghcn paid any attention to the moves. None of the Ghcn changes matches up with them. The 1998 move is closest, but is still earlier by 2 years to the single year adjustment Ghcn made in 2000.
Here is a new graphic to replace the screwed up Barrow/Barter one I posted earlier.
http://i62.tinypic.com/2i7pf89.gif
Zeke:
A potential research path may be to investigate the improvements you could achieve in the grid initialization if you use kriging and a “local grid refinement” in areas where mountain chains and possibly sea currents create sharp breaks in weather behavior?
I’m not an expert in this field, but I’ve worked with a group of researchers who really sank their teeth into kriging, and of course the match of existing historical data to arrive at a family of “realizations” which could be used in a forecast ensemble.
I should be clear that in hindsight the models lack the ability to make a long term prediction. This is why it’s important to use multiple realizations in the oil and gas modeling exercise. I’ve also concluded we just never seem to have the tails described properly, so there’s a tendency for models to fail when things go way out there where we thought they would never go. In our case this would happen a lot in areas in which we knew less about (such as the changes in the earth’s stress field when we drop pressures). However, we also found that in some circumstances the most fundamental equations we used were erroneous, they broke down, and in some cases it took a long time to figure out the reason they did.
How about this Berkeley station – Amundsen Scott at the south pole.
23 quality control failures identified by the algorithm despite the fact this is supposed to be one of the highest quality weather research stations on Earth. Tell that to the scientists freezing their butts off in -60.0C temperatures.
http://berkeleyearth.lbl.gov/stations/166900
The actual quality-controlled raw data is reported here and it has virtually no trend despite Berkeley having revised it to +1.0C over 50 years.
http://www.antarctica.ac.uk/met/READER/surface/Amundsen_Scott.All.temperature.html
A couple of months out-of-date chart of the above raw temps.
http://s13.postimg.org/6w98pvd8n/Amund_Scott_90_S.png
“Averaging absolute temperatures when the station network composition is changing is not fine.”
– Zeke
Unless I missed a later response from Zeke, when he was asked about the elephant in the room – why does dropping stations and adding stations always result in an increase in the warming trend – his reply was that it’s all just an amazing coincidence.
A guy like this would never get an engineering job at my company, that’s for sure. Not impressed.
Zeke
What is the actual adjustment or trend figure that you input into the infilled stations. Do you do the infills first on these trends and then back change the real temperatures that are outliers. Do you only adjust the low readings because the high readings must be right (confirmation bias). Or do you remove both equally based on a median around the infilled figure averages.
How can you ever get a lower reading unless over 75 percent of the real stations show a lower figure per month?
Will Nitschke, the exact nature of the error is derived on Nick’s blog. If you’re really an engineer, it won’t be any problem for you to immediately see why Goddard is being particularly boneheaded, even for Goddard.
Bill Illis,
Interesting catch; that looks like it might actually be a bug in how the quality control code deals with extremely cold temperatures. I’ll bring it up next meeting for us to look at.
“Zeke (Comment #130058)
June 7th, 2014 at 11:45 am
Mosh,
Actually, your explanation of adjusting distant past temperatures as a result of using reference stations is not correct. NCDC uses a common anomaly method, not RFM.
The reason why station values in the distant past end up getting adjusted is due to a choice by NCDC to assume that current values are the “true†values. Each month, as new station data come in, NCDC runs their pairwise homogenization algorithm which looks for non-climatic breakpoints by comparing each station to its surrounding stations. When these breakpoints are detected, they are removed. If a small step change is detected in a 100-year station record in the year 2006, for example, removing that step change will move all the values for that station prior to 2006 up or down by the amount of the breakpoint removed. As long as new data leads to new breakpoint detection, the past station temperatures will be raised or lowered by the size of the breakpoint.
An alternative approach would be to assume that the initial temperature reported by a station when it joins the network is “trueâ€, and remove breakpoints relative to the start of the network rather than the end. It would have no effect at all on the trends over the period, of course, but it would lead to less complaining about distant past temperatures changing at the expense of more present temperatures changing.”
Zeke, that would be just debating which side of a bad coin to look at. The problem (which ever way you adjust it) is the lack of quality control on the breakpoint adjustments. The algorithm doesn’t check if the break point is introducing an error or correcting one. And from what we have seen the climate scientists don’t check either, because the algorithm gives them the answer they expect or want.
Take two examples, there was an airport thermometer in Seattle, WA that was reading 2C warmer than surrounding areas due to instrument error. So the thermometer gets replaced by a valid one and the 2C bias goes away. So what does the breakpoint algorithm do with this station? It adjusts past temperatures downward to correct for the cooling bias in this breakpoint. When a quality measurement system would determine when the sensor went bad, and subtract out the error but only from the era when the thermometer was bad. Instead the breakpoint system, homogenizes the error into neighboring stations and then adjusts 100 years or more of the past downward. Without validating each breakpoint, or confirming the breakpoints do not change the distribution (they do) the method is only lowering past temps to ‘correct’ for current warming biases.
Another example is the painting of Stevenson Screens, peer reviewed work shows that repainting a screen causes a breakpoint because weathered stations have a warming bias as the screen absorbs heat. So the breakpoint method, rather than rezeroing after a fresh coat of paint and removing the warming bias that was introduced between paintings, instead adjusts the past downward to incorporate the warming bias into the historical record. So if the bias is 0.3C between paintings and you paint it 5 times over 50 years, the past will be adjusted down 1.5C with the breakpoint method to account for 5 sections of 0.3 bias in the opposite direction.
Goodard is trying to explain the bias to the adjustments, when the real bias is right there in front of his face and yours.
“Nick Stokes (Comment #130183)
June 9th, 2014 at 11:51 pm
BobD (Comment #130182)
“I presume we should ignore the large spikes at zero in each case, as they reflect zero adjustments.â€
No, you shouldn’t ignore them. But the mean adjustment differential is written underneath, and it is about 0.3C/cen.
That is a reasonable estimate of the difference to land temperature. It would make a difference of less than 0.1°C/century to land+ocean.”
That would require assuming the ocean temperatures were not adjusted too. And even Phil Jones admitted that they adjusted the ocean temperatures downward in the past, because it was not warming as fast as the (UHI influenced) land temps. He said it wasn’t physically possible for the land to warm that much and the ocean to not warm, so rather than finding the error in the land, he just adjusted the oceans to match the bad land data.
Theodore (Comment #130278)
+0.9
I would give you +1 if you managed to write °C
Theodore
Please could you provide a link or citation to your source for Phil Jones words as the fail here would appear to be beyond belief
Gras Albert (Comment #130285)
“Theodore
Please could you provide a link or citation to your source for Phil Jones words as the fail here would appear to be beyond belief”
Yes, I also don’t believe it. Phil Jones does not manage any SST dataset.
http://www.youtube.com/watch?v=9rQUII8PyiA
He discusses it a little after the 4 minute mark with more specifics after the 6 minute mark.
Says the sea and air temps “can’t really differ that much as a global averageâ€.
So they had to adjust the oceans or they would “have great differences in sea and air temperatures that just couldn’t happen naturallyâ€. Which is true, but he missed that UHI was causing the land temps to go up and cause the divergence.
For the background of how they did invented the sea surface adjustments in the ‘literature’. http://wattsupwiththat.com/2013/05/25/historical-sea-surface-temperature-adjustmentscorrections-aka-the-bucket-model/
Theodore:
Regardless of providence, this is just nonsense.
Land versus ocean trends (CRUTEM vs SST database).
So not adjusted to match.
It is of course a fact that when you have identified systematic effects in your measurement process, you should adjust for them.
Why that simple observation makes some peoples’ brains explode is itself a curious puzzle.
“Regardless of providence, this is just nonsense.”
http://wattsupwiththat.com/2014/01/17/phil-jones-2012-video-talks-about-adjusting-sst-data-up-3-5c-after-wwii/
Will, as a somebody who claims to be engineer, I assume you can read graphs.
Do SST and land trends match up? Is land-only trend different that sst? Was SST therefore adjusted to match the “bad” land data? [self snip]
ANS: No. Yes. No. No comment.
“Bill Illis,
Interesting catch; that looks like it might actually be a bug in how the quality control code deals with extremely cold temperatures. I’ll bring it up next meeting for us to look at.”
– Zeke
Since you don’t know what this “bug” is, how do you know it only effects extremely cold temperatures? That is to say, you are making very specific technical claims about something you admit to not understanding at this point in time.
This is why I keep referring to the partisan nature of this argument.
It seems to me that your time would be better spent investigating even just the significant discrepancies and concerns raised in the comments section of this blog, concerning the methods of the temperature products you endorse, instead of wasting your time attacking some semi-anonymous and largely unimportant amateur climate blogger.
Will:
Given how partisan you have been on this thread, I’d say that’s somewhat self-referrential.
“Given how partisan you have been on this thread, I’d say that’s somewhat self-referrential.”
Yes Garrick, I’ve pointed out a few times now that I have no confidence in Goddard, nor his his attackers. And I’ve explained exactly why that is the case. That does make me biased, towards separating nonsense from truth.
Carrick (#130291):
“Regardless of providence, this is just nonsense.”
“providence” should be “provenance”?
Bad typo day, I guess. 🙂
Will:
Creating false dichotomies is in no way biasing yourself towards “separating nonsense from truth”. Some facts are easy to check, and when you glaringly miss these simple tests, it is obvious to the rest of us.
HaroldW: More like “malapropism Wednesday”. Right spelling, wrong word.
I haven’t read all the comments because it’s is getting me down. Has anyone realised that collating together trends from absolute temperatures at each station is the same as getting a trend from collating the anomalies ie. 2(x-y)=2x-2y? Of course, this assumes that the way the data is collated is kept simple and is correct.
The real problems show up when you look at individual stations. How can adjustments that account for changes to equipment or positioning be justified for every month? Effects of UHI would be gradual but that means adjustments to show a smaller rate of warming, not larger.
http://kenskingdom.files.wordpress.com/2014/05/amberley-tmin.jpg?w=450&h=256
Hi Zeke, am hoping you (or other knowledgeable people) can answer a question about the accuracy of climate models. Adrian Ocneanu is an accomplished mathematician at Penn State (No. 1 in the world at International Math Olympiad in his youth) who frequently criticizes climate models at Dotearth. About 2 weeks ago at DE he stated:
x
“The climate models had a very wide, 95% confidence band. That means that with 95% confidence they expected the measured temperature to be INSIDE that band.
The actual measured temperatures are OUTSIDE that band. They are, in fact, getting close to exiting the 99% confidence band, wide like a football field, as well.” See approximately middle of comments at this link http://dotearth.blogs.nytimes.com/2014/05/22/gavin-schmidt-on-why-climate-models-are-wrong-and-valuable/?module=BlogPost-Title&version=Blog%20Main&contentCollection=Climate%20Change&action=Click&pgtype=Blogs®ion=Body
x
In so doing he linked to an article you wrote at the Yale Climate Connections. See http://www.yaleclimatemediaforum.org/2013/09/examining-the-recent-slow-down-in-global-warming/ [climate models and observations section I believe]
Do you think his comments are accurate or inaccurate? And what would be the reasons for your answer. Hope you can reply.
Best wishes,
JD
“Creating false dichotomies is in no way biasing yourself towards “separating nonsense from truth—
A false dichotomy is when someone suggests that the solution is either X or Y. Clearly, I’m stating I have issues with X and have serious concerns about Y. Which is a proposition that has nothing to do with dichotomies.
But let me tell you why I have little confidence in “Team Zeke”. Bill Illis provides an example of why an officially endorsed methodology is broken. Zeke acknowledges this is probably a “bug” in the methodology, but then immediately goes into damage control mode by asserting that the bug only effects extremely cold temperatures. How does he know this? Of course he doesn’t know this. Until he read that post, he presumably didn’t know the bug existed, or didn’t care, or hoped nobody would notice. It’s impossible to qualify a software engineering problem that has not been studied and is therefore not understood.
Which is why I’m pointing out that the BS comes in thick and fast, whether we’re dealing with “Team Zeke” or “Team Goddard.” But if I have a bias it’s this: I hold those who are involved in the creation of an authoritative temperature product to a much higher standard than I do an amature climate blogger.
The thing is, these adjustments have to continue widening over time. The algorithms can’t be fixed now unless the “fix” results in an ever increasing trend.
The “adjustment” warming needs to increase from the current about 0.3C to about 2.8C in the next 86 years in order to reach the target.
I’m just saying that this is never going to end. The Blackboard of 2098 will be asking why version 29.856 made change X.
Actually a dichotomy is “a division or contrast between two things that are or are represented as being opposed or entirely different”.
Dividing this debate into “Team Zeke†or “Team Goddard” is what I am referring to as the false dichotomy:
I’ll grant that “Team Goddard” is simplistic, driven as it appears to be by a desired political outcome.
However the scientific issues and controversies surrounding reconstructing global mean temperature trend from an imperfect array of instruments not intended to be used for that purpose is not definable as a single “camp”.
(Therein lies the false dichotomy.)
There are people who are convinced the problem is solved as well as needed and nothing of interest left to study (possibly Eli), others who are driven by the desire to understand how to make the measurements as good as possible (science for the sake of science, e.g. Robert Rohde), and people who are convinced that the problems are serious enough in the current reconstructions that they affect policy-based outcomes (possibly Anthony Watts).
While I agree with the need to hold people up to high standards, I think it’s equally important to become knowledgable enough so that you can understand the reasons why people made the decisions they made in the various stages of the analysis.
In other words, skepticism is good, but it needs to be tempered by some insight on the part of the skeptical examiner. Just assuming that people did things wrong because you didn’t take the time to understand the problem (e.g., your initial severe judgement of Zeke) doesn’t cut it with me.
@Carrick
You seem to be muddling yourself up. If I list the problems I have with McDonalds hamburgers and then list problems I have with Burger King burgers, I am not implying that you can’t go off and eat pizza. Again, no dichotomy. I’m guessing you threw in this accusation because you thought it sounded clever. Anyway, doesn’t matter.
As for Zeke, if he doesn’t talk nonsense then he won’t be criticized. Certain aspects of the method he endorses is clearly absurd, and I don’t think Zeke refuses to acknowledge this. But there is a lot of hand waving going on too. That doesn’t mean Zeke isn’t a smart guy. That doesn’t mean he doesn’t get a lot of things right. And that doesn’t mean that a log of sceptical criticism won’t turn out to be unjustified. I’d be very surprised if all of it was justified. And maybe this consistent pattern we see in relation to station mix is just some sort of amazing coincidence after all, as Zeke is suggesting. But wouldn’t it be great if time was spent answering these tougher questions?
Thank you Zeke for putting this post up. Hopefully it will result in greater openness and sharing of information though you may not be feeling this yet you are trying, which a lot of your colleagues do not want to do. The level of vitriol reflects the extreme importance of dong the data collection and models openly so all sides can feel confident that their arguments are on standard ground. As you know this is not the case at the moment and has not been the case for skeptics for a long time.
Incidentally Mosher described the principle that if site A is closer to site B than site C then site A is more likely to be similar to site B than C is a fundamental theorem of geo statistics at JC. 10.50 12/6/2014 asymmetric responses of arctic and Antarctic.
My question to you is that Robert Way has stated at Skeptical Science that this is not true when calculating the Arctic infilling as used in Cowtan and Way and my understanding is that this faulty principle may now be being used in your current Arctic infilling. Can you assure us if you use Steven’s fundamental principle or Robert Ways new improved principle.
See “how global warming broke the thermometer records” by Kevin Cowtan at Skeptical science 25/4/2014 speaking of his and Robert Way’s finding that the Gisstemp conundrum was due to actual GCHN Arctic data and infilling showing a cooling “bias” when compared to their model only method.
This occurred supposedly by violating the assumption that neighbouring regions of the planet’s surface warm at a similar rate.
angech,
The problem in the arctic is one of station density; Cowtan and Way actually discovered the problem with GHCN’s adjustments by comparing them to Berkeley’s results, which are more accurate for those stations given the denser network. There is always a challenge in very sparsely sampled areas of misclassifying abrupt changes (in this case an abrupt warming trend) as local biases rather than true regional effects. Larger station networks can help ameliorate this.
.
Will Nitschke,
Some sort of automated homogenization is necessary. We’ve been working on ways to test to ensure that homogenization is not introducing bias. The Williams at al paper makes a compelling case, for example: ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/papers/williams-etal2012.pdf
Our recent UHI paper also looks at this by redoing homogenization using only rural stations to detect breakpoints/trend biases.
The reason I suspect that the Amundsen-Scott results are a bug due to the extreme cold is that they are flagged as regional climatology outliers. I’ll suggest that the Berkeley team look into it in more detail next week.
.
JD Ohio,
Using that approach, observations are still within the 95% CI for models, though they are close to the bottom. As I mention in the article you reference, the next few years will be key in seeing if they become clearly inconsistent. I have an updated graph here: http://www.yaleclimatemediaforum.org/2014/02/the-global-climate-in-context-2013-in-review/
.
For other folks: sorry for being slow in responding; other things in life have been pulling me away from blogging, and I’m about to head out on a camping trip with no internet access for the weekend.
Thanks Zeke
USHCN estimated final versus non-estimated final by year and month. No anomalies.
http://sunshinehours.wordpress.com/2014/06/11/ushcn-2-5-estimated-warming-by-year-and-month/
This is all ‘very interesting’ in the supercilious way the phrase is often used. I’m not sure why Steven Goddard posts what he does, but I know why I find his posts so refreshing: (a) he hoists the media on its on petard in a way that anyone who is at all interested in climate science should appreciate and (b) he continually calls attention to the Humpty Dumpty nature of surface temperature reconstructions.
Yes, I understand that Nick and Zeke and Mosher etc. have a lot invested in the belief that a surface temperature record ‘designed’ to record local maxima and minima can somehow be magically modified into a true record of temperature change over land over a long period. Great, if you all think that is true and that your constantly modified estimate of temperature change over time for a quarter of the Earth’s surface gives you visions of the future – go for it, but do it in private please. Anyone who approaches this data set without your preconceptions would have to conclude that it is busted. And you know what they say about all the King’s horses and all the King’s men.
Zeke,
Thanks a lot for your 6/12 response. Still, I don’t have a good grip on this though. Hope that you or others can follow up on this.
What I am asking is whether in a practical sense, the models are grossly underperforming or not. Is Ocneanu cherry picking? Is it too short of a period of time to evaluate the efficacy of the models? The reason for asking the question is that if the models are performing nearly as badly as Ocneanu states, there is very little reason, at this time, to attempt substantial reductions in CO2. As I have made clear on this blog before, I do not have a good background in statistics or math, and I would like to have a good understanding of this very important issue.
JDOhio,
There are lots of question in that single comment. Aside from the finer points (rainfall patterns/amounts, ocean heat uptake) that most models don’t do well on, the most glaring discrepancy is that even considering “replicate runs” from the many models, virtually all runs of all models run above the measured rate of warming. This means either the post 2000 period is an extreme outlier, or the models are wrong. There is no certainty that the models are much too sensitive to radiative forcing, but the weight of the evidence indicates that they really are too sensitive. The high sensitivity of the models is mainly due to strong assumed positive cloud feedback ( clouds are paramererised, not modeled), and the hottest models assume the strongest cloud feedbacks. Absent that positive feedback, the models would a actually be close to the observations, at least for average temperatures. The other thing to be aware of is that the modeling groups have been free to choose whatever historical level of aerosol offsets they want…. Which means the models can be made to simulate the past reasonably well; they just kind of suck at making predictions, which is another indication that they have serious problems.
SteveF Thanks a lot for your response. I would like to simplify my question.
What I would like to know is whether the models are in a basic practical, sense, by a 95% chance, failing by their own terms or whether Ocneanu is overstating their failure or taking the 95% figure out of context. I realize that there are many matters that cannot be modeled (for instance clouds) but my narrower question is simply whether the models are, in a real functional way, failing according to their own terms. Ocneanu seems to be arguing that they are, but he can sometimes be a strident critic, and I would like the objective take.
Thanks for your help.
JD
JD,
Based on the >95% of model runs which are running above observations, the simple answer is that it is very unlikely the models can accurately predict future temperatures. As I said, that is not 100% certainty, but a bettor would be a fool to bet the model projections are right. Those who are concerned about warming will, of course continue to offer ‘explanations’ (excuses) for why the recent warming is well bow model projections, with the implication always being that really really truly the future projected warming rates (very fast) will come true. It is IMO nothing but motivated reasoning; those folks understand that the kind of public policies they desire will never happen if climate sensitivity is determined to be in fact low. There will be a lot of kicking and screaming as reality imposes itself on those folks. It is and has always been primarily a leftist/green/Malthusian POV which has motivated the hysteria. It will take a while, but reality will not be denied.
JD:
The models cannot ever fail because they are not predictions singly or as an ‘ensemble’. That is the quasi-official warmist position, as I understand it. The models purport to incorporate the sum of accepted science. Unless and until unqualified types (I.e., those who (a) don’t appear among senders and recipients in the Climategate emails or (b) are recognized by those involved with said emails as acceptable) establish a new paradigm that the qualified accept, the models are science and nothing contrary is.
You might take the simplistic approach that the models call for far more warming than is actually taking place and are therefore, well, wrong. No. They are simply incomplete because we have not yet found out what is suppressing the heating we know is happening even if it isn’t. It’s science.
For further reference, I suggest Googling “phlogiston” “ex cathedra” “Lysenko” and “epicycles”.
“Yes, I understand that Nick and Zeke and Mosher etc. have a lot invested in the belief that a surface temperature record ‘designed’ to record local maxima and minima can somehow be magically modified into a true record of temperature change over land over a long period”
Huh.
That is not what I believe
1. there is no such thing as a true record.
2. there is only the best estimate given the data and the method.
3. The index created by this procedure is not a ‘temperature’ in the physical sense. Its a metric composed of SST and SAT.
4. That metric has two uses
A) to test models
B) to give a VERY GROSS indication of changes in the system
it is one slice through very complicated system. As a system diagnostic its not that great, OHC is better, But we have a short
OHC record and a long temperature index.
it is NOT what i would do if I could go back in time.
but, given the data, given the need to create some useful metric,
it is a useable tool.
so. its not a “average temperature” its an index composed of SST and SAT.
its not an average, its an expectation that minimizes error given the data.
its useful. I use it to criticize models.
Steve, as I understand it the various anomaly records indicate global warming and that this is also manifest in the USA.
However, let us take one local that has been assigned as belonging to a temperature field identified by BEST, and other methodologies, as having warmed since the 1950’s.
I we examine a local, say for the sake of argument Roswell, New Mexico, and find that there is no change in the daily temperature, 1-365, between 2013 and 1950, does this mean we cannot trust absolute temperatures recorded by thermometers in the past, the present or either?
As I see it, if we are able to show there is no statistically significant between Tmin and Tmax, over 30 day periods, between 6 decades, of a station in an assigned temperature fields, then no warming.
Can we use actually station recordings, separated by 60+, to test your reconstruction?
Thanks a lot to Steve F and George T for responding to my questions.
Also, some people may not be aware that Christopher Booker has a column, relying on Goddard, with the headline that “The scandal of fiddled global warming data.” The lead-in sentence states:
“The US has actually been cooling since the Thirties, the hottest decade on record.” See http://www.telegraph.co.uk/earth/environment/10916086/The-scandal-of-fiddled-global-warming-data.html
Don’t have a lot of confidence that Goddard’s numbers are correct.
JD
JD Ohio – While I would tend to go along with the view that Steven Goddard is making errors in how he goes about his calculations I find it hard to get away from the conclusion that his work is a symptom of an underlying lack of cofidence in the USA temperature history calculation.
There are 2 main reasons.
1. Over time GISS has introduced a significant change of trend in the order of 0.7C for the position of 1998 relative to 1934.
See
http://rankexploits.com/musings/2014/how-not-to-calculate-temperature/#comment-130132
http://web.archive.org/web/20071008100432/http://www.climateaudit.org/?p=2077
These are not Steven Goddards conclusions but from their own output based on GHCN input. There has been an incremental change over time and as far as I can tell no explanation for the changes has ever been forthcoming. They can only come about if either the data or processing has changed. It would be perfectly reasonable to restate the record if for example an error was identified or the need for an adjustment was identified but I would expect a clear statement of what error or what adjustment with a before and after visualisation. As far as I can tell that never happens.
The GISS team did try to improve matters by eventually releasing code and that led to projects like Clearclimatecode but GHCN are still rather opaque.
2. Data gets homogenised and if people look at that individually per station then the results can appear very strange. Back in 2007 there was work done by a number of people. Steven Mosher was one of them but a JohnV was heavily involved. I am using webarchive links due to format and retention of at least some of the graphics.
http://web.archive.org/web/20080126190946/http://www.climateaudit.org/?p=2048
The main conlusion was there was a difference in trend between the good surface stations and the bad one. The adjustments performed by GISS did match the good stations and people like Tim Lambert over at Deltoid portrayed this as justification of the GISS output.
Zeke et al also seemed to recognise some difference in trend in 2011 if I understand this correctly
http://wattsupwiththat.com/2011/12/05/the-impact-of-urbanization-on-land-temperature-trends/
On the other hand and apparently based on a similar subset of the good stations the Menne paper came to a different conclusion and that there was no difference between good stations and bad stations. This paper has been used as a justification for homogenisation. At the time Anthony Watts however suggested that the paper actually used homogenised records for the comparison so the justification itself may be circular reasoning.
http://wattsupwiththat.com/2010/01/27/rumours-of-my-death-have-been-greatly-exaggerated/
The recent changes done by GISS apparently took out an urban adjustment and input the homogenised record. The original urban adjustment may or may not have been valid but it did apparently offset some issues caused by changes to siting over time.
http://icecap.us/index.php/go/new-and-cool/big_greens_untold_billions1/
The majority of the US stations are sited poorly so it is likely that if there is a difference between poor stations and good stations then homogenisation will have introduced a trend. The value of homogenisation is still contentious.
http://wattsupwiththat.com/2014/01/29/important-study-on-temperature-adjustments-homogenization-can-lead-to-a-significant-overestimate-of-rising-trends-of-surface-air-temperature/
Not very happy about the comment on using ocean heat content as a better system diagnostic.
While it is trivially true that if we could measure OHC accurately it would be the best estimator of heat in the system we fail because no one has a way of measuring the heat in the oceans accurately and the capacity of the oceans to absorb heat is so large that the amount of change is vanishingly small.
This leads to the ridiculous fact that people can claim to have have huge amounts of heat vanish in the depths of the ocean ,with a straight face.
A surface temperature measure is indeed a much more useful object apart from the fact that it is refusing to toe the party line on climate sensitivity.
Please do not insult intelligence by describing OHC as a better diagnostic indicator when it is impossible to get a diagnosis with it.
JD: “Don’t have a lot of confidence that Goddard’s numbers are correct.”
A lot of the “Final” data is estimated in the monthly USHCN as I’ve noted in earlier posts on this thread.
But even old data changes from day to day.
This post was just comparing the June 21 2014 file to the June 22 file.
http://sunshinehours.wordpress.com/2014/06/22/ushcn-2-5-omg-the-old-data-changes-every-day/
About 1900 changes were made for each month in just one day.
For Dec it was 1934 changes. Of those only 7 were for 2013.
“Steve, as I understand it the various anomaly records indicate global warming and that this is also manifest in the USA.
However, let us take one local that has been assigned as belonging to a temperature field identified by BEST, and other methodologies, as having warmed since the 1950′s.
I we examine a local, say for the sake of argument Roswell, New Mexico, and find that there is no change in the daily temperature, 1-365, between 2013 and 1950, does this mean we cannot trust absolute temperatures recorded by thermometers in the past, the present or either?”
No. one thing that people dont appreciate is that the temperature FIELD generated by BEST represents an expectation.
Here is roswell
http://berkeleyearth.lbl.gov/stations/170013
Now, one thing to note is that the station is actually multiple stations. It has moved several times.
The raw anomalies show a trend of 1.8
After QC the trend is 1.7
in 4 cases the months were removed because the had less than 20 days. on 4 days values were removed because the same value
( say 10C) was repeated day after day after day.
So QC cooled the record.
So much for adjustments that only warm the record
The next step is to find breakpoints. there are two kinds of breakpoints
A) a physical change to the station, like a station move, or an instrument change. Rather than adjust the station before or after the change we say ‘its a new station’. it used to be at 100 meters
ASL, now its at 500 meters, thats a different station measuring temp at a different location and since we model temperature as a FUNCTION of location, its a new station
B) empirical breaks. Where all the neighbors go up and it goes down. Again, we dont adjust the record to fit its neighbors, we
merely break the record. Empirical breaks can happen because of
undocumented changes to station.
After breakpoint the trend is down to .8C
The regional expectation is .76.
What does that mean. It means you take all the stations in the region. You then break the records when stations move to get the real count of stations. Note that GISS does the exact opposite.
They try to average stations together into long records. they also
adjust station records fiddling them up and down to adjust for
instrument changes, TOBS changes etc. We just split the record and say ” treat this one times series as 2 or 3 etc”
Next we use that data to create a surface for the region. That surface minimizes the distance (error) of the surface to the raw sliced data. So there isnt any detailed looking at individual sites and trying to fiddle one up or another down.. there is just the data, the raw data, sliced into time series where we have evidence that something material changed at that station.
Then a surface is fit. This surface is the regional expectation.
This is our best estimation of what the temperature would be at
all the UNSAMPLED locations in the area.
So, to your question. What does this do to trust? Well trust can be tested pretty simply. Take the 40000 stations. use 5000. create the field. then check the held out data versus the expectation.
The field gives you a prediction about what you will see in the held out data. your predictions will be accurate. That is all one can say.
“As I see it, if we are able to show there is no statistically significant between Tmin and Tmax, over 30 day periods, between 6 decades, of a station in an assigned temperature fields, then no warming.
Can we use actually station recordings, separated by 60+, to test your reconstruction?”
Not sure I understand your question. However, one thing you find is that the field or expectation is a estimate with an interval.
When you look at actual local stations you must find those that deviate from the prediction. That is even with a 95% confidence interval you’ll find at least 2000 stations that look ‘wonky’. of course the law of large numbers ensures that this wonkiness is small, but there is no getting rid of it. Since temperature is modelled as a function of location, you will always find weird locations where things like inversion layers, a nearby body of water, persistent wind, drive the temperature of that site as opposed to latitude and altitude driving the temp.
:Not very happy about the comment on using ocean heat content as a better system diagnostic.”
go eat some chocolate. you’ll feel better.
“Now, one thing to note is that the station is actually multiple stations. It has moved several times.”
Steve. I do not care about moves or the particular way you generated the anomaly field; I only care is there is a method to test the field generated, in an unambiguous way or not.
If the thermometers used in 2014 and in 1950 were calibrated then you should see >1.5 degrees in the absolute temperature, in the form of (Tmin+Tmax)/2.
If that is not the case then the field generated is probably not real and is an artifact or we have a case where we have managed to pick an artifactual local. So all I have to do is find a station in a BEST field, and see how many locals match the BEST result, in diverse areas.
If the change in daily absolute temperatures of thermometers in a field in 1950 and 2014 is trivial, when the field says it should be large, it suggests you have missed a systemic bias.
angech,
We don’t measure ocean heat content, we measure the rate of change in ocean heat content. This is not a distinction without a difference. Currently we probably are measuring the rate of change of heat content in the upper 700m of the world ocean more precisely and more accurately than we are measuring the metric we call the global average temperature. I have somewhat less faith in the data below 700m, but the system can only get better.
I suggest you read Cazenave, et.al., 2009 to see how ARGO OHC data, GRACE mass data and satellite sea level altimetry correlated from 2003-2008.
Over time I think ARGO and OHC will tell us a great deal about heat in the system and global temp anomaly but I don’t think we are there yet. Too small a sample size, too short of a historical record period and too wide of error bars as a result. For my money, Global SST offers better sample size, a long historical record, and being boundary-coupled ocean/atmosphere it represents the most likely proxy for global climate change and heat in the system.
It is not perfect and there are still problems associated with historical data collection, current movements, and short term solar effects but I think it is the most workable of the lot and is reasonably consistent with RSS and UAH over the sat record period. If I had to pick only one it would be SST.
Dear Zeke
I want to let you know of my new book for your perusal.
‘The Deliberate Corruption of Climate Science’
Thank you.
Tim
Historical Climatologist
PS My website is
“Steve. I do not care about moves or the particular way you generated the anomaly field; I only care is there is a method to test the field generated, in an unambiguous way or not.”
Yes.
take the US for example.
for the US there are some 20000 stations.
Build the field using 5000 stations for example.
Here is what you do to build the field.
1. Regress out the effect of latitude on temperature.
2. Regress out the effect of altitude
3. Regress out the seasonality.
What that gives you is the deterministic component of temperature. This is a continous surface, that is you can pick any
arbitrary point in space and time and produce the expected
temperature.
Next,
subtract the deterministic component from the observations.
This leaves a residual. The residual is the weather.
interpolate the weather.
Add surface 1 to the weather and you have the expected temperature.
take your 15000 out of sample stations and test your prediction.
You can also test when you find new data that has been recovered.
The rest of your comment doesnt make any sense, perhaps you can re-state
May I ask a simple question.
If a temperature field suggests that there is has been an increase of 1 degree at a local, between 1950 and 2013, then what would one expect the difference in the actual temperature averages of two stations, within that field, in 1950 and 2013?
De Witt, Steven did say OHC , not rate of change in OHC and most other people who comment on this issue seem to use the term OHC, not rate of change. The surface temperature record is not a rate of change record, though a composite. As you say the true OHC measured under 700 m is extremely unreliable.
GRACE is a particular bugbear (eats lots of chocolate) as it is the best way to try to measure sea volume and ice volume that we have but it is extremely prone to the same Heisenberg problem. The more we micromanage the data input the wider the uncertainty in the actual output.
The fact that they had widely differing models of interpretation by the various groups using their data over the last 10 years, the adjustments for glacier rebound that conveniently help boost the level of estimated sea rise ar a time when the purported increase was reducing, the rapid recalculation when one set of data said the Antarctic was gaining ice a few years ago do not help.
As said in the past GRACE is the best model we have but need a lot of adjusting.
For the average of Temps doc it would depend on the elevation of the two
Stations and the latitude of each. The US spans
25 degrees of late with a difference of around 10-15c or
. 5c per degree.
The monthly error in temp is around 1.6c two sigma
But your question as posed is still confusing.
You specify a trend for the grid and the ask about
The average of two temperature series.
Re: angech (Jun 24 17:35),
Everybody says OHC, but if you look at plots, you don’t see a number that reflects the total heat content like you can with temperature (unless it’s been converted to an anomaly). OHC is always an anomaly because we don’t, and likely can’t, know the total heat content. You would need to know in detail how the heat capacity varies starting from absolute zero. Good luck with that.
DocMartyn (Comment #130463)
June 24th, 2014 at 4:54 pm
May I ask a simple question.
If a temperature field suggests that there is has been an increase of 1 degree at a local, between 1950 and 2013, then what would one expect the difference in the actual temperature averages of two stations, within that field, in 1950 and 2013?
——-
It took me a couple of readings of Moshers reply to figure it out, but the answer is no. Here “subtract the deterministic component from the observations.
This leaves a residual. The residual is the weather.”
The deterministic component, 1Ëš, is too small to be seen in two samples. “Weather’ variations swamp the deterministic component. They are measuring under the curve…..
Right, so we don’t know the true OHC , which is a very large nebulous figure. We measure one small substrate of it imperfectly with Argo floats which are rapidly diminishing in number but can be modelled and no doubt are from their last legitimate working day.
Then we take the perceived change which is a very small fraction of the total OHC, say millions of times smaller, measure it in Hiroshima
Units , not 0.00001 of a degree Celsius, and say we think this is a better system diagnostic?
Got it, thanks.
Whatever the result of all the study of the numbers today and the way in which calculations are done, what errors were made the thing you are all missing is that the fundamental ongoing issue is data quality! Assuming we debate and eventually conclude what the correct methodology for handling the data are in terms of computing averages, etc the fact remains that every day as new data are entered and things change (however those changes may come about for whatever reasons) if you are depending on those numbers for serious work you need to have tools to insure data quality.
What does that mean? It means that the NOAA and other reporting agencies should add new statistics and tools when they report their data. They should tell us things like:
a) number of infilled data points and changes in infilled data points
b) percentage of infilled vs real data
c) changes in averages because of infilling
d) areas where adjustments have resulted in significant changes
e) areas where there are significant number of anomalous readings
f) measures of the number of anomalous readings reported
g) correlation of news stories to reported results in specific regions
h) the average size of corrections and direction
i) the number of various kinds of adjustments, comparison of these numbers from pervious periods.
What I am saying has to do with this constant doubt that plagues me and others that the data is either being manipulated purposely or accidentally too frequently. We need to know this but the agency itself NEEDS to know this because how can they be certain of their results without such data? They could be fooling themselves. There could be a mole in the organization futzing with data or doing mischief. Even if they don’t believe there is anything wrong and everything is perfect they should do this because they continue to have suspicion of their data by outside folks who doubt them.
This is standard procedure in the financial industry where data means money. If we see a number that jumps by a higher percentage than expected we have automated and manual ways of checking. We will check news stories to see if the data makes sense. We can cross correlate data with other data to see if it makes sense. Maybe this data is not worth billions of dollars but if these agencies want to look clean and put some semblance of transparency into this so they can be removed from the debate (which I hope they would) then they should institute data quality procedures like I’ve described.
Further of course we need to have a full vetting of all the methods they use for adjusting data so that everyone understands the methods and parameters used and can analyze, debate the efficacy of these methods. The data quality data can then insure those methods appear to be being applied correctly. Then the debate can move on from all of this constant doubt.
As someone has pointed out if the amount of adjustment is large either in magnitude or number of adjustments that reduces the confidence in the data. Calculated data CANNOT improve the quality of the data or its accuracy. If the amount of raw data declines then the certainty declines all else being the same. The point is that knowing the amount of adjustments, the number of adjustments helps to define the certainty of the results. If 30% of the data is calculated then that is a serious problem. If the magnitude of the adjustments is on the order of magnitude of the total variation that is a problem. We need to understand what the accuracy of the adjustments we are making is too. We need statistical validation continuing (not just once but over time continuing proof that our adjustments are making sense and accurate).
In academia we have people to validate papers and there is rigor applied to an extent for a particular paper for some time on a static paper. However, when you are in business applying something repeatedly, where data is coming in continuously where we have to depend on things working we have learned that what works and seems good in academia may be insufficient. I have seen egregious errors by these agencies over the years. I don;t think they can take many more hits to their credibility.
i wonder did steve goddard ever get the apology he was due from the op and a few commentators on this thread regarding actual data used vs what was claimed ?
bit chilly,
If you want to make a point, your going to have to make it directly. This thread is 3 years old….
Still a good thread