I am marveling at inline responses to comments at RC. Get this one:
# B.D. Says:
12 November 2008 at 9:46 AMYou would have been far better off simply saying that you found an error, are working to correct it, and the new data will be posted at that point. End of story. Instead, while complaining about much ado about nothing, you actually make much ado by: 1) incorrectly placing the blame on NOAA instead of your own processing algorithm. You can accurately model the atmosphere into the future but you can’t detect that a +13C anomaly might be a red flag? Once you take a product and use it to produce a new product, YOU take responsibility for that new product. And 2) ludicrously blaming “temperature observers†for heavily wanting to find something wrong. There are nutcases on both sides of the AGW issue, but legitimate temperature observers want the data to be correct, whether is pro- or anti- AGW. Maybe the “auditors†made your molehile into Mt. Washington, but your blame-deflection game elevated it to Mt. Everest.
[Response: I’m finding this continued tone of mock outrage a little tiresome. The errors are in the file ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean.Z, not in the GISTEMP code (and by the way, the GISTEMP effort has nothing to do with me personally). The processing algorithm worked fine. Multi-degree anomalies are not that unusual (checks are made for unphysical outliers which wasn’t the case here). I daresay they’ll put in more checks now to deal with this specific issue, but you can’t check for all possible kinds of corrupted data in the input files ahead of time. Science works because people check things – against expectations, against alternate records, against other sources of data – this is what occurred here. – gavin]
Yes. Gavin does seem hellbent on turning the molehill into an even bigger mountain. If he keeps this up, maybe the mountain can turn into a volcanic eruption of Krakatoa like proportions which would then lower the GMST. . .
But, back to the comment.
I wonder what sorts of outliers Gavin considers “unphysical”? What does Gavin mean by, “The processing algorithm worked fine?”
It may have worked as designed. But the design must not catch even the most egregious data glitches. Based on other factoids Gavin posted, before GISS yanked the October data and their October anomaly report, as many as 10% of the stations used to compute the October anomaly had held over temperature values since September.
Here’s a news flash: Station’s don’t move around. The glitches affected stations in Siberia (among other places.) Multi-degree outliers in, for example, Turuhansk are highly unusual.
I downloaded the full set of Turuhansk monthly temperatures since 1881:
- Fraction of observed anomalies that are larger than 10C: 0.4%
- Fraction of observedanomalies less than -10C: 1.4%
- October 2008 Turuhansk anomaly pre-correction: 13.53 C.
- Fraction of observed with absolute values greater than 13 C: 0.00%. (That is none out of 1480 observations.)
- Temperature change from Sept 2008 to Oct 2008, pre-correction: 0C. Full range of temperature changes from Sept-October: -2.8C to -19.1C. Note: A change of 0C has never happened.
In short, the data processed by GISS was chock-ful of unprecedented events.
Had these unprecedented events occurred, that would have been real news.
As it happened the files contained errors. These errors could have been caught with quite rudimentary QA tests, involving a few “if/then/else” test comparing the data in the files to readily available statistics describing the range of anomalies and temperature changes associated with a particular station or a particular region of the earth.
If a data record is showing numerous totally unprecedented temperature anomalies along with numerous unprecedented 0.0C temperature changes from one month to the next, any halfway decent script ought to catch that.
BTW: Does anyone have any clue why Gavin thinks the tone is mock outrage? The tone I hear is amazement and laughter!
===
As I was getting ready to publish, I got a note from SteveM. He also appears to perplexed by the idea the algorithm worked fine.
in spanish:
¡¡¡¡¡que papelón!!!!!
Ya Lucia I checked the Russian site SteveMc referred to before the Giss site was taken down. The average delta between sept and oct was 17C with a 1 sig
of 3C ( or so). Quality control is an ongoing process. Especially when you throw a big pile of data into a old convoluted piece of code ( gisstemp is mess)
It astounds me that gavin treat this so flippantly. It’s like he never heard of IQC. Now the funny thing is if you look at the sites which Hansen and Giss EXCLUDE in the USA, they are all sites that show too much cooling.
coincidence of course.
At my blog I noted the same bit of whining Steve did in re (lack of) resources. To me this whole thing sounds remarkably similar to the Microsoft vs Open Source debate.
http://www.di2.nu/200811/12.htm
They have a serious quality management problem. They don’t understand production environments. They think errors are ok if you correct them. They don’t understand your reaction should be to track down the source and eliminate the cause – and without giving rise to more. It confirms one’s view that GISS is run by production environment amateurs, and the more Gavin digs, the more he proves this to be the case.
Try running anything that has legal liabilities and practice insurance like this, and you’d be fired, sued, and unable to get practice insurance in very short order. All Gavin is proving is that neither he nor GISS understands this.
Re: #6596 I believe you meant Gavin not Steve.
On the subject of OpenSource, I was thinking last night that perhaps this would be a good time to create an OpenSource version of the temperature database(s). If one could get the raw data I think there would be enough interest and expertise to maintain an alternative to the GISS and HADCRUT data sets and perhaps the satellite data as well.
I continue to be amazed that Gavin’s comments can easily be taken to mean that all GISS/NASA work relies on persons outside the organization for Quality Control. What about the years of published GISSTemp results that haven’t been looked at?
Surely the results of GCM runs aren’t treated in this manner? The time scale needed to actually check that huge bunch of numbers is enormous.
Not to mention the thousands of times we’ve been told that we aren’t Certified Climatologists because we haven’t Published in Proper Peer-Reviewed Certified Climatologist Journals.
Schnoerkelman (Comment#6602) November 13th, 2008 at 4:12 am
Yes, any number of people / groups could easily make data storage and reduction codes. It’s a straightforward process.
The problem is all these would very likely get different answers because of different averaging methods. GISS/NASA, and maybe all other organizations, do not report simple area-averages of the raw data. Models representing the organization’s ideas about temporal and spatial and ‘local’ effects are instead applied to the raw data.
The present situation would appear to be very rational compared to the case of having even more ideas about what The Global Mean Surface Temperature should mean.
Dan Hughes,
I don’t know why, but when I think of GISTEMP, “rational” is NOT the first word that pops into my head.
Mike
I remain equally amazed that GISS/NASA employees, and others at RC, almost always throw out presumptive motives against anyone who dares attempt to discuss data quality issues. It happens every time. Back when Watts started the surface stations work it got really ugly.
How can any engineer or scientist not be concerned with the quality of the data? It’s beyond my ability to comprehend. And for me data quality applies also to the numbers calculated by all software; from the most simple and direct data analysis programs, the GCMs, and even the post-processing software.
For me, I don’t care if the temperature is going up or going down. I just want to know that it’s been correctly and accurately calculated.
What physical significance any Global Mean Surface Temperature might mean is another issue.
Dan–
I assume you are referring to Gavin saying this:
He didn’t provide a link, but evidently, the structure of the paragraph would seem to suggest that the authors of the blog post (aka “we”) identified this motive in some past scholarly work they posted at their blog.
How about this
Here, of course, presumptive disappointment is assigned to the “many”.
Of course it’s good if this error does not recurr. What does he think people have been complaining about? The series of errors in GISSTemp! But somehow, the idea that people are correct to complain about the frequent errors in GISSTemp! Sheesh.
So why does NASA have any credibility?
We have their own admission that their quality control is complete garbage. We have an admission that their management skills are terrible. There is an ongoing, long term problem with their unwillingness to have anyone else check their work or even know what methods they use.
Every time someone actually gets a look at what goes on behind the curtain the revelation is extremely unsettling. When do honest, rational people conclude that perhaps the omnipotent, omniscient wizard is not all he claims to be?
Why aren’t people asking the obvious questions?
Lucia,
Gavin can’t make the difference between what he thinks he’s hearing and the actual tone of the reactions all around because he is in full cognitive dissonance. GISS’ credibility is now seriously in question, and having taken on the role of Hansen’s guard dog, any and every reaction is threatening to his deeply held views on AGW. My guess is that GISS was so slow in reacting because in good part is because the data appeared simply to further support Hansen’s positon that we are facing runaway temperatures, only more so…
Considering all the placement problems that Antony Watts and volunteers have identified for surface stations in the U.S., I wonder about the credibility of stations in Russia and other countries. Has NASA ever visited these sites and calibrated their devices? I doubt it. Hell, it is doubtful this has been performed in NASA’s own back yard. Yet we may be losing our freedoms and jobs because of these readings.
Further, at CA and WUWT it was pointed out that many Russian and other stations had 999 (missing) readings, and further that the number of stations
in use has declied greatly.
Can the skeptics come up with their own temperature analysis? Since they think they can do it better and all 🙂
Come on guys, Skeptemp.com is available!
Boris,
“Can the skeptics come up with their own temperature analysis? Since they think they can do it better and all ”
Will you give me half-a-million bucks?
That’s what Gavin figures it ought to cost.
Boris– Erhmm… Christy? Spencer?
GISS is funded to do a particular job. It happens to be a visible one; if they mess up people notices and talke about it.
When Spencer and Christy have messed up or RSS for that matter people notice but it attracts much less flack though. Is GISS actually funded to do GISTEMP, i.e. is it a line item in their budget?
Phil–
Do RSS and UAH mistakes attract less flak? I’ve read unfair slams suggesting either Spencer or Christy skew their data. Some of those slams suggest religious views are involved. (I can’t remember which one is the evangelical. I guess I could google, but I’ll admit I don’t want to invest the time.)
Also, it appear the problems with the satellites were of a more complex nature than this month’s problems at GISSTemp. This affects the character of the complaints.
That said: Yes, some of the specific complaint about GISS have been unfair. This was a blunder; they should institute at least quarterway decent QA. Gavin needs to stop posting excuses that just make people’s eyes roll. But, there is no great conspiracy: It’s just a garden variety screw up.
On funding: I don’t know if NASA has a line item specifically for GISSTemp.
Maybe NASA is given a block of money and granted wide discretion to spend as they wish or maybe they are directed to create GISSTemp. Either way, NASA GISS spends US taxpayer nickles. So, people get to observe and comment on their product.
If taxpayer money is spent under a program that gives NASAGISS wide latitude, but NASA GISS managment doesn’t have their heart in it, and don’t wish to do QA, maybe they should drop the task and focus on other things. After all, if you are granted wide discretion, why take on a task you don’t value, do it half heartedly, and get flak when things go wrong. (I think Gavin suggested NASA could elect to use the money on other task. Maybe they will. Then climate blog addicts will have to rely on Hadley, NOAA and the satellites. )
If NASA/GISS is directly instructed to do this by an outside funding agencies, then NASA/GISS should also do the task correctly.
Either way, if they take this task on, I don’t see why onlookers shouldn’t point out NASA-GISS bungled this month. I realize this might be painful for some at NASA-GISS, but that’s what comes from working in an area that is highly visible.
“Boris– Erhmm… Christy? Spencer?”
Yes, and Christy and Spencer’s dataset was in error for YEARS..by an enormous amount.
“I don’t see why onlookers shouldn’t point out NASA-GISS bungled this month.”
Gavin’s whole point is that mistakes are made and that they are corrected quickly. But remember that the data is also released quickly. Point it out, but “These errors could have been caught with quite rudimentary QA tests” is just meaningless rhetoric, and I’m afraid you know it.
The criticism of GISS is a way to criticize mainstream scientists without having to do any of the work that they do. GISS is also a proxy for AGW in general–if GISS is wrong, then we can safely ignore AGW. But logic was never an AGW skeptic’s strong suit…
I await skeptemp.org and the skeptics who know how to do it better and faster than GISS. Ask Heartland for the $500K.
Boris–
Gavin’s point may be that mistakes are corrected quickly. The difficulty is that the mistakes were so obvious, and noticed by outsiders.
So, logically, this is no credit to GISS.
Boris,
Does Heartland have $500K to spread around? I figure they can afford the conference once a year and that’s about it.
Oh… also, how long was that year 2K issue lingering in GISSTemp before SteveM found it? I honestly don’t remember. By the time I jumped into the whole climate blog addiction syndrome, the discover and resolution were in the past. But I know that was more difficult to tease out and found longer to diagnose.
I more or less trust Hadley, GISS, HadCrut, NOAA, RSS and UAH equally. None are perfect at what they do. With luck, the ARGOS floats will soon have a monthly product and we can gripe about that too! 🙂
Boris,
You’d love Hearland to fund 500K for temperature monitoring.
Think of the whining opportunities in the AGW camp!
They’d all be in a constant state of rapture.
Anyway, as I’ve grown older, I’ve come to the conclusion that analogies are generally a waste of time, since no situation perfectly mimics another, but I’ll do one anyway.
Your humorous little suggestion that skeptics fund and do their own temperature monitoring is sort of like telling Michael Moore that if he doesn’t like the way the military operates, he ought to use his movie royalties to fund his own Army.
Boris (Comment#6704) November 14th, 2008 at 5:56 pm,
True. But the reason was that the source of the error had not been identified. Very soon after the orbital drift problem was identified, UAH revised their algorithm and recalculated the data. They didn’t spend time whining about how they were picked on and they didn’t have the budget for proper QA/QC. The competition between UAH and RSS has been fruitful in identifying errors in each others methods. That’s because they are both relatively open with their methods and they don’t try to shoot the messenger when an error is identified, they just fix the problem. Converting microwave brightness temperatures to real atmospheric temperatures is a far more daunting problem than compiling a surface temperature record from actual measured temperatures.
Boris,
I have no argument with Nasa or anyone else publishing preliminary data.
Marking it as preliminary data subject to change. And provided they issue change notices. Change notice is critical and can, with the simple addition of RSS, be made automatic. Change notice is important for people who used the data previously. Finally, nobody should be doing press releases on preliminary data. Blog about it if you have to.
I also have no problems with preliminary data provided clear notices are made. Based on what we’ve learned during the kerfuffle NASA should consider a footer with
1) A timestamp.
2) A notice indicating the total number of stations used in the anomaly computations for each of the previous 12 months.
3) A link to archived reports. (This would permit people to check.)
4) RSS feed and a subscription button to inform people of the update.
All this is possible by small additions to the code, and by creating a data base. Heck, 1, 3-4 could be provided by posting the updates using WordPress! Number 2 requires them to add a line of code to the file reporting the number of stations.
Of course, they aren’t required to do any of this. But it would provide context.
From the GISS site
So what you had was never meant as a production product, only an internal one to check their models. It’s a hack in the old sense of the word. And one done post facto which leads to the problem of having an expected answer (but not ground truth) in hand while your building the model. It’s morphed into something else. Trying to go back and “do it right” is extremely difficult. I’ve been in that bind a number of times. Even if you did everything on a shoestring, management assumes the product was perfect because it “works” and doesn’t want to spend their funds on “old stuff”. Hansen’s use of the product to support his pronouncements makes it even worse because he would have to admit that they are based on a shaky foundation. That would be close to professional suicide at this point.
GISS could certainty improve the product. As long as they rely on publishing monthly products before checking, they will continue to experience PR problems.
If GISS as a whole, Gavin commenting at RC or those in the peanut galleries of blogs, forums etc. don’t understand this, then …. so be it!
The specific problem experienced this week could have been avoided easily with routine QC scripts that
1) Compared individual station temperatures to their September values.
2) Flagged all “no change” temperatures.
3) Compared anomalies to extreme values for each station. (GISS already uses all station data to create the product, so computing the max/min/ std. deviation is trivial.)
4) Set flags when a station shows either a) no change since last month, b) the smallest or largest monthly change ever or c) the maximum or minimum monthly change ever.
Gavin, who tells us he has nothing to do with GISSTemp claimed GISS checks for unphysical values. Values well outside the range of anything ever observed are were not detected.
I would suggest that if Gavin has nothing to do with GISSTemp, and he is making claims that are contradicted by the evidence, then we should conclude that its quite plausible he has no idea what GISSTemp does or does not do to ensure quality of their product.
However, based on what Gavin reveals, and what we have seen, it appears they do very little to ensure quality. They don’t even seem to do consistency checks we would expect of undergraduate students performing lab experiments.
This may be unfair to the hardworking computer coders making the GISSTemp product, but it’s the picture Gavin paints for them.
Lucia, I found Gavin’s remarks the most remarkable of the whole incident. As you state, it is hard to imagine a better (or worse way) to demonstrate that GISS did not have the QC that has been repeatedly claimed. I remember posting on RC when Watt’s had just started surfacestations. The flack I got for daring to sugest that even the most miniscule of data integrity problems meant I was an unscientific heretic. This is not a large exaggeration on my part, especially looking at the recent fiasco. The good people at RC assured me I knew nothing of data quality control of GISS by even suggesting something might be amiss with GISS. I just pointed out that with the anomoly of about .6C in a century, we needed to have accurate and precise numbers if the world was going to engineer some solution. The patrons of RC assured me that 0.01C was it. Pretty funny claims, I dare say.
A description of GIStemp QA. Please dont make me read the code again.
PLEASE for the love of god.
Result: v2.mean_comb
Step 1 : Simplifications, elimination of dubious records, 2 adjustments (do_comb_step1.sh)
———————————————————————–
The various sources at a single location are combined into one record, if
possible, using a version of the reference station method. The adjustments
are determined in this case using series of estimated annual means.
Non-overlapping records are viewed as a single record, unless this would
result introducing a discontinuity; in the documented case of St.Helena
the discontinuity is eliminated by adding 1C to the early part.
After noticing an unusual warming trend in Hawaii, closer investigation
showed its origin to be in the Lihue record; it had a discontinuity around
1950 not present in any neighboring station. Based on those data, we added
0.8C to the part before the discontinuity.
Some unphysical looking segments were eliminated after manual inspection of
unusual looking annual mean graphs and comparing them to the corresponding
graphs of all neighboring stations. See CLEANING NOTES for further details.
Result: Ts.txt
I have raised issues with this process BEFORE. I asked gavin for a description. ( WRT the elimation of northern california stations in the early part of the 20th century and the total elimation of Crater Lake station in Oregon.. no explination or code was forthcoming )
But lets look at the “CLEANING PROCESS”
http://data.giss.nasa.gov/gistemp/sources/cleaning.html
As noted the “cleaning process” resulted in no difference. It appears however that they rely on a manual process.
NOAA on the other hand have an automated process. Note this is for
USHCN ( us data)
http://www.ncdc.noaa.gov/oa/climate/research/ushcn/
First, daily maximum and minimum temperatures and total precipitation were extracted from a number of different NCDC data sources and subjected to a series of quality evaluation checks. The three sources of daily observations included DSI-3200, DSI-3206 and DSI-3210. Daily maximum and minimum temperature values that passed the evaluation checks were used to compute monthly average values. However, no monthly temperature average or total precipitation value was calculated for station-months in which more than 9 were missing or flagged as erroneous. Monthly values calculated from the three daily data sources then were merged with two additional sources of monthly data values to form a comprehensive dataset of serial monthly temperature and precipitation values for each HCN station. Duplicate records between data sources were eliminated. Following the merging procedure, the monthly values from all stations were subject to an additional set of quality evaluation procedures, which removed between 0.1 and 0.2% of monthly temperature values and less than 0.02% of monthly precipitation values.
GHCN.. the data GISS relies on has QA. described in this 1998 paper
http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/images/ghcn_temp_qc.pdf
Historically, the identification of outliers has been the primary emphasis of QC work (Grant and
Leavenworth, 1972). In putting together GHCN v2 temperature data sets (hereafter simply GHCN) it was
determined that there are a wide variety of problems with climate data that are not adequately addressed
by outlier analysis. Many of these problems required specialized tests to detect. The tests developed to
address QC problems fall into three categories. (i) There are the tests that apply to the entire source data
set. These range from evaluation of biases inherent in a given data set to checking for processing errors;
(ii) this type of test looks at the station time series as a whole. Mislocated stations are the most common
problem detected by this category of test; (iii) the final group of tests examines the validity of individual
data points. Outlier detection is, of course, included in this testing. A flow chart of these procedures is
provided in Figure 1. It has been found that the entire suite of tests is necessary for comprehensive QC
of GHCN.
Look at figure 1. It indicates that there is a check done to make sure the reported temps comport with history of the region.
Time for NOAA to FREE THE CODE. that way many eyes can check it
Steve moscher:
The NOAA text you post mentions outlier detection, and then mentions the special stuff they do.
If NASA has done nothing beyond outlier detection on the output from NOAA, and thrown flags, they would have avoided this month’s embarrassment.
I don’t expect NASA to do everything. But they should at least do cursory, routine checks on data they use as input.
Oh well…
I think Gavin is neglecting a small but important distinction, and this goes to Phil’s concern about unequal treatment…
The situation here is more than the incorporation of some bad data. Sure that incorporation was an automated process, but the decision to then make a posting on the GISS site highlighting that the Earth was REALLY warm (despite the sun etc, etc) changed the nature of the story. This is not just an automation failure; it passed into human hands. Human hands at GISS saw “big news” and then rather than sniff around, they assumed that the news was real and posted about this record breaking event.
Jon–
There was no written announcement at GISS discussing the October anomaly. Yes, it would have been a record for October– but they didn’t say anything about that. They just posted the results.
But, of course, we all know they are looked at.
lucia (Comment#6780)
This is, I assume, the quality checks that Hansen does if the are still doing it as he described in “GISS analysis of surface temperature change”.
If I’m following it correctly it wouldn’t have caught this type of error, since if occurred at adjacent stations also it wouldn’t be seen as an error
lucia (Comment#6780)
This is, I assume, the quality checks that Hansen does if the are still doing it as he described in “GISS analysis of surface temperature change”.
If I’m following it correctly it wouldn’t have caught this type of error, since if occurred at adjacent stations also it wouldn’t be seen as an error
BarryW– 5 σ ?!!! Five sigma from the long term mean is happens something like 1 in 10 million times. There is at most 200 months of data. If that’s the criteria, they thought about the issue and then concocted a criteria that is absolutely nuts!
Plus, yes, if when all the stations make a mistake, they wouldn’t even catch that.
Seriously, they didn’t sit down and think about what is required to detect blunders while minimizing false positives.
I think the intent, such as it was, was to catch scribal errors (a 89 read as opposed to a 9 for example). There are some other checks for discontinuities but they seem to be more concerned with discontinuities from the mean as opposed to testing against specific months values. Maybe due to an implicit assumption of a trend in the data. There also assuming that QA is on the GHCN side.
Interesting note in the GHCN QA description:
So two months slips through, even when there is a significant change expected in the data.
BarryW–
On the one hand, I think it fine to assume some amount of QA on the GHCN side. On the other hand, Oh well….
DeWitt,
I wasn’t trying to slam the UAH analysis–and I think C&S are doing a good job with it–they helped to identify an error at RSS last year. My point was that errors crop up from unexpected areas. Perhaps GISS could have foreseen such an error, then again one could spend a lot of time thinking up ways that the data could come in bad–and still miss the next mistake.
Lucia and mosh,
I think your suggestions are good and I hope GISS will implement them.
Just as complaints have been made about the lack of statistical expertise or use of statistical experts in the climate science field, there is obviously a lack of engineering/process management expertise. I’ve done the types of processing that they’re trying to do and what is acceptable for a one-of is not the same as for a continuing process. And that applies to both organizations.
Boris–
It’s true GISS could spend a lot of time and still miss the next mistake. But that’s a red herring. The reason is the recent mistake falls in the category of mistakes any halfway decent QA script would flag. There are so many different sorts of errors that result in “frozen data” that it’s almost impossible to believe someone writing a script wouldn’t check and catch it.
But in this case, NASA didn’t implement the equivalent of “test 1” in a script driven QA system: are readings frozen? Even in the tropics where temperatures don’t vary much, having every single station in, say, all of central america, display frozen readings would at least warrant flagging for human attention.
The narratives steve m posts suggest NASA-GISS didn’t really spend much time trying to come up with a rational script driven QA system.
I’m sure GISS will implement one now, with luck it won’t cost $500K. 🙂
Obviously, GHCN has problems too. But one of the aspects of science is that more than one entity checks anything important to a result. The checks are supposed to be independent. I know the data processing isn’t a journal article– but independent checking is important in science, and NASA/GISS suggesting they don’t need to check because GHCN does the checking is beyond odd.
you hoi polloi are pathetic.