The Blackboard

E.M. Smith:

My major point has simply been that much of the available data is not used. It is dropped on the floor. You can call it “deleted” or “dropped” or “ignored” or whatever. It is still NOT in the GHCN data set. The pattern of these “droppings” is that high latitude and high altitude stations are dropped, while low altitude and low latitude stations are kept (with an ever increasing percentage at warm heat islands of airports). There is a clear ’survivor bias’ toward warmer stations (and with warming trends, like airports), with warmer winters, more heat islands, and lots of tarmac.

My response:
GHCN: High Alt, High Lat, Rural Feb 1, 2010, Ron Broberg
http://rhinohide.wordpress.com.....lat-rural/

Global Gridded GHCN Trend by Seasonal Offset Matching Feb 26, 2010, Jeff Id
http://noconsensus.wordpress.c.....-matching/

I also verified the dropout issue seemed to have little effect back when this was being discussed. No posts on it but if anything the stations cut short seemed to have slightly greater warming trend.

Ron–
That’s good! Too bad more people weren’t reading you in January when you first started posting.

Climate Audit had a number of post several years ago on the issue of station drop out. It’s left to the reader to do a search there…

GIStemp

“Hansen Frees the Code” Sept 2007 GIStemp
http://climateaudit.org/2007/0.....-the-code/

I followed up with my first recompile in May 2008
http://climateaudit.org/2008/0.....ent-148178

Lost interest until Nov 2009 when someone claimed that no one had ever run GIStemp to completion. (Guess, he is partially correct, I haven’t run the Steps4,5 which add the ocean data)
http://www.realclimate.org/?co.....ent-145586

Currently hosted here:
http://rhinohide.wordpress.com/gistemp/

Ron–
Getting Hansen to free the code was a good thing. One of the consequences is places like ccc were able to write emulators and can now play around and test theories like “the march of the thermometers”.

Other than that, I don’t see any connection with the 2007 Climate audit post and “the march of the thermometers” issue which relates to massive drop out of thermometers in the 80s-90s. Are you suggesting a connection exists?

I know that there are some folks working on CRUTEM from published descriptions, but I can’t find them at the moment. For some reason, I’m thinking of a French blog …

Anyway, here is a small piece:
http://iv-6.blogspot.com/2010/.....m3_04.html

Ron– Someone found an error in computation of CRU uncertainty intervals. I think they found that by examining CRU code and it’s been admitted to be true. I’ll look through my email.

Lucia: Are you suggesting a connection exists?

Sorry if I’m wandering off topic. The general temperature recompiles, reconstructions (GISTEMP, CRUTEM), and new constructions are addressed more to the second half of your post: Spin Off

These different levels of effots (recompiles, reconstructions, new constructions) were going on long before Smith’s claims were made and will continue long after they are laid to rest.

Now that I think about it, that description maps on to my 3-levels of confirmation post a few days back.

Level 1: Recompile the original sources
Level 2: Reconstruct from published descriptions
Level 3: New construction

When we create temp trends using new methods and new data (for instance: synop rather than climat), we have moved to Level 3.

The genesis is much further back.

If you look at the SPPI document page 11, it goes back to Ross McKitrick, circa 2000. He made the plot of station number overlaid on the simple mean of all the absolute temperatures. Perhaps this inspired EM Smith to go around making more of the same plots, years later.

But most bizarrely, there is an antecedent in the published literature. See Figure 2 in Willmott et al, GRL 18: 2249-2251 (1991). It’s the exact same sort of plot, in a paper about spatial sampling error. At least Willmott used spatial sampling, but why was he not using anomalies? How did this get published? By 1991, anybody in the field should have known better.

Meanwhile, it must be noted that EM Smith’s original claims were based on absolute temperatures, not trends. He was talking about hot/cold places, not warming/cooling places. You don’t need much quantitative analysis to dismiss that; only a knowledge of how anomalies are calculated. Dr Nielsen-Gammon weighed in forcefully on that, here: http://tinyurl.com/ykfy8aa

Tracing EM Smith’s evolution is interesting in itself. From his comments at WUWT, it’s clear early on that he looked at the GISS code for the reference station method but couldn’t figure out what it was doing, and that he didn’t read the relevant papers. He also had trouble working out that the choice of baseline when displaying the results doesn’t affect the trends. (An overly short baseline when using the CAM can yield poor results, but that’s an entirely different matter). And he thinks the slight changes the do occur when you add or remove a few stations is vindication of his idea, when they are not.

Lucia: I’m pretty sure you’re incorrect on the CRU error error.

The mistake was found without looking at anybody’s code. They simply took the published errors and data, and found they could not replicate the errors by using the equations given in Brohan (2005).

http://www.jgc.org/blog/2010/0.....rrors.html

Again, you can find errors like this by just reproducing the calculations for yourself, without proofreading or copying the other person’s code.

I posted at Chiefio in November and was particularly trying to establish what mechanism(s) were at play. I was not able to get E M Smith to describe the mechanism then and my own suspicion was that he was looking at correlations and had not actually established one.

However I am quite prepared to believe he has now found a mechanism but will want to see it described before I will accept it as a real issue.

What Lucia and other commentators are missing is that EM Smith is effectively investigating the impact of an unstable baseline on the GISS code and it will require people to run that code with data selections to be able prove or disprove his assertions.

As an illustration we are led to believe that a limited number of stations with lights = 0 have a significant influence on the remainder of the stations. What happens if those stations with lights = 0 change over time?

Obviously if all stations are dropped that would effect the calculation so where is the cut-off that allows no effect.

Andy: It’s a question of how over/undersampled you are in any given part of the world.

In the US, you’ve got more thermometers than you really need to describe the anomaly field, so you could randomly remove a few to little effect.

In some parts of Africa or the poles, you already have a low coverage of reporting weather stations, so if you remove a few, you could lose a lot of information. There’s little redundancy in the measurements. Hopefully the ones you have are not overly plagued by station moves and other discontinuities.

GISS explicitly considers this, and its error bars are computed solely on the basis of sampling error. CRU error bars include all sorts of other error sources.

clivere: if the reference station method leaves GISS somehow vulnerable to something, then it ought to show up when Zeke uses the CAM instead. But it doesn’t. The RSM can hiccup a little when a station drops or is added, but this doesn’t show up in the big picture.

Re: Ron Broberg (Mar 2 12:21), Fair enough! I don’t mind people going OT. I just thought you might be suggesting a connection.
Re: carrot eater (Mar 2 12:32),
I think there is another blog post somewhere. I seem to remember some guy used some weird consistency checking software (something I’d never heard of) and found something. I have no idea how to google to discover that.

Re: clivere (Mar 2 12:33),

What Lucia and other commentators are missing is that EM Smith is effectively investigating the impact of an unstable baseline on the GISS code and it will require people to run that code with data selections to be able prove or disprove his assertions.

I think that no one can prove or disprove his assertions because, to a large extent, we don’t know precisely what he intends to assert.

As an illustration we are led to believe that a limited number of stations with lights = 0 have a significant influence on the remainder of the stations. What happens if those stations with lights = 0 change over time?

Who knows? But this isn’t precisely the station drop out issue, is it? To answer, you have to check.

Re: Andy Krause (Mar 2 12:34),

Obviously if all stations are dropped that would effect the calculation so where is the cut-off that allows no effect.

Yes. This is why the issue is a big nuanced.

No one is saying that it is flat out impossible for dropping stations to affect the result. But some very, very strong statements have been made some intimating fraud. Meanwhile, we know that if there are plenty of thermometers, the first order effect of cutting thermometers is to make the monthly data noisy, not biased. This is particularly so if the anomaly method is used.

So, the first guess bias is likely to be small– but if not, we aren’t going to be able to easily guess the direction. Showing the bias is going to take some very careful work.

We also see that absolutely no work is shown to explain why bias would rear it’s head, in this case knowing which thermometers were dropped, which that remain, and the knowing the anomaly method is used.

Quick investigations by people accessing actual data and computing “pre” and “post” cut-off thermometers suggest that when the anomaly method is used, dropping these thermometers as occurred in the 80-90s is not resulting in bias.

If Smith and D’Aleo have done analyses that detect the bias, they now have a very tough row to hoe. They need to explain how any bias arises and show it does. So far, they have not. So, until they show it… well…. We are we we are: No particular evidence the march of the thermometers caused any upward bias in the surface record.

There’s a very interesting reconstruction here:

http://justdata.wordpress.com/.....te-change/

by Eugene Zeien.

He started with entire database of stations, selected only those with a history from 1900 to 2009, and graphed the raw data. He found a 1/10 degree C rise over 110 years, which is to say, no trend. There was a 60 year sine-wave peak to peak. Worth a look at this chart:

http://go2.wordpress.com/?id=7.....09sine.jpg

re: Andy Krause (Comment#35950) March 2nd, 2010 at 12:34 pm

Yes, it’s a continuum. If it makes a difference under some condition, ever at the end points, it has a potential to make a difference under other conditions. Don’t the different baselines make a small difference in even the trend? Aren’t GISS and CRU/UEA different because one includes data that the other doesn’t?

Even the area-weighting average seems to be a function of the grid size. These aren’t converged until the numbers are independent of the grid size.

There’s got to be some kind of implicit constraint at work here if dropping / adding data as a function of space and time doesn’t make a difference.

How would you design the scan of the steady-state fluid temperature distribution in a 3-D flow field in order to ensure accuracy, for example ?

All corrections appreciated. I’m way outside my baseline here.

Eugene Zeien’s analysis of raw data from long lived stations is worth a look.

I think you missed Roy Spencer’s previous post. Near the end of it, he says that he doesn’t think the dropped thermometers have much of an impact.

I’ve argued with Chiefio in the past, that he is claiming to disprove AGW with his findings. Namely that warmer winters and nights and higher latitudes is to be expected by global warming theory.

Reading his Hypothetical Cow post, I’m convinced that his statements have not been disproved, and that his analysis is a bit different.

Roy Spencer Feb. 20, 2001. Roy Spencer computes trends using data drawn from the NOAA-merged International Surface Hourly (ISH) dataset,

That Roy Spencer was ahead of his time. Nine years in fact.

I accept from the analysis posted above that the dropping of the thermometers had no effect. But I still do not understand why it is being done. Why throw away perfectly good data.
If I understand the science correctly the only way you could drop thermometers and still not affect the results is if you do a Principal Component Analysis and see which thermometers are redundant. But why do all that. Why not just log the data.
Also I would be interested in knowing what all these drops do the error bars in the resulting data. The reduction in spatial resolution has to have some effect.
Would be grateful for any input. Especially if my assumptions are wrong.

Peter,

Those results are… odd. It also worries me that the first half of his post is trying to calculate a global temp from absolute temperatures instead of anomalies, which is a bad idea for a number of reasons.

There are 782 stations in GHCN with a start date prior to 1900, an end date after 1990, and at least 100 years of data (note: this only counts a years where stations are actively reporting data, and uses the more restrictive wmo_id imod instead of just wmo_id). Spatially weighting the results based on station location (see previous post for details), we get:

http://i81.photobucket.com/alb.....ure134.png

Note that this chart uses v2.mean raw GHCN data.

If I measure the average deposits of ‘all the members’ of my bank, (to give the high ones a bonus or something) dropping 1/3 of the membership in my analysis may not do much to the average…

…but then I would not be analyzing ‘all the members’ of my bank anymore.

I think that’s an issue. For a deposit not to get consideration (for whatever reason) would have to be justified in some way.

(restating Ani’s post in different words)

Andrew

Peter Dunford and Zeke:

It looks very much to me that Eugene took the absolute temperatures, averaged them together, and only then subtracted out the baseline. He may or may not have gridded them on a 1×1 grid first, but it doesn’t matter. If so, that’s just a basic conceptual error, and similar to the that in the SPPI report.

Ani: Not all weather stations report on a regular monthly basis, in real-time. The WMO has picked out a number of good and well-distributed stations and encouraged the countries to report at the very least those results on a regular basis, but this is not the entire set. The countries could report more, and the US does, but not all countries go above and beyond in this respect. Some countries do less than they’ve been asked. So now and then, researchers go around and collect archived data from the non-reporting stations. Hence, the station count history has that hump.

If you look at Hansen’s 1987 paper, there is a huge station drop-off around ~1970. Of course, all that missing data got filled in later, so that the dropoff moved to ~1990. In the upcoming re-release of GHCN, you may see another round of fill-in, but I’m not sure to what extent.

Carrot,

His results actually looked similar to raw U.S. temps at first glance. Using a grid size that small (1×1) and a filter for long-lived stations would tend to result in U.S. stations dominating the results.

Think of it this way. If UHI exists, and it does, and you drop mostly rural stations, and you underestimate UHI when making corrections, then there has to be an effect.

Almost all (if not all) GISS stations are now airports and non-rural.

Unless the UHI corrections are perfect, the GISS temp will too warm.

Re: BillyBob (Mar 2 14:32),

Unless the UHI corrections are perfect, the GISS temp will too warm.

Not quite. Suppose UHI corrects are too large? Then GISStemp will be too cool.

Whenever you discover a bias due to an identifiable factor (e.g. UHI, TOBS etc.), and then try to correct, the correction itself could be too large or too small. The goal is to reduce the original bias due to that factor. But you really can’t ever know if you over corrected or undercorrected.

BillyBob,

GHCN was more rural (as a % of stations) post-1992 than pre-1992. Your airport reference makes me think you are referring to the U.S.; there are no stations dropped in the U.S. per se, so that is rather a moot point.
.
Carrot,

You might find this interesting. Looks like I was wrong

http://i81.photobucket.com/alb.....ure135.png

Yeah, I think the guy (Eugene) just doesn’t know how to calculate anomalies.

It’s a little sick, just how insensitive the basic picture is to these changes in processing. The WUWT crowd hates the word ‘robust’ for some reason, but that’s the appropriate word. So long as you don’t do something totally wrong, you’ll get about the same answer.

At most, you just change how noisy the series are.

Re: carrot eater (Mar 2 15:01),

The WUWT crowd hates the word ‘robust’ for some reason,

I think this is because sometimes people seem to suggest “robust” means “accurate” or “correct”. All it means is that given the range of choices that seem reasonable and information available, you don’t get wildly different answers. However, it’s entirely possible the information is just too limited. That’s a distinct possibility in climate science because if every pours over the same uncertain data 100 different ways, you could all get roughly the same answer, and all could be wrong. The bias could be in the data– but there isn’t a way to know it.

The other problem is sometimes the use of “robust” seems rather laughable. People can make a few changes in assumption; the result goes away. But that doesn’t seem to suggest lack of “robustness” to those claiming “robustness”.

WUWT started hating “robust” about here:

http://wattsupwiththat.com/200.....l-warming/

It only got worse as it became increasingly obvious that this “robust” test was code for “the answer we want”. For instance Mann et al’s reply to McIntyre and McKitrick’s PNAS comment on Mann et al 08. “Robust” is right in the title.

On a separate note, IJOC rejected McKitrick’s response to Gavin’s paper saying that his and DeLatt and Maurellis’ results were “spurious”:

McKitrick, Ross R. and Nicolas Nierenberg (2009). Correlations between Surface Temperature Trends and Socioeconomic Activity: Toward a Causal Interpretation.

What is really bizarre is that he showed that Gavin’s objections were just wrong, but the reviewers made all kinds of crazy demands but Gavin’s paper got a pass on the same problems.

http://sites.google.com/site/r.....edirects=0

Effectively IJOC readers will be left with the distinct impression that M&M 07 was proven incorrect, when in fact it was not, and one of the authors is being denied an opportunity to show that it was not.

good article, good discussion so far. i had a short exchange with chifio some time ago, and it was obvious that he does not understand anomalies, trend and the gridding process. he thought placing thermometers on mountains would cause the record to cool.
.
i just want to add two minor aspects.
.
the “Smith” affair shows a serious problem of the “free the data, free the code” approach:
people with little understanding of the subject have come to a complete believe in this mantra. they draw the (no logic) conclusion, that sources that provide code and data are telling the truth.
this is one of the reasons, why the barely readable chifio pages get so much support. all that data and code cluttered into the text, together with constant repetition of the “free data, free code” idea make him appear as a good scientific source.
(a significant percentage of the comments boils down to : “i don t understand what you are doing, but it is very good!”)
on the other hand pretty legitimate reasons to hold back data or code are seen as signs of “corrupt” or “fraudulent” sources.
.
.
.
the connection to the Science and Public Policy Institute (SPPI) is another point. these institutions are more important to the sceptic and denialist movement than most realise. this is not as much about giving financial or other support, but also mostly in making since look “legit”.

Zeke: “GHCN was more rural (as a % of stations) post-1992 than pre-1992.”

Yet … my post said GISTEMP, not GHCN.

” all of the US temperatures – including those for Alaska and Hawaii – were collected from either an airport (the bulk of the data) or an urban location.”

http://wattsupwiththat.com/200.....-than-row/

So … GISTEMP has ZERO rural stations in the USA.

The UHI correction better be perfect … right?

Do you think it is perfect?

Do you think dropping all rural stations in the USA is a good idea?

Thanks Lucia,

My initial reaction was to agree with Nick stokes. Anomalies should fix everything. I could never make heads or tails out of the “basket to apples basket of organes” stuff and just gave up trying to understand what CheifIO was saying. That said, I thought the best approach would be to run GISS and compare:

1. The whole series using only surviving stations.
2. The whole series with the bulge of stations followed by the drop out.

That would answer the question about the effect and how the code actually worked. If CCC did that that would settled the matter in my mind.

sod (Comment#35994) March 2nd, 2010 at 3:51 pm

It doesnt show a problem with free the code; free the data.

people who merely read papers can ALWAYS misunderstand the approach. That misunderstanding doesnt go away.

When you share code and data people will STILL misunderstand.
but with this difference. other people will be able to show them
there mistakes.

Like when Tom P took SteveMc code, changed it. screwed it up.
posted results. Gavin picked those up. posted them without checking.

the result? a loss of credibility.

What code and data release do is give people enough rope.

What they do with that rope is up to them.

Billybob: GISTEMP uses a combination of GHCN and USHCN.

Your source doesn’t seem to be very good. At a glance, there are about 130 US stations current in GHCN, that’s true, but your source forgets that there are a ton more in the USHCN. There are rural stations in there.

lucia (Comment#35945) March 2nd, 2010 at 12:20 pm
Ron– Someone found an error in computation of CRU uncertainty intervals. I think they found that by examining CRU code and it’s been admitted to be true. I’ll look through my email.

Its a mail i sent to you and judy. can’t recall the guy.

I think they did a clean implementation according to the paper.
couldnt match results, had the code to check..

Mind you, I knew this meme was out there, but since there has never been any convincing evidence the march of the thermometers actually caused any large bias in the reported surface temperatures, I never expected it to get quite as much play as it did.

So ‘why did it get so much play?’ will be a very interesting question when all the analysis has been done, especially if this analysis shows that all kinds of allegations and insinuations have shown to be calumniatory. What would be the motive of giving something that’s false so much play?

And this could be silly old subjective little me, but I have this feeling Chiefio is slowly being set up to be the ‘fall guy’? But who gave the Chiefio his platform? And what were that person’s motives to do that? Will he be called out on it here and elsewhere?

Re: sod (Mar 2 15:51),

the “Smith” affair shows a serious problem of the “free the data, free the code” approach:people with little understanding of the subject have come to a complete believe in this mantra.

How does anything Chiefio does show the weakness of free the code, free the code? He’s not basing his ideas on other people’s code. He’s looking at available data (and not even necessarily the stuff people wanted freed), applying his own thinking, running his own scripts, and posting his own code. Chiefio is not the end result of “free the code”.

So, if Chiefio’s style shows anything, it’s that people who have conceptual errors will still have them when they write their own code. Re: steven mosher (Mar 2 16:09), Does CCC have to use GISSTemp code? Not their emulator? I’m pretty sure whatever tests they do, they would rather use their emulator. If you need someone to specifically run GISSTemp, you going to have to get someone who is going to run GISSTemp itself. Maybe you?

sod, Mosher:

Ultimately, the issue is Watts’ quality filter, or lack thereof. There will always be people out there who don’t understand what they’re doing. But they’re ignored unless somebody like Watts gives them a platform, via his website, the SPPI report and the TV show. (I know Watts didn’t produce or edit the show, but I’m guessing Smith’s stuff wouldn’t end up there if Watts and d’Aleo weren’t using it).

Maybe EM Smith will eventually come to some sort of useful insight, but we haven’t seen it yet. He should have been toiling in obscurity until he had something that made any sense. And whatever insight he eventually reaches, it’s rather unlikely to be the one that’s already been publicized.

I’ve been messing with the GHCN data and what I have found is that the spatial weighting does not make much difference.

It is the gridding which decreases the influence of the North American stations.

I’ve looked at plots for other sub-regions of the world and the different patterns are interesting.

Sod,

“the “Smith” affair shows a serious problem of the “free the data, free the code” approach:”

Please explain to me the professional credentials of Albert Einstein when he published the ‘Special Theory of Relativity”.

What University did Thomas Edison graduate from? I’ll make it easy, Thomas Edison had no formal education whatsoever, didn’t even graduate the 1st grade.

What University did the Wright Brothers attend? I’ll make this one easy as well, they were both high school drop outs.

The problem with excluding ‘those not qualified’ is that sometimes, those who aren’t qualified are actually the ones that come up with the answer the ‘qualified ones’ couldn’t find.

Lucia,
One d.Aleo/Smith submeme that does seem to have stuck like glue is the notion that stations were actively “dropped” in the first place. GHCN is an entity with two distinct phases. In the early ’90′s it was indeed a “historical” project. With funding from DoE, CDIAC and Univ Arizona, a whole lot of historical records were collected in batches, surveyed for quality, and put in the database. It didn’t matter if the records were then up-to-date, and they seemed to have no special requirement for spatial representativeness.

That’s where it sat for a while. Peterson, in his 1997 paper, said that planned updates were limited to the US. Then NOAA decided to put some resources into a continuing, updated-by-the-month database, with a rationalised selection of bases to continue with. The requirements for getting and checking a single data set vs arranging for regular submission and processing of CLIMAT forms, are clearly very different.

Most of the “dropped” stations were never reporting on a monthly basis.

I have a question. What set of thermometers define the ‘global temperature’?

Andrew

Thanks again Lucia for the fine analysis. I think E.M.Smith’s hypothesis of bias introduced by selective culling of thermometers is a good hypothesis, but data will confirm or deny the hypothesis. We advance either way.

On the data, the UHI correction aspect is a sore spot for me. A way to maybe get a better grip on it may be to deconstruct the data into these categories:

A. Rural pristine (I hope there is such a thing)
B. Rural land use adjusted (eg see http://tinyurl.com/y8jvo37 which is via R Pielke Snr, 19 Feb)
C. Rural, UHI contaminated (ie close to a city)
D. Urban (possibly split between urban small & urban large)

Each of B-D have a UHI-like component, whereas A is the only one with a potentially ‘clean signal’ (well, cleaner). The differences then give more quantification of UHI.

The fifth category would be SST, which itself is going to have data quality issues, but will have only a tiny downwind cross-contamination from UHI (I’d hope this anyway). You have already been doing the ENSO thing I know – similar principle.

I’m not saying that this would be a quick or easy analysis to do…

I’ve said before this is not my field, but in understanding analogous datasets I’ve always wanted to tease them apart in this type of manner. Sometimes you go down blind alleys, sometimes its gold. When I know how a dataset ticks, then I have more confidence that the aggregated results are not artefact, and most importantly can then defend the analysis to my customer.

Nick Stokes: That’s the whack-a-mole aspect of the meme.

This other 1997 paper is of interest, in terms of the history of monthly CLIMAT reports.

http://www.ncdc.noaa.gov/oa/ho.....ection.pdf

Fig 1 shows that in 1995, 1600 stations sent at least one montly CLIMAT report. Not participating at all were Bolivia, Peru, Afghanistan, Pakistan, Angola, Namibia, maybe Zimbabwe. DR Congo was pretty much empty.

That’s actually a pretty good number of stations, but they weren’t distributed optimally. So they came up with station locations they’d like to see report, to get a more uniform global coverage, with a preference for rural and reliably reporting stations.

Note they explicitly wanted to include some stations on mountains – not because they are cold, but because some mountain locations may not correlate well with stations in the valley below.

If every country had actually sent reports from every station marked on Fig 3, the global coverage would be awesome. I hope some problem areas are being back-filled now from archives and SYNOPs.

Bruce of Newcastle wrote:

I think E.M.Smith’s hypothesis of bias introduced by selective culling of thermometers is a good hypothesis, but data will confirm or deny the hypothesis. We advance either way.

Will we truly have advanced if Chiefio’s hypothesis turns out to be wrong, but has been used to undermine the public’s trust in climate science? I don’t think so. I think that the damage has already been done, and that it was intended to be so.

Um, haven’t you got things upside down a bit? The (core) issue is not about ‘expertise in climate science’ – the ‘March of the Thermometers’ meme is about processing data by a computer programme. How it is processed and what the processing does to it. EM Smith’s profession was computer programming. He may not currently actively work in that field, but he is the professional here – not just some amateur with a possible insight.

He has been running the actual GHCN data through the actual GIStemp programme and seeing actual effects. That was BEFORE he started writing bits of his own code to explore the issue further. The ‘effect’ is something I was not convinced about and I have argued with him about it (partly off-blog), however I now understand what he means and I am listening.

My own very different analysis has also shown something which may back up what he is saying, but I will not be posting that until I am sure one way or the other.

Think about it this way “If you do what you’ve always done, you’ll get what you’ve always gotten.” (can’t remember the attribution). Now GIStemp and the CRU code etc are doing what they are expected to do and showing the climate scientists what they are expecting to see (confirmation of their theories), so there is merit in looking with a new pair of eyes – purely “how does this code process the numbers?” As the input data is not uniform from 1880 to present, does any of that inherent bias (in the data) leak through processing? EM Smith’s dissection of what the programme does and how it does it is about just that – using professional skills learned in the software industry. Incidentally, he regards those who wrote the code as the amateurs.

Vjones, you are clouding the issue with facts, but yes, it is always best to seek answers to the questions actually asked.

“The WUWT crowd hates the word ‘robust’ for some reason, but that’s the appropriate word.”

I’m not sure hates is the right term but I don’t think robust is particularly good either. Insensitivity to changes in the input data or how it is filtered can also be an indicator of poor processing. Are your successive graphs similar because you have the right theory or are they similar because you have the same inherent bias no matter how the data is sliced?

The oracular pronouncement “he doesn’t understand anomalies” unfortunately reminds me of hand-waving statements used to dismiss scientific criticism — and is particularly comical when directed to applied and theoretical physicists because they’re not “climatologists”.

As Bruce says, E.M. Smith’s hypothesis seems good but data will confirm or deny it — as long as the data isn’t locked up or altered. Even if he turns out to be wrong, the hypothesis reinforces the case to free the data and free the code rather than refuting it.

Neven (Comment#36021)
March 2nd, 2010 at 5:43 pm

Will we truly have advanced if Chiefio’s hypothesis turns out to be wrong, but has been used to undermine the public’s trust in climate science?

I thought the science was “self-correcting”.

That ought to mean that science doesn’t depend on the “public’s trust”.

Now, if it’s the publics opinion of policy that you’re worried about, I guess we can talk about all the sources of hyperbole.

vjones:

On your last sentence, that’s the nature of code written by scientists. Programmers always think it’s bad, but research grants don’t usually come with money to hire a guy just to do the programming. In the end, it works well enough, even if it isn’t terribly well structured or efficient. As for quality of code, I noticed some original and recent EM Smith code that used line numbers and gotos, so take that for what it’s worth.

The problem is when a programmer jumps in without understanding the surrounding math or science. Read a few papers first, so you know what’s going on. It’s pretty clear EM Smith didn’t initially understand anomalies, trends, baselines, or what he was looking at. Maybe he’s gotten past that point (hard to tell), but he didn’t exactly leave a first or second impression of competence. As for his great insight, we’re still waiting for it, yet his claims are already well publicized. Doesn’t that seem just a little backwards?

Re: steven mosher (Mar 2 16:09), I do think that anomalies are meant to take out the effect of mean temp, and I think they would. But even if they didn’t, perfectly, the gridding provides a second line of defence.

Suppose they were actually calculating a global temp (not anomaly), and supposed they were on a planet without mountains or oceans, where this might make sense. And suppose a thermometer did decide to march South, from the Arctic to the tropic. What would happen to the global gridded average?

Well, as long as it stayed within it’s original grid cell, that cell would report a warming average. But when it crossed the cell boundary, the cell it left would lose its warmest member, and its average would drop back to the level it had when the errant was at mid-cell (all else being equal).

And what about the cell to the S. That has a new thermo colder than average (on its N border). So its average suddenly goes down too.

Then, of course, the S cell warms back to its original level by the time the errant gets to the cell’s mid-lat, and so on.

Net effect – a sawtooth variation of the average, with no nett trend. You might say, well, there’s that initial uptrend. But the nett effect depends on where the errant started and finished. If it finished further N relative to its grid than where it started, the movement has a cooling effect. And that will happen about half the time.

Of course, what is claimed to happen is that stations are lessened in the cold and added in the warm. But whether that adds to the temp (not anomaly) average depends on whether it warms the grid cell where it happens. And that depends on whether the cold station dropped was below average for its cell, not globally. Same for added warm.

Zeke: “Note that this chart uses v2.mean raw GHCN data.”

Could you chart max and min as separate charts.

UHI biased measurements would show night time warming and no daytime warming, so the mean might go up … but it would all be UHI and nothing to do with CO2.

The sampling issue may or may not turn out to be null irt to showing a big piece of AGW evidence being invalid.
But having followed this pretty closely as a lay person, I find it improbable that massaging data and reducing sampling points ends up doing more than getting you a picture you were already wanting to see.
The explanations that you can accurately know what something that is not there is telling you do not pass the smell test.
Add to that the UHI issues, and I think that the lack of confidence in GISS/CRU surface temps are well earned. Calling on a vast conspiracy for this is simply dissembling by AGW partisans.
Additionally, showing graph after graph that purports to *prove* that a climate apocalypse is at hand, when all that shows are 0.0X to 0.X o’s of change that are well within historic ranges of variation is not credible either. Add to that the missing hot spot and OHC that declines to cooperate.
Then add to that the secretiveness, self dealing, and the consistent over statement of peril, and the flat out wrong predictions irt storms, sea levels, etc., along with the non-falsifiable claims of snow, storms, heat and cold, and I think the collapse of public confidence is well explained.
Chiefio’s being right or wrong on his critique of surface temps is a small road bump either way.

@carrot eater
that’s the nature of code written by scientists. and …research grants don’t usually come with money to hire a guy just to do the programming. Unfortunately! (I mean that sympathetically). Two points.

1. I review research grants from time to time. I LIKE to see professional services (e.g. for web design) written in as subcontracts if the funding allows it. It shows the applicant has thought about their strengths and weaknesses and best use of the researcher’s time. And it usually represents value for money. Just an example.
2. A climate researcher understanding the data and communicating effectively with a programmer to write and test the programme seems logical in hindsight, but so do most things.

With reference to comments by carrot eater and Lucia. I am looking out for a number of things. I am trying to position myself to be able to do my own analysis but for the moment have to rely on those produced by other people. I am looking at those produced on both sides of the debate and am looking at how they confirm or contradict other analyses that are out there.

At this point I have not seen anything in the analysis by E M Smith that is definitely wrong and clearly disproven by other peoples runs. I do have an issue with his rants where he asserts things and I suspect he has his own confirmation bias so I just read past that and focus on what he produces which sometimes means browsing the extensive listings he produces.

E M Smith has done a couple of analysis runs using his own code but much of his output appears to be concerned with how GISS handles data from an unstable baseline. He is doing this by reviewing how GISS handles various subsets. So far the people running the Clear Climate Code version are not doing any equivalent runs so there is nothing to compare against. He is not really asserting there is a bias at input so the various people who do analysis runs on that input are not disproving his results. His assertion is there is a bias at output from GISS based on how it handles data that is unstable and the “migration” of thermometers amongst other issues is apparently leading to a bias.

I still do not understand the mechanism at play and will not accept based on assertions alone he is correct until that mechanism is explained. He is providing reasonable outputs to demonstrate his findings and I have not seen anything that would lead me to dismiss him out of hand. However at some point he will need to put up or shut up!

From my own review I am aware of some issues particularly in that there is not a clearly defined raw dataset and that both NOAA and NASA GISS are processing the data and that the processing is progressively changing. How well do NOAA and GISS talk to each other?

For example we know from one of Zeke’s outputs that USHCN2 appears to be running in the order of 0.2C per century hotter than USHCN. This appears to be the result of their recently added homogenisation processing. Zeke’s output confirms what other people have been finding.

My understanding is that GISS ignore estimated values where data is infilled but will pick up the homogenised values.

To the best of my knowledge NOAA do not perform an equivalent to the GISS Lights = 0 processing. It is therefore quite likely they have homogenised some or all of the reference lights = 0 stations. It is also quite possible they have dropped some reference stations (perhaps they may be replaced with different ones).

The reference stations are supposed to have a significant impact and are relatively few in number. They are supposed to correct for urbanisation by reducing the values in urban stations.

If the homogenisation has raised the values of the reference stations or if many of them have been dropped over time then it is quite possible that this will result in an upward bias for the USA. I dont see any of Zeke’s runs disproving this so far.

JohnM wrote:

I thought the science was “self-correcting”.

That ought to mean that science doesn’t depend on the “public’s trust”.

Now, if it’s the publics opinion of policy that you’re worried about,

.
Yes, that’s what I’m worried about. I’m worried that the Watts/d’Aleo/Smith papers promoted by the Heartland Institute and SPPI are not about the science, but about impacting AGW policy in such a way that nothing happens.
.
Like carrot eater says: “As for his great insight, we’re still waiting for it, yet his claims are already well publicized. Doesn’t that seem just a little backwards?”
.
Is it so far-fetched to believe all of this is ideology driven?

Is it so far-fetched to believe all of this is ideology driven?

“The charge … is double-edged, and cuts both ways.”

clivere: In all of that, I still don’t see a concisely expressed assertion by EM Smith, so it’s hard to even know what we’re dealing with. Yes, dropping a rural station will have some impact at the local gridpoint, especially if there aren’t a lot of other rural stations around, and particularly if they don’t all correlate well with each other. That isn’t surprising; it’s obvious that it can happen. But we aren’t talking about little deflections at a local grid point; we’re talking about the big picture. And none of this is the same as the claims in the SPPI.

The well publicised statements in the SPPI report are just blown out of the water. They weren’t backed up by any analysis, in the first place. Removing a cold station doesn’t, in itself, make the remaining average hot. Removing a cooling station would make the remaining average trend warmer, but that’s not what the SPPI said. Second, the SPPI says quite definitively that stations were intentionally and purposefully dropped to create a warming bias in the global average. Forget the conspiracy theory part of that; the effect simply isn’t there in the global average, as has been shown clearly by Zeke, Tamino and ccc.

As for baselines: The ccc guys can confirm or contradict this, as they know GISS best, but I don’t think the baseline comes into play anywhere in GISS, until the display steps after the stations are already combined. Monkeying with the baseline on the GISS website won’t do anything to change the trends; in fact when you display a trendmap, you aren’t even allowed to choose a baseline. It’s entirely irrelevant.

If by baseline, you mean the period of overlap between stations during the station combining step, well yes, that period of overlap is different from station to station. Is he worried about that? It’s a fixed baseline for Zeke; he gets consistent results (even when changing that baseline), so it would not appear to matter much for the global record. GISS’s method does (presumably) allow them to use more stations than Zeke.

Talk about Robust!

I downloaded V2.max and V2.Mean. I live in Canada so I thought I would unpack the data, import it in Access and count the number of records for one month like July.

V2 Mean: After maxing out at 815 in 1974, the record count slowly dropped to 623 in 1989, then 393 in 1990, then averages 40 since 1992.

If you can measure the temperature in Canada (the 2nd largest country in the world) whether you have 815 data points or 40, then the alogrithms are Robust in the worst meaning of the word.

Neven (Comment#36034) March 2nd, 2010 at 7:01 pm
Yes, that’s what I’m worried about. I’m worried that the Watts/d’Aleo/Smith papers promoted by the Heartland Institute and SPPI are not about the science, but about impacting AGW policy in such a way that nothing happens.

.
Too late.
.
You may have noticed Cap and Trade failed before Smith’s paper, before SPPI started gaining notice, and before ClimateGate.
.
The believers like to blame all their failures on some vast skeptic conspiracy, but the fact is AGW fails without any help at all from us. Even when there’s a Democratic super-majority in the House and Senate and a Democrat in the White House.

I am not sure that this counts. Is this the march of the thermometers or cherry picking? I could only follow the links to an English version of a Russian newspaper. The original work would be in Russian from the Institute of Economic Analysis, Moscow.
December 16, 2009
http://blogs.telegraph.co.uk/n.....l-warming/
—————————-
“On the whole, climatologists use the incomplete findings of meteorological stations far more often than those providing complete observations.

IEA analysts say climatologists use the data of stations located in large populated centers that are influenced by the urban-warming effect more frequently than the correct data of remote stations.”
——————————

When I read comments like “the ccc guys could confirm this” I’m wondering why them? Why can’t the GISS guys confirm it? Aren’t they the scientists with the funding? Seems weird that the “guys” who built it couldn’t do the “confirm this” part.

Andy Krause (Comment#36047) March 2nd, 2010 at 8:52 pm

When I read comments like “the ccc guys could confirm this” I’m wondering why them? Why can’t the GISS guys confirm it? Aren’t they the scientists with the funding? Seems weird that the “guys” who built it couldn’t do the “confirm this” part.

They thought they had, no one believed them.

vjones (Comment#36023) March 2nd, 2010 at 5:53 pm

EM Smith’s dissection of what the programme does and how it does it is about just that – using professional skills learned in the software industry. Incidentally, he regards those who wrote the code as the amateurs.

See, that’s what I’m talking about, real science.

FWIW, as a professional programmer, I rate Smith as an amateur, both at programming and understanding climate.

Extremely OT, but should not people be very concerned about possibility of major Quake in West USA, Canada etc?Etc, considering recent activity Chile, Haiti and current Sun status?

“They thought they had, no one believed them.”
I was referring to a “March of the thermometers” bias test sorry I wasn’t very clear on that.
.

Listen to NOAA’s own words.
From Coleman’s special (don’t choke on your tofu)
Watch from 2:20 to 3:24
http://www.youtube.com/watch?v=A7W4-50n1HE
So, they are adding heat to calibrated thermistors.
These are powered weather stations.
Nothing to see here, move along.

One of the issues I haven’t seen mentioned is that one set of thermometers is apparently used to compute the baseline and another overlapping set is used to compute the actual anomalies.

Nick, do you see any possible issues with that??

How about thermometers “lost” and thermometers with higher trends used to compute values for the now “missing” thermometers grid cell. You see any possible problem with that Nick Stokes??

The ancillary issue is that using GHCN data is using homogenised data with its built in added trend.

On the first Spencer post I suggested his data wasn’t raw. As he used a data set I was unaware of I retract that.

Bugs,

“FWIW, as a professional programmer, I rate Smith as an amateur, both at programming and understanding climate.”

Then you should be able to point out the issues with the GISS code for us?? Please proceed.

[quote Andy Krause (Comment#36055) March 2nd, 2010 at 10:37 pm ]
I was referring to a “March of the thermometers” bias test sorry I wasn’t very clear on that. [/quote]
.
I have to admit I haven’t been following Smith’s work as closely as I’d like. Mostly because I’ve been busy with my own. But I think NASA has responded that the bias Smith talks about doesn’t exist. But since NASA is one of the parties that stands accused, their denial carries little weight.
.
From what I can tell from reading this thread, it’ll take some time for these complexities to be fully understood, for both sides to develop language and techniques that have credibility with those who disagree with them, and to reach a widely held conclusion.
.
From what I’ve seen from my own work on the Aqua satellite AMSU (for example, similar raw readings being published as very different anomaly values), the work of the professionals absolutely needs to be checked.

Almost forgot one of the weirdities of GHCN/GISSTemp. GISS backs out part of the GHCN adjustments before adding its own, which are smaller!!

Yet, y’all are tellin’ me that you are getting the same answer, using that GHCN data, as GISS is getting through its fancy program??? But you claim the fancy program isn’t adding trend??

HAHAHAHAHAHAHAHAHAHAHAHAHAHA

One question I do have is has Smith made his code available?
.
If he has, the first step is to go through his code and see if his results can be reproduced.
.
Saying things like “So and so couldn’t reproduce the results with their own code” isn’t meaningless, but it’s not very conclusive either. Even if 5 or 6 so and sos couldn’t reproduce the results. It’s very easy for there to be some small difference in someone else’s code that changes the results. You really want to start with Smith’s code.
.
And to folks calling him an amateur, just stop it. You make yourselves look biased. If he’s mucking around in someone else’s FORTRAN climate code and not breaking things, he’s not an amateur.

Kuhnkat

have you looked at ClearClimateCode? They have done some pretty good work at replicating GISS. Doesn’t seem to be anything particularly wrong with it. Why not go and disucss the issues you have with GISS with them. You can download the ‘easy to read’ version they made from their site.

Re: carrot eater (Mar 2 14:16),

“It looks very much to me that Eugene took the absolute temperatures, averaged them together, and only then subtracted out the baseline. He may or may not have gridded them on a 1×1 grid first, but it doesn’t matter. If so, that’s just a basic conceptual error, and similar to the that in the SPPI report.”

Why is this a conceptual error? Isn’t calculating an anomaly simply subtracting a constant? An example: suppose I have a temperature series 14,12,10,10,12,13,13. Suppose I designate the two values of 10 as my baseline. I average the temperatures (av. = 12). Then I deduct my baseline. Result: average anomaly = 2. Alternative method: I deduct the anomaly first. My anomaly values are 4, 2,0,0,2,3,3. I average them. Result: average anomaly = 2. What’s the difference?

This may seem like a silly question, but please humor me and tell me what you are actually saying.

Zeke,

Got a question for you. I saw over at The Whiteboard you posted a link to a CSV file that you got from Frei for the snow cover data for A1B. Did he explicitly say that all the data is for A1B? I’ve downloaded all the original gridded data and my emulation shows significant divergence for the post-20C3M period for a couple of models. I think some of the data may not represent the A1B scenario. I also found a few computational errors on top of that.

Re: kuhnkat (Mar 3 01:51),
KK, you’d need to be more specific. But I don’t think what you say is correct. As I understand, stations in the neighborhood of a grid point reporting in the current month are combined in a specific way, with each being shifted to match the current mean before combining. Then the mean shifts are undone to restore the average to the 1951-80 mean for those stations. There is no mixing in of extra stations.

Big Governement, Big NGU, Big Media, Big Soros and even Big Oil is behind AGW, so a lot of money is aviable. Computers can measure temperature for us, the internet can get data anywhere in no time. So one would expect a growing number of modern, rural stations in all parts of the world.
Why is this not happening?

Random que

Does the process of transmuting temperature series into temperature anomaly series introduce or remove any information? i.e what is the anomaly referenced to and does this change with number / siting of stations?

Is there a difference in anomaly between stations that are being dropped and stations being retained? I.e obviously there is a difference between Bolivia and coastal Brazil (one of Smithy’s big areas on interest) in actual temperature – is it valid to assume similar anomalies despite this?

Alexej Buergin (Comment#36085) March 3rd, 2010 at 4:14 am

Big Governement, Big NGU, Big Media, Big Soros and even Big Oil is behind AGW, so a lot of money is aviable. Computers can measure temperature for us, the internet can get data anywhere in no time. So one would expect a growing number of modern, rural stations in all parts of the world.
Why is this not happening?

So the truth won’t come out. Shhhhh.

bugs (Comment#36093) March 3rd, 2010 at 5:21 am

30 stations are being built or named to measure world-temperature with balloons. That is an improvement, but a small one.

Since SD of a mean is s/SQR(n), a greater number (=n) is better.

Chris Polis

When working with anomalies, there is information lost: the original absolute temperature of the station. What is preserved is how the temperature at that station changes over time. This basic concept is what EM Smith/d’Aleo/Watts clearly did not understand at the time of the SPPI report, which is why it’s received such utter derision. The report is just conceptually nonsensical. Dropping a cold station doesn’t matter; it’s dropping a station with a different trend that matters. So Zeke, ccc and Tamino have shown that globally, the dropped stations have the same trends anyway.

Alex Heyworth:

Correct, converting to an anomaly is just a matter of adding or subtracting a constant. The key is, you have to do this for each station *before* you combine it with other stations to form a regional or global average. From his graphs, it appears to me that Eugene first combined the absolute values of the stations in a simple mean, and then subtracted the constant afterwards. If that’s what he did, that’s backwards, as it misses the point of using anomalies.

In the case of CRU or Zeke, the constant is the average temperature at that station over some fixed baseline. In the case of GISS, the constant is an offset that accounts for the fact that different stations have different absolute temperatures.

For all the people worried about dropped stations being included in the baselines: Globally, the remaining and dropped stations were doing the same thing before 1990, so what are you worried about? You might find individual grid boxes here and there where there is more divergence, but it doesn’t matter in the global picture.

For all the people who are worried about choice of baseline:
The choice of display baseline does not affect the trends, which is what matters. You can set the baseline to be the temperature of the dark side of the moon or the surface of the sun, for all I care, and GISS is still going to show a recent warming trend of +0.16 C/decade or whatever it is.

As an addendum: the choice of what I call ‘computational’ baseline does matter, if you use one (GISS does not). Zeke used a baseline of 1960-1970 before combining his stations. If he’d used a single year, like 1960, then his results would be very noisy and probably not usable. He says he tried a longer period (1940-1980) and got about the same result.

But after having combined the stations, he can recenter the average on whatever zero point he likes, and still get the same trends. That’s what I’m calling the ‘display’ baseline.

[quote carrot eater (Comment#36102) March 3rd, 2010 at 6:29 am]
This basic concept is what EM Smith/d’Aleo/Watts clearly did not understand at the time of the SPPI report, which is why it’s received such utter derision. The report is just conceptually nonsensical. Dropping a cold station doesn’t matter;
[/quote]
.
Not necessarily. Dropping a station with high surface reflection and replacing its data with data from a station with high surface absorption will not only raise the temperature, it’ll raise the rate at which the temperature increases.
.
You can’t just go throwing out stations willy nilly and expect anomalies to correctly handle all changes due to the laws of physics.
.
I’m not saying Smith is right. I really don’t know if his idea is valid or not. But the idea that anomalies handle all problems is wrong.

Thanks, carrot eater. I can see the sense of doing it the way CRU and Zeke have done. I’m a bit dubious about the GISS approach, without seeing more detail. Anywhere this is explained? I had a bit of a hunt around the NASA GISS website, couldn’t find anything to explain.

magicjava: That’s great if true*, but that isn’t what the SPPI report said.

The SPPI report simply said that dropping a cold location would make the remaining average warm, just because the dropped station was cold. Period. Not a word about temperature trends; just absolute temperature. Which is simply wrong.

Yes, if the dropped station had a different trend from the remaining stations, then you’d have a problem. That isn’t what the SPPI said, but it’s an interesting thought, so Tamino, ccc and Zeke had a look at the trends, and found globally, no difference between the remaining and dropped stations. But to refute the exact concept given in the SPPI report, you don’t have to do any work at all. You just have to know that people work with anomalies.

* It isn’t necessarily true, anyway. The Arctic has a high albedo (in the visible wavelength range), yet is warming faster than the rest of the world.

I’m not terribly surprised that Watts is still confused about anomalies. It was just about a year ago where he made a hash of comparing the magnitudes of the anomalies between CRU, GISS and the satellites, without realising they were on different display baselines. He even noticed the trends were similar, and somehow didn’t put two and two together.

carrot eater, yeah the example I gave was just that: an example. It wasn’t meant as an exhaustive list.

Have Tamino, ccc and Zeke contacted Smith to check why they seem to be getting different results?

Alex Heyworth (Comment#36105)

Read this 1987 paper to see what GISS does. Free access. GISS has changed some aspects of GISTEMP since this paper, but the basics of the method are here.

http://pubs.giss.nasa.gov/abst.....edeff.html

In particular, see equations 1-3b, and Figure 5. I wasn’t convinced myself when I first read it, so I went and played around with it, using made-up data. I made the data more and more ridiculous, to see what mattered, and under what conditions you get different results from the CRU/Zeke method.

Also note Figure 4, which shows a station number history. Note that at this time, there was a huge station rise around 1955 and a drop around 1970 (if my eyeball serves me). If EM Smith and Watts were around back then, we’d have been having the same discussion 20 years ago.

Re: magicjava (Mar 3 06:59),

Have Tamino, ccc and Zeke contacted Smith to check why they seem to be getting different results?

Who says they are getting different results? To get different results from Smith, Smith has to have published results. Has Smith published any results comparing trends based on the early thermometer set and the later set? My impression is he has not. But if he has, someone can find a link and we can just go look to see what Smith found.

As far as I can tell, Smith has discussed a purely hypothetical issue, and has explained ideas of what could go wrong if people processed data in ways that no one processes it.

magicjava (Comment#36107)

“carrot eater, yeah the example I gave was just that: an example. It wasn’t meant as an exhaustive list.”

Which is why Tamino, ccc and Zeke took the trouble to look at the big picture, to see if the dropped stations had the same trends globally as the remaining ones. Turns out, at least in the global average before 1992, they do. If you really go fishing, you’ll probably find a region where there is a divergence, but it’s not affecting the global picture. Of course, there was some sampling error even before 1992, so that remains. Also, there is some chance that the dropped stations would have started diverging from the surviving ones after 1992, but that requires a time machine in order for the Watts/Smith conspiracy theory to work.

But again, the SPPI report didn’t even go as far as talking about trends. It was talking about absolute temperatures, so it’s wrong from the start.

“Have Tamino, ccc and Zeke contacted Smith to check why they seem to be getting different results?”

What results does Smith have? Looking through the SPPI document, I see no relevant analysis. They published a think-tank report without doing the work to back up the assertions.

“You can’t just go throwing out stations willy nilly and expect anomalies to correctly handle all changes due to the laws of physics.”

First, the stations weren’t intentionally thrown out. Not every station gets reported in real-time, or near real-time. The others get collected at irregular interval. Second, as I’ve made painfully clear through repetition, no, using anomalies doesn’t save you if the dropped stations had different trends from the surviving neighbors in the same grid box. Which is why Tamino, ccc and Zeke bothered to do what they did, to find out if the trends differed. Globally pre-1992, they don’t.

Re: magicjava (Mar 3 06:41),

You can’t just go throwing out stations willy nilly and expect anomalies to correctly handle all changes due to the laws of physics.

You’re right. This is why overall checks like Tamino’s, ccc’s and Zekes are important.

Part of some discussions at Smith’s discuss the main part of Smith’s notion that a positive bias will be introduce by the march of the thermometers southward. Every now and then, issues that might matter are discussed.

So, yes, even if the anomaly method is used, if stations that had been exhibiting less dramatic warming trends had been culled, while those exhibiting more warming were retained, then this could cause a problem. As it happens, thermometers further north show more warming. So at least with regard to “the march of the thermometers south”, if northern areas drop out of the area average, then we’d get a cold bias not a warm bias.

Other things could happen. So it’s worth checking over all — as Zeke, Tamino and ccd did.

Are more detailed checks worth pursuing? Sure. But Smith needs to show that overall, the bias he thinks exists can even be detected. I don’t think he has.

lucia & carrot eater,

As I’ve said, I’ve not spent much time looking at Smith’s work, so I’m probably not the best person to defend it. But I did take a quick look at his site and found this link: http://chiefio.wordpress.com/2.....ined-ghcn/

Where’s he has a methodology and a result, both of which differ from mainstream work.

Anyway, I’ve had a difference of opinion or two with scientists in the past. And when I do, I write them to see if they have an explanation. If folks think Smith is wrong or vague, why not ask him to do a guest post here so he can explain his position?

[quote lucia (Comment#36117) March 3rd, 2010 at 7:33 am
Re: magicjava (Mar 3 06:41),]
So it’s worth checking over all — as Zeke, Tamino and ccd did.
[/quote]
.
Yes, but as we both know lucia, in climate science the results of “checking things” often seem to line up very well with one’s predisposition to the issue.
.
But if folks are saying Smith hasn’t made any claims, how could other folks have checked those claims?
.
Again, it might be a good idea to have him do a guest post to present his side of the story.

magicjava:

“Yes, but as we both know lucia, in climate science the results of “checking things” often seem to line up very well with one’s predisposition to the issue.”

These are cheap words – easy to say, but what does it mean? You can see what Tamino, zeke and ccc have done. You can see how they did it. What would you have done differently? As it is, the audience has asked Zeke to do all manner of variations to see what happens, and he’s graciously indulged those requests.
—-

“But if folks are saying Smith hasn’t made any claims, how could other folks have checked those claims?”

See the timeline Lucia made above. I’m going off the very explicit claims in the SPPI report. It clearly says that cold locations stations were deliberately and purposefully removed, and that removing a cold station necessarily makes the remaining average warmer. It ‘ensures’ it, in fact. This is just wrong, on the face of it.
—
“I write them to see if they have an explanation.”

Watts asked him to give some response at WUWT. His response was to ramble, as he generally does, before deciding to ignore Tamino and continue with whatever he’s currently working on (which was re-inventing something like the FDM, if he’d bothered to read the literature before he started working). He made one substantive point: even if the dropped and surviving subsets were consistent upto 1992, they could have diverged afterwards, in the period where there is no data in GHCN to check. While this is possible, it requires a time machine for his conspiracy to work, and it isn’t a defense of the original idea that dropping cold stations makes it hot.
—-
As for that link to EM Smith’s page:

Just utter confusion from Smith. He graphed Dec 2009 vs a baseline of 1991-2006, and decided the world was getting cooler. Why? Because in one single month (Dec 09), the US, Europe and parts of Russia were colder than the Decembers over that time period? (I’m not sure if he even noticed that the global anomaly was still positive, up in the corner, even without interpolating over the areas with lower station coverage).

If he was curious whether the world was warming or cooling, he’d be making trend maps, not anomaly maps for random months. When you make a GISS trend map, the baseline is utterly and totally irrelevant. This is a concept he’s been struggling to get, if he ever has.

The fair question is whether the trends of the dropped and surviving stations are the same. Zeke, ccc and Tamino have answered that question, on the global scale. EM Smith hasn’t even tried to answer that question, so far as I can tell.

“Yes, but as we both know lucia, in climate science the results of “checking things” often seem to line up very well with one’s predisposition to the issue.”

“These are cheap words – easy to say, but what does it mean?”

carrot eater,

It means that there are some people (you are likely one of them) who are committed to perpetuating the idea of AGW, and they promote analyses that they think support that claim.

Now if you presented material that examined all sides of the issue, pro and con, you would appear to be objective, but I haven’t seen any such offerings from you, so we can only conclude that you are not objective.

Andrew

[quote carrot eater (Comment#36128) March 3rd, 2010 at 8:29 am:]
These are cheap words – easy to say, but what does it mean? You can see what Tamino, zeke and ccc have done.What would you have done differently?
[/quote]
.
It means people take a look at a data set, massage a little here, filter a little there, and Presto! the data proves they are right. Someone else comes along, uses the same data set to prove they are wrong. Happens all the time.
.
What should be done differently is a question to ask Smith, not me.
.
I’m just saying, I’m reading this thread and hearing different things about what Smith did or did not say, or even if he’s said anything at all. Give the guy a chance to explain himself.
.
And just in case you’re wondering, I’m not saying this because Smith is a skeptic. Dr. Spencer (a skeptic) just did an analysis on Phil Jones’ data. Spencer was saying he couldn’t see why Jones did this or that. I’m thinking, why not ask Jones why he did it. He’s not dead.
.
Ask Smith to do a guest post. It’s just common courtesy. If he says no, he says no. But at least you gave him the chance.

Magic–

Where’s he has a methodology and a result, both of which differ from mainstream work.

I don’t see any “methodology” or any “result”. Maybe you can see something I can’t see. In 200 words or less could you summarize:

1) What specific question do you think he is investigating?
2) What method is he using to discover the answer. (Bullet points plz.)
3) What’s the result does he provide to the reader (i.e. answer to the question.)

I curious why do we need more than two stations?
.
When it comes to temperatures from more than 10000 years ago climate researchers are perfectly happy using ice cores from Antarctica and Greenland as proxies for the GMST.
.
If this limited covered led to a large and unpredicatable error in GMST then that would mean that all of our knowledge of the ice ages is suspect as are the estimates of climate sensitivity which have been derived from those estimates.

[quote lucia (Comment#36132) March 3rd, 2010 at 8:54 am]
Magic–
Where’s he has a methodology and a result, both of which differ from mainstream work.
I don’t see any “methodology” or any “result”. Maybe you can see something I can’t see. In 200 words or less could you summarize:
1) What specific question do you think he is investigating?
2) What method is he using to discover the answer. (Bullet points plz.)
3) What’s the result does he provide to the reader (i.e. answer to the question.)[/quote]
.
lucia, ask Smith these questions.
.
The question here isn’t what I think of Smith’s work. I’ve already said I’m not familiar with it. He made some changes with station selection (procedure) and produced a graph (result). Whether or not it has any validity, I don’t know. I didn’t read it.
.
I understand your skepticism here. I think you make a good point that dropping stations is more likely to produce noise than a trend.
.
But if you don’t give the guy a chance to defend himself, it just looks like you’re trying to beat him up.

I think VJones said it very clearly of E.M. Smith yesterday –

‘He has been running the actual GHCN data through the actual GIStemp programme and seeing actual effects.’

So, to repeat –

He has attempted to document the effect of changes (e.g. reduction in number of stations) on the output of the NASA GisTemp software.

Re: magicjava (Mar 3 07:47),

I”m asking you because you said:

But I did take a quick look at his site and found this link: http://chiefio.wordpress.com/2.....ined-ghcn/

Where’s he has a methodology and a result, both of which differ from mainstream work.

So, presumably, you think that specific post shows a methodology and a result. I don’t see any such thing at that page. I could ask Chiefio to clarify, but a bunch of us are discussing what we think Chiefio has or has not shown. I could certainly invite Chiefio to guest post. (If I do, I’m going for a question and answer format.) But before trying a guest post, I think it’s useful to know what people who think Chiefio has shown something to explain what they think the post where he showed this “something” actually shows.

Lucia: One more thing for the timeline. GISS made a temperature record without the high latitudes. I’m not sure if they did this in response to EM Smith, or to help explain the differences between GISS and CRU (which are largely due to the Arctic).

Last datestamp: Jan 22, 2010
http://data.giss.nasa.gov/gist.....egArea.pdf

As you see, removing the Arctic doesn’t warm the trend, it cools it. What’s relevant is that the Arctic is warming quickly, not that it’s cold. Again, direct refutation of the words in the SPPI.

Andrew_KY (Comment#36129)

More vague talk. Where’s the beef? If there was an analysis to support the claims in the SPPI report, we’d look at it. So far as anybody can tell, it does not exist. Watts couldn’t point to one; he referred readers to EM Smith. Which got us nowhere.

magicjava (Comment#36131)

That’s nice, but in this case, different sets of people have analysed this data in several different ways. The conclusion here is not sensitive to doing the analysis in some particular way. Hence, ‘robust’.

“What should be done differently is a question to ask Smith, not me.”

Why not you? Other readers here have asked Zeke to make changes, to see what would happen. You can, as well.

There is probably some undersampled grid box somewhere, in which the change in stations made an appreciable difference to the trend. Maybe in Africa. But globally, no, and the claims in the SPPI were on the global record. Even in northern Canada, Zeke shows no, though it’s be good if he calculated the trends in case the eyeball deceives; the data are very noisy.

Are you really going to make me read that thing, lucia? I hate station data. It just looks like spaghetti to me.

“Andrew_KY (Comment#36129)

More vague talk. Where’s the beef? If there was an analysis to support the claims in the SPPI report, we’d look at it. So far as anybody can tell, it does not exist. Watts couldn’t point to one; he referred readers to EM Smith. Which got us nowhere.”

carrot eater,

I don’t think it’s vague talk. You could just say, “Yeah Andrew, I believe in AGW and I’m trying to promote the idea”, and we wouldn’t have to go round and round.

I don’t need any other analyses to conclude that yours are subjective.

Andrew

Re: Chuckles (Mar 3 09:08),

I think VJones said it very clearly of E.M. Smith yesterday –

‘He has been running the actual GHCN data through the actual GIStemp programme and seeing actual effects.’

So, to repeat –

He has attempted to document the effect of changes (e.g. reduction in number of stations) on the output of the NASA GisTemp software.

Has documented the results of any analysis to show how the reduction in the number of stations affects a computed surface anomaly record over any significant area, for example: US, Northern Hemisphere, or World. If so, do you have a link so we can read that? I haven’t been able to find anything; it would really help people trying to understand what’s going on if we could read:

1) A brief synopsis of any specific claim.
2) The best analysis or argument to support that claim.

[quote carrot eater (Comment#36139) March 3rd, 2010 at 9:15 am]
That’s nice, but in this case, different sets of people have analysed this data in several different ways. The conclusion here is not sensitive to doing the analysis in some particular way. Hence, ‘robust’.
[/quote]
.
Asking Smith for his input to ensure you’re actually checking the things he’s looking at would make it even more robust. It would be the robustest of all.

Chad,

I was incorrect in asserting they were all A1B. Frei told me that “They are the concatenated 20c3m and a1b scenario, ensemble mean time series for 9 models.”

torn8o,

Nice work. Its not too huge a surprise given the relatively large number of stations in the U.S. vis-a-vis the rest of the world.

kuhnkat,

I’m not sure where you are getting this notion that GHCN and USHCN data is somehow always homogenized. Apart from basic QC checks, GHCN v2 and USHCN raw is the direct readings from the stations (averaged over the month of daily readings). There are no adjustments in those data sets.

Re: magicjava (Mar 3 09:25),

Asking Smith for his input to ensure you’re actually checking the things he’s looking at would make even more robust. It would be the robustest of all.

Sure. But it’s also useful to try to discover what people who believe Chiefio think he’s claimed and which arguments they think support those claims. It would be nice to have someone make connections like:

claim 1 =>Summary of argument in fewer than 40 words is “blah, blah, blah’. Discussed in blog posts a, b, c.
claim 2 =>Summary of argument in fewer than 40 words is “blah, blah, blah’. Discussed in blog posts d, e, f.

Then, if possible, we might be able to evaluate whether or not what Chiefio claims supports the documents D’Aleo wrote as sole author or with Anthony Watts. It’s the D’Aleo arguments that seemed to have triggered the Television coverage and the “Chiefio” links for support seem to be contained in the “D’Aleo” portions of the joint documents. ( At least that’s my take. Anthony’s surface station bits are in the latter portion and link to his own stuff. This is not to say that I think Anthony objects to anything in the SPPI document, but I don’t think he’s the source. Chiefio appears to be the source on “the march of the thermometers” notion.)

Raven:

I curious why do we need more than two stations?

If you had good enough climate models, you might only need one.

The climate model can’t predict weather (especially in the absence of detailed knowledge of the forcings), but it should be able to give you the “teleconnections” function that maps the mean weather in one region to the weather over the entire globe.

Lucia,

All of the stuff of his that I have read has been NASA GisTemp based, in terms of getting the NASA code compiled and running and looking at results.
I’ll have a look around and see if I can find anything, but there seems to be a lot of fairly random refuting going on at the moment.

Andrew_KY/magicjava

If you don’t trust the results, take a look at the source code, play around with it, and test it yourself. See what happens if you only use rural stations, only use stations with long histories, use different grid sizes, etc. Or ask me to take a look at a particular combination of factors and I’ll do it when I have a chance (though I already have a bit of a backlog of things I need to look at, especially now that I need to rerun all my U.S. USHCN work at 2.5×3.5 grid boxes…)

Andrew_KY (Comment#36142)

Still no beef, Andrew. Can you point me to any analysis that shows that including or not including the stations that dropped off around 1990 makes a substantial difference to the global or hemispheric trends?

Where is the substance?

Again, it isn’t enough to show that there is some slight difference in the values; we can see slight differences in Zeke’s/Tamino’s/ccc’s graphs. If you remove stations, there will be some slight changes. Especially if you remove them from already undersampled areas. What we’re looking for is a real global bias in trend. Or, if we were literally sticking to what the SPPI report says, what I’m looking for is analysis that shows that removing cold stations (not cooling or less warming, but *cold*) necessarily makes the average warmer. Yes, I suppose it does if you don’t use anomalies, but that’s why everybody uses anomalies.

[quote Zeke Hausfather (Comment#36150) March 3rd, 2010 at 9:37 am]
If you don’t trust the results, take a look at the source code, play around with it, and test it yourself.
[/quote]
.
Zeke, I’m not saying you’re wrong.
.
All I’m saying is it would be an appropriate gesture to formally allow Smith the opportunity to respond. This is not just something I say other people should do. I do it myself when I disagree with someone.

Zeke Hausfather (Comment#36150)-One thing that can’t be done is see the impact of stations which were never included in the record at all. I’ve seen studies which have found orders of magnitude more stations than are used for specific areas in the popular datasets.

lucia (Comment#36146)

Now this is a thought. Do we have the possibility that d’Aleo/Watts wrote the SPPI report using graphs and blog material from EM Smith as reference, but made claims that EM Smith himself would not make?

I don’t think so; I think the claims in the SPPI are repeating claims that EM Smith himself has made, but that would require a more close reading of his blog. Sadly, it’s hard to search for comments at WUWT (so far as I can tell), where EM Smith has also had input.

Chuckles (Comment#36149) March 3rd, 2010 at 9:36 am

It isn’t random refuting. One can see what is written in the SPPI report. It’s pretty simple. And easy to refute.

Re: Chuckles (Mar 3 09:36),

I’ll have a look around and see if I can find anything, but there seems to be a lot of fairly random refuting going on at the moment.

The difficulty is that
a) Claims are made in the SPPI D’Aleo & Watts document. The claims could be broken down into those that are less important to assessing AGW or the whole ‘fraud’ meme; some are more important. The analysis supporting these claims link back to Chiefio. Does Chiefio agree with all the claims in the D’Aleo Watts document? Just the more narrow ones? Or are the broad claims D’Aleo’s interpretation of more narrow claims made by Chiefio?

b) Some broad claims similar to those appearing in D’Aleo’s do seem to be made directly or indirectly at Chiefio’s. For example, the whole “march of the thermometers” meme does appear at Chiefio’s. But actual analyses to show uncertainty and bias tend to be limited to things like showing there is a lot of uncertainty about climate changes in Bolivia.

c) Lots of these claims have hit the mainstream press— including KUSI television. So, we are past just blogging.

At this point, people want to know the answer to “Has anyone shown the march south of thermometers results in a bias in GISSTemp, CRU or NOAA/NCDC temperaruture anomaly series?” If a bias has been show, has it been quantified? And how large is it?”

Because Chiefio is the source linked in the D’Aleo documents, and he has appeared on TV, people want to know whether Chiefio thinks he has shown this, and they want to know his answer to this.

Andrew_FL (Comment#36154) March 3rd, 2010 at 9:45 am

Finally, some substance, though it’s not backing up the SPPI claims.

Yes, you can say that even the pre-1990 data set had some sampling error. It did; you will always have some sampling error. That’s precisely what GISS error bars are for. You can estimate how much information you are missing, due to not having thermometers absolutely everywhere.

As for databases that have more stations than GHCN: yes. It’s a matter of reporting. The weather websites pick up METAR or SYNOP reports, but you need somebody to calculate monthly averages and compile a CLIMAT report for it to be used by GHCN. That’s why Roy Spencer’s project is interesting, I think he’s using archived SYNOPs (I’m not quite sure).

There’s also GHCN-D (daily), but I don’t think anybody’s used that for anything, in this context. As you can read here, “The coverage is a bit uneven, as some countries still are reluctant to freely share daily data (this is improving steadily). ” http://climexp.knmi.nl/help/ghcnd.shtml

Zeke, have you done v2.max and v2.mean graphs yet?

Do they come out the same?

Andrew_FL,

“One thing that can’t be done is see the impact of stations which were never included in the record at all.”

This is akin to the question I asked yesterday about a list of what stations currently represent ‘global temperatures’ according to the experts.

It seems to me that if this list is undefined or dynamic, we have some issues. (deliberate understatement)

Andrew

Re: carrot eater (Mar 3 09:46),

Do we have the possibility that d’Aleo/Watts wrote the SPPI report using graphs and blog material from EM Smith as reference, but made claims that EM Smith himself would not make?

That’s possible. We would have to ask Smith. But some of the claims in the SPPI report do seem to be things Chiefio also intimates. So… who knows?

BillyBob,

Still working on max/min analysis for the USHCN data. I’ll look at max/min for GHCN after that.

Chuckles:

All of the stuff of his that I have read has been NASA GisTemp based, in terms of getting the NASA code compiled and running and looking at results.

Most of what he has done (that I’ve seen) involves computing an unweighted average of raw station temperatures without understanding any of the issues in doing that. That’s not even close to what GISTEMP does.

This error has given rise to his erroneous claim that dropping stations of high latitude during the 1990s has lead to an artificial warming (actually it leads to decrease in the temperature trend if you don’t spatially average the data).

IMO he has no credibility at this point. Why should anybody take him seriously, unless you just like his answers and don’t care how he got them?

Carrick,
.
Your statement confirms what I suspect is a bit if circular logic.
.
i.e. if a climate model is used to determine the GMST from a single location then the temperatures from that location cannot be used as evidence that the climate model is accurate.
.
It also assumes the patterns of teleconnections is constant over time. Something that cannot be true over timescales where continental drift is a factor. Over shorter time scales the advance, retreat of ice sheets and changes to vegetation would affect these teleconnections.
.
The more I learn the less I suspect we really know about past climate. So much of the ‘common knowledge’ appears to be built on unverifiable assumptions.

Entirely OT.

I always have a fun experience trying to post over at Open Mind.

anyways. There ought to be some place where the following can be discussed by all, maybe MT or Lucia or DR. Curry

http://www.plos.org/

http://www.plos.org/about/principles.php

http://www.plos.org/oa/index.php

http://www.plos.org/oa/definition.php

http://www.plosone.org/static/policies.action

http://www.plosone.org/static/.....on#sharing

http://www.opensource.org/docs/osd

http://www.plosone.org/static/review.action

very OT. no need to comment on, but read if you are interested.

Raven (Comment#36165)

When you go back that far, there are of course pretty good uncertainties. But you’ve got more than just the polar ice cores. If nothing else, geological evidence of glaciation is left whereever it occurs, so we have a good idea of when ice ages happen, and you can see how well that correlates with the polar ice cores measures. I’m not really into paleo, so I can’t comment on the applicability of other indirect methods.

Carrick,
.
I don’t doubt that the ice ages occurred. But a lot of science is based on the assumption that we know the magnitude of the GMST changes during these transitions. I am not certain that we know that to any degree of accuracy.

I ran this through the translator:

“pretty good uncertainties”

It came back with:

“unknowns”

Andrew

Some additional material from Chiefio’s blog so you can draw your own conclusions.

—

http://chiefio.wordpress.com/2.....-artifact/

“1) There is a pronounced seasonal variation in the “Global” average temperature. This means that the GAT is decidedly biased to the Northern Hemisphere. Hemispherical changes can easily bias the “GAT”. (i.e. effects of change of axial tilt, of precession (which pole is close to sun at perigee) etc.)”

About averaging:

“2) In calculating these “Global Average Temperatures” I found that the exact method of calculation strongly changes the result. Do you average all the separate valid records and then divide by the count of valid records? Or do you calculate a yearly GAT, then average those to get a decade or total data series GAT? This implies that the number of thermometers active in any particular period of time has a strong impact on the GAT in that time.”

Smith’s Eureka moment:

“3) Finally, this lead me to the idea of selecting only those stations with a long history (I now have a FORTRAN program that takes the GHCN format files, counts the records for each station ID, then sorts them in rank order by total years of data and lets you select a “cutoff” value. I chose to use a 3000 station cut off (that gives about 64 years for the “short lifetime” stations) but similar results happen with 1k, 2k, and even 4k stations. The result is that almost all of the AGW “signal” goes away. The conclusion is that the AGW “signal” is an artifact of the arrival and departure of thermometers from the scribal record. The addition of more thermometers in the Southern Hemisphere followed by the loss of Siberian thermometers with the collapse of the Soviet Union. The thermometer count rises from 1 in 1701 to over 9000, then drops back to under 3k today. That has an impact… I calculated “decade Global Average Temperatures” from a data set reduced to the 3000 longest lived thermometers. A sample of the records are below. The next to last field is an average of all data for the decade, while the last field is the number of thermometers (station IDs) active in this group from that 3000 in the data set. You can see that the GATs don’t change much from decade to decade anymore.

DecadeAV: 1890
0.6 2.2 5.8 11.9 17.0 20.9 23.2 22.4 19.0 13.2 7.2 2.9 12.2 1174
DecadeAV: 1940
0.3 1.3 5.4 10.4 15.3 19.1 21.6 20.9 17.5 12.3 6.2 2.0 11.0 2851
DecadeAV: 1990
-0.1 1.4 5.7 10.8 15.4 19.3 21.6 20.9 17.2 11.9 6.1 1.0 10.9 2186
DecadeAV: 2009
1.7 2.2 6.6 11.7 15.9 19.9 22.3 21.7 18.0 12.3 6.9 2.1 11.8 209
I have no explanation for why the long lived thermometers drops to 209 in the last data.”

—

http://chiefio.wordpress.com/2.....mmers-off/

What is Global Average Temperature:

“So I decided to calculate a couple of different “Global Averages of Temperatures” as my benchmark. For each year, I would add up the temperature records for each month in the GHCN set and divide by the number of cells with valid data. A simple average. But I would then add those ‘averages by month’ together, and divide by 12, to get an annual average GAT. Then all years in a given decade would be averaged together to get a “Decade Average of Temperatures”.

One can also add all those “monthly averages by year” together and divide by the number of years to get a “Grand Total Global Average Temperature by Month” and could then average those together to get GAT for ALL months of ALL years. Or one could simply take all the temperatures for all valid records, add them together, and divide by the number of records to get “The One True Number” of the grand total GAT for all time for all records.

I did all of those things.

Why? To see how the different ways of calculating things impact that GAT as reflected in the data.

I do feel compelled to point out, once again, I believe that the Global Average Temperature no matter which way you calculate it, does not really mean much. This is only a way to “characterize the data” and “assess the sensitivity of the data to ways of calculating”. The fact that everyone is all excited about a “Global Average Temperature” does, to me, raise the issue of exactly how one chooses to calculate that One True Number. The fact that it is used is distressing, but I’m stuck with what the world has chosen to do. And the world has chosen to think that averaging several thermometers together (or the same one over long time intervals) might mean something… OK, I must accept that premiss. So what happens when we “characterize the data” in this way?”

—

http://chiefio.wordpress.com/2.....g-problem/

About seasonal signal in raw data:

“Now you want to turn that signal into an “Annual Global Average Temperature”, then you would benefit from grids, boxes, etc. But since we’ve already shown that the signal isn’t global and that it has a strong seasonal component: I’m left to wonder the wisdom of making a “global” thing that isn’t and an “annual” thing that can’t be. So the “Annual Global” part of “Annual Global Average Temperature” seems to me to be hiding truths that ought not to be hidden…”

Some additional comments:

“Further, I’ve read every single line of GIStemp. I’ve ported it to Linux and have it running in my living room. I’m painfully aware of what it does and exactly when and how it does the grids and zones. An analysis of those bits is “in the queue” for a couple of weeks from now.

For now, I’ve run the temperature data all the way through STEP3. What I see is pretty simple. The zonalizing and gridding behviours are, IMHO, likely to reduce the impact of added thermometers, but can not eliminate it entirely. It manages to take a 10C rise in January and reduce it a lot, but still gets some signal through. Enough to cause folks to get worked up over a fractional to 1 or 2 C “rise”.”

“There must be a “warming signal” in the raw data to find “Global Warming” and a rise of the Global Average Temperature (whatever that is, or means).

That signal is not present in summer.
That signal is not present in the long lived thermometers.
That signal is strongly present only in the short lived thermometers added, by inspection, largely in warm places during NH winter.

And no amount of gridding, zoning, or other manipulation can change those facts.”

“If there is “global warming” a stable set of thermometers ought to show some warming, even if unevenly distributed in space. They didn’t. That just can’t be resolved with the AGW by CO2 thesis.

The last link in the chain of events was this posting. I realized I could very easily invert the logic of my sort / selection and see the “negative space” of the good thermometers. The seasonal and trend effects are dramatic in the data selected this way.

Substantially ALL the “global warming” signal is carried by the records from these stations (or a subset of them … more to do…). So the warming isn’t “Global” at all. And the station arrival dates have a strong correlation with the arrival of more warming. That strongly suggests that the warming signal is an artifact of station addition to the record. (Though that needs a bit more proof to make it a rock solid case. But that the warming signal in the whole body of data makes it through STEP3 and that the signal only exists in this subset of the data strongly imply that the whole of AGW comes down to these selected stations.)

Now you want to turn that signal into an “Annual Global Average Temperature”, then you would benefit from grids, boxes, etc. But since we’ve already shown that the signal isn’t global and that it has a strong seasonal component: I’m left to wonder the wisdom of making a “global” thing that isn’t and an “annual” thing that can’t be. So the “Annual Global” part of “Annual Global Average Temperature” seems to me to be hiding truths that ought not to be hidden…”

—

http://chiefio.wordpress.com/2.....arm-globe/

About Reference Station Method:

“And what will happen when we use the Reference Station Method to fill in those empty (undoubtedly cold) boxes with fictional temperatures from 1000 or 1500 km north in the warmer band? (There not being many thermometers to the South, either…)”

“So what we seem to have discovered in the AGW “process” is that if you add a pot load of thermometers in warm places, you get more warmth in your average of thermometers. Who knew…

Want a Nobel Prize? All you have to do is demonstrate that fact.”

—

http://chiefio.wordpress.com/2.....arm-globe/

About spatial averaging:

“Leaving aside the question of just how do you make a “Global Average Temperature” for comparisons from 1839 or even 1889 with 91.5% of thermometers in the cold north and only 8.3% in the 60 degree band of the planet from near Cairo to Cape Town South Africa …
We are still left with the fact that we move about 1/4 of the thermometer records from the cold places to the hot places. That not only increases the impact of the hot places, but reduces the impact of the cold places.
Given that the cold places are a physically smaller area, if we “area adjust” our thermometer records by making zones and boxes for our geographical bands, we will even further reduce the impact of the “cold thermometers” on the Global Average Temperature and increase the impact of the “warm thermometers”.”

Looking around EM Smith’s page, in the time before the SPPI report was published (end Jan 2010):

You see a ton of graphs of different regions, with absolute temperatures simply averaged together. No sign of anomalies or gridding, anywhere. None. These went into the SPPI. The SPPI recognises the need for gridding, but neglects to grid anyway; no mention of the need to use anomalies.

One example is here.
http://chiefio.wordpress.com/2.....australia/

Here he talks about latitude
http://chiefio.wordpress.com/2.....arm-globe/
And here elevation
http://chiefio.wordpress.com/2.....hat-andes/

He’d have something of a point if he said that neighboring mountain and valley anomalies might not correlate well with each other. But I see no sign of that point. I just see simplistic discussion of hot and cold.

As for charges of intentionally choosing stations to somehow enhance a warming signal:

Here he acknowledges that people have told him how the GHCN was put together, such that there was a dropoff in station counts between historical archiving and subsequent continuous reporting. So he finds a station that stopped reporting in 2005, not 1990, and sticks to his story of manipulation. It doesn’t strike him as a possibility that Madagascar simply stopped submitting CLIMATs on time. At this point, I don’t think he’d heard of a CLIMAT report; I don’t think he ever mentioned them until I tried to tell him about them. So from here, I think it’s fair to say he’s in agreement with the intentional fraud charge, though it’s hard to find it laid out as clearly and unambiguously as the SPPI report puts it.
http://chiefio.wordpress.com/2.....scar-muse/

So maybe we can say that d’Aleo/Watts used some stronger language, but it’s hard to say Smith actually disagrees with anything in the SPPI. You’d think he’d say so if he did disagree, but one could double check with him.

carrot eater (Comment#36157)-”though it’s not backing up the SPPI claims” More accurately it isn’t backing up EM Smith’s claims, which, it happens, I strongly disagree with to begin with.

“Yes, you can say that even the pre-1990 data set had some sampling error. It did; you will always have some sampling error. That’s precisely what GISS error bars are for. You can estimate how much information you are missing, due to not having thermometers absolutely everywhere.”

This only amounts to random error and thus “error bars” worthy if the stations that are excluded are random and those included are random. The number of stations in some places is so small that them being representative is highly unlikely.

The specific papers I know of that have found actual unused stations in places in the US as well as Africa, where the stations tend to be extremely sparse, have found that the use of a few stations biases the trends.

http://ams.allenpress.com/perl.....2.3.CO%3B2

http://ams.allenpress.com/perl.....1&ct=1

http://ams.allenpress.com/perl.....JCLI2726.1

ds: You can go through all that, and he’s just doing the same thing, over and over and over. He’s taking simple means of absolute temperatures. Nobody does this, for good reasons.

He keeps talking about hot and cold, not warming and cooling. Through all those posts, he is showing zero recognition of the reason why you’d want to use anomalies for this type of analysis, instead of absolutes.

The whole body of work can be dismissed on this basis alone. I’m sorry; he just doesn’t know what he’s doing. Based on his comments, it is apparent that he looked at the GISS code, and didn’t at all understand what the offsets do in the RSM. So he felt justified in making all these graphs of absolute temperature averaged together. But GISS does nothing of the sort.

“high altitude stations are dropped, while low altitude … stations are kept (with an ever increasing percentage at warm heat islands of airports)”

Winter in Switzerland was “cold” at high altitudes and “normal” at low altitudes (where most of the population lives and where most of the thermometers are in urban surroundings). The reason: MeteoSwiss does not adjust for UHI effects, and that hides the fact that it was probably cold at low altitudes, too.
So if a high station is replaced with a low station, one gets ASW (S for Swiss).

“He’s taking simple means of absolute temperatures. Nobody does this, for good reasons.”

CE,

What are the good reasons?

Andrew

Questions Cheif has asked and answered;

1 a. Is California getting warmer?
b. Note that his tomato plants are not setting fruit.
c. report the divergance between tomato and NASA.

This indeed is based on absolute temperatures, as tomato plants are incapable of calculating with sophisticated anomaly methods.

2. a. What is the effect of dropped thermometers in GISTemp?
b. Test GISTemp code with subsets of GHCN.
c. GISTemp is very brittle to deviations from its usual diet.

Cheif has also pointed out the need for a variety of manual interventions within a GISTemp run. The implicit conclusion (to me anyway) is that GISTemp has not been subjected to modern Software Quality Assurance testing methods, as it would be too difficult.

3. a. How many Months does GISTemp need to declare an annual average temperature?
b. Dissect gistemp code, publish same.
c. At minimum, 3 ‘seasons’ containing at least 2 months.
4. a What does Gistemp to fill in missing months.
b Dissect code.
c. If missing month is middle of 3 in a ‘season’, interpolate.
If end month, do something complex.

5 a. Are there non-linear ‘steps’ within GISTemp processing?
b. Examine code.
c. Yes, the ’20-year’ rule will cause records from short or intermittant stations to enter/exit the baseline.

It may not be immediately obvious, but Cheif has been working with extremely modest computer hardware for the majority of his effort on these issues, hardware that I suspect many here would have declared obsolete and donated to a truly desperate organization long ago. He recently got a slightly less obsolete machine, with a disk that was slightly larger than 10GB.

The hardware constraints have clearly limited his ability to provide some of the requested amenities such as graphic presentations, etc.

While his style might be more significantly idiosyncratic than the majority of significantly idiosyncratic bloggers in this arena, I think it is a big misteak to underestimate his comprehension.

I also think his unique form of (down and dirty) data engagement has intrinsic merit. Where many of us rely on assumptions of ‘reasonable rational actor’ behaviour within scientific software, his approach (Be The Data) to comprehension of the trip through the anomalizer/homogenizer allows him to detect non-egalitarian treatments.

I’ll admit I enjoy the novelty over there.
RR

Raven:

Your statement confirms what I suspect is a bit if circular logic.

Which I interpret as you not understanding the argument.

i.e. if a climate model is used to determine the GMST from a single location then the temperatures from that location cannot be used as evidence that the climate model is accurate.

Of course that’s an accurate statement.

But if one could obtain (hypothetically) a climate model that could reliably compute the telecommunications function for the period where we have reasonably extensive measurements, that could be used to verify the models.

In the end, the models are based on fundamental physics that are well understood and extensively tested and verified. The challenge is the scale of the Earth climate problem is too big to fit on any computer using the “current” (read 30 year out of date) algorithms used for solving the CFD problem.

Write a program good enough, know the forcings and feedbacks precisely enough, and you should be to compute properties like mean temperature without any thermometer readings, at least in some well characterized average sense.

Temperature is an outcome of forcings + feedback, you don’t need it to infer either of these.

It also assumes the patterns of teleconnections is constant over time. Something that cannot be true over timescales where continental drift is a factor. Over shorter time scales the advance, retreat of ice sheets and changes to vegetation would affect these teleconnections.

We know the pattern of drift of continents, we know the changes in solar forcing due to astronomical factors. We have proxies for solar activity and CO2 content that are pretty darned reliable. And changes in the biosphere can be included in the climate models too.

In my opinion, there’s no reason that in time we won’t have a very good picture of the paleoclimate.

I don’t doubt that the ice ages occurred. But a lot of science is based on the assumption that we know the magnitude of the GMST changes during these transitions. I am not certain that we know that to any degree of accuracy.

Science is a boot-strap process.

So if you can “nail” the surface temperature record for an extended period (say 150 years) and you can “nail” the climate models (including any caveats you need for comparing the two), then this allows you to progressively get by with fewer and fewer measurements. The “holy grail” of paleoclimate science is to have the model be good enough you can directly invert the proxies to give you prehistoric climate reconstructions.

[Note I didn't say proxy-based temperature, I envision climate models that directly incorporate the proxy readings into their inputs.]

Re: Ruhroh (Mar 3 12:02),

It may not be immediately obvious, but Cheif has been working with extremely modest computer hardware for the majority of his effort on these issues, hardware that I suspect many here would have declared obsolete and donated to a truly desperate organization long ago. He recently got a slightly less obsolete machine, with a disk that was slightly larger than 10GB.

He’s a programmer by profession. Why is his hardware so limited? I’m sure Chad, Tamino and Zeke don’t have supercomputers in their living rooms.

Andrew_FL (Comment#36173)

You gave me three papers to read. The first had nothing to do with the topic at hand, which is spatial variations in anomaly fields, and the resulting error from undersampling. The paper was just Christy sitting there and homogenising the stations in his backyard by hand, based on painstakingly collected historical metadata.

Ruhroh (Comment#36177)
“I think it is a big misteak to underestimate his comprehension.”

His lack of a good computer isn’t keeping him from using anomalies, instead of averaging together absolute temperatures. That’s why I have a dim view of his comprehension.

Re: Carrick (Mar 3 12:13),

In the end, the models are based on fundamental physics that are well understood and extensively tested and verified. The challenge is the scale of the Earth climate problem is too big to fit on any computer using the “current” (read 30 year out of date) algorithms used for solving the CFD problem.

Write a program good enough, know the forcings and feedbacks precisely enough, and you should be to compute properties like mean temperature without any thermometer readings, at least in some well characterized average sense.

It’s a good thing I wasn’t drinking something when I read that or you would owe me a new keyboard. 100 km grid size and fundamental physics does not compute. Jerry Browning and Tom Vonk, among others, strongly argue that it’s not even possible to correctly model the problem with a grid size orders of magnitude smaller. When you try to couple an atmosphere model with an ocean model with time constants orders of magnitude longer than the atmosphere, the fun really begins.

You do know that, for example, when the lapse rate in a grid box exceeds the stability criterion, they just move things around in the entire grid box until it’s stable again. This is called a kludge, not fundamental physics. Now let’s talk about clouds and fundamental physics. Or rather, let’s not, because those physics are far from well understood, tested and verified, not to mention occurring on a scale far smaller than the grid box. One of the reasons models are useless for regional analysis is that modeled cloud coverage doesn’t begin to match reality. Aerosols are another fudge factor to tune the models to at least approximately hindcast rather than being based on fundamental physics.

There’s also a reason you only see anomalies in model output. It’s because the range of calculated absolute GMST’s of an ensemble of models using the same data is several degrees. Write a program good enough with good enough data… If I had some ham I could have some ham and eggs…if I had some eggs.

Carrick (Comment#36178) March 3rd, 2010 at 12:13 pm

“In the end, the models are based on fundamental physics that are well understood and extensively tested and verified.”

Really?? Why are people still actively researching turbulence? Do we really know all the multiphase processes that govern, say, cloud physics? Can we express them mathematically with reasonable accuracy for all possible states? The answer is no.

And we are even more in the dark when when it comes to the numerical algorithms for many climate codes, given that the equations being solved are non-linear and highly coupled. Some people don’t even write their equations down…

Andrew_KY (Comment#36176)

Why use anomalies? Many reasons. First, lots of local factors can affect absolute measurements, but not anomalies. You can put your thermometer at 2 m above the ground, or 8 m above the ground. This will affect what absolute temperature you measure, but not the anomaly. So instead of stressing about the exact details of where the thermometer should be, you just calculate the anomalies. This is why some microsite factors matter less than people think. Yes, the station might be reporting 1 C higher or lower than the temperature measured 50 feet away, but so long as it has the same trends as the spot 50 feet away, it doesn’t matter.

Anomalies also correlate over distance, not just up and down. New York and Philadelphia may have different temperatures, but in any given month, they’ll probably both be about the same amount above or below the long term average (however you want to define that). So you can generate anomaly maps without having an infinite number of thermometers.

This becomes important when you combine different stations to get a spatial average. Suppose warming is uniform across a region, at the rate of 1 C per century.

Station A goes linearly from 20 C to 21 C, from 1900 to 2000.
Station B goes linearly from 30 C to 31 C, from 1900 to 2000.

Maybe station B is on a hill, I don’t know. If you take a simple average, you get a ramp from 25 C to 26 C over this time period. In that case, you actually get a reasonable result.

But say, for instance, that station B burned down in 1980, and wasn’t replaced. Using absolute averages, you’ll get an abrupt discontinuity at 1980, as the simple average of A and B is replaced by A alone. But using any of the anomaly methods (CAM, RSM), you’d still get a nice smooth line increasing at 1 C/century.

Now, as it happens, if Station A and B had very different trends from each other, you can still get slight discontinuities when using RSM, when one station drops off or picks up. But this isn’t what EM Smith has been going on about, nor does this effect affect the global trends.

Bleh, and as an addendum: if station A and station B had very different trends, then losing station B would leave you undersampled. You’d lose information that you would have wanted, in order to accurately describe the region.

Hence, people quantitatively study what sort of thermometer density you need to have good sampling.

Lucia;
Perhaps this is a good example of ‘re-examine the assumptions’.
Cheif’s current circumstance of substantial frugality is one that many more of us may soon face. I was amazed to learn that he did all the work on a shared ‘family’ computer which had been declared obsolete and was discarded by a CA public school. (!)
Perhaps you have a better sense of the HW used by the names you mentioned. What’s the vintage of your machinery?

TIA
RR

DeWitt:

100 km grid size and fundamental physics does not compute.

Of course I never said they did.

What I said was “The challenge is the scale of the Earth climate problem is too big to fit on any computer using the “current” (read 30 year out of date) algorithms used for solving the CFD problem”.

Are you sure you weren’t drinking?

And opinions are what they are… opinions. The same goes for Browning and Vonk as anybody else.

Frank:

Really?? Why are people still actively researching turbulence? Do we really know all the multiphase processes that govern, say, cloud physics? Can we express them mathematically with reasonable accuracy for all possible states? The answer is no.

When any of this research changes Newton’s Second Law, drop me a note.

Cloud dynamics is applied science, not fundamental. And I do research that involves atmospheric turbulence, so you needn’t lecture me on what’s known and not known there. As far as climate related processes go, the problem is basically solved.

Again my caveat “The challenge is the scale of the Earth climate problem is too big to fit on any computer using the “current” (read 30 year out of date) algorithms used for solving the CFD problem.”

There’s a reason I included that sentence. It has meaning and implications.

And we are even more in the dark when when it comes to the numerical algorithms for many climate codes, given that the equations being solved are non-linear and highly coupled. Some people don’t even write their equations down…

This is addressing the current state-of-knowledge of climate modeling, which is not related to the point I was addressing.

Zeke,

I was incorrect in asserting they were all A1B. Frei told me that “They are the concatenated 20c3m and a1b scenario, ensemble mean time series for 9 models.”

I assumed that the 20th century period was 20C3M and after that was A1B. He incorrectly concatenated the two experiments. I’ll have a post up soon on this. Just have to cross some t’s and dot some i’s.

Thank you all for a most informative thread. But why the bunnies?

Carrot Eater, Bugs?

carrot eater,

Thanks for taking the time to reply. My concern is that things happen at specific temperatures, so I’d rather know that the temperature at a particular site was being accurately measured.

I don’t know that a ‘trend’ is meaningful until it runs into some kind of reality.

Andrew

Re: Ruhroh (Mar 3 13:22),

Perhaps you have a better sense of the HW used by the names you mentioned. What’s the vintage of your machinery?

I don’t specifically know what they use– but I doubt they have supercomputers in their living rooms because almost no one does.

I use a mac. The “about this mac” info reveals

Processor: 2.8 GH Intel Core 2 Duo.
Memory 2 GO 667 MHs DDR2 SDRAM
I’m using MacOSX v 10.5.8.

My machine doesn’t limit anything I want to do home. Any limitations are self-imposed, or knowledge imposed.

I’m perfectly willing to believe Chiefio is hardware limited. But it’s also odd for a programmer to not have kept up with home computers while employed. They tend to scrimp on other things rather than computers. So, I am surprised to learn the “family” computer wasn’t a bit more up-to-date even if he is currently unemployed.

lucia (Comment#36113) March 3rd, 2010 at 7:23 am

As far as I can tell, Smith has discussed a purely hypothetical issue, and has explained ideas of what could go wrong if people processed data in ways that no one processes it.

He just using the standard level of denialist proof. It is one that Willis also uses, with the slight variation of “If I can’t understand what’s happened here, it must be something nefarious”.

The recent transfer of a 400MHz G3 128M 40G machine was a huge step up. Also, no more sharing it with the kids…
Beware of ‘rational world’ models…
Cheif is an unusual guy.
He seems to be closer to the earth than a lot of us, certainly me.
RR

Carrick (Comment#36192)
“When any of this research changes Newtons Second Law, drop me a note.”

“Cloud dynamics is applied science, not fundamental. And I do research that involves atmospheric turbulence, so you neednt lecture me on whats known and not known there. As far as climate related processes go, the problem is basically solved.”

But you said…

“In the end, the models are based on fundamental physics that are well understood and extensively tested and verified.”

Oh, so as long as we know Newton’s second law, we know everything! Well that’s good…why didn’t I think of that?

“Again my caveat The challenge is the scale of the Earth climate problem is too big to fit on any computer using the current (read 30 year out of date) algorithms used for solving the CFD problem.”

I agree here, but it’s NOT just scale, but non-linearity, coupling, well-posedness, and other mathematical inconveniences…

I do my modeling on two computers.

My work computer
Model: 21-inch IMac
Processor: 2.66 GHz Intel Core 2 Duo
Memory: 2 GB ram

My home computer
Model: 27-inch IMac
Processor: 3.06 GHz Intel Core 2 Duo
Memory: 4 GB ram

Things do run noticeably faster at home.

Andrew_KY (Comment#36198)

Well, absolute temperatures do matter for local events (like growing seasons), but the anomalies and trends also get you back to that – if there’s a trend in the anomaly, there’s also a trend in the absolute temperature measured in your back yard.

Look at daily experience. You turn on the radio, and they say the temp is 92 F. But if you pull out your thermometer, in any given spot you can measure all sorts of things. But if you leave your thermometer in one location, then it will trend the same as the radio station’s thermometer. Unless there is something really messing with it, nearby.

The problem is when one area is not trending the same as another area within the same grid box, and you don’t have a thermometer there to know it. If you lose a weather station and get that scenario, you get sampling error.

I think it would improve the S/N if Carrick would tighten up his nomenclature a bit. Several of the characterizations of the GCMs are clearly in conflict with what is known to be the state of these models / methods / codes / application procedures

Recently over on RC the GCM problem was stated to be nothing more than a computational physics problem. I take that to mean things like DNS for the Navier-Stokes equations, accurate numerical solutions of the Blotzmann equation with the collision effects based on first principles, and similar investigations into equations that contain information only about the materials of interest. Carrick seems to be attempting to propagate this fundamentally flawed characterization of the GCMs.

At Carrick (Comment#36192) March 3rd, 2010 at 1:29 pm,

When any of this research changes Newton’s Second Law, drop me a note

The GCMs do not solve the fundamental equations generally associated with Newton’s Second Law. Instead, rough, approximate models of these laws are used. The Navier-Stokes equations, for example, contain only material properties that describe the response of the material to forces acting on the material. They do not include empirical data measured for a particular state that the material has attained. These data are generally strictly valid only for the state under which they were measured.

DeWitt Payne (Comment#36185) March 3rd, 2010 at 1:02 pm

You do know that, for example, when the lapse rate in a grid box exceeds the stability criterion, they just move things around in the entire grid box until it’s stable again. This is called a kludge, not fundamental physics.

It’s an algebraic switch that can be implemented by use of an IF-THEN construct; I call these, Physics by Fortran. If the correct equations corresponding to Newton’s Second Law were being solved, such statements would not be necessary.

Something that puzzles me (and I may have just mis-read things). If we are asking does “dropping stations effect the result” then the test is to run the result with and without the dropped stations. It was stated up-thread by lucia and others that this has been done at least 4 times. The puzzle being where did the “dropped station” data come from?

Andy

Something that puzzles me (and I may have just mis-read things). If we are asking does “dropping stations effect the result” then the test is to run the result with and without the dropped stations. It was stated up-thread by lucia and others that this has been done at least 4 times. The puzzle being where did the “dropped station” data come from?

Different people did different test.

One of the tests is this:
1) There are series 1 that goes from year A to year C.
2) There are series 2 that only go only from year A to year B, with B< C.

You can’t get data for years B through C from series 2. But you can compare trends for years A-B for both series 1 and series 2. If the there is a bias due to thermometers, you should see it.

Roy Spencer did something different: He went out and found an even bigger thermometer set and did trends with that, comparing to other trends.

You could do other things.

But whatever it is Smith thinks happens, it would be useful to see something concrete. Math? Runs? Something.

What precisely he thinks happens needs to be described because the anomaly method is supposed to compensate for changes in actual thermometer locations. There is uncertainty associated with these changes, but there is no reason to expect it to be a bias.

Re: Dan Hughes (Mar 3 14:34),
“The GCMs do not solve the fundamental equations generally associated with Newton’s Second Law.”
That’s not true. The first of the Navier-Stokes equations (momentum equation) is an expression of Newton’s second law. The inclusion of a turbulence model just modifies the force term and replaces the velocity with an average, but still satisfies conservation of momentum for that average..

Andy Krause (Comment#36216)

Everybody except Roy Spencer is comparing the pre-drop data from the dropped and surviving stations. (I don’t like saying ‘dropped’, because there wasn’t somebody at NOAA deleting stations, but that seems to be the terminology here). If the two subsets coincide so well up to 1990 or 1992, we have no particular reason to think they diverged appreciably afterwards. They conceivably could have, but no evil person sitting at NOAA in 1992 would have known that yet.

So how do you know what happened after 1992? Roy Spencer’s analysis is useful in that respect, since it ditches the GHCN altogether and uses a different source of raw data. Though I don’t think anybody’s carefully compared area coverage in his set, vs the GHCN.

As archived data gets collected, some of the ‘missing’ data will get filled in; that’s how the hump ending at 1990 was made in the first place. The hump used to end around 1970 or so.

Re: Nick Stokes (Mar 3 14:52),

Oooohhh! I get to agree with Nick for once:

The first of the Navier-Stokes equations (momentum equation) is an expression of Newton’s second law. The inclusion of a turbulence model just modifies the force term and replaces the velocity with an average, but still satisfies conservation of momentum for that average..

Both the Navier-Stokes equations and approximate forms of Navier-Stokes used in GCMs do apply conservation of momentum.

What GCMs do not do is solve the Navier-Stokes. This is not the same as not applying Newton’s second law. Other than the possibility of computational error, or some formulation making discretizations error of some sort but F=ma is applied.

The turbulence just might not correctly describe how momentum flows from one grid box to another or it may mis-specify the forces acting on the surfaces of any grid boxes. It might also get the frictional force between air and dirt or air and water wrong. But the equations are formulated to apply F=ma.

Similar things happen with conservation of mass.

Nick Stokes (Comment#36219) March 3rd, 2010 at 2:52 pm

Yes, I said that. But I also said:

Instead, rough, approximate models of these laws are used.

Are you stating that the GCMs solve the complete turbulent form of the fully 3-D compressible Navier-Stokes equations, including the effects of compressibility on turbulence?

I know that some GCMs use only the hydrostatic balance in the vertical direction. That would be the steady-state form of one component of the momentum equations containing only an accounting of the pressure gradient due to the hydrostatic head for the fluid. Hardly a complete statement of Newton’s Second Law. There won’t be any turbulence accounting in that form, either. Other GCMs solve a simple transient form of this equation in the vertical direction. There’s a name for it, but I don’t have in at hand.

carrot eater

“But if you leave your thermometer in one location, then it will trend the same as the radio station’s thermometer. Unless there is something really messing with it, nearby.”

Not necessarily. It depends on the how close my thermometer and the radio station’s is.

If they are right next to each other, you would think so, all things being equal. The farther apart they are, the more likely it becomes they could be different.

Andrew

Re: Dan Hughes (Mar 3 15:10),
Are you stating that the GCMs solve the complete turbulent form of the fully 3-D compressible Navier-Stokes equations, including the effects of compressibility on turbulence?
I don’t think Newton mentioned any of that. All I’m saying is that GCM’s require the balance of force and rate of change of momentum, which is Newton’s second law. And they can’t do that without taking account of dynamic pressure variations.

Dan Hughes:

Are you stating that the GCMs solve the complete turbulent form of the fully 3-D compressible Navier-Stokes equations, including the effects of compressibility on turbulence?

Over time scales at issue for climate, you need not explicitly include compressibility in the models. Typically it can be replaced by a term called “turbulent viscosity”.

But I think you are getting confused though over the questions ofwhat is “state of the art” for climate models and what is theoretically possible for them to achieve, which is what I was driving at.

Andrew_KY (Comment#36228)

Yes, the correlation depends on distance. But the correlation remains further than you might expect.

Go to the GISS page, click on the map wherever you live, and download the raw data for some neighboring stations. Station moves, instrument changes and UHI, etc will mess with this, but find the correlation between pairs of stations.

The 1987 Hansen paper did this sort of thing, and found good correlation out to hundreds of kilometers, especially at high latitudes; less so in the tropics.

So most likely, if the radio station thermometer is having a warmer than usual day, then so will your backyard. Unless you’re using shortwave, and listening to something from across the world.

Well, I guess I need to tighten up my nomenclature a bit.

For me, for fluid motions, the fundamental equations generally associated with Newton’s Second Law are the fully complete Navier-stokes equations.

In my opinion, anything less is a model of Newton’s Second Law.

I’ll go so far as to say that if you insist that you can characterize any model of Newton’s Second Law as representing the fundamental equations for fluid motions, then we’ve all got a communication problem. Kind of dumbs down Newton’s work, too, if we can label anything as that law. How will we ever know what each of us has in mind?

In the case of using a steady hydrostatic balance in one direction, all of us can agree that something has been left out ( actually all the hard parts have been left out ). Note, too, that the physics described by this approximation does not correspond to the physics actually occurring in the flow. F=ma is not what is contained in this approximation; the physical momentum of the flow is not balanced. While the fluid satisfies completely, in all respects, Newton’s Second Law, this model does not. This approximation does not describe what the fluid is experiencing. It is an incomplete statement of Newton’s Second Law. It is a model of that law.

Nick Stokes (Comment#36230) March 3rd, 2010 at 3:27 pm

The complete, unaltered, unmodified, forms of the Navier-Stokes equations, i. e. complete statements of Newton’s Second Law, do in fact contain all the information necessary to calculate all aspects of compressible turbulent flows. No rough, approximate, models needed.

Carrick (Comment#36234) March 3rd, 2010 at 3:49 pm

I’m not confused. I know what the state of the art is. This discussion stared with someone mentioning Newton’s Second Law. All I said was that the GCMs use rough approximate formulations of Newton’s Second Law. And no one yet has shown this statement to be in error.

If we allow the nomenclature to become too loose, then almost any problem can be classified to be a computational physics problem. The GCMs are process models, based on approximations to some fundamental concepts and containing a host of ad hod, heuristic, EWAGS, and many parameterizations almost all of which are known to describe critically important physical phenomena and processes at only very crude levels.

Carrot Eater:

Yes, the correlation depends on distance. But the correlation remains further than you might expect.

I think this is another thing that could be quantified further. I’ve seen claims in both directions on this.

If it were true in most cases (but not in some) then that may be an automated way of spitting out problematic stations for further examination (such as micro-siting and instrumentation issues).

Re: Dan Hughes (Mar 3 16:08),

In my opinion, anything less is a model of Newton’s Second Law.

Even the NS are a model of Newton’s second law. They contain a constitutive model relating the stress tensor to the strain rate. It’s quite good in what we call Newtonian fluids, but does not always work for some other fluid-like substances like, say, paint or plastic.

Still, obviously, solving the NS directly is different from using a turbulence model.

If all you want to say is that GCM’s are highly parameterized and that probably “matters”, I suspect few here will contradict you. Certainly, those who can write the Navier Stokes equations, and explain the meaning of each term will agree with you.

Re: Carrick (Mar 3 16:10),

I think this is another thing that could be quantified further. I’ve seen claims in both directions on this.

I also think that it’s worth nothing that it’s thought the spatial correlation works better on anomalies than absolute temperatures. For example, the top of a mountain is generally cooler than a valley directly below. But if we found the average temperature of both and compared anomalies, we’d tend to find the anomalies of these two locations are well correlated.

Dan Hughes:

I’m not confused. I know what the state of the art is. This discussion stared with someone mentioning Newton’s Second Law. All I said was that the GCMs use rough approximate formulations of Newton’s Second Law. And no one yet has shown this statement to be in error.

I don’t doubt you know the state of the art. And like the others I agree that the GCMs satisfy Newton’s equation, probably Euler’s equation, but certainly not the full Navier-Stokes equation.

My point is I wasn’t actually talking about the state-of-the art GCMs (which are mostly limited by the state of the art of computer science and CFD algorithms), but what the limit the physics imposes on the models.

. I took Raven’s (probably sarcastically meant) question literally…with an ideal model, how many temperature measurements do you need to know the temperature field of the entire Earth?

My claim is “none” if you can measure the forcings and feedbacks precisely enough. We need temperatures in practice because we don’t know the forcings and feedbacks, we use the temperatures to help us infer information about these measurable but unmeasured unknowns.

For paleoclimate, there is no need to convert from a proxy to temperature, since what you are really trying to get to (for the climate models) is the variation in the various forcings and feedbacks over time (this could include things like AO oscillation indexes).

Finally for people who appear unclear what the role of solutions to the “fundamental equations” is, it is a short cut when you don’t want to study all of the intermediate applied science. If what you want is the temperature teleconnections function for the Earth…definitionally this is T(latitude, longitude)/T_mean…you don’t want to have to write down detailed models of cloud dynamics.

Large eddy simulations can (or soon will) produce all of the correct details for the turbulent structure in the atmospheric boundary layer, mesoscale development of super cells and even model the development of tornadic vortexes within those supercells. At this moment, they can include compressibility (since that has been brought up) using Lighthill’s equation, and can even compute the long-range infrasound associated with convective storms (the most detailed can only go to 1-Hz at the moment, but that is real progress over older codes that were strictly incompressible).

….

Anyway, if what you want to study is cloud dynamics, solutions to mesoscale LES codes will largely get the actual dynamics right, but it isn’t going to tell you anything new that you couldn’t get by going out and taking measurements. The main advantage is it allows you to run a lot more “numerical experiments” that you can use to test your theory of cloud formation.

But you don’t need a theory of cloud formation to get it to fall out of the fundamental model.

It’s a bit like solving a physics problems using an analog computer. It’ll tell you what will happen, not why. But if you don’t care about “why” for a particular application, that is a real blessing, because it’s one less thing you have to fuss about in the process of getting answers you actually are interested in.

Re: Dan Hughes (Mar 3 16:08),
In the case of using a steady hydrostatic balance in one direction, all of us can agree that something has been left out
Yes. What you’ve left out is any evidence that any GCM does this. That is, ignored the vertical gradient of dynamic pressure.

Carrick (Comment#36242)

It’d take a different sort of script from the ones being developed here, but it’d be pretty easy to re-do the distance-correlation analysis of Hansen 1987.

If you did it by grid box instead of just latitude band, you could isolate problem areas, where things don’t correlate as far as usual.

I think the Western US could be a good test bed. Mountain, desert, forest, maybe even coast: all these things you could find within a radius of 1200 km.

carrot eater (Comment#36181)-If you actually read the papers or even asked John, you’d know that when he compared his data with USHCN and CRU he had 1. More stations and 2. Lower trends.

Evidently you didn’t read them. Well, how about this: In the case of East Africa John found 18 stations in GHCN for Kenya and Tanzania. He came up with 242 different records, some of which might not be separate, but even then there would still be at least 131 at least. Now, what trend is there over the study period from GISS and CRU?

+0.14/decade
+0.17/decade

He found:

+0.04/decade

Germane enough for you?

Andrew: OK, I’ll read the paper about Africa, then. The one about Alabama had nothing to do with spatial variations and undersampling, and that’s what I started with.

From your description this sounds like a raw/adjustment thing, more than a spatial sampling thing, but I’ll see.

“The 1987 Hansen paper did this sort of thing, and found good correlation out to hundreds of kilometers, especially at high latitudes; less so in the tropics.”

You mean Hansen the Famous Political Activist Climate Celebrity?

If this is the paper you mean, some of the intro sounds like a Global Warming propaganda piece:

“Although it is safer to restrict temperature analyses to
regions with dense station coverage, there is a great
incentive for trying to obtain estimates of long term global
temperature change. Such global data would provide the
most appropriate comparisons for global climate models and
would enhance our ability to detect possible effects of global climate forcings, such as increasing atmospheric C02″

http://pubs.giss.nasa.gov/docs.....bedeff.pdf

Andrew

Lucia, Carrick, et al.

I guess I don’t follow this line of argument; (1) The Navier-Stokes equations are the complete and correct statement of Newton’s Second Law when applied to fluid motions, but, (2) I can replace the full and complete statement of Newton’s Second Law with approximations and I still get Newton’s Second Law.

The terms that describe the material, the constitutive description of the material, do just that and nothing more, describe the material. For fluids that exhibit a linear response to rate-of-strain, the part that Navier and Stokes did, the equations are exact. We have to agree to accept the given equations at some level, otherwise we’ll be off in second-coefficient-of-viscosity-land and molecular gas dynamics and that never never land at the intersection of continuum and particles. I think it’s best if we stick with Euler, Cauchy, and Newton, probably a couple other of those pioneers.

The mathematical properties of constitutive equations must satisfy universal principles of continuum mechanics. It is interesting that sometimes when the complete constitutive equations are replaced with a model, the terms that replace the constitutive equations often violate these principles. Notably this occurs whenever the fundamental constitutive equations are replaced by algebraic equations. We do this replacement all the time because we can seldom resolve the gradients that appear in these terms. And equally notable that certain models of turbulence violate these principles. The problem most frequently encountered is that the models are not frame indifferent. And because the equations that describe the material are an important part of the dissipation term in the energy equation, that accounting, too, gets screwed up. When the terms that describe the material are replaced with algebraic model equations, these latter are no longer constitutive equations. They are generally empirical descriptions of measured data obtained under certain states that the material has attained and describe these states, not the material.

According to Gavin Schmidt, in an e-mail that I no longer have, Equation 2 on page 612 is the equation used in the GISS/NASA ModelE GCM. Washington and Parkinson state explicitly that this is the hydrostatic formulation. You can refer to almost any climate-modeling textbook and find that this approach is part of the geostrophic wind equations model. There are variations on this theme including a quasi-geostrophic modeling.

Andrew_KY (Comment#36268)

For Pete’s sake, just read the paper. It’s a critical paper to read, if you want to understand what GISTEMP does (though some things have changed since then).

You’ll see some plots of correlation coefficient vs distance, for different latitude bands. That’s what’s relevant to this discussion; and it should be pretty easy to reproduce/update using today’s data set. These charts are what GISS use to justify their use of interpolation.

carrot eater,

I will read the rest of the paper, but I’m not so sure I can flippantly dismiss

“there is a great incentive for trying to obtain estimates of long term global temperature change.”

Oh, you bet there is, Jimmy Boy. Don’t we know it. And how.

Andrew

To whomever it was, who wanted to know what stations go into the GHCN:

This guy has made a map for you. There are 1200+ stations which report on a timely and regular basis. The map on the top shows them.

A couple hundred more filter in late, but within a year; adding them gives you the third map.

So that’s the electronic reporting via CLIMAT. Every few years, some stations are added from other efforts; that last happened on a big scale around 1990, and there may be another bump this year; not sure yet.

http://climatewtf.blogspot.com.....month.html

____

Andrew_KY:

For goodness sake. The point of GISTEMP is to collect the surface record, so you know what’s happened. How can you possibly see malicious intent in that?

“For goodness sake. The point of GISTEMP is to collect the surface record, so you know what’s happened. How can you possibly see malicious intent in that?”

carrot eater,

I don’t see any malicious intent at all in simply collecting temperature readings for the record.

However, if that’s all you can see that’s been going on Climate-Wise, I submit that you have been looking at squiggly-lined graphs too long. Dr. Hansen in his own words has injected his own paper with a disclosure of ‘great incentive’. I’m just observing that he has done so. He wrote it, not me.

Andrew

Andrew_KY (Comment#36283)

When you write a paper or give a talk, you have to provide the motivation – why you bothered to do this work, why the audience should listen, why it should be published.

In this case, it’s pretty obvious – you need a global surface record, in order to know what has happened over time. It’s just weird that you don’t like this sentence.

Re: Carrick (Comment#36248) March 3rd, 2010 at 4:29 pm

Large eddy simulations can (or soon will) produce all of the correct details for the turbulent structure in the atmospheric boundary layer, mesoscale development of super cells and even model the development of tornadic vortexes within those supercells.

I find this to be an interesting claim. Do you perceive a cutoff where the “large” eddies are sufficient to produce “all the correct details for the turbulent structure…”?

Re: Dan Hughes (Comment#36269) March 3rd, 2010 at 5:47 pm

According to Gavin Schmidt, in an e-mail that I no longer have, Equation 2 on page 612 is the equation used in the GISS/NASA ModelE GCM. Washington and Parkinson state explicitly that this is the hydrostatic formulation. You can refer to almost any climate-modeling textbook and find that this approach is part of the geostrophic wind equations model. There are variations on this theme including a quasi-geostrophic modeling.

I found the challenge to the (fact of the) use of the hydrostatic approximation to be puzzling as well. How else does almost all of linear theory work?

carrot eater,

I was under the impression that science papers were about science only. You know, dispassionate, like an instruction manual. Instruction manuals don’t disclose that the author(s) are geeked about an unknown reader like, putting a bicycle together or something. There are no ‘authors.’ There is no intro and it doesn’t doesn’t say that Huffy or Schwinn has ‘great incentive’ in putting these instructions together.

Andrew

Andrew: You have to justify why your work is important, worth reading, and worth publishing. In a typical paper, you should find some statement of motivation or relevance in the abstract or introduction. You can be dispassionate, and still explain that your work is important; if you didn’t think it was important you wouldn’t have bothered doing it. Well, sometimes you get sidetracked by insignificant details, but..

In this case, the importance of the work appears self-evident, although from Hansen’s wording, the reviewers might have asked for more justification – the CRU already does this, so what’s so great/new/different about yours? That’s the feel I got, when I read the paper.

Further thoughts,

In the intro it says:

“Such global data would provide the
most appropriate comparisons for global climate models and
would enhance our ability to detect possible effects of global climate forcings, such as increasing atmospheric C02″

Words mean things. This isn’t just a paper about how to compile a temperature record. This is a piece of a grand narrative that he is already thinking about. He’s looking ahead. He tells us so.

In light of what has transpired since 1987, and in the light that this quote contains Global Warming Talking Points verbatim…

…we are very far removed from the dispassionate Instruction Manual.

Andrew

oliver:

I find this to be an interesting claim. Do you perceive a cutoff where the “large” eddies are sufficient to produce “all the correct details for the turbulent structure…”?

Yes, definitely.

Typically they use 10-m resolution, then “sub-grid” the turbulence at smaller scales. This turns out to be sufficient to reproduce measured turbulence spectra for daytime boundary layer measurements, but a finer grid is used when they do nocturnal simulations.

As I understand how this sub-gridding works, they attempt to “resolve” the spatial wavelengths associated with the turbulence “source region” then assume a Kolmogorov spectrum [e.g., power goes as 1/f^(5/3)] to extend this into the sub-scale domain (the idea being that you have a turbulent cascade, in which larger scale turbulent vortices breakup in into successively smaller scale (and higher frequency) vortices.

Since all you want is the statistical properties of this spectra, it isn’t necessary to reproduce the turbulence all the way to the turbulent viscosity limit.

Dan
There seems to be a vertical pressure gradient in eqs T6 and T7. It’s true that they don’t have an explicit vertical velocity, but I think vertical convection is handled through the potential temperature, which includes pressure and it’s vertical gradient also appears.

So much noise and heat over nearly nothing. I’ve only had time to scan about 1/3 the comments (and won’t have time for them all) so here’s a couple of responses.

The March Of The Thermometers meme came from me as a picturesque way of describing the change of the instrumentation over time. Other folks had observed that we start with one thermometer in northern Europe and spread out over time, but I put the picture on it. That picture was only meant to show that thermometer locations change over time, and the change is from high latitude to low latitude. What has happened with the picture after that point was not up to me.

That is entirely disjoint from what impact it has on an anomaly process.

“Chiefio (E.M. Smith), Feb 22, 2010: elaborats on the KUSI – Coleman TV show discussion which covered the story that dropping thermometers from the temperature record results in a warming bias in surface temperature anomoalies reported to the public. ”

WRONG. I have gone out of my way to say “warming bias in the DATA” and not in the “anomalies”. I make no claim about warming in the anomalies since they can be calculated a few different ways so it will vary by product. I do claim that instrument change can leak through GIStemp as I’ve run a benchmark that shows that it does. (Method posted too). I’ve also said I have a theory I’m working on (that will stay private pending decisions on publishing) about how bias could leak through the anomaly process, but I’ve never said that anomalies ARE biased.

@Steve Mosher: The “Apple basket to Orange basket” is a simple imagery way of describing how GIStemp calculates anomalies. It is not “New York Central Park in 1955 to New York Central Park in 2009″ it is “A bit of N.Y.C.P. and some Jersey in 1955 to “More Jersey, some Delaware, and a bit of Upstate NY in 2009 with Laguardia”. (As a fictional example) That is, it puts one set of thermometers in the baseline and computes a grid cell anomaly to a different set of thermometers in the various years. This is, IMHO, the major way that the the bias in the data could get into the product (but yes, I have to measure it. I’ve benchmarked that it DOES get through, now I have to measure exactly how much and show the “filter Q” figure).

@Carrot: Yeah, you hate me. I got that. That’s why you don’t get free run at my place. Quelle Surprise… So your going around “talking dirt” about me elsewhere. What a guy. Real class. /sarcoff>

And people wonder why some postings get snipped at my place…

Per averaging temperatures instead of anomalies: Folks keep asserting that I must be an idiot for doing this and I can’t possibly understand that anomalies are the only right way to do things. This is completely ignoring what I’ve said about why I’m doing it. My purpose in doing it is NOT to find a temperature (or a temperature anomaly) but to MEASURE the DATA. To find how much bias is in the data and to see how that changes over time. To observe, for example, that the Pacific basin has a big drop out of warm stations in WWII and that it starts with one cold station down in Hobart. To see that this bias has about 10 C of magnitude and that we’re looking for 1/100 C of “climate change” precision inside that noise. (GIStemp anomaly reports are in 1/100 C) If you are going to measure noise rejection of a filter, you have to know the magnitude of the noise and of the signal. GIStemp has to have a 1:1000 rejection factor to have that 1/100 C come through that 10 C noise. Think like an engineer, not an academic.

If you don’t measure, you are guessing. So I’m measuring and then saying, roughly, “My God, that’s a lot of noise!”. That does not say it is impossible to build a filter that can reject it, just that it’s going to be hard.

Then I look at GIStemp and the code there is not, er, “robust”. (And folks have grown weary of “robust” due to it’s over use on very non-robust things… It’s become tainted by association.)

GIStemp does a laundry list of calculations with the temperatures before they make their grid / box anomalies (and then they do them “Basket of one set of thermometers in one time” vs “Basket of a different set of thermometers later”) and each of those steps is an opportunity for bias to leak past PRIOR to the anomaly calculation and due to the “Basket A vs Basket B” method.

And for the folks who have enjoyed tossing rocks at what I’ve posted: Well, it isn’t for everyone. It isn’t pre-processed pap and it isn’t political food fights. It is technical work in progress pimples warts and all. I’d originally started with the intent of it being a place for programmers taking a look at GIStemp code and what it does to blocks of data. So yes, it has that flavor. Code, data, all out there for folks to see.

Every so often I put in an “everyman” posting that recasts the observations for the average guy to get an idea what I’m doing. Often with non-technical picturesque language. Neither of these is an academic paper. Get over it. It’s going to be like visiting the car repair shop, not the showroom out front. If you don’t like oil and pistons, it isn’t for you. And if you want to spill coffee in the work area and complain about oil on your shoes, you will be shown the door to the waiting room – just like in any other shop.

If you want academic papers and lots of political spin? Go somewhere else. You want to see what I’m looking at today, and how I did it, then come on in. But if you don’t LIKE what I’m looking at, well, that is just not my problem.

If, for example, I’ve gone through GHCN and posted a table of numbers showing that we now have about 92% (IIRC) of GHCN thermometers at Airports today in the USA. Well, that’s what we have. And that’s what I’ve posted. I may then SPECULATE (often in the comments section) that I would expect this to lead to a warming bias IN THE DATA due to airports having an airport heat island. But that’s all it means. (Though the GIStemp benchmark implies that bias would leak through to the anomaly maps).

And no amount of Hypothetical Anomaly Processes can change that. In fact, I’m working on a different anomaly process to show what the actual trend is (one I call dT/dt that is like the First Differences method, but without a reset on any old data gaps – it will preserve real trend better, but with higher tendency for equipment change to show through) for the purpose of using it to measure the GIStemp filter Q and bias leakage. So I fully expect that DATA BIAS can be largely removed. (And once the code is stable it will be tested and measured on a selected set of known data to validate it.)

But before I run off and CLAIM that dT/dt removes DATA BIAS I have to know how much there is. And that closes the circle back to the simple averages of temperatures. So I started with simple averages TO SEE THE DATA and the biases in it. And I ran a test on GIStemp to see if any gets through (and it does). Now I’ve got a tool that shows more nearly what OUGHT to be the trend, and I can compare that to the original data bias (to see what comes through) and to GIStemp (to see how much it diverges). I have the original NOISE figure, I have a baseline for post noise removal, and I have a way to measure the GIStemp performance against it. Not a Hypothetical Cow in sight.

And that is why it does not interest me how many other folks Hypothetical Cows (programs that are not used to produce policy results or data series that are used for same) show there exists some way to remove the bias in the data. It just does not matter. Show that GIStemp is a perfect filter or with a Q giving a 1000:1 rejection factor and I’ll be happy to examine the work.

Why make a new anomaly process? Simple. To avoid any systematic issues in someone else’s code and to keep it simple. (I.e. no UHI, no homogenizing, none of the stuff that provides leakage paths).

If all of this sounds alien to you, good! I want it to. I am deliberately using different techniques from the usual to remove systematic bias. Like the space shuttle had one team write the same control software in complete isolation so a software bug would not take out all their computers. So I’m using “self to self only” anomaly calculations but not exactly First Differences. And I’m avoiding The Reference Station method. I’m also looking first at where is the “warming signal” carried before I try to calculate “how much”.

One of the standards of QA testing is that at least one person who did not write the software ought to do some of the testing. They are the person who can do the unexpected things since they don’t know what the code expects. I believe the same thing can be done in here. Come at it from a new direction and surface new issues. That is why I’m staying away from the way everyone else has done things. To maintain the Martian View.

So do not expect that I’ll do it the way everyone else has done it (the design goal is to do it a different way) and do not expect me to approach this as an academic exercise. I’m coming at it as an Electrical Engineer would. That isn’t a matter of not knowing or not being able to do it the ‘regular way’, it’s a matter of knowing that you find more interesting things if you come at it fresh. To find the real errors, you simply must take a different path.

Well, it’s now 2:45 am and I’m to be up a 6 am. So I’ll not be back to this thread any time soon (and perhaps not at all). I’ve got things to do and, frankly, listening to folks miss-quote me, cast aspersions on my character, and not ‘get it’ about the intent of what I’m doing is rather low on my list of “must do” things.

You may now resume your food fight and rock tossing.

E.M.Smith (Comment#36344)

“WRONG. I have gone out of my way to say “warming bias in the DATA” and not in the “anomalies”. I make no claim about warming in the anomalies”

I guess you can do that if it’s interesting to you, but one needs to get Watts/d’Aleo to stop selling your work as something that it simply isn’t. You might want to tell them. There are many reasons nobody else looks at absolutes averaged together, and these are among them. If you’re working on your own way of calculating anomalies, fine; we’ll see what you have when you finish that.

This tends to confirm that the worst overstatements are coming from either Watts or d’Aleo, and not EM Smith, which is useful to know.

” That is, it puts one set of thermometers in the baseline and computes a grid cell anomaly to a different set of thermometers in the various years. This is, IMHO, the major way that the the bias in the data could get into the product (but yes, I have to measure it. I’ve benchmarked that it DOES get through, now I have to measure exactly how much and show the “filter Q” figure). ”

And ccc/Tamino/Zeke already measured it and showed that globally, it doesn’t matter. For individual grid points, you might see more of a deflection due to the loss of a station. In the big picture, nothing.

“So your going around “talking dirt” about me elsewhere. ”

You get to criticize the methods and integrity of others, and nobody gets to criticize you for calculating simple means of absolutes and acting as if they mean anything. Right.

Andrew_KY (Comment#36293)

You’re being paranoid. So he wrote that a temperature record is needed, in order to see the effect of different forcings, including CO2. Exactly. What on earth is wrong with that? That’s the whole point of collecting a surface record – to see what’s happened.

“You’re being paranoid.”

carrot eater

“The trains carrying coal to power plants are death trains. Coal-fired power plants are factories of death.”

http://wattsupwiththat.com/200.....l-and-co2/

Now who’s being paranoid?

Andrew

carrot eater (Comment#36346)
“You might want to tell them. There are many reasons nobody else looks at absolutes averaged together, and these are among them.”

Could you explain, please, why averaging anomalies is thermodynamically meaningful? Doesn’t your “reference” (baseline) temperature have spatial variation by definition?

Frank K: What’s important is the changes in temperature. Which is what working with anomalies gets you, without all the measurement headaches of working with absolutes, which I’ve alluded to throughout this thread.

In my opinion, the focus on specifics is most helpful (data integrity, methods, quantitative results, background as to concepts, accuracy/precision/uncertainty). Other things, not so much. There are many places on teh intraweb to go for runaway threads on the topic of this post. No points deducted if one chooses not to respond to a perceived ad hom or similar challenge, quite the opposite. To this reader.

“Frank K: Whats important is the changes in temperature. Which is what working with anomalies gets you, without all the measurement headaches of working with absolutes, which Ive alluded to throughout this thread.”

Right, but how does this relate to the thermodynamics? Again, isn’t your baseline temperature different for each location? For example, let’s say I want to get the average anomaly in a region of space. The point value for the anomaly is:

delta(x,y,z,t) = T(x,y,z,t) – [T(x,y,z,t)]

where [T(x,y,z,t)] = Tref(x,y,z) is the time average of the temperature at point (x,y,z). This is your reference temperature and varies in space since each point (“station”) has a different time-average temperature. If I now compute a spatial average, we get:

delta_avg(t) = T_avg(t) – T_ref_avg

Where f_avg = volume integral average of function f(x,y,z).

The first term on the right hand side is simply the spatial average of the absolute temperature. The second term is a constant.

Of course, this assumes that we know T(x,y,z,t) at all points (x,y,z) and for all time t. Where the anomalies appear to be useful is in developing an interpolation scheme to fill in missing data in the temperature records.

“That’s the whole point of collecting a surface record – to see what’s happened.”

You already said that. I already agreed that was OK.

Hansen introduced (literally) the other issues and the factories and trains of death, not me. You brought him into the conversation, not me.

Andrew

AMac:

Point taken.

Frank:

Getting there. To speak meaningfully in absolutes, you’d need a nearly infinite network of thermometers. For each square foot of surface, and then also for each foot of elevation above the surface, upto to some point. Maybe also an IR sensor for the ground itself. And all of these thermometers would have to be in service all the time, with no adds or drops over time.

But we have nothing like that, so we work with anomalies. Which we can measure with a finite number of thermometers. It gets you all the information you want, with none of the impossible measurement headache above. If you took your perfect imaginary absolute temperature field and calculated the trends over time, you’d get the same trends as in the measured anomaly field, within some sampling error.

Thermodynamically, the changes are all you really need. If you really want to crudely simplify, if the heat capacity in a box is constant, then if you put in some thermal energy, then the temp will rise by some delta T. It’s the change that’s relevant. Moisture complicates the crude cartoon a bit.

Some basics:

I found Chiefio’s response interesting. Seems to me he brushed aside almost all of what was being said about him. Seems also at least a few threaders here, high-end brilliant people as far as I can tell, got what he was doing wrong. Now, you may say you think what he is doing is irrelevant. It doesn’t sound that way to me, though. I want him to soldier on. The love you express for anomalies. I have to confess: it gives me the hives.

Anyone here recall Plato’s Allegory of the Cave?

Reading through these threads, and I do find them illuminating, but perhaps not so illuminating on the central problem here, which is politics, not science, I was reminded of Swift’s Battle of the Books, or Tale of a Tub. Chiefio says he’s–I interpolate–tired of academics. I agree. What we have with the likes of Santer, Briffa, Jones, Mann, Hansen et. al. are a bunch of tenured academics who think they have figured out how the world works. Worse, they want to use that information to save us from ourselves. Anyone here have a sense of history?

Heck, what is temperature? You’re talking about energy, right? Here I have a question: if the Earth has warmed by about .07 C over the last 130 years, which I think is the agreed upon number, what’s the big deal? Are we going to melt?

The issue now is the hockey stick, isn’t it? Is it valid or not? Can one of you whizzes please let me know? My sense is the diagram has been discredited. Am I wrong? Stupid? Biased? Blind?

Another question. Do we even know how much of a role human produced CO2 has played (& plays) in any temperature increase? Has the relationship between CO2 in the atmosphere and rising (if they are rising) temperatures been causally established, with the lines of force going in one direction? Yeah, I know what happens in a jar in a lab. But the atmosphere? Aren’t all these Hansen CO2 forcings THEORETICAL? Or is everyone here on board?

You can talk all you want about the physics behind a tsunami or a tree limb that’s about to come down. My primary interest is in what’s going to happen to my car. That’s how Chiefio thinks. I want him in the boat with me when it springs a leak–not one of you whizzes with a Blackberry.

Tone. I read Chiefio and I just appreciate the heck out of his tone. He approaches problems the way I would (former electrical engineer). He doesn’t listen to the academic chatter, and chatter is what it mostly is. He reminds me of Pirsig in Zen and the Art of Motorcycle Maintenance. Maybe his new path goes nowhere; maybe it leads to something. He isn’t afraid to get his hands dirty. And you know what else? He’s doing it on his own dime.

Every time I visit RealClimate I get depressed. Know what’s depressing? They have an answer for everything. Everything! They are NEVER wrong. I’m going to go ask them if there’s a God because I know that if anyone knows, they do. The smugness on that blog is thick as lard. And it makes me suspicious. Know why? Because the only people I have ever met in life who have an answer for everything are liars.

Does everyone posting here believe that Mann, Santer, Briffa, Jones and the rest of their crew are honest men? That’s a key point to me. So step outside your self-imposed boxes. Take off your night-vision goggles. If you think these men, who after all, have a considerable vested interest in a particular outcome–a notion that is or ought to be like Kryptonite to a serious scientist–have never banged on the data to get it to arrive at the right shape, please. Tell me. I need to know!

I mean, Jones’s remark about not showing his data b/c the data requester might try and prove him wrong was astounding. And it totally, utterly, irrevocably annihilates his credibility. Send him a copy of Cargo Cult Science. Then fire him.

Everyone here appears to know his or her stuff. Thank you for the time you take on the various facets of this AGW issue. I may sound obtuse, but believe me, I’m learning from what you say. You are all light years beyond me in your grasp of physics. But with only a few exceptions, most of you are on board with the AGW hypothesis–or so it seems. But none of you has disowned the liars (or are they not liars?) in your midst. What’s up with that?

“Thermodynamically, the changes are all you really need. If you really want to crudely simplify, if the heat capacity in a box is constant, then if you put in some thermal energy, then the temp will rise by some delta T. Its the change thats relevant. Moisture complicates the crude cartoon a bit.”

Based on my analysis, whether you use anomalies or absolute temperature should not matter.

And, thermodynamically, what is your reference temperature when you refer to delta T? It is really the *** initial *** temperature of the system and not it’s time average over the time interval being examined. The first law for a system is:

dE/dt = Q-W

where E can be simplified to mCp(T-Tref) if you neglect kinetic, potential and other energy changes, and assume spatial uniformity. Neglecting the work term, you get (note Tref is a constant):

mCpdT/dt = Q

where m is the mass of the system and Q is the heat transfer rate (in watts). So it is clear that “warming” occurs if the temperature increases from an *** initial *** state. I do not see where the time average temperature has any relevance, since you can choose Tref arbitrarily.

I don’t follow your difficulty. Right there, you can see that dT/dt is what’s relevant. And working with anomalies delivers you the dT/dt that you want. The absolute scale of T is not important, so you can plot anomalies on whatever arbitrary scale you want. It’s the changes with time that you care about.

“Based on my analysis, whether you use anomalies or absolute temperature should not matter.”

Only if you had the perfect thermometer field I described above. Thermometers absolutely everywhere, with all of them operational all the time. But you don’t have that. You do, however, have sufficient thermometers to describe the anomaly field, and this gives you the dT/dt that’s thermodynamically important.

Titan28 (Comment#36367)

> The issue now is the hockey stick, isn’t it? Is it valid or not? Can one of you whizzes please let me know?

No, the HS is likely not valid, but no, that is not the issue here. This thread concerns data integrity and analysis of the instrumental temperature record ~1850-present.

“I dont follow your difficulty. Right there, you can see that dT/dt is whats relevant. And working with anomalies delivers you the dT/dt that you want. The absolute scale of T is not important, so you can plot anomalies on whatever arbitrary scale you want. Its the changes with time that you care about.”

How are “anomalies related” to dT/dt?? dT is the change in the ** absolute ** temperature. You can certainly insert your reference temperature temperature (constant) and that will not change dT/dt, that is d(T-Tref)/dt = dT/dt. Your statement “the absolute scale of T is not important” is incorrect when applied to the first law of thermodynamics. Again, my contention is that anomalies are only important for interpolating data to fill in missing temperatures in the historical records.

Carrot Eater:

I don’t follow your difficulty

Me either.

dT/dt eats offsets for breakfast, lunch and dinner.

Only if you had the perfect thermometer field I described above

I think it’s possible to very possible to compute the real temperature field (at least a spatially and temporally smoothed version) assuming there was a sparse network of C1 or C2 level instrument sites, or if one uses satellite IR data to help infill missing and identifying microsite issues (which would show up as offsets relative to the global field)..

Frank:

dT is the change in the ** absolute ** temperature.

Right.

T(final) – T(initial)

The only way that gets shifted is if your effective local temperature calibration is wrong. [That can happen, separate topic.]

Absolute temperature is utterly irrelevant to whether climate warms or to the the relationship between temperature change and heat energy transfered.

I would like to suggest to Carrot Eater that he make it clear to E.M. Smith that he is not Eli Rabbett. Perhaps a private mail? I think there is unnecessary conflict here due to an unfortunate conflict of Internet handles.

bob

“Absolute temperature is utterly irrelevant to whether climate warms or to the the relationship between temperature change and heat energy transfered.”

My contention is that it doesn’t matter if you average absolute temperatures or anomalies. Do what you like – you’ll get the same result (again noting that we assume that temperature records for the data being averaged are complete and don’t need “filling”).

In any case, it is also clear to me that the global mean surface temperature anomaly is just a proxy and does not necessarily say anything (thermodynamically speaking) about the level of “warming” or “cooling” since we do not have temperatures measured throughout the atmosphere over the historical period of interest. (Note – satellite temperature measurements are better suited to estimating warming/cooling since they can measure temperatures through the depth of the atmosphere).

I found Chiefio’s response interesting. Seems to me he brushed aside almost all of what was being said about him. Seems also at least a few threaders here, high-end brilliant people as far as I can tell, got what he was doing wrong.
.
you got this one wrong. Chifio is misleading people. he talks of warming bias, without any warming bias. he talks of manipulation, without any manipulation.

Re: Frank K. (Mar 4 09:21),

The measurement of the global mean surface temperature anomaly of the earth (GMST) is not intended to quantify changes in enthalpy in any particular volume. The fact that some later use the value to estimate the change in enthalpy despite its shortcomings in that regard doesn’t make the measurement GMST meaningless for all purposes. It only means that it’s has deficiencies when used to estimate the change in enthalpy of some particular volume. GMST is still a good metric for testing model predictions of GMST. It’s till a good metric for detecting whether or not changes are occurring.

These questions are not thermodynamic in nature; the fact that GMST is not a measure of enthalpy is not a shortcoming when we use it to answer these questions.

Frank K. (Comment#36392)

First, on the thermo: Correct that surface air temp does not give you the total energy content of the earth system. But it’s part of that picture, and relevant to human life. But this is why there are also satellites and weather balloons and ocean measurements, to try to get the big picture of the total thermal energy. Alternately, one could try to measure the actual radiation imbalance at the TOA, though this is not easy to do directly with accuracy.

Second, you absolutely don’t get the same thing, when working with only absolute temperatures. Just take a glance at EM Smith’s random looking graphs; that’s what you get. Stations coming in and out of operation are going to really mess with such an approach. When using anomalies, stations coming in and out don’t matter so much, so long as they were giving redundant information about the trends. If you only had one thermometer in a part of the world that was cooling, and you lost it, then you’d have a problem.
_____________________
schnoerkelman

Does Eli have any history with EM Smith? I don’t think EM Smith particularly cares who I am. I am harshly critical of his methods; that is all. Though perhaps it’s Watts/d’Aleo I should be more critical of:
_____________________
Titan28 (Comment#36367)

The problem seems to be this: EM Smith muses about certain things; those things have various possible implications that he has admittedly not demonstrated. d’Aleo/Watts are then taking that and making all sorts of unsupportable claims, in no uncertain terms. The SPPI report says, with certainty, all sorts of stuff that nobody has been able to actually show. Somebody has to take responsibility for that.

carrot eater (Comment#36396)

“Second, you absolutely dont get the same thing, when working with only absolute temperatures.”

This is not true as I’ve demonstrated. However, that is NOT the same thing as using absolute temperatures to interpolate data and filling in missing data. I agree anomalies can be used to interpolate data, but the method for doing so is not unique. But again, if you like this proxy, fine…

Frank K. (Comment#36398)

Your algebra exercise didn’t take into account a station dropping off and not reporting anymore, nor a station starting up. Go back and look at it. If there are two stations, both trending at +0.2 C/decade, but one is at the cold mountaintop and the other at the warmer valley below, then you get a spurious mess in the absolute average if one station stops reporting at some point. Using anomalies, no mess.

Using anomalies, you only get spurious error if you don’t have a thermometer in a location that’s trending differently from the neighboring thermometers.

As for filling in missing data: depends on what you mean by that. The GHCN raw v2.mean has no infill of missing data points. None. That’s why you have all the -999s in the raw file. If that’s not what you meant, then nevermind.

lucia (Comment#36395)

“GMST is still a good metric for testing model predictions of GMST. Its till a good metric for detecting whether or not changes are occurring.”

“These questions are not thermodynamic in nature; the fact that GMST is not a measure of enthalpy is not a shortcoming when we use it to answer these.”

I fully agree with your assessment, Lucia. There’s nothing wrong with using GMST as a metric, as long as it is calculated consistently in the temperature data analysis and the climate codes. It is what it is. My only objection is to folks who say that you should never average the absolute temperature in climate science…my response being that it really doesn’t matter, you can do it either way (anomalies or absolute temperatures) and come up with a valid proxy, again with the proviso that you have a sensible method for filling in and/or correcting the temperature records – that’s where GISTEMP et al. add “value” :^)

“Your algebra exercise didnt take into account a station dropping off and not reporting anymore, nor a station starting up. Go back and look at it. If there are two stations, both trending at +0.2 C/decade, but one is at the cold mountaintop and the other at the warmer valley below, then you get a spurious mess in the absolute average if one station stops reporting at some point. Using anomalies, no mess.”

Mr. Carrot – at the risk of repeating myself for the tenth time, my analysis assumes complete temperature records. Your contention is that you need anomalies in order interpolate and correct the historical records – and that’s fine! You can do it as you wish. The “reference station” method embodied by GISTEMP is a reasonable approach, but again it is not unique. As for getting spurious “messes”, it probably matters more if your thermometer is located next to a sewage treatment plant or not versus valley/mountain location…

FrankK

My only objection is to folks who say that you should never average the absolute temperature in climate science…my response being that it really doesn’t matter,

Clearly, which method is best depends on what information you are trying to discern and also the data available.

I’d say that given the surface temperature data available back to 1800, if you want to detect whether or not the earths surface is warming on average, you should not average absolute temperatures. These leaves open the possibility that averaging absolute temperatures might make some sense in some other context. If you want to find that context, use it!

Clearly, you can’t just make context free declarations about how to average.

Of course, the fact taht we can’t make context free declarations about “bias” or “meaning”, is precisely why EM Smith’s insistence that he is just talking about bias in DATA is meaningless. What does bias in the DATA that even mean? He doesn’t seem to mean in any single individaul thermometer or thermistor. He doens’t seem to mean any single individual station. Bias in those cannot be diagnosed by inspecting GISSTemp code or running cases.

If he means that if one computes the simple average over all thermometers, that average is biased relative to the change in the earth’s surface temperature: Yes. Everyone knows this. That’s why NOAA, GISS and CRU don’t try to detect the change in the earth’s surface temperature that way. So, as a practical matter– you know, engineering, not academic– telling people DATA are biased in that way is pointless. It’s like arriving at a blog and writing posts telling people they can’t fry eggs by cracking them on a snow bank. No one plans to fry eggs that way; the observation is pointless.

Frank K. (Comment#36406)

At the risk of repeating myself for the nth time, you can only get away with using absolutes if you have a perfectly continuous set of stations. You do not, and never will, so you can’t use absolutes. Some stations start in 1760. Some in 1920. Some end in 1950. Some in 1990. This is reality. Reality says you can’t use absolutes. EM Smith recognises that using absolutes has this shortcoming, but then spent months making such graphs anyway, saying it will help him investigate better. Well, he can investigate away; the objection is in he or d’Aleo or Watts making completely unfounded statements based on his absolute average graphs.

Now, the more subtle point: even if all stations had exactly the same lifespan: you might get reasonable trends from the absolute average, but you should still use the anomaly terminology, because you aren’t measuring the true absolute average temperature of the earth. You’d need an infinite number of thermometers to say you did that. Rather, what you have is the absolute average of thermometers placed in the particular places they are placed.

However, if you said you had the average anomaly of the earth, give or take some sampling error, you’d be on more firm ground. And since dT/dt of the average anomaly is the same as dT/dt of the true absolute (if it were possible to measure), that’s all you need.

Carrot Eater:

At the risk of repeating myself for the nth time, you can only get away with using absolutes if you have a perfectly continuous set of stations. You do not, and never will, so you can’t use absolutes

I think you’re being too absolutist here.

You can work with absolute temperatures, there might even be something one could learn by doing it carefully.

However, I do think if you aren’t extremely careful that you will end up with much nosier and probably biased data when you try and compute the global mean trend from it.

And if you are extremely careful, you’ll probably end up with something akin to GISTEMP for how you compute the global mean temperature from your reconstructed surface temperature field.

Carrick (Comment#36426)

For this to have meaning, we have to isolate what this careful treatment could possibly be. In the context of trying to find out temperatures have changed, at least.

In my mind, you’d have to do it with a selection of stations with zero missing months over the time period being studied.

If you do that, you can end up with a reasonably shaped graph, if you had enough stations (which you won’t, unless you infilled missing points). And in any case, you still would not want to name the final product the absolute temperature of the earth or even that region; it’s just be the absolute temperature of the particular thermometers that you had.

Carrick (Comment#36426) March 4th, 2010 at 1:20 pm

“I think youre being too absolutist here.”

I agree – Thanks Carrick.

Mr. Carrot doesn’t want to read what I’ve written, so I’ll let him be. Good luck with those anomalies.

lucia (Comment#36416) March 4th, 2010 at 12:32 pm

“Clearly, which method is best depends on what information you are trying to discern and also the data available.”

I agree, and I think we all agree that the fundamental problem is that the historical records are imperfect and require interpolation and corrections (e.g. TOB, UHI, siting changes, etc.) as a prerequisite to deriving any metric like GMST. Perhaps what my questions are illuminating is that the concepts of “warming” and “cooling” and their relation to a global temperature metric should be studied a bit more carefully.

Carrot Eater, I can imagine a procedure whereby one retains the offsets and trend corrections that get made by the adjustment software. I don’t think the missing station issue is going to be as big as you seem to think—could be wrong. My suspicion is that the absolute temperature field will be a lot noisier is all.

Carrick (Comment#36440)

I don’t follow what you have in mind, could you spell it out a bit more?

As for missing stations: It is a big issue; it’s part of the reason (the main reason?) why EM Smith’s plots look nothing like the anomaly plots.

Carrot Eater, EM Smith’s big problem is that he isn’t area weighting his stations. Most of the effect of the station drop-out disappears by area weighting.

In principle you start with

T_station(t) = C_station(t) * T_measured(t) + Offset_station(t) + Noise(t)

Here T_station(t) is the “true” temperature of the station, C_station(t) is the “calibration error” and Offset_station(t) is the offset error of the station. In practice, the offsets for a given station would be represented by a number of discrete steps, as would be the temperature trends.

The only big difference with using absolute temperature is that you have to calibrate across temperature sensors that are in the same geographical region. This presumably could be done using satellite surface infrared measurements.

It’s main weakness, that the offsets are constant over time, is actually shared by anomaly based analyses too. That’s why trying to reconstruct the full temperature field might be illuminating.

It might also impact our understand of how to reconstruction temperature from a limited number of temperature proxies for the pre-instrument period.

But to emphasize, if what you are interested in is the global trend in temperature using the ground-based measurements, there’s no reason to go to the trouble of calibrating individual sites.

Yuck, you’re making this unduly complicated. Let’s keep surface and satellite measurements independent. Is there any existing satellite measure of the land surface, anyway? I didn’t think so.

I’m not entirely sure what you’re trying to capture with your first and zero order correction terms, but I think I’m correct in saying they can’t be gotten at from the surface record alone?

As for area weighing: if you take spatial averages, but neglect to use anomalies, I think you get the plot made by Eugene somebody-or-other, linked in one of these threads. It looks better, but still not quite right. Though in a grid box with a wide variation of conditions, it could look a right mess.

Likewise, a plot with anomalies but not spatial averages looks better, but not quite right. Maybe take the global and the US, smear them together, and that’s maybe roughly what you’ll get. This is where Zeke started, if you look in the timeline above.

So maybe both are equally important.

Carrot Eater:

Yuck, you’re making this unduly complicated. Let’s keep surface and satellite measurements independent. Is there any existing satellite measure of the land surface, anyway? I didn’t think so.

I didn’t say it was simpler, and as I pointed out there’s no reason to do it if all you want is global mean temperature trend. Nor do I plan on doing this myself, but it can be done.

I’m not sure how much can be done with the satellite measurements, but if I had to do it, that’s what I would want to use. (Think Santer’s reconstruction of Antarctica.) It could be done in ares with good regional coverage, but in sparsely samples areas, you’re pretty much stuck with satellite measurements.

To be honest, I was guessing as to which was most important.

In order to which corrections mattered most, you’d have to subtract that effect from the “best” global mean temperature reconstruction. I do know that a lot of the effect of dropped stations goes away simply by area weighting. The remainder would disappear if the individual station data were correction calibrated (I’m guessing that the offset correction is the larger of the two).

Carrot Eater:

Yuck, you’re making this unduly complicated. Let’s keep surface and satellite measurements independent. Is there any existing satellite measure of the land surface, anyway? I didn’t think so.

I didn’t say it was simpler, and as I pointed out there’s no reason to do it if all you want is global mean temperature trend. Nor do I plan on doing this myself, but it can be done.

I’m not sure how much can be done with the satellite measurements, but if I had to do it, that’s what I would want to use. (Think Santer’s reconstruction of Antarctica.) It could be done in ares with good regional coverage, but in sparsely samples areas, you’re pretty much stuck with satellite measurements.

To be honest, I was guessing as to which was most important.

In order to which corrections mattered most, you’d have to subtract that effect from the “best” global mean temperature reconstruction. I do know that a lot of the effect of dropped stations goes away simply by area weighting. The remainder would disappear if the individual station data were correction calibrated (I’m guessing that the offset correction is the larger of the two).

By changing about three lines in ccc-gistemp, one can remove the “anomalizing” steps and make it compute absolute temperature means rather than anomalies. The GISTEMP algorithm uses absolute temperatures up to the very end of the gridding step (step 3): station records are combined, the urban adjustment is applied, and gridded temperatures are all calculated in absolute temperatures. *Then* a reference annual series is computed for each sub-box, and subtracted from the sub-box series to give anomalies. This calculation (and ones like it in step 5) could simply be omitted. The story for ocean temperatures is a little different: historical ocean temperatures are stored as anomalies, and recovering the absolute temperatures would require a little more work.

But (a) I suspect absolute temperature means just aren’t very interesting, from the point of view of climate science, and (b) the numbers computed would be very dependent on station conditions, far more so than the anomalies.

Nick Barnes:

The critical part is before that, I think. The important part to using anomalies is in the variable biask in step 3. So you’d just have to set that equal to zero. And again in step 5, I think?

The second you use the offsets in biask to nudge different stations up and down before adding them to the growing mean, you’re really using anomalies of a sort; they just aren’t centered on zero yet. Centering it on zero is just cosmetic.

Re: Nick Barnes (Mar 5 03:48),

But (a) I suspect absolute temperature means just aren’t very interesting, from the point of view of climate science, and (b) the numbers computed would be very dependent on station conditions, far more so than the anomalies.

I think statement (a) is overbroad. Absolute temperatures are a poor way to detect or report trends warming in the observational series. This is owing to the fact that thermometers have never been optimally distributed, stations move, stations drop out etc.

However, absolute temperatures, absolutely precip, absolute snow pack are interesting if we want to evaluate whether physics in models is at all reasonable. To give a reductum ab adsurdum example: if a model predicted the earth was currently a large frozen snowball with an average temperature of -60C or that the earth was a firey inferno with average temperatures at 110C and with all water evaporated from the oceans, we would have reason to doubt they correctly capture physical processes sufficiently well to improve our understanding of climate.

Models are certainly not that that bad, but the example does illustrate that absolutes can be interesting in come context. Comparisons of absolutes not so splendid though, and I suspect I think modelers would rather convince themselves and the public believe that absolutes are un-interesting per se. In fact, absolutes are interesting and proper discussion of the uncertainties in models should be showing the obvious discrepancies between between predictions of absolute temperatures and observations.

That said: we should be able to detect changes in the surface temperature anomaly with greater precision than we can detect the absolute temperature itself. To those outside research communities it’s important to point out that this situation is not uncommon in science or engineering. Those doing experiments in labs used relative measurements quite frequently.

Re: Nick Barnes (Mar 5 03:48),

*Then* a reference annual series is computed for each sub-box, and subtracted from the sub-box series to give anomalies.

It’s difficult to understand Chiefio’s gripe. But I suspect this is the step where Chiefio thinks the bias creeps in. Buried in all the “basket A/basket B” and “spherical cow” stuff, I think he is saying the problem is that the baseline is not computed for each station individually.

So, suppose during the baseline period, the grid box contains a hot station and a cold station. You compute the baseline temperature for the grid based on those.

Then after the baseline ends, you drop the cold station. Going forward, you measure only using the cold station. However, you still use the baseline based on the “hot+cold” station.

In this case, you could introduce a bias because you are no longer using the “pure anomaly” method. Does this happen? If it does, is there an “adjustment” step?

lucia (Comment#36518) March 5th, 2010 at 9:14 am

The relevant code is in step 3.

First, there isn’t a true fixed baseline in GISTEMP, in this sense. Stations are combined, based on their period of overlap with grid point average that’s being built up. So in that sense, the station’s entire history is part of the baseline; no more, no less.

When you add each station in, you add a bias (biask in the code) to account for it being warmer or colder than the growing reference set. So when you add it in, the fact that the station was hot or cold is completely lost. When you add biask, it’s no longer the absolute temperature; it’s a relative temperature. It’s this part that I don’t think EM Smith understood for several months. Hard to tell; I don’t see him mentioning that step explicitly.

So the risk for bias comes from the dropped station having a different trend from the survivors. Nothing to do with its absolute temperature.

Carrot–
I don’t want to read code to get an overview of an algorithm. If you think I think code is a efficient way communicate an algorithm, you are mistaken. I think code– even clear code– is one of the most inefficient ways to communicate the overview of what is done. I suspect many do– otherwise, many journal articles would say “I have submitted this code. It’s cool. Down load, it run it, play with it and see what it does!”

Code is only good for checking pesky details. I want words, examples of what the process does in idealized circumstances or a flow chart.

Do you have a link gives an overview words or even a flow diagram what step 3 does? Are there any ‘toy’ examples showing what the process does with a small number of stations dropping in and out? So, suppose we had a case where:
1) A grid contained data from 2 thermometers providing series 1 and 2.
2) The two sub-regions provided data from 1900-1980. One drops out. The second continues until 2010.
3) The baseline is chosen to be 1950-1970.

Now:
Given the raw data, what would the individual time series look like?
Given the raw data, what would the anomalies for series 1 look lie?
Given the raw data, what would the anomalies for series 2 look like?
Given the raw data, what would the anomaly for the “region” look like?

Or is this sort of simple toy problem not impossible to do?

The reason I’m asking this is that I think I might know what Chiefio thinks goes wrong. If he has identified something that contains even an element of truth, it’s worth trying to understand what that might be.

Lucia:

I don’t like reading code for such things either. The relevant material is in Hansen (1987), and so far as I can tell, the guts haven’t changed.

So here, I’ll provide a case study. A spherical cow, perhaps, but the ccc guys can correct me if the real cow does something different.

I’ll ignore months. Let’s work with years only. Follow along on your spreadsheet, for it to make sense.

You have a grid point. Equidistant from this grid point are two stations, A and B.

Station A is impervious to climate change, or even weather. From 1900 to 2000, it showed 15 C the whole time. So the average over any period is 15 C.

Station B is cold, but getting warmer. From 1900 to 1980, it went linearly from 0 C to 0.8 C. +0.01 C/year. In 1980, the caretaker died. The average value over its lifespan is 0.4 C.

What does GISS do? The stations are equidistant, so they’re evenly weighted. So forget the weighting. Station A is longer lived though, so it starts with A.

Now, it wants to add in station B. First, it corrects for the fact that B is colder than A. The difference in the mean values over the period of overlap (1900-1980) is 14.6 C. You use the entire overlap period, not a fixed baseline. So you add 14.6 C to every single value of station B. [This is the key part; at this point it's no longer an absolute temperature]. So now, the offsetted B goes from 14.6 C in 1900 to 15.4 C in 1980.

So now, GISS will average A with the offsetted B.

So the combined average will start at 14.8 C in 1900, rise to 15.2 C in 1980, then have a spurious jump down to 15 C in 1981, and remain at 15 C until 2000. If you want to express it as an anomaly, now you can pick whatever baseline you want and subtract the average. The shape of the curve doesn’t care.

You can see that in the period of overlap, the trend is the correct average of +0.005 C/year. After 1980, no trend (mathematically correct, but presumably a sampling error). If you zoomed in at the step in 1980-1981, that step is indeed spurious. But it’s much, much less serious than the step you’d get if you weren’t using the offset.

Now. Let’s go back and change station B to being a hot station, getting warmer. From 1900 to 1980, it went from 40 C to 40.8 C. The average over this period is 40.4 C. So to get the offset, you’d subtract 25.4 C from station B.

Go through the motions, and you get the exact same result.

GISS *does not know* the difference between a station going from 0 to 0.8 C, and a station going from 40 to 40.8 C over the same period. They get treated the exact same way.

This is why absolute hot and cold don’t matter. The trends do matter. If the trends are different from station to station, then when you drop a station, you get the incorrect trend, and you may also get a little spurious step. With enough stations in the mix, this step would generally get washed out so it doesn’t make any difference, but it is in theory a flaw.

Now, the First Difference Method is designed to avoid that spurious step, and it does. But it has other flaws.

It is somewhat possible that EM Smith has detected an example of the discontinuity that is actually visually apparent. I doubt it, but maybe.

Carrot–
Thanks. What you describe makes sense. Of course, the off sets are key.

One of the difficulties with Smith is that it is very, very difficult to figure out what precise problem he means. But it does sound like he is concerned that station A is not anomalized with station A. But if it’s done the way you say, it wouldn’t matter.

I’m going to go ahead and write “anomalies for dummies” — because there actually has been a request along that line. With some luck, as Chiefio elaborates, these ‘toy’ explanations will help use figure out where in the heck he thinks the problem comes up.

If there is a problem, it should be possible to figure out what step of combinations of steps cause the problem and it should be possible to demonstrate it on a toy problem.

I hope I got that all in correctly.. by the way, the climate anomaly method can suffer from the false jump as well. so if you want to be sure your record is free from that problem, look at a NCDC generated chart. and they get the same result as anybody else.

McKitrick, 2003

“Figure 3 shows the total number of stations in the GHCN and the raw (arithmetic) average of temperatures for those stations. Notice that at the same time as the number of stations takes a dive (around 1990) the average temperature (red bars) jumps. This is due, at least in part, to the disproportionate loss of stations in remote and rural locations, as opposed to places like airports and urban areas where it gets warmer over time because of the build-up of the urban environment.”

As seen on Lambert @ Deltoid
http://scienceblogs.com/deltoi.....itrick.php

Looks the timeline moves back well before the recent SSPI publication.

Nick Barnes – in this thread at Chiefio You entered into some dialogue with E M Smith.

http://chiefio.wordpress.com/2...../#comments

He posed the following questions to you.

“BTW, at this point I think that the ‘code issues’ in GIStemp are ‘the small fish’ and that the massive changes of thermometers used in GHCN are ‘the big fish’. If you can, it would be interesting to know if your code will accept “subsets” end to end: then feed it the “surviving 1176 locations” in GHCN (i.e. remove survivor bias) with the (deleted after about 1989) 7k or so bolus in the baseline period removed. If you are uncomfortable publishing such a benchmark, just knowing that subsets of data will run end to end would be helpful to know.”

Are you able to verify if your CCC version would be capable of performing this analysis without significant intervention?

@clivere: Yes CCC can accept subsetted inputs. This is exactly what I did for this article. Except that I split the input into those stations reporting in 1992 and after, and those not (instead of a 1989 cutoff, but whatever).