In previous comments Robert Way referred to some AIRS work that I’ve been doing ,so I thought I’d share that with Lucia’s readers. Some background first. Over the course of the last 6 months or so I’ve been collating a variety of satellite data sets. There are several projects I’m working on some are related to testing R packages but for the most part I am looking at ways to improve our temperature products during the 1979 to present period. As you know we solve the temperature field by first regressing out latitude and altitude and then krigging the residual or weather. There are two primary concerns: How high can we push the resolution ( we are at ¼ degree) and are we over smoothing the field. To improve the resolution I’ve been looking at regressing out other geographical features that drive temperature, some of which change over time.  Here is a short list of some of the geographical features we’ve been looking at: distance from coast, percent of water in the grid cell, terrain aspect and slope, cold air drainage, local vegetation, albedo, impervious area. More long time series of these features are coming on line. For example, for impervious area I should be able to use Landsat and create a monthly global gridded series of impervious area or use monthly albedo ( white sky or black sky). The issue of local smoothness is a related issue and in our AGU poster we compared the smoothness of the field to other products: Prism, UAH, RSS and several reanalysis. The problem with UAH and RSS is that their resolution is 2.5 degrees. I swapped mails with Roy about the possibility of re gridding down to 25km, but it would be a lot of work. At AGU I ran into the AIRS team and they had some products at 1 degree and another product at 45km. So, I got curious.
The AIRS data is in a nasty format called HDF. A single month of AIRS data comes in a 500mb HDF file and that file has over 700 SDS ( scientific data sets). As luck would have it the guys I worked with on the MODIS project in R were also working on a new utility (gdalUtils) that makes reading and unpacking HDF in R really easy. So, I started working on AIRS. To give you a hint with AIRS I’ve got cloud layers at 12 different pressure levels (1018 hPa up to 22 hPa or something like that ) so that allows me to look for effects on low clouds in conjunction with changes in GCR. Plot spoiler: no effect. When Cowtan and Way came out and folks wanted to question their use of UAH my first thought was to look at AIRS as it gives me temperatures from the skin up through multiple layers. And if I really want to push the envelope they have some swath level product (45km) at 100 pressure levels. Working with swath data would require another terabyte drive and for this preliminary probing I stuck with the gridded 1deg data. It’s days of download time if anybody wants the code.
The data set I’ve selected is AIRS Version 6, level 3 data. In particular I’ve selected a few interesting files from the over 700 climate data files that sensor delivers.  I showed a few charts in my post on Judiths , I’ll repeat those here:
Given the differences in methodologies for each of these products and given that they measure slightly different parameters, you can’t simply compare them. Below I show what the differences look like. AIRS skin temperature estimate the temperature of the surface, whereas Berkeley measures the air above the surface. Consequently we see that show up in the mean difference maps.
For a sense of whether it will work or not to use these satellite retrievals to inform our surface estimation, I look at the correlation. The effects of clouds is pretty evident, but over land we have good correlations
In terms of trends it looks like this
The Berkeley Average falls in between the Skin trend and the SAT trend as we would expect.
Finally, I’ll show the results for the Arctic (60N-90N). How well does our Baseline estimation of the arctic compare with AIRS Skin and AIRS SAT. Here is what the correlation looks like for SAT followed by the anomalies.
And this is what the difference series looks like. The spike in late 2003 I’m betting will be cloud related
At this stage what I’m seeing is that over the 2002 to present period that our method captures the trend in the arctic pretty well and the ground work here would get passed on to others to see if it was helpful or held up under examination. I haven’t done any comparisons with Cowtan and Way or Hadcrut. From my standpoint it made more sense to pass this data onto Robert and Kevin and let them delve into it deeper. However, I’ve yet to see any data from any source that would make me change my assessment that HADCRUT is biased low. I suppose one could argue it is the lower bound. In addition to this data there will also be some ice surface temperature series coming available that cover the entire 1980ish to present.
My concluding remarks come down to this. Cowtan and Way, Berkeley Earth and GISS all estimate what the non sampled areas in the arctic would look like. The way to test these predictions is by looking at out of sample data where we can find it ( Buoys for example) and to compare the trends we see from different products, reanalysis and satellite data sets. AIRS is one more data set to look at, the answer seems pretty clear.
The last note I will make is that the similarity in trends between Berkeley and AIRS should give one pause about claiming large UHI effects over the time period covered.
If folks what the time series, just say so in the comments and I’ll upload to Google Docs
Is this a trend in daily average temperature, max, min, or something else?
AIRS has two passes. One ascending and one descending.
with a equatorial crossing time of 130AM AND 130pm
As a first pass I average the ascending and descending for a Tave.
Of course that calculates Tave a bit differently than berkeley which averages the min and max. Its one reason why you cant do a strict comparison of absolutes.
To compare absolutes you’d have to calculate the hourly local time for every grid and then find hourly data from a surface station.
The more interesting thing is the correlation and the smoothness.
With my browser, I have to double-click on each map to see the scale on the right.
For the AIRS SAT versus Berkeley comparisons: The T(ave) maps look nicely similar, including near the equator. But on the correlation map, while most land and sea is pleasingly high — 0.6 to 1 pretty much everywhere else — it falls to near zero at places on the equator. Atlantic, Indian Ocean, Western Pacific.
Er… what does “correlation” refer to? The time-series trend of Berkeley vs. SAT, for the duration of the AIRS data?
Yes it falls to zero in those areas where you have persistent clouds. The 1 degree monthly product is built up from daily 45km swath data. So the monthly cells are sampled irregularly and that shows up in the areas that have high cloud counts.
Also this time period includes 2010.
and its more like .8 to 1, every else or rather in the areas we care about.
Correlation is on the absolute temperature.
So, basically the charts indicate that the information in AIRS may be usable to improve predictions in unsampled areas.
By the time its done flying ( say 2020 or so ) then I’d probably put the time in work more on the absolute series.
Steve: Yes I would love any time series that you can publish. I would also like a satellite (AIRS) to satellite (UAH & RSS) comparison, especially for the poles.
Do you have banded monthly series that would allow direct comparison to the ‘normal’ data sets, of Arctic/Antarctic, Ex tropical, Tropics, Hemisphere, etc. to other data sets?
The first three graphics seem to show a clear division between the USA and Canada, and since seeing a political division on a temperature chart seems counter-intuitive, there should be a plausible explanation. Is it
a) differences in measurement methodology between the USA and Canada makes the border show up?
b) differences in measurement density between the USA and Canada makes the border show up?
c) differences in climate between the USA and Canada is the reason that the border was drawn there in the first place?
Steve Ta
Only if you count Alaska as Canada. I think the issues might just be “latitude” and the degree to which one’s eye really pick up contrast when the color clicks over from ‘orange/peach’ to ‘blue’. Possibly better color choices could be made. Right now they seem to do “red– full intensity”, “orange= red+yellow but with lowered intensity”, “blue with low intensity”, “blue with high intensity”. That doesn’t mean this is the way the additive light goes– just how it looks to my eye.
For me, the transition from “very light orange” to “blue with no yellow or red in it” looks dramatic and happens near the latitude for the US Canada border. But if I look closely, I see similar fairly strong ‘latitudinal’ banding in most parts of the globe. (There are some deviations– but you can see red horizontally across the equator and so on.)
I suspect Mosher and the Berkely team have tried many color choices, so I’m not going to say any other choice would be better. But I think the strong “US Canada break” may be somewhat of an illusion.
Amac
I took the liberty of changing the width tag for display on the images. I hadn’t noticed that last night when I told mosher to go ahead and publish. People can still double click if they wish to see greater detail.
SteveTa
Blow up the picture and zoom in.
“To improve the resolution I’ve been looking at regressing out other geographical features that drive temperature, some of which change over time. Here is a short list of some of the geographical features we’ve been looking at: distance from coast, percent of water in the grid cell, terrain aspect and slope, cold air drainage, local vegetation, albedo, impervious area. More long time series of these features are coming on line.”
Would you expect these details (and with a monumental task to incorporate I would think) increase/decrease the confidence intervals/error bars for the temperature series?
Do you have CI/error bar estimates for your current work, i.e. are the series trends you are comparing statistically different?
I have been comparing the Cowtan Way Hybrid against GISS in filled temperature series for the period 1979-2012 in the Artic polar region using how well the mean temperature series follow the sea ice extent. If we assume that the regression of sea ice extent versus temperature should be linear then the CWH has a better correlation than GISS – but not by a lot. CWH R^2=0.86 and GISS R^2=0.78. Looking at year over year correlations by differencing you get CWH R^2=0.48 and GISS R^2=0.45. Sea Ice area does not correlate as well as sea ice extent, and in fact the year over year correlations are quite low.
I am currently looking at other aspects of the Arctic polar amplification in comparing CWH and GISS and to attempt to better understand a mechanism for the amplification. CWH gives a significantly higher amplification than the GISS CMIP5 climate models. With CWH, and to a lesser extent GISS observed, you have a plateaued mid latitude warming over a relatively long time period recent past with an Arctic warming continuing at the same (CWH) or nearly same (GISS) pace.
series
https://drive.google.com/folderview?id=0B9IIrBCUsRhIa2ZvMFQwRnRUMEU&usp=sharing
“Would you expect these details (and with a monumental task to incorporate I would think) increase/decrease the confidence intervals/error bars for the temperature series?”
marginally perhaps since they would reduce the residual that has to be krigged.
####################################
Do you have CI/error bar estimates for your current work, i.e. are the series trends you are comparing statistically different?
I haven’t checked the CI. The work flow goes like this.
I do preliminary investigations of data sets to assess their
useability (data coverage ) reliability ( look at validation
reports) theoretical assumptions ( Check the ATBD)
and applicability ( does it contain information that might
refine our estimate). For example I would take DEM
generate slope and aspect, then take a sample of data, say
the US, run regressions on lat alt and slope and if I see
something interesting, I pass it on and move to the next pile of data. If I find nothing, first pass, I move on and circle back later.
So with AIRS I find that it merits further investigation.
############################
I have been comparing the Cowtan Way Hybrid against GISS in filled temperature series for the period 1979-2012 in the Artic polar region using how well the mean temperature series follow the sea ice extent. If we assume that the regression of sea ice extent versus temperature should be linear then the CWH has a better correlation than GISS – but not by a lot. CWH R^2=0.86 and GISS R^2=0.78. Looking at year over year correlations by differencing you get CWH R^2=0.48 and GISS R^2=0.45. Sea Ice area does not correlate as well as sea ice extent, and in fact the year over year correlations are quite low.
My sense is that improvements over GISS will be marginal, but
technically interesting.
Understand that IDW and Krigging will produce the same result
roughly if the underlying terrain ( as modelled) is the same.
where the terrain is different ( look at greenland) you’ll see differences between the GISS approach which just interoplates over the grid and a krigging approach which models temperature as a function of physical geography.
To the extent that the arctic ‘terrain’ is homogenous an IDW and Krigged approach ( which models the terrain) will be very similar.
##########################
I am currently looking at other aspects of the Arctic polar amplification in comparing CWH and GISS and to attempt to better understand a mechanism for the amplification. CWH gives a significantly higher amplification than the GISS CMIP5 climate models.
The models are all over the map with regards to amplification.
http://berkeleyearth.org/graphics/model-performance-against-berkeley-earth-data-set#regional-amplification
######################
With CWH, and to a lesser extent GISS observed, you have a plateaued mid latitude warming over a relatively long time period recent past with an Arctic warming continuing at the same (CWH) or nearly same (GISS) pace.
############
As a non tech I found it interesting that the mountain ranges eg west side South America etc do show up as a lighter colour due to the coldness at elevation and that Greenland is so much colder almost similar to Antarctica. Japan and several other areas seem to have a large abundance of black ink not colour coded. Is this showing high temps i.e. An urban heat effect?
Do the time series you have confirm anomalous Arctic warming or just expected Arctic warming.
Steve Mosher:
Can you split this into Tmax versus AIRS SAT (day) versus Tmin versus AIRS SAT (night)?
I’d expect the nocturnal data to have worse correlation than the daytime, which is the thrust of my question.
If you can’t do it directly, then looking by season would be another way to approach it. E.g., I’d expect correlation between AIRS SAT and BEST for high-latitude winter should look worse than high-latitude summer. (It’s also possible that with the resolution you have, the differences might be smeared out though.)
Carrick < yes. I can get around to that.
In fact I started by comparing min and max. In terms of bias
Min is much better. Max had large northern latitude bias.
I started to look at sample counts and then switched to what is called the "joint" product were QC is done jointly.
So give me a bit and I can compare the Joint Ascending and Descending againt min and max
The files Im working with are at the link above.. in netcdf
Berk min and max, airs SAT min and max for the Joint product (J)
Thanks for this – it got me looking at AIRS again. The HDF was a pain – they use HDF4 and the R hdf package is for HDF5. I used a conversion utility, but h5r wouldn’t read the output. I looked at GDAL, but it’s going to take a while.
I found their Giovanni useful. I can make a plot of any variable over a period. Then if I ask to download data, it lets me download all the data files used to make it. If I ask for a 1-year average, I get 12 monthly files, gzipped and tarred. There’s a 200Mb limit for each request. A verbose ascii format if I want, or NCD or HDF. The ncd gives me all variables – not sure how to download a subset.
I looked at daily SAT. There’s a huge amount of missing data, reflecting I suppose what it sees in a day.
I expect BEST and AIRS SATs are a lot different. Most of BEST is really SST. And I don’t imagine AIRS is 1,5m above the surface. So I don’t know how well they should compare.
Which Berkeley temperature series are you using?
Steven Mosher (Comment #127420)
Steven, thanks for the replies. I should note, since I am a stickler for CIs, that the trends and correlations I noted for CWH and GISS are not significantly different using 95% confidence limits – or less.
“As a non tech I found it interesting that the mountain ranges eg west side South America etc do show up as a lighter colour due to the coldness at elevation and that Greenland is so much colder almost similar to Antarctica.
##########
That is one of the benefits of the kriging approach. We model temperature as a function of the physical geography. The residual is the weather.
Japan and several other areas seem to have a large abundance of black ink not colour coded. Is this showing high temps i.e. An urban heat effect?
###########
Thats a resulotion of my graph problem.
Do the time series you have confirm anomalous Arctic warming or just expected Arctic warming.
You see arctic amplification as theory predicts
Thanks for this – it got me looking at AIRS again. The HDF was a pain – they use HDF4 and the R hdf package is for HDF5. I used a conversion utility, but h5r wouldn’t read the output. I looked at GDAL, but it’s going to take a while.
###############
When I get hone tonight I will post the code for you.
The h5r ( from bioconductor) wont work thats why folks were doing gdalUlilities
The work flow goes like this
1. Download and install gdal ( not R gdal but the gdal libraries)
make sure the drivers support hdf4 ( like version 1.92 or something)
2. install RGdal
3. Install gdalUtilities
RGdal is basically just drivers and gdalUtilties wraps the gdal library with R. gdal is a tool that allows you to reproject, translate, warp etc all sorts of GIS canned proceedures
4. Download the AIRS data .. monthly daily whatever
5. Now write R scripts to call gdal. you use the gdalUtilities
to query the hdf find the SDS you want, and then gdal_translate() to pull
that SDS and transform it to geotiff.
6. After translating all the SDS you want into geotiff you are good to go
#####################################
I found their Giovanni useful. I can make a plot of any variable over a period. Then if I ask to download data, it lets me download all the data files used to make it. If I ask for a 1-year average, I get 12 monthly files, gzipped and tarred. There’s a 200Mb limit for each request. A verbose ascii format if I want, or NCD or HDF. The ncd gives me all variables – not sure how to download a subset.
I’ve tried Giovanni and all the other web tools. Its hard to make the work reproduceable so in the R community we have focused
on ( in Modis R for example) building tools to download /local archive/update from FTP. Then take the command line tools supplied by NASA and others ( MRT reprojection, gdal) and wrap those tools in R. As described above we work to get everything in a common format like geotiff.
########################
I looked at daily SAT. There’s a huge amount of missing data, reflecting I suppose what it sees in a day.
I expect BEST and AIRS SATs are a lot different.
1. Depends whether you re looking at ascending/descending etc
2. Depends on the location
Most of BEST is really SST.
1. Yes, one of the things I’ve looked at is using AIRS Skin for the
SST part.
And I don’t imagine AIRS is 1,5m above the surface. So I don’t know how well they should compare.
AIRS SAT as far as I can tell ( still plowing through documents)
is constructed from the other temperatures . There are a pile
of validation studies that im plowing through.. cool bias here, warm bias there.. etc
“Which Berkeley temperature series are you using?”
There are two versions of berkeley global: baseline and alternative
The alternative uses a prescribed SST under ice in the arctic, the baseline uses SAT over ICE.
because I work with satillite data ( gridded) I work with our gridded product. The gridded product lags somewhat in the production
chain. In any case for this work I take the gridded product ( 1 degree) and then to compile a time series I load it into R,add back in the climatology
and do a simple area weighted average of every time slice.
Mosh,
“Yes, one of the things I’ve looked at is using AIRS Skin for the
SST part.”
Skin can be a degree or so cooler.
Thanks for the advice on GDAL. I agree it’s worth making the investment for automatable tools.
Nick.
Thanks
Ive looked at a couple validation studies. My sense
Was the useable information would be the corelation
There can be no correlation found between temperature data and carbon dioxide, because it cools only by about 0.1 degree. You need to come to grips with the new 21st century paradigm shift in climate science based upon the gravito-thermal effect.
This is how absurd the old 20th century paradigm of greenhouse radiative forcing gets. They claim that you can work out Earth’s surface temperature by adding together the radiative flux from both the Sun and the colder atmosphere, and then bunging this total value into the Stefan-Boltzmann equation and out pops your answer 287K or 288K. Well it might well do if you fiddle the back radiation and then use the emissivity value instead of the absorptivity.
But there’s absolutely no physics to support the calculations. When you consider that about 70% of the surface is a thin transparent water layer, it is obvious that the solar radiation which mostly (like over 99%) passes through this layer into the thermocline is not what is determining the temperature of that thin surface layer. In fact the mean temperature of the thermocline is obviously less, and the mean temperature of all the ocean water is less again.
Oh, and the back radiation doesn’t even enter the surface layer – it just raises electrons between quantum energy states momentarily, and then those electrons immediately emit another photon which climatologists think is energy coming from the kinetic energy in the surface molecules, but it’s only electro-magnetic energy from the back radiation being thrown back in their red faces.
Can you provide a link to the Berkeley data you used because I cannot find a dataset which matches what is presented above.
Bill see the link above. Its in the google drive.
You will have to work with the netcdf files
Min and max. Apply the climatologies then
Area weight using the raster package.
Steven Mosher (Comment #127487) thanks, angech
Doug Cotton; Lucia with her heart of gold has created an entire thread for your posts, so please post there.
Steve Mosher, if my post here is too far off topic you can delete it. I have been doing some work on Arctic amplification and the thread where it might be better suited for posting is for all practical matters dead.
I have nothing conclusive and as a matter of fact have used rather simplistic methods on my first pass through the data and literature. What I find interesting is the differences in conclusions of the several published papers on the amplification source being primarily albedo feedback effect or from other sources. I have attempted to find evidence for albedo feedback and with my methods have failed. I find that Arctic open sea (from sea ice extent) produces a very linear trend with Cowtan Way Hybrid temperature anomalies for the 60N-90N region. The highest correlation is for the 70N-90N zone (r=0.93) but is also nearly as high for the 60N-90N region (r=0.87). The year over year changes also correlate well (r=0.68). There is no correlation of year over year changes for the product of monthly average open Arctic sea and monthly average solar radiation versus winter mean temperature anomalies. These are all based on the 1979-2012 time period. The indication here is that rising Arctic temperatures predictably cause more open sea but that feedback on temperature cannot be detected in this manner.
As an aside I am thinking that work on the Arctic amplification has been more in terms of climate model results than observations and whether that might change with the advent of newer temperature series like Cowtan and Way that claim to track the polar regions better than those existing sets. Also not much use has been made of the satellite troposphere data – and that is what I am looking at currently.
We had a rather lengthy discussion on Arctic amplification on another thread, where, as I recall, DeWitt Payne rather emphatically was pointing to the known mechanism of heat transport to the Arctic, and, I think, implying or saying that that should be a starting point for understanding the amplification mechanism. On doing more literature research on my own I think I better appreciate what I think he was saying. My current prospective on the matter is that while oceans play a role in the transport of heat from the tropics to the polar regions most of the heat is transported in the troposphere through the Hadley (30S-30N), Ferrel (mid latitudes) and Polar Cells (60S-90S and 60N-90N) circulation. Some global circulation patterns can change with the seasons and some are affected over longer periods of time by changes in multi-decadal oscillations. I believe the heat from the lower latitudes to the Arctic would appear in the upper regions of the TLT/TMT regions of the troposphere where that air would eventually sink near the pole to the surface. The global circulation and as it could affect amplification is much more complicated than I have summarized here and much of which I do not have anything near a complete understanding. What I think I need is sufficient knowledge to look for gross effects that might favor a particular mechanism.
I have been looking at various time period trends on a latitudinal and longitudinal bases from 30N to 90N with surface and troposphere temperature series. I see at some longitudes a “leakage” of warmer trending temperatures into the polar region in the UAH TLT series. I need to do more analyses and look at the TMT also.
Kenneth,
Not sure what I can add. However if I were looking for transport in the atmosphere I would what to look at various levels, not just TLT and TMT. with AIRS you can look at individual pressure levels.
sadly the series is not that long, but you can always wait
I see where KNMI Climate Explorer now has the Berkeley 1 degree temperature data set. That is my go to site for temperatures.
So, just to be sure I am following this: You are saying that your preliminary analysis suggests Cowtan and Way’s new estimate of the arctic contribution is unlikely?
What are you referring to when you talk about UHI? Was their analysis calling the missing heat they thought they found in the arctic (due to unsampled grid cells as I recall) a UHI effect?