AGW alarmists, led by the “hockey team”, have dismissed criticisms that urban heat-island effects have been distorting surface temperature measurements upwards. Now Vincent Gray, a reviewer of the 2007 IPCC report, says this: not only is the single paper on which this dismissal is based fraudulent, the hockey team knows it’s fraudulent and keeps citing it anyway!
Paleoclimatologist Eduardo Zorita writes: “I may confirm what has been written in other places: research in some areas of climate science has been and is full of machination, conspiracies, and collusion, as any reader can interpret from the CRU-files.”
A Franco-Russian geomagnetics research group who was rebuffed when it tried to get primary temperature datasets from the CRU has assembled its own series of average temperature efforts by going back to ground-station measurements that the hockey team has never had an opportunity to “correct”. The result?
Aside from a very cold spell in 1940, temperatures were flat for most of the 20th century, showing no warming while fossil fuel use grew. Then in 1987 they shot up by about 1 C and have not shown any warming since. This pattern cannot be explained by rising carbon dioxide concentrations, unless some critical threshold was reached in 1987; nor can it be explained by climate models.
The report on this is well worth reading, as it goes into some detail on how the geomagneticians’ statistical methods produced a different — and much higher quality — result than the IPCC did. Among other things, they used daily rather than monthly averaging and avoided suspect techniques for statistically inferring temperature at places it hadn’t actually been measured.
Interestingly, their calculation of average temperature in the U.S. says “The warmest period was in 1930, slightly above the temperatures at the end of the 20th century. “. Could this inconvenient warm spell be what the VERY ARTIFICAL correction was intended to suppress?
I can almost pity the poor AGW spinmeisters. Perhaps they still think they can put a political fix in to limit the damage from the CRU leak. But what’s happening now is that other scientists who have seen the business end of the hockey team’s fraud, stonewalling, and bullying are beginning to speak out. The rate of collapse is accelerating.
Eric,
when I first saw the hockey stick, it was obvious the paleoclimatologists weren’t statisticians; statistics, as we all know, is a very rigorous science in and of itself; my very scant exposure to stats (b-school marketing) was to deeply incorporate the axiom: if the stats confirm your model too well, then you have imported your bias into the model; anything (hockeystick graph) showing the kind of 1:1 correlation of AGW “research” HAS to be biased because stats aren’t that finely powerful on a model of global scale; unfortunately this sort of stats-ignorance pervades our society, try reading “The Last Well Person” by Dr. Nortin Hadler to see how this plays out in the BigPharma marketing blitz on medical trials science.
Don’t even mention the sum of squares equals a negative issue…how much noob fortran programming like that is actually in the the works?
I find a lot of the comments on other blogs (e.g. Boing Boing) to be rather telling. There’s been some huffiness about the “theft” of the data, largely from the same camp that cheers whenever the NY Times or Washington Post publishes something leaked from the Pentagon. (Personally I think the data came from a whistleblower inside the CRU, and haven’t we been taught that whistleblowing is always a good thing?)
Similarly, many normally pro-transparency/anti-DRM/”information wants to be free” folks are making excuses for scientists ignoring FOI requests and keeping raw climate data secret because the scientists say the governments they got it from asked them to. So apparently if the Pentagon or Apple wants to keep some things secret it’s scandalous and an impediment to Truth and progress, but when climate scientists do it while claiming the sky is falling, well, we shouldn’t pester them to show their work because it distracts them and they made agreements to keep some raw data secret. Well alrighty then! I guess it’s OK to remake the world’s economy on their say-so.
Maybe I’m just a right-wing denialist, but isn’t ‘urban heat-islands distorting the temperatures upwards’ pretty close to ‘urban heat islands are an obvious, redundantly provable Anthropomorphic cause of Global Warming’?
Unless your budget depends on Al Gore’s Carbon fraud.
And ‘consensus science’ is looking like proof of Ronald Syme’s axiom that ‘when consensus is seen, previous conspiracy must be assumed’.
But then I’m the goober who can’t tell one fake hockey stick from another.
What PaypayaSF said.
Great article: been following your AGW coverage for the past few days. Blogged it on Spin, Strangeness, and Charm (click on my handle for link). [Hopefully this copy won’t be held for moderation as it has no link embedded: Eric, feel free to whack this copy if you already approved the 1st!]
Is this the “Lincoln moment†of AGW? (“You can fool some of the people all of the time, all of the people some of the time, but not all the people all of the time.â€)
I personally regard myself as an AGW agnostic. It may well be that AGW is indeed a major perturbation on climate; or it may be a minor perturbation compared to some man has no control over; or it may get buried in the noise. Studying it more and deeper, and in more detail, will only benefit us all. And there are plenty of non-AGW reasons to get serious about looking for clean and renewable energy resources. (Among other things, finite oil supplies will one day be far more valuable as feedstocks for the chemical industry than as fuel.)
The behavior of the ’strong AGW’ camp, however, is increasingly becoming a negative role model for any intellectually honest scientist. “Dear G-d, bring me in the company of those who seek the truth and spare me from those who have found it.â€
Nice find!
So doing some searching around netted the actual Le Mouel, Courtillot et al paper at this location.
Still looking for data and models.
Oh and if you want more shenanigans… New Zealand is getting warmer… apparantly
>AGW alarmists, led by the “hockey teamâ€, have dismissed criticisms that urban heat-island effects have been distorting surface temperature measurements upwards.
Note that “hockey team” refers to the paleoclimatologists; i.e. the people making hockey-stick shaped climate reconstructions. It’s not a catch-all epithet for all climatologists, and it’s confusing to use it as one.
>The report on this is well worth reading, as it goes into some detail on how the geomagneticians’ statistical methods produced a different — and much higher quality — result than the IPCC did.
Confirmation bias again. These results are only “higher quality” because they agree with your preconceptions. Look carefully at the graphs: global temp, and piecewise linear smoothed Europe and N America temps are compared to Europe and US based on a subset of the data. How can you even tell that they’re saying different things?
>Interestingly, their calculation of average temperature in the U.S. says “The warmest period was in 1930, slightly above the temperatures at the end of the 20th century. “
I’m pretty sure GISTEMP says the same thing. That’s the difference between global warming and local warming.
When I was a kid a 25 years ago, I was convinced that by 2010 we would have conquered the Moon and be involved in the ‘terra-forming’ of Mars and Venus. I had many of the “numbers” those enterprises would depend on memorized. I have forgotten quite a bit since then, including ‘this’ percentage. Upon stumbling across ‘this’ percentage I smacked myself in the head thinking “how could you have forgotten ‘this’*”? As I walked around work I asked my fellow employees ‘this’ question. “Got a second? Good cause I’m going to ask you a question and don’t worry about getting the answer exactly right as all I’m looking for is the immediate answer that pops into your head, kind of like the random word association studies. Ok so first number that pops into your head ‘What percentage of the Earths atmosphere is Carbon Dioxide.
So what number did YOU think of? I should be online again beginning of next week and if no one has posted the answer I’ll post it then. My little survey, which is in no way a large enough sample, has returned 30 answers so far with 4 answers being with in 1%.
*my other head smack that day was ?”wait a minute, What Free Market Health Care”?
@ JonB >Oh and if you want more shenanigans…
Debunked here. Looks like it’s the deniers at the NZ CSC that are involved in shenanigans.
Note that this is the same NZ CSC founded by Vincent Gray quoted above.
@bruce >Maybe I’m just a right-wing denialist, but isn’t ‘urban heat-islands distorting the temperatures upwards’ pretty close to ‘urban heat islands are an obvious, redundantly provable Anthropomorphic cause of Global Warming’?
No, because we have no cities in the ocean.
Doesn’t look debunked to me, pete. They put forward a few cherry picked counter examples while calling the critics ignorant and dishonest. All they have to do here is release the data and show the adjustments, then people can see if they make sense. But they won’t release the data. It’s not as if adjustments are automatically correct. GISS, for instance, had a temperature in NY City that jumped up about 2 degrees F around 2000. Seems that the adjustment for population was off because they were using the population from 1900. And the form of the adjustment itself was a power of the population; my guess is that someone plotted points on log-log paper and drew a line through them. Now suppose that formula is off, then all the adjusted data becomes biased.
Ideally, a location change will be made with the previous instrument in place so that the average temperature difference can be determined. But that isn’t always the case and then much depends on how the adjustment is made. Consequently the adjustment becomes open to bias on the part of the adjuster, bias of which they may not even be aware. In the case of NZ, it is strange that the adjusted and raw temperatures should differ so much, it is even stranger when you consider that the primary effect of urbanization is an increase in temperature. When raw and adjusted data differ to that degree any reasonable person should be suspicious and ask for clarification.
More on “value added” data here.
Amplifying on what Pete says, shockingly little of the Earth’s surface is actually city. I mean this with no sarcasm whatsoever: Because most of what we hear about takes place in a city or some other place where civilizations rests heavy on the land, it’s easy to think that we already live in Asimov’s Cities of Steel, but for a few meadows somewhere out in the Western US. The truth is that very little land is actually city, and consequently, the urban heat island effect can’t be having much effect on the macro climate, in terms of how much heat the whole planet is retaining.
Moreover, when 69%-ish of your temperature stations are within ten meters of an artificial heating source, it’s easy to get a skewed view of what’s going on due to urban effects. In a more perfect world, 69% would have been a number I pulled out of my ass, but alas, I did not (search for “station quality ratings”).
>Confirmation bias again. These results are only “higher quality†because they agree with your preconceptions.
No, they’re “higher quality” for the exact reasons I cited. Finer temporal grain, and no noise introduced by dodgy interpolation methods.
>>Confirmation bias again. These results are only “higher quality†because they agree with your preconceptions.
>No, they’re “higher quality†for the exact reasons I cited. Finer temporal grain, and no noise introduced by dodgy interpolation methods.
I was suggesting that, had they not matched your preconceptions, you might have noticed that they’re based on a much smaller sample size.
ESR says: You’re wrong.
Murph:
Oooh oooh I know the answer…(waves hand excitedly) I won’t blow it for everyone else though. (but most people will be very surprised)
I actually discussed the CO2 levels in a post I wrote that discusses most of the issues about global warming in a pretty detailed manner.
http://gahrie.blogspot.com/2009/11/some-inconvenient-truths.html#links
>They put forward a few cherry picked counter examples while calling the critics ignorant and dishonest.
The critics were ignorant and dishonest. The claimed there was no reason for the adjustments, despite having the reasons explained three years earlier.
>All they have to do here is release the data and show the adjustments, then people can see if they make sense.
They have. Their critics they proceeded to lie about the data.
>Ideally, a location change will be made with the previous instrument in place so that the average temperature difference can be determined.
Ideally, yes but often you can’t keep the instrument in place for the same reason it was moved in the first place.
>In the case of NZ, it is strange that the adjusted and raw temperatures should differ so much.
I’m not sure if you know the geography of Wellington, but large temperature differences between the harbour and the hills should come as no surprise.
>I’m not sure if you know the geography of Wellington, but large temperature differences between the harbour and the hills should come as no surprise.
pete is dead wrong about most of these issues, but he gets points for this one. NZ in general is a place with a lot of vertical – the ruggedness of the landscape creates odd little microclimates everywhere.
As to who if anyone is lying in the NZ dispute, I have insufficient data and therefore reserve judgment. I will only note that cooking the NZ temperature data with convenient “adjustments”, always upwards, would fit the pattern of pro-AGW fraud.
Moreover, when 69%-ish of your temperature stations are within ten meters of an artificial heating source,
It’s worse than what you are stating..actually 89% of the stations are sited within 100 feet of a heat source.
>ESR says: You’re wrong.
You wouldn’t have noticed the different sample sizes in either case?
Looks like everyone may get to see the data after all:
In a statement welcomed by climate change sceptics, the university said it would make all the data accessible as soon as possible, once its Climatic Research Unit (CRU) had negotiated its release from a range of non-publication agreements.
The publication will be carried out in collaboration with the Met Office Hadley Centre. The full data, when disclosed, is certain to be scrutinised by both sides in the fierce debate.
ESR: The code needs to be open-sourced, too.
Would you care to make any explicit, falsifiable predictions about the eventual results of the CRU leak, ones which can be easily and definitively tested within the next five years? I’d like to submit them to WrongTomorrow.
ESR says: Thank you, not playing, don’t care to be nitpicked to death in N years by people resenting that I was right about the fraud.
@ Murph / Gahrie
Crud … I was off by 0.02%
>don’t care to be nitpicked to death in N years by people resenting that I was right about the fraud.
Don’t worry, from what I’ve seen here there’s very little chance of that.
Thanks for the link to the counter article.
Actually there was a slightly less detailed release from NIWA that was linked in the article I gave and then a counter-counter-article given just below that link at this location. A separate acknowledgement of the counter you linked (found through link surfing) is at this location.
Just for those playing at home.
Would you care to make any explicit, falsifiable predictions about the eventual results of the CRU leak, ones which can be easily and definitively tested within the next five years? I’d like to submit them to WrongTomorrow.
ESR says: Thank you, not playing, don’t care to be nitpicked to death in N years by people resenting that I was right about the fraud.
This is, by the by, the root cause for the following process when requesting data sets from climatologists.
0) Request comes in. Has a name attached
1) Name is run through Google looking for what sort of climatology research you do.
2) If climatology posts are largely political in nature (or deemed to be political), the result is foot dragging.
3) If the climatology record shows that you got data before and then misquoted it (like not reporting the documented adjustments that are in the metadata), your next request is denied.
4) If the Google-stalking shows that you can’t discuss data sets with a pattern of A is caused by B, C, D and E, with probability R, and known issues X, Y and Z meriting further study, the request is usually treated as a crank.
Steve Easterbrook makes excellent points in his comments on the prior post about the importance of domain expertise, and how people who work in the Open Source community can be blinded by their domain expertise. When the majority of your social or peer group are Open Source contributor, or in Eric’s case, Open Source project leads, it’s easy to assume that everyone can mentally shift from C to FORTRAN to Python without noticeable effort.
Pete: Debunked here. Looks like it’s the deniers at the NZ CSC that are involved in shenanigans.
Pete, are you under the impression that your continued ad hominem attacks are helping your case?
Because they are not.
As I wrote last night, I find it quite revealing that the climate science community is convinced that the belief state of their critics determines the results those critics obtain, while somehow they (the climate scientists) remain immune to any bias themselves. Ever hear of projection?
I’m sure that your description of the process is accurate, Ken, but it fails to explain why they don’t just throw their data up on a webserver.
Russell:
At a rough estimate, somewhere around 80% of the data used for IPCC AR4 IS publicly available on a web server. Somewhere around 30% of the models (there are 23 cited in IPCC AR4) have their source code available as well.
A list of available data sets and models (from Steve Easterbrook’s web site) shows that the data is there if you know where to look at it. One model even has public access to the TRAC system for code viewing…
Why doesn’t CRU release all their data?
Largely because CRU has licensing agreements with the meteorological organizations to get the data in the first place. Those meteorological organizations sell that data to companies that do weather forecasting. They provide their data for climatology research for free to people who sign the agreement, one of the terms of which is ‘no redistribution’.
In the US, most of the data appears to be available, though I’ve not done real serious digging to find out if that’s true across the board.
Many times that data can be gotten again, if you’re willing to dig around and find the right points of contact, and send out emails.
One of my projects (assuming it’s feasible at all) is to try and get CCMS 3.0 up and running and run data through it. CCMS is an open source modeling program from UCAR; you have to create an account there to get the software. I’ve got a friend who used to do climate modeling as a code monkey, and Steve Easterbrook is also willing to help.
The goal here is to give a narrative of our experiences going through the process; I don’t doubt we’re going to make some horrible errors along the way, and there’s, at best, a 40-50% chance we can make it work at all.
It’s my hope that by documenting what we did and how we did it that we can help the climate scientists (note the careful choice of terms there – I did not use ‘alarmist’ or ‘denier’) improve transparency. One of the most important QA steps is having a village idiot use your software.
You’re looking at a volunteer to be a village idiot. Follow along (from morbid curiosity if nothing else) at:
http://data-n-demagogues.blogspot.com/
Eric Raymond is also part of the project, and in my ideal world, Larry and Eric would both have this software running and would independently cross check each other.
FYI: Eric – you may want to download the compiler and related information for a statistical package/programming language called ‘R’.
>FYI: Eric – you may want to download the compiler and related information for a statistical package/programming language called ‘R’.
I know enough about it to have no doubt that I can learn to use it effectively when the time comes.
A thought on ‘continuing’ a temperature station’s data when it’s discontinued at one location and a new station established some distance away, but still claiming it’s the “same station”.
If a Dr. takes Bill’s rectal temperature then takes Bob’s oral temperature the next day, does the Dr. write Bob’s oral temp on Bill’s chart under rectal temp? Of course not!
So why are there many temperature station records on the books as being the same station when the sensors have been moved to new locations with very different conditions? Ie. moving from near a harbor in NZ to a mountainside.
The code with the ARTIFICIAL in it is tree ring code designed to obscure the divergence between the tree rings and the instrumental temperature record since 1960. Yes I know it started doing the change in 1930 this seems designed to make the merge less obvious
(Note that the instrumental record being compared with is almost certainly the Hadley/CRU one)
@Galana,
a large chunk of the CRU temperature code and data management practices are designed to adjust for station movements and the like.
I haven’t deciphered the CRU stuff enough yet but the GISS code tries to make these adjustments automatically by “automagically” detecting the change and correcting for it based on the records of nearby stations. There are a lot of problems with this as has been documented by climateaudit (IIRC one of the y2000 bugs in the code was to do with this) and I have no doubt that similar junk is to be found in the cru code
Whoever it was that obtained and transmitted the CRU e-mails has probably saved millions of lives. The left wing was perfectly happy to throw the economies of every third-world country under the bus for the sake of their power lust.
You people seem to think it’s some kind of revelation that there are dickheads, sloppy workers, and petty rivalries in climate science. News flash, sometimes we even suffer from confirmation bias too! I’ve only been reading this blog for a few months, didn’t realise you were quite this nutty…. Anyone can write convincing invective given an archive of personal emails and files to mine.
Your Zorita quote is out of context – he explicitly states he doesn’t think AGW is a fraud.
“a large chunk of the CRU temperature code and data management practices are designed to adjust for station movements and the like.”
But the practices themselves -still- haven’t changed to request that they actually run the two stations in parallel for a bare minimum of a month.
Physical data trumps height lapse estimates. Easily.
Al, if you’ve got a time machine to help remedy that problem, I’ve got some lucrative opportunities for you. ;)
Another reason the CRU hasn’t released their data: they threw a lot of it away. “We do not hold the original raw data but only the value-added (quality controlled and homogenised) data.â€
Ken,
The -current- policy doesn’t include “please perform overlap.” That’s something that can be improved going forward. This is ignoring the massive station closures and citing issues.
The goal (from a climatology standpoint) of a station isn’t the temperature of the city anyway. It is the average temperature of a gridcell.
Well, calibrating a gridcell or three might be a decent start to the issue. Knowing what your current data means is crucial to unraveling what in the heck your historical data might mean.
Well, calibrating a gridcell or three might be a decent start to the issue. Knowing what your current data means is crucial to unraveling what in the heck your historical data might mean.
Can you tell me what would satisfy you for that calibration?
What accuracy do you want out of this hypothetical instrument?
I’d start with 31 truly random cites in a gridcell for a year. If you pick a depopulated cell, you can at least start making a rock-solid measurement with which to cross-calibrate satellite measurements.
I don’t have any problems with the current standards for instrumentation and situation – if followed.
@Al
All the weather station issues make more sense when you realise that they’re weather stations. Climatologists use the data, because often that’s all that’s available, but that’s not what the stations are designed for.
NOAA are setting up a climate reference network, with much stricter standards.
“Historical Climate Network”
I agree with you – you use what you’ve got. But recognize the limitations.
Al, most of those limitations are acknowledged. It is, in fact, the acknowledgement of those limitations that causes the fudge factors that everyone’s up in arms about.
Eric, why do you disallow comments that are critical of you and your positions?
ESR: I don’t. You landed in the spamtrap.
True – but the scattered papers discussing the issue in any depth do it primarily from the theoretic and modeling aspects.
“Well, we can model UHI by counting nightlights…” Recognition of an issue – excellent. Plausible resolution – excellent. Hard calibration data – hmmm.
The brute force method of strapping a time-and-GPS-logging thermometer on the roof of the car and driving through Phoenix several times is left to a high-school science fair project (IIRC). This isn’t anywhere near as hot a topic as thinking up a new method of paralellizing a circulation model though.
Uh? There are programmers who can’t do that?
Climatologists are using weather stations? And they’re probably wondering why there are other scientists that aren’t taking them seriously… *sigh*
Just boggle Morgan.
The main page there has pictures of 89% of the US Historical Climate Network. The vast majority aren’t insane. But the number of “pristine” sites is disheartening.
>“Historical Climate Networkâ€
I think that’s the old weather station network.
They’re working on a new network, The USCRN (US Climate Reference Network), and supplementing that with the USHCN-M (M for modernized), which should provide climatologists with higher quality data in the future.
It is the old weather station network. It is also the source of basically all the instrumental records we’ve got prior to the satellite era. There are some slightly repackaged versions of the same instruments, but the only modernization of the last decades has involved switching sites to MMTSes without performing a proper cross-calibration. And dropping a whole lot of stations.
@Al:
You asked about improvements “going forward”. NOAA clearly agrees with you that the current situation isn’t ideal. So they’re working on the USCRN and USHCN-M.
@ Galane (November 29th, 2009 at 2:32 am)
> A thought on ‘continuing’ a temperature station’s data when it’s discontinued at one location and a new station established some distance away, but still claiming it’s the “same stationâ€.
> If a Dr. takes Bill’s rectal temperature then takes Bob’s oral temperature the next day, does the Dr. write Bob’s oral temp on Bill’s chart under rectal temp? Of course not!
> So why are there many temperature station records on the books as being the same station when the sensors have been moved to new locations with very different conditions? Ie. moving from near a harbor in NZ to a mountainside.
FIRST – love the analogy between patient #1 anal temperature and patient #2 oral temperature, recording both on same patient’s chart on different days. Very Funny. Brings it down to a level everyone can understand.
SECOND – back in the day, I did some tedious keypad entry of weather station data from hardcopy weather station records into an ASCII text file (80 columns wide). Similar to “HARRY_README.TXT” there is missing data in the official hard copy record I held in my hands (e.g. gov’t holiday, weather checker ill, too cold to venture outside, etc). I was told to enter as -99999999 or some such ridiculous number.
THIRD – Every official US Weather Station had a 3-letter acronym (4?) with certain letters reserved for different countries. Back in the day of punch cards and ASCII 80-column text files, the technology literally FORCED the well meaning weather checker to call the ‘new’ station, located several miles from the the ‘old’ station, the ‘same’ station. No deep dark national treasure free-mason illiminuti conspiracy. Just the way it was due to technology. Recall Y2K and two-digit years. Recommend confirming this new-station acronym same as old-station acronym with the ‘weather geeks’ over at SurfaceStations. Could also see some local rivalry as to weather or not a town has an ‘official’ weather station. Even if station has moved, keep calling it the old one so no one at the local level gets bent out of shape.
For all those following along at home, realclimate.org has commenced a project to aggregate links to climate datasets, models (including source) and tools.
No doubt this is still not adequate for some – in particular it is not currently a well organised set of links and has no “forge” structure at all – but it is a demonstration of the amount of data already available. Most of this, I understand, has been available for some time, too.
So, Eric, seeing as you are interested in the Truth and in Good Science (and not at all interested in cherry-picking quotable quotes to support your long-held and public opposition to AGW theories) , we are all eagerly awaiting your assessment of the data available for your analysis.
(I for one would be happy if you did this over at Ken’s new site so you could leave the politics back here at “armed and dangerous”)
.
ESR says: Ken, myself, and a couple of domain experts are discussing ways and mens in email now.
That is an absurd assertion. Without even thinking five seconds about this, I arrived at this simple solution:
Take the 4-character ICAO “call sign” and append to it a character for the “version” of the station. So the weather station at Auckland, NZAA, was originally NZAA0. They move it, and it’s NZAA1. There may be a period of overlap during which we get readings from both of these, or a gap with none at all. Lather, rinse, repeat. If they move it a tenth time, it goes from NZAA9 to NZAAa. This allows a weather station to be moved 35 times and still have a unique 5-character code. After that, we can make case significant, and use upper-case letters, allowing each weather station to be moved 61 times.
Tom Says:
“For all those following along at home, realclimate.org has commenced a project to aggregate links to climate datasets, models (including source) and tools.â€
@The Monster (November 30th, 2009 at 10:27 am)
> That is an absurd assertion. Without even thinking five seconds about this, I arrived at this simple solution:
> [blah]
> Take the 4-character ICAO “call sign†and append to it a character for the “version†of the station.
> [blah]
> still have a unique 5-character code.
> After that, we can make case significant, and use upper-case letters,
> allowing each weather station to be moved 61 times.
Things were pretty absurd back in the days (stone knives and bear claws, according to Spock), you were limited to:
(a) an 80-column file format (holdover from punch cards)
(b) Uppercase Only
(c) No Internet, No calculators, Slide Rulers, and such
Because you have arbitrarily (grin) widened the station ID, the 1950’s weather checker community must now sacrifice a character elsewhere in the weather data record. Furthermore, their teletypewriter (circa 1950s) had no “lower-case”. Imagine reading everything in Courier font, all uppercase. Pretty absurd I admit.
Although you can discuss WHAT COULD HAVE BEEN done with today’s 21st century buck-rogers technology (500GB hard drive ~ $50); I think the reader would agree we should learn WHAT WAS DONE and WHY IT WAS DONE
Although you and I can go back & forth about this ‘n that, someone should convince the ‘weather geeks’ associated with http://www.surfacestations.org/ (primary) and http://wattsupwiththat.com/ (secondary) to do a writeup on the ins & outs of weather station siting and movement during the 20th century. What did a hard-copy log page look like? How was it transmitted to Washington DC? etc
What is ‘obvious’ today to you was not even a weather checker’s absurd wet dream five-decades ago. Compare what WAS RECORDED five-decades ago to what IS BEING or COULD BE RECORDED today.
Yes, I believe that you didn’t think 5 seconds about it. What part of ” 3 character acronym” did you not understand?
Monster, three obvious problems with your solution. One, if you have a 4-character field for your data, 4 letters and a number won’t fit. Two, they didn’t do it that way when the system started. I’m sure you can probably backtrack in many cases, but it’d probably be nightmarish, and likely error-prone. Third, even if you got “version numbers” attached to every single data point, what then? You don’t have any overlap in the records, so you can’t tell very effectively what bias is introduced with the location change. I mean, you can interpolate from the records of nearby stations, but that gets you deep into “sketchy interpolations” territory again.
I’m not sure how many problems it solves, and I can’t imagine the people arguing for AGW would be eager to open themselves up to that many attacks on methodology when it doesn’t even improve the data very much.
Why interpolate at all? When a station is moved, the record is broken at that point and started fresh at the new point. Get all the *original* data that it’s possible to get and find out when all station moves happened.
Enter all that data AS-IS into a database and retroactively assign/append new identifiers to all moved stations that were not assigned new codes when they were originally moved. Since there’s no need to be restricted to ye olde punch-card formats, the station ID field can be expanded without taking space away somewhere else.
When the raw data is done, then go back through whatever records are available on how the quality of the station sites changed over time. Surfacestations.org has most of the *current* conditions of the American sites, but the slightly iffy to completely out of spec sites weren’t always that way.
This would take a lot of time and research, digging around in public archives, to discover the progress of urban encroachment on the sites, but it’s vital to know how stations that are bad today got that way. It’s mapping quality deterioration over time in order to know when stations *were* good.
That’s where expandability of the data space is very handy, to add a quality indicator that can indicate as close as possible exactly when changes happened.
With the quality data, the database could be filtered to plot temperatures only from stations that have been in full compliance their entire existence, or to only include the times of all stations when they were in full compliance with the siting guidelines.
Experiments need to be conducted to get a more exact idea of how much different types of siting problems affect the instrument readings. THEN the site quality data can be used to create reasonably accurate corrections for out-of-spec stations BUT such corrected data must always be treated as lower accuracy.
That’s a proper scientific approach to this mess. Of course it would be tedious and difficult and take a long time to do. Good science is tedious and difficult and takes a long time to do. That’s how you get the answers and results that should be the most accurate.
What’s been a problem with how the data has been used, is the assumption that the weather stations were useful for collecting data about an *area* when what they’ve always really been are *point source* instruments spaced too far apart to generalize what conditions exist between them.
What’s been a problem with how the data has been used, is the assumption that the weather stations were useful for collecting data about an *area* when what they’ve always really been are *point source* instruments spaced too far apart to generalize what conditions exist between them.
The generalization of how much a station’s readings map to a given area is a known quantized variable in climate science, and is effectively a ‘solved’ problem. Is the data clean? No – it will always be noisy.
Is the current monitoring situation good? Not at all – we went from 1850 monitoring stations in 1968 to 126 today. There is a definite movement afoot to get better monitoring stations out there.
Moving the goalposts.
The assertion was the word “FORCED”. The use of 4-letter ICAO codes for weather stations was a decision. It was not “FORCED”. I am painfully aware of the limitations of an 80-column format; I use them on a daily basis and have to find clever ways to encode a maximum amount of information into a small amount of space. What is absurd is the bald assertion that there was no alternative but to conflate instrument readings from several different locations that used the same ICAO codes at different times. I proposed but one simple way to disambiguate those stations. You may object that it involves a tradeoff of some kind, but all decisions involve tradeoffs.
There are many other ways to achieve that disambiguation. Even if the 80-character punch card was already filled up, the program processing the data could have a table of moved stations that produces the digit I propose based on the date of the reading. And because we can do that now, if the primary data are available, renders the objection that I’m engaging in revisionism by advocating a 21st century solution to 20th century data moot.
That no disambiguation was implemented indicates that the people doing the work didn’t think it was worth the effort.
A nice set of references for the Copenhagen Diagnosis report:
http://agwobserver.wordpress.com/2009/11/28/the-copenhagen-diagnosis-references
there is missing data in the official hard copy record I held in my hands (e.g. gov’t holiday, weather checker ill, too cold to venture outside, etc). I was told to enter as -99999999 or some such ridiculous number.
Here’s another source of upward bias; if extreme lows (“too cold to venture outside”) are recorded as nulls, while extreme highs are included in the data, then the overall figures will be too high.