Nov 23

Open-Sourcing the Global Warming Debate

The email and documents recently netjacked from the Climate Research Unit at the University of East Anglia raise serious questions about the quality of the research being used to underpin major public-policy decisions.

In the open-source software community, we understand about human error and sloppiness and the tendency to get too caught up in a pet theory. We know that the most effective way way to combat these tendencies is transparency of process — letting the code speak for itself, and opening the sources to skeptical peer review by anyone.

There is only one way to cut through all of the conflicting claims and agendas about the CRU’s research: open-source it all. Publish the primary data sets, publish the programs used to interpret them and create graphs like the well-known global-temperature “hockey stick”, publish everything. Let the code and the data speak for itself; let the facts trump speculation and interpretation.

We know, from experience with software, that secrecy is the enemy of quality — that software bugs, like cockroaches, shun light and flourish in darkness. So, too. with mistakes in the interpretation of scientific data; neither deliberate fraud nor inadvertent error can long survive the skeptical scrutiny of millions. The same remedy we have found in the open-source community applies – unsurprisingly, since we learned it from science in the first place. Abolish the secrecy, let in the sunlight.

AGW true believers and “denialists” should be able to agree on this: the data get the last word, because without them theory is groundless. The only way for the CRU researchers to clear themselves of the imputation of serious error or fraud is full disclosure of the measurement techniques, the raw primary data sets, the code used to reduce them, and of their decisions during the process of interpretation. They should have nothing to hide; let them so demonstrate by hiding nothing.

The open-source community has many project-hosting sites that are well adapted for this sort of disclosure. If they require assistance in choosing one and learning how to create and manage an open-source project, I and many others in the open-source community will be happy to provide it.

For the future, we need to restore the basic standards of science. No secrecy: no secrecy of data, no secrecy of experimental methods, no secrecy of data-reduction or modeling code. Such transparency and accountability are especially vital when the public-policy stakes are large. This is among the excellent reasons that both the US and UK have Freedom of Information Acts, and the logic of those acts has perhaps never applied more pressingly than it does here.

Nov 21

Hiding the Decline: Prologue

According to the summaries I’ve seen, the 61 megabytes of email and documents net-jacked from the Climate Research Unit a few days ago do not — quite — reify conservatives’ darkest fantasies about “the team” (as the network of professional anthropogenic-global-warming alarmists communicating through CRU likes to style itself). To do that, they’d have to contain marching orders from the Socialist International.

However, the excerpts I’ve seen are already quite damning enough; among other things, they are evidence of criminal conspiracy to violate the Freedom Of Information act. And I no longer have to speculate about the rest; I’ve downloaded the documents from Pirate Bay and will study them myself.

For those of you who have been stigmatizing AGW skeptics as “deniers” and dismissing their charges that the whole enterprise is fraudulent? Hope you like the taste of crow, because I do believe there’s a buttload of it coming at you. Piping hot.

Am I going to blog about it? Heh…try to stop me…

UPDATE: I’ve read about 10% of the material and started a file of notes on it, but been delayed by preparing for a major release on one of my projects. In the meantime, read this excellent summary with links to the original emails.

Nov 16

Barbecue kings!

John Birmingham writes from Australia:

Even, and this is gonna hurt, the Americans have it all over us when it comes to cooking with fire, iron and tongs. In fact it’s arguable the American barbecue, or rather its plethora of regional variations on barbecue, set the gold standard worldwide for applying heat to meat while out of doors. While the popular image of American cooking, at least as practised by average Americans, involves squeezing a plastic sauce packet over something nasty in a chain restaurant, the truth is their barbecue specialists would put ours to shame. Undying, unutterable shame.

Alas, John, it is so. I have eaten barbecue all over the U.S. and the world, and the kings of the genre are in this country. Not in my part of it; I’m a Boston-born northerner and most barbecue where I live is as bland and bad as you describe. As a general rule in the U.S. the further south you go the better the barbeque gets, with the acme reached in south Texas. (Though the area around Memphis, further north, is a contender.)

Internationally, almost nobody even competes with the Southern U.S. for the barbecue crown. Brazilian churrasco is the one exception I can think of – that stuff can give good ol’ Texas ‘cue a run for its money. But you ex-British-Empire types aren’t even properly in the running. I’ve been to a backyard braai in South Africa and, while the spirit was there, the seasoning and cooking technique was sadly lacking, much like what you describe.

American cooking in general gets a bad rap internationally that it doesn’t deserve. It’s as though foreigners think it’s still 1965 here or something. I can remember a time in my childhood when the slams were richly deserved — heck, I remember returning here from Europe in 1971 and having to wait more than a decade before I saw a decent piece of bread! But Americans got a clue about food in the 1980s and haven’t lost it since. I learned this when I was traveling intensively around the turn of the century; most places I visited, even the “high-quality” food was inferior to what I ate at home.

UPDATE: I suppose it’s worth noting that Brazilian-style churrascarias have become the most recent high-end restaurant fad in the U.S., suggesting that other Americans generally agree with me that the style competes well with native ‘cue. Sadly, Korean barbecue failed to become naturalized here when that was tried in the 1990s.

Nov 15

The pragmatics of webscraping

Here’s an amplification of my previous post, Structure Is Not Meaning. It’s an except from the ForgePlucker HOWTO on writing code to web-scrape project data out of forge systems.

Your handler class’s job is to extract project data. If you are lucky, your target forge already has an export feature that will dump everything to you in clean XML or JSON; in that case, you have a fairly trivial exercise using BeautifulStoneSoup or the Python-library JSON parser and can skip the rest of this section.

Usually, however, you’re going to need to extract the data from the same pages that humans use. This is a problem, because these pages are cluttered with all kinds of presentation-level markup, headers, footers, sidebars, and site-navigation gorp — any of which is highly likely to mutate any time the UI gets tweaked.

Here are the tactics we use to try to stay out of trouble:

1. When you don’t see what you expect, use the framework’s self.error() call to abort with a message. And put in lots of expect checks; it’s better for a handler to break loudly and soon than to return bad data. Fixing the handler to track a page mutation won’t usually be hard once you know you need to – and knowing you need to is why we have regression tests.

2. Use peephole analysis with regexps (as opposed to HTML parsing of the whole page) as much as possible. Every time you get away with matching on strictly local patterns, like special URLs, you avoid a dependency on larger areas of page structure which can mutate.

3. Throw away as many irrelevant parts of the page as you can before attempting either regexp matching or HTML parsing. (The most mutation-prone parts of ppsages are headers, footers, and sidebars; that’s where the decorative elements and navigation stuff tend to cluster.) If you can identify fixed end strings for headers or fixed start strings for footers, use those to trim (and error out if they’re not there); that way you’ll be safe even if the headers and footers mutate. This is what the narrow() method in the framework code is for.

4. Rely on forms. You can assume you’ll be logged in with authentication and permissions to modify project data, which means the forge will display forms for editing things like issue data and project-member permissions. Use the forms structure, as it is much less likely to be casually mutated than the page decorations.

5. When you must parse HTML, BeautifulSoup is available to handler classes. Use it, rather than hand-rolling a parser, unless you have to cope with markup so badly malformed that it cannot cope.

Actual field experience shows that throwing out portions of a page that are highly susceptible to mutation is a valuable tactic. Also, think about where in the site a page lives. Entry pages and other highly visible ones tend to get tweaked the most often, so the tradeoffs push you towards peephole methods and not relying on DOM structure. Deeper in the site , especially on pages that are heavily tabular and mostly consist of one big form, relying on DOM structure is less risky.

Nov 09

Ego is for little people

When I got really famous and started to hang out with people at the top of the game in computer science and other fields, one of the first things I noticed is that the real A-list types almost never have a major territorial/ego thing going on in their behavior. The B-list people, the bright second-raters, may be all sharp elbows and ego assertion, but there’s a calm space at the top that the absolutely most capable ones get to and tend to stay in.

I’m going to be specific about what I mean by “ego” now, because otherwise much of this essay may seem vague or wrongheaded. I specifically mean psychologial egotism, not (for example) ethical egoism as a philosophical position. The main indicators of egotism as I intend it here are are loud self-display, insecurity, constant approval-seeking, overinflating one’s accomplishments, touchiness about slights, and territorial twitchiness about one’s expertise. My claim is that egotism is a disease of the incapable, and vanishes or nearly vanishes among the super-capable.

It’s not only scientific fields where this is true. For various reasons (none of which, fortunately, have been legal troubles of my own) I’ve had to work with a lot of lawyers. I’m legally literate, so a pattern I quickly noticed is this: the B-list lawyers are the ones who get all huffy about a non-attorney expressing opinions and judgments about the law. The one time I worked with a stratospherically supercompetent A-list firm (I won’t name them, but I will note they have their own skyscraper in New York City) they were so relaxed about recognizing capability in a non-lawyer that some language I wrote went straight into their court filings in a lawsuit with multibillion-dollar stakes.

This sort of thing has been noted before by other people and is almost a commonplace. I’m bringing it up to note why that’s true, speaking from my own experience. It’s not that people at the top of their fields are more virtuous. Well…actually I think people at the top of their fields do tend to be more virtuous, for the same reason they tend to be be more intelligent, less neurotic, longer-lived, better-looking, and physically healthier than the B-listers and below. Human capability does not come in nearly divisible chunks; almost every individual way that humans can excel is tangled up with other ways at a purely physiological level, with immune-system capability lurking behind a surprisingly large chunk of the surface measures. But I don’t think the mean difference in “virtue”, however you think that can actually be defined, explains what I’m pointing at.

No. It’s more that ego games have a diminishing return. The farther you are up the ability and achievement bell curve, the less psychological gain you get from asserting or demonstrating your superiority over the merely average, and the more prone you are to welcome discovering new peers because there are so damn few of them that it gets lonely. There comes a point past which winning more ego contests becomes so pointless that even the most ambitious, suspicious, external-validation-fixated strivers tend to notice that it’s no fun any more and stop.

I’m not speaking abstractly here. I’ve always been more interested in doing the right thing than doing what would make me popular, to the point where I generally figure that if I’m not routinely pissing off a sizable minority of people I should be pushing harder. In the language of psychology, my need for external validation is low; the standards I try hardest to live up to are those I’ve set for myself. But one of the differences I can see between myself at 25 and myself at 52 is that my limited need for external validation has decreased. And it’s not age or maturity or virtue that shrunk it; it’s having nothing left to prove.

Continue reading

Nov 05

Maybe if moral cowardice cost money, it would be less common?

Heh. State representative Fred Maslack of Vermont has proposed a bill under which non-gun-owners would have to register and pay a fee. Entertainingly enough, there is actual justification for this in a careful reading of the Vermont state constitution.

The Hon. Rep. Maslack is joking. I think. And I’m against requiring people who don’t want to bear arms to do so. But gad, how tempting – because underlying his argument is a truth that the drafters of the Vermont and U.S. constitutions understood. People who refuse to take arms in defense of themselves and their neighbors are inflicting a cost on their communities far more certainly than healthy people who refuse to buy medical insurance (and yes, I do think that proposed mandate is an intended target of Maslack’s jab). That externality is measured in higher crime rates, higher law-enforcement and prison budgets, and all the (dis)opportunity costs associated with increased crime. And that’s before you get to the political consequences…

I’ve never made a secret of my evaluation that refusal to bear arms is a form of moral cowardice masquerading as virtue. Real adults know how precious human life is, when they are ethically required to risk it on behalf of others, and when killing is both necessary and justified. Real adults know that there is no magic about wearing a police or military uniform; those decisions are just as hard, and just as necessary, when we deny we’re making them by delegating them to others. Real adults do not shirk the responsibility that this knowledge implies. And the wistful thought Rep Maslack’s proposal leaves me with is…maybe if moral cowardice cost money and humiliation, there would be less of it.

Nov 05

Structure Is Not Meaning

So, I announce ForgePlucker, and within a day I’ve got some guy from Y Combinator sneering at me for using regular expressions to parse HTML. Says it’s “crappy code”. The poor fool…he has fallen victim to a conceptual trap which I, fortunately, learned to avoid decades ago. I could spout a freshet of theory about it, but instead I’m just going to utter a maxim: Never confuse structure with meaning.

Continue reading

Nov 04

Announcing ForgePlucker

I’ve been strongly hinting in recent blog entries that I planned to do something concrete about the data-jail problems of present open-source hosting sites. Because I believe in underpromising and overperforming, I decided at the outset not to announce a project until I could not only show working code, but code with wide enough coverage to make it crystal-clear that the project goals are achievable with a relatively modest amount of effort.

That time has arrived. I am very pleased to announce ForgePlucker, a project aimed at developing project-state extractor software for backup, offline analysis, and (eventually) re-importation. The proof-of-concept code can extract complete issue-tracker state from Berlios, Gna!, or Savane — and issue trackers are probably the hardest part of the job. I expect extraction of repository histories and developer permissions tables to be easier. Extraction of mailing-list state is probably a bit trickier than either of those, but doable.

Continue reading

Oct 29

The future of software forges

I’m still not going to talk about my attack on the forge infrastructure problems quite yet; the software is coming along nicely, but I intend to announce only after it handles its fourth forge type (yes, that was a tease). But I will say this: I now think I know what the future of forges looks like. It’s called Roundup, and it is astonishingly elegant and potentially more powerful than anything out there. Anything, not excluding the clever decentralized systems like Fossil or Bugs Everywhere.

Here are the big wins:

1. Mailing lists, issue trackers, and online forums unify into *one* message queue that can be filtered in various ways.

2. Scriptable via XML-RPC or an email responder ‘bot.

3. Small base system with good extensibility – just three base classes (User. Msg, File) and the ability to define new classes. ‘Issue’ is a class built on top of these.

4. Arbitrary attributes per issue is basically free, with baked-in support for defining controlled vocabularies.

5. There’s a uniform way, called “designators”, for messages and other objects to refer to each other in text.

6. Small, clean implementation written in Python.

There are some things it needs, though… (Read the Roundup design document before continuing.)

Continue reading

Oct 26

Hacker superstitions about software licensing

Hackers have a lot of odd superstitions about software licensing. I was reminded of this recently when a project maintainer asked me whether he needed to get a sign-off from each and every one of his contributors before switching from Apache v1 to Apache v2. Here’s what I told him:

My opinion is this. Under U.S. law — and I believe European codes are not different in this respect, because both are controlled by the Berne convention — a license change on a collection is grounds for protest or legal action only if the rights of the contributors are materially affected by the change. That is, a court would have to be persuaded that the change caused a monetary loss or at least damage to a contributor’s public reputation. If there is no such possibility, then there is no harm and no grounds for complaint.

Continue reading

Oct 23

Fearing what might be true

I am not generally unhappy with my model of how the universe works. Yes, it would be pleasant if there really were a beneficient creator-god and an afterlife; it would be nice if “good government” were actually a sustainable possibility rather than a fond but deluded hope in the minds of statists; it would be just peachy if wishes were horses and everybody could be happy and rich. But I’m reconciled to these things being not true because it seems to me they are necessarily not true – that is, for example, I cannot imagine a universe in which the actual existence of an intervening creator-god is actually compatible with the observed regularity of natural law, nor a universe in which scarcity and agency problems do not imply that governments are subject to the iron laws of political economics.

Other people can imagine theism or statism to be true, and so could I at one time – before I understood enough to grok the contradictions they would entail. But I’m not actually writing to argue for anarchism or atheism today; I brought those up as examples to make a more subtle point. It seems to me it’s actually more difficult to deal with unpleasant possible truths that are only contingent, because you can imagine how things could be different if you lived in a universe that was slightly different but recognizable.

I’ll give two examples of this that I’m still struggling with. One: I fear feminism may be dangerously wrong. Two: I fear that the crowd calling for the abolition of all forms of intellectual property may be dangerously right.

Continue reading

Oct 21

Public Space in Cyberspace

Following my post Condemning Censorship, Even of Werewolves, a great many people took me for task for trying to make a principled distinction between public space and private space, and tangled that up with a lot of confusion about the distinction between “public” and “state-owned” space. Here I’ll attempt to shed some light on the matter. In doing so, I will not actually be retailing a private theory of mine, but touching on principles with a long history in ethics and Anglo-American common law.

Continue reading

Oct 19

Condemning Censorship, Even of Werewolves

Thomas Paine once wrote: “He that would make his own liberty secure must guard even his enemy from oppression; for if he violates this duty he establishes a precedent that will reach to himself.” Paine’s truth is not idealistic handwaving, it is brutal pragmatism. Justifications for censorship, even the best-intentioned kind, have a way of expanding until they become instruments of abuse. Therefore, if we truly care about freedom of speech, it is not sufficient to defend that freedom when it is comfortable to do so — when the censors are ugly and the victim is appealing. It is necessary, sometimes, to speak up in defense of ugly victims of censorship.

I have found myself placed under that necessity in the last week. A member in good standing of the open-source community, one Beth Lynn Eicher, had sought and achieved the suppression of public speech by one Mikhail Kvaratskhelia, aka ‘mikeeeUSA’, aka serveral other aliases. When I first approached her privately on the matter, she refused to apologize or retract. In my judgment, she was committing a crime against our community’s future by setting a precedent which might one day reach to all of us.

This put me in a difficult position. I had received an explicit appeal for redress in my capacity as one of our tribal elders, and I felt the appeal was in the right and Ms. Eicher in the wrong. But I knew — for various reasons which will become very clear — that making that case would involve me in a batter and divisive wrangle. I was prepared to do it anyway, because my conscience would not permit otherwise, but I knew it was going to be hell.

Fortunately, after several days of debate among myself and some friends of mine who leapt to Ms. Eicher’s defense, Ms. Eicher proved to be cleverer than either them or me. While Ms. Eicher’s defenders were still flaming me for intransigence on the free-speech issue, she designed a solution which I consider totally appropriate, and which I actually hope will set a precedent.

Continue reading

Oct 12

How Not To Tackle the Mess around Forges

In my previous two posts I have diagnosed a significant weakness in the open-source infrastructure. The architecture of the code behind the major SourceForge-descended hosting sites is rotten, with all kinds of nasty consequences — data seriously jailed, poor or completely absent capabilities near scripting and project migration. I said I was going to do something about it, and I’m working the problem now — actually writing code.

The rest of this post is not an announcement, because it will be mostly about things I’ have figured out I should not try to do. Yet. But it is a teaser. I see a path forward, and shortly I expect to have some working code to exhibit that shows the way. Actually, I have working code that attacks the problem in an interesting way now, but I’m still adding capabilities to make it a more impressive demonstration.

Here are some approaches I’ve considered, or had suggested to me by others, and rejected:

Continue reading

Oct 09

Looking Deeper into Forges, And Not Liking What I See

In my previous post, Three Systemic Problems With Open-Source Hosting Sites I identified some missing features that create serious brittleness in or project-hosting infrastructure. The question naturally arises, why don’t existing hosting systems already have these facilities? I have looked into this question, actually examining the codebases of Savane and GForge/FusionForge, and the answer appears to go back to the original SourceForge. It offered such exciting, cutting-edge capabilities that nobody noticed its internal architecture was a tar-pit full of nasty kluges. The descendants — Savane, GForge, and FusionForge — inherited that bad architecture.

Continue reading

Oct 08

Three Systemic Problems with Open-Source Hosting Sites

I’ve been off the air for several days due to a hosting-site failure last Friday. After several months of deteriorating performance and various services being sporadically inaccessible, Berlios’s webspace went 404 and the Subversion repositories stopped working…taking my GPSD project down with them. I had every reason to fear this might be permanent, and spent the next two days reconstructing as much as possible of the project state so we could migrate to another site.

Berlios came back up on Monday. But I don’t trust it will stay that way. This weekend rubbed my nose in some systemic vulnerabilities in the open-source development infrastructure that we need to fix. Rant follows.

Continue reading

Sep 30

Raymond & Polanski vs. “Mr. Society”

In my previous post, Why Artists Defend Roman Polanki, I analyzed the flap over the Roman Polanski arrest as a case of artists arguing for a privilege to behave like shitheels without being held to account for it. I advanced this as an explanation because I think it covers the facts better than some of the culture-war political narratives being bandied about, especially by conservatives, but I deliberately did not take a position on the rights and wrongs of the arrest or whether I think Polanski should be prosecuted at this late date.

Now I’ll do so. I expect it will startle almost all of my regulars and offend a good many of them, but I think Polanki should be let go.

Continue reading

Sep 29

Why artists defend Roman Polanski

In 1977, Roman Polanski drugged, raped, and sodomized a 13-year-old girl. When he believed a sort-of-plea-bargain was about to come unstuck, he took it on the lam. He lived the high life in this self-imposed exile for thirty years, until busted in Switzerland recently. Now various of the usual suspects on the right wing’s enemies list are campaigning to block his extradition.

There’s a good deal of perplexity being expressed about this, and some predictable chuntering from right-wingers about lefties being moral degenerates. But this flap isn’t really about politics at all — it’s much simpler than that. It’s about people who think of themselves as “artistes” reserving themselves a get-out-of-jail card when they feel like behaving like repellent scum of the earth, too.

Continue reading

Sep 28

Mighty aches from little ACORN’s fall

In all the foofaraw surrounding the ACORN scandals, there is a huge important consequence of them understood – but not spoken – by everyone who follows politics as a blood sport. This story describing conditions in Michigan and tallying up some recent ACORN convictions for electoral fraud is an indicator. And, on top of Obama’s plummet in the polls, the attention now being focused on ACORN has got to have any thinking Democratic strategist deeply worried about the 2010 midterms.

No, I’m not talking about the mere aura of scandal, the prospect that some of the smell coming off of ACORN might cling to the general run of Democratic politicians. That’s not going to happen, not while most of the mainstream media seems ever more intent on operating as an unpaid auxiliary for the Democratic National Committee. No stench will be allowed to adhere, not even if the stalwart partisans of the Fourth Estate have to lick it off with their own tongues. No, the real problem is this: ACORN was the linchpin of the Democratic electoral-fraud machine. Without it, the party’s position going into the next round of elections may be seriously weakened.

Continue reading

Sep 24

Uncivil society and the collapse of the nomenklatura

A few moments ago, I read a review of a new book, Uncivil Society: 1989 and the Implosion of the Communist Establishment, and the following sentences jumped out at me:

This is less a story of dissidents, so-called civil society, than of the bankruptcy of a ruling class–communism’s establishment, or “uncivil society.” The Communists borrowed from the West like drunken sailors to buy mass consumer goods, then were unable to pay back the hard-currency debts and so borrowed even more. In Eastern Europe, communism came to resemble a Ponzi scheme, one whose implosion carries enduring lessons.

I found myself wondering “And this differs from our political class…how?”

Continue reading