These are comments on policy for sharing information from the
U.S. Coast Goard’s NAIS, a network of 140 AIS receivers covering
U.S. coastal waters, as solicited by Federal docket USCG-2009-0701.
(The docket request for comments is here.)
There’s a wonderfully tongue-in-cheek project called the The Alliance for Code Excellence (“Building a better tomorrow — one line of code at a time.”) that sells Bad Code Offset certificates. They fund open source projects to produce good code that will, in theory, offset all the bad code out there and mitigate the environmental harm it does. They’ve asked software authors to write essays on how their projects drive out bad code, offering $500 dollar prizes.
I sat down to write an essay about GPSD in the same vein of high drollery as the Alliance’s site, then realized that GPSD actually has a serious case to make. We really do drive out bad code, in both direct and indirect ways, and we supply examples of good practice for emulation.
Your handler class’s job is to extract project data. If you are lucky, your target forge already has an export feature that will dump everything to you in clean XML or JSON; in that case, you have a fairly trivial exercise using BeautifulStoneSoup or the Python-library JSON parser and can skip the rest of this section.
Usually, however, you’re going to need to extract the data from the same pages that humans use. This is a problem, because these pages are cluttered with all kinds of presentation-level markup, headers, footers, sidebars, and site-navigation gorp — any of which is highly likely to mutate any time the UI gets tweaked.
Here are the tactics we use to try to stay out of trouble:
1. When you don’t see what you expect, use the framework’s self.error() call to abort with a message. And put in lots of expect checks; it’s better for a handler to break loudly and soon than to return bad data. Fixing the handler to track a page mutation won’t usually be hard once you know you need to – and knowing you need to is why we have regression tests.
2. Use peephole analysis with regexps (as opposed to HTML parsing of the whole page) as much as possible. Every time you get away with matching on strictly local patterns, like special URLs, you avoid a dependency on larger areas of page structure which can mutate.
3. Throw away as many irrelevant parts of the page as you can before attempting either regexp matching or HTML parsing. (The most mutation-prone parts of ppsages are headers, footers, and sidebars; that’s where the decorative elements and navigation stuff tend to cluster.) If you can identify fixed end strings for headers or fixed start strings for footers, use those to trim (and error out if they’re not there); that way you’ll be safe even if the headers and footers mutate. This is what the narrow() method in the framework code is for.
4. Rely on forms. You can assume you’ll be logged in with authentication and permissions to modify project data, which means the forge will display forms for editing things like issue data and project-member permissions. Use the forms structure, as it is much less likely to be casually mutated than the page decorations.
5. When you must parse HTML, BeautifulSoup is available to handler classes. Use it, rather than hand-rolling a parser, unless you have to cope with markup so badly malformed that it cannot cope.
Actual field experience shows that throwing out portions of a page that are highly susceptible to mutation is a valuable tactic. Also, think about where in the site a page lives. Entry pages and other highly visible ones tend to get tweaked the most often, so the tradeoffs push you towards peephole methods and not relying on DOM structure. Deeper in the site , especially on pages that are heavily tabular and mostly consist of one big form, relying on DOM structure is less risky.
So, I announce ForgePlucker, and within a day I’ve got some guy
from Y Combinator sneering at me for using regular expressions to parse HTML. Says it’s “crappy code”. The poor fool…he has fallen victim to a conceptual trap which I, fortunately, learned to avoid decades ago. I could spout a freshet of theory about it, but instead I’m just going to utter a maxim: Never confuse structure with meaning.
I’ve been strongly hinting in recent blog entries that I planned to do something concrete about the data-jail problems of present open-source hosting sites. Because I believe in underpromising and overperforming, I decided at the outset not to announce a project until I could not only show working code, but code with wide enough coverage to make it crystal-clear that the project goals are achievable with a relatively modest amount of effort.
That time has arrived. I am very pleased to announce ForgePlucker, a project aimed at developing project-state extractor software for backup, offline analysis, and (eventually) re-importation. The proof-of-concept code can extract complete issue-tracker state from Berlios, Gna!, or Savane — and issue trackers are probably the hardest part of the job. I expect extraction of repository histories and developer permissions tables to be easier. Extraction of mailing-list state is probably a bit trickier than either of those, but doable.
I’m still not going to talk about my attack on the forge infrastructure problems quite yet; the software is coming along nicely, but I intend to announce only after it handles its fourth forge type (yes, that was a tease). But I will say this: I now think I know what the future of forges looks like. It’s called Roundup, and it is astonishingly elegant and potentially more powerful than anything out there. Anything, not excluding the clever decentralized systems like Fossil or Bugs Everywhere.
Here are the big wins:
1. Mailing lists, issue trackers, and online forums unify into *one* message queue that can be filtered in various ways.
2. Scriptable via XML-RPC or an email responder ‘bot.
3. Small base system with good extensibility – just three base classes (User. Msg, File) and the ability to define new classes. ‘Issue’ is a class built on top of these.
4. Arbitrary attributes per issue is basically free, with baked-in support for defining controlled vocabularies.
5. There’s a uniform way, called “designators”, for messages and other objects to refer to each other in text.
6. Small, clean implementation written in Python.
There are some things it needs, though… (Read the Roundup design document before continuing.)
Hackers have a lot of odd superstitions about software licensing. I was reminded of this recently when a project maintainer asked me whether he needed to get a sign-off from each and every one of his contributors before switching from Apache v1 to Apache v2. Here’s what I told him:
My opinion is this. Under U.S. law — and I believe European codes are not different in this respect, because both are controlled by the Berne convention — a license change on a collection is grounds for protest or legal action only if the rights of the contributors are materially affected by the change. That is, a court would have to be persuaded that the change caused a monetary loss or at least damage to a contributor’s public reputation. If there is no such possibility, then there is no harm and no grounds for complaint.
In my previous two posts I have diagnosed a significant weakness in the open-source infrastructure. The architecture of the code behind the major SourceForge-descended hosting sites is rotten, with all kinds of nasty consequences — data seriously jailed, poor or completely absent capabilities near scripting and project migration. I said I was going to do something about it, and I’m working the problem now — actually writing code.
The rest of this post is not an announcement, because it will be mostly about things I’ have figured out I should not try to do. Yet. But it is a teaser. I see a path forward, and shortly I expect to have some working code to exhibit that shows the way. Actually, I have working code that attacks the problem in an interesting way now, but I’m still adding capabilities to make it a more impressive demonstration.
Here are some approaches I’ve considered, or had suggested to me by others, and rejected:
In my previous post, Three Systemic Problems With Open-Source Hosting Sites I identified some missing features that create serious brittleness in or project-hosting infrastructure. The question naturally arises, why don’t existing hosting systems already have these facilities? I have looked into this question, actually examining the codebases of Savane and GForge/FusionForge, and the answer appears to go back to the original SourceForge. It offered such exciting, cutting-edge capabilities that nobody noticed its internal architecture was a tar-pit full of nasty kluges. The descendants — Savane, GForge, and FusionForge — inherited that bad architecture.
I’ve been off the air for several days due to a hosting-site failure last Friday. After several months of deteriorating performance and various services being sporadically inaccessible, Berlios’s webspace went 404 and the Subversion repositories stopped working…taking my GPSD project down with them. I had every reason to fear this might be permanent, and spent the next two days reconstructing as much as possible of the project state so we could migrate to another site.
Berlios came back up on Monday. But I don’t trust it will stay that way. This weekend rubbed my nose in some systemic vulnerabilities in the open-source development infrastructure that we need to fix. Rant follows.
I’ve been doing some serious redesign work on GPSD recently. I had planned to do a blog posting about lessons learned, but the result grew enough length and structure to turn into an actual technical paper. You can read it here; comments and criticism will be welcomed.
Note, everything described in the paper has already been implemented in gpsd. There’s work still to be done; for those of you familiar with the software, I still need to do equivalents of the old–protocol commands B C J N R Z $. I do not expect these to pose any significant difficulties.
Well, yes. I’m kind of wondering why they didn’t figure that out sooner. They had the data; I’ve probably been quoted in The Economist more often than any other single magazine. Except maybe Linux Journal.
Six years ago, in The Art of Unix Programming, I observed some interesting trends in the deployment of programming languages. One Christer NyfÃ¤lt mailed me this morning reporting that he had followed up by collecting the analogous statistics from SourceForge for present time. Here’s what he said (lightly copy-edited as his English is a bit shaky):
Well, now. This is interesting: A study of corporate Linux adoption polling 1,275 IT professionals says:
Linux desktop roll out is easier than expected for properly targeted end-user groups
Those with experience are much more likely to regard non-technical users as primary targets for Linux. The message here is that in practice, Linux is easier to deploy to end users than many imagine before they try it.
It’s become fashionable lately to be pessimistic about Linux’s future on the desktop, but I have to say this matches my experience pretty well. The handful of Ubuntu deployments I’ve done in the last couple years for end-users have indeed been easier than one might have expected.
I enjoy a game called “Commands and Colors: Ancients”, which I’ve blogged about here before. It’s a simulation of tactical ancient warfare that uses special dice to resolve battles. In any given battle you can consider each six-sided die to have faces labeled with the symbols Miss, Miss, Hit, Sword, Helmet, Flag. To improve my play, I decided to generate and study a table of the odds of getting a specified number of hits when a specified number of dice is rolled. I set out to write a Python program to do this.
There are special circumstances under which flags and helmets convert to hits, so the program actually needs to print out several tables and is not entirely trivial. Still, it is computing on a mathematically simple model with strictly bounded computational cost – except for war elephants. These units have the special ability that when they roll a sword, the sword is counted as a hit and then rerolled. (This may make more sense if you think of a sword roll as representing impact damage.) You keep rolling and marking hits as long as the die keeps coming up swords.
To calculate the non-elephant probabilities I enumerated eight entire state spaces for each of 1 to 8 dice (the most you can ever roll under the rules – takes Alexander the Great or Julius Caesar leading a Heavy unit with the the Clash of Shields +2 bonus active), then counted up instances of each distinct outcome (so many misses, hits, swords, flags, banners) to assign a probability mass to each.
(Statisticians often think in terms of probability mass or probability weight, which has to be conserved as a distribution changes. It’s analogous to thinking of electricity as a fluid.)
I then had to write code to mutate a copy of each distribution according to the elephants’ sword-reroll rule. To do this, each outcome containing a sword hit needs its probability mass divided by six and reallocated to itself and five other outcomes with one fewer sword apiece; you stop reallocating when the probability mass on a sword-containing outcome drops below a very low noise level.
That particular piece of code gave me more trouble than the rest of the program put together. At one point I grumbled to a friend who had been following the project “I’m having persistent bugs in my probability mass reallocator.”
He looked right back at me and said, with a perfectly straight face, “Have you considered reversing the polarity of the neutron flow?”
The reactions to my posting on the economic case against the GPL reminded me yet again why failure to understand basic economics often becomes more toxic in people who think they understand a bit about the subject. In this mini-essay, I’ll take a look at the most important (and misleading) of the superficially clever arguments I saw in responses.
In a recent O’Reilly interview, Richard Stallman utters an anathema against software-as-a-service arrangements, calling them “non-free” and saying “you must not use it!” It would be easy to parody RMS’s style of uttering grave moralistic sonorities as though he were the Pope speaking ex-cathedra, but I’m going to resist the temptation because I think in this case his concerns are quite valid.
Is open-source development a more efficient system of software production than the closed-source system? I think the answer is probably “yes”, and that it follows the GNU GPL is probably doing us more harm than good.