I nearly issued a forking threat a few minutes ago. Only the second time I’ve felt a need to do that and the first was in 1993, so this is not something I do casually. And I drew back from the brink.
But I may have to if the maintainer I’m dealing with doesn’t clean up his act. His library is critical to one of my projects, but his behavior has been increasingly sloppy and erratic lately. He made a serious design mistake which he’s been trying to paper over with kluges; the kluges have made the code unstable and the latest shipped version is actually broken to the point of unusability without a patch.
My title is, of course, a reference to the 1984 paper End-to-End Arguments in System Design by Reed, Saltzer, and Clark. They enunciated what has since become understood as perhaps the single most central and successful principle of the design of the Internet. If you have not read it, do chase the link; it well deserves its status as a classic.
The authors wrote mostly about the design of communications networks. But the title referred not to “network design” but to system design, and some of the early language in the paper hints that the authors thought they had discovered a design rule with implications beyond networking. I shall argue that indeed they had – there is a version of the end-to-end principle that applies productively and forcefully to the design of (non-networked) software. I shall develop that version, illustrating it with a case study from experience.
I’ve just shipped irker 1.8, and I think this brings the wild ride I’ve been on for the last eleven days approximately to a close. I consider this release feature-complete; it achieves all the goals I had in mind when the CIA service died and I decided it was up to me to rescue the situation. I expect the development pace to slow down a lot from the almost daily release I’ve been doing.
The last really major feature was irkerhook support for Mercurial repositories. I’d be mildly interested in a bzr extractor class if anyone wanted to contribute one, and that probably wouldn’t be hard – the git and hg extractors are about 70 lines each. But with git, hg, and Subversion covered it’s good enough.
Uptake of irker continues at a pleasingly rapid pace. There’s now a second symbiote application, a poller daemon that watches the log of a specified Subversion repository and uses irkerd to ship notifications from it. This can be useful if you don’t have write access to the repo hooks and thus cannot install irkerhook.
Time for a pause and some reflection on lessons to be learned.
It was just three days ago that I shipped irker 1.0, but the project is already a huge hit out there in hackerland. It’s clear from traffic on the freenode #commits channel that irker installations are springing up everywhere. There’s already one symbiote, a proxy that takes XML-RPC requests in the CIA format and passes them to an irker instance (you have to supply your own mapping of projects to IRC channels for it to use). And at least one custom hook already written and in production – by the Python development list, as it happens.
I’m a bit boggled, actually. I don’t think I’ve ever had a project go from launch to all over the freakin’ landscape this fast before. Guess that’ll happen when you step up with a clean replacement for a service that lots of people were habituated to and have suddenly lost.
There’s more work to be done, of course. (There’s a public repository, and an #irker IRC channel, for people interested in following development.)
The CIA commit-notification service abruptly died two days ago, a development that surprised nobody who has been paying attention to the recent history of the codebase and its one public server site. A screwup at the cloud service hosting the CIA virtual machine irretrievably destroyed the instance data; please don’t ask me for details, I don’t know how it happened and don’t care. The CIA codebase is so messed up that even reconstituting a virgin instance would be way too much work – and that I will talk about a bit later in this post.
Fortunately, I saw this coming and had started work on a CIA replacement in late August. I had been holding off releasing it because there was some effort going on to salvage the CIA code, but that possibility effectively vanished when the only instance was erased. I shipped my replacement just a few minutes ago, and expect to spend much of the next week helping forge-site operators install it so we can have our notification service back.
The remainder of this post is a finished version of a design analysis of CIA I started a couple of weeks ago when the death of the service was still only a theoretical possibility. Since that theory has become actuality, the message should be heard loudly and clearly: this was a truly classic case of over-engineering, code bloat, excessive centralization, and bad practice. Read on for the cautionary tale.
OK, I’ve been hacking intensely for most of the last 24 hours and here’s the payoff: irker-1.0 is shipped. Code and documentation are at http://www.catb.org/esr/irker/.
Out of the starting box we have a hook script with tested support for git and (rather clumsily) Subversion; hg should be a piece of cake for anyone who wants to step up. Forge-site operators can begin installing the relay daemon and the repo hook immediately, and should do so.
Coming soon: a long essay I’ve been sitting on analyzing the now-dead CIA service as a case study in over-engineering. It’s not really very surprising that it collapsed under its own weight.
Also note that there is an XML-RPC proxy for people who have limited ability to change their hook scripts. I haven’t looked at the code myself but there’s a pointer in the irker README file.
Coverity simplified their remote-submission procedure. Because of this, I have been able to remove the ugliest bits of configuration cruft from coverity-submit; you no longer have to specify either a public drop directory for your results tarball or a URL that advertises it.
Get your remote-static-checking goodness here.
Sometimes good terminology, by making a distinction that wasn’t easily articulated before, can be very clarifying. I was in an IRC conversation about software engineering with A&D regular HedgeMage earlier today and found myself inventing a term that I think may be useful: the ground-truth document.
I shipped an updated version of the ciabot hook scripts for git to the git maintainers this morning.
The cool new thing in this release is that the script no longer needs to be modified for installation as a hook. You can install one copy where any number of git repositories can see it; when it’s run it will collect the information it needs either by autoconfiguring or by looking at variables set in each project’s .git/config file.
UPDATE: Once I started looking at the code…I found a way to make it completely self-configuring in the normal case. So I’ve shipped 3.6. This will be helpful for forge sites like Savannah, because it means they’ll be able to install one standard git hook that Just Works.
I’ve released doclifter 2.9, and as part of that process I’ve been testing it on the entire collection of manual pages on my system again. Because doclifter does mechanical translation of troff-based markups to DocBook-XML, one of the side effects of testing it is that I find lots of broken markup. I’ll ship over 700 fix patches back to maintainers this time, though maybe not until after I get back from World Boardgaming Chapionships next week.
Release here, report on markup bugs found is here. Yes, over 700 patches, but that’s actually a drop from previous passes.
Comes the news that Nvidia just lost an order for 10 million graphics cards to AMD because it wouldn’t open the source for its driver. At a very conservative estimate, that’s north of $250 million in business Nvidia just threw to a major competitor because it couldn’t get its head out of its rectum. Somebody’s quarterlies are going to suck.
The really interesting aspect of this isn’t the amount of money Nvidia’s idiotic secrecy fetish just cost it, but why it happened – and why it’s likely to happen again, soon and repeatedly, to other hardware companies with equally idiotic secrecy fetishes.
In response to a bug report that was relatively easily fixed, I’ve just shipped release 2.8 of doclifter, a program that takes troff-based document markups – including man page markup – and lifts them to DocBook XML.
Yes, two software releases in a day is an unusually rapid tempo even from me. But freecode-submit is part of my release machinery for other projects, and when I shipped GIFLIB 5.0.0 I discovered it had gone all pear-shaped on me. Problem turned out to be an unannounced change in freecode’s JSON interface. I hate it when that happens…
I’ve just shipped the 5.0.0 release of GIFLIB, a graphics service library that is deployed pretty much everywhere that throws pixels on a display. Older versions live in your browser, your game console, and your smartphone. I have written about what it was like to go back to this code after 18 years previously, in The Long Past of C; also in my 4.2.0 release announcement.
Some people are obsessive about never using closed-source software under any circumstances. Some other people think that because I’m the person who wrote the foundational theory of open source I ought to be one of those obsessives myself, and become puzzled and hostile when I demur that I’m not a fanatic. Sometimes such people will continue by trying to trap me in nutty false dichotomies (like this guy) and become confused when I refuse to play.
A common failure mode in human reasoning is to become too attached to theory, to the point where we begin ignoring the reality it was intended to describe. The way this manifests in ethical and moral reasoning is that we tend to forget why we make rules – to avoid harmful consequences. Instead, we tend to become fixated on the rules and the language of the rules, and end up fulfilling Santayana’s definition of a fanatic: one who redoubles his efforts after he has forgotten his aim.
When asking the question “When is it wrong (or right) to use closed-source software?”, we should treat it the same way we treat every other ethical question. First, by being very clear about what harmful consequences we wish to avoid; second, by reasoning from the avoidance of harm to a rule that is minimal and restricts peoples’ choices as little as possible.
In the remainder of this essay I will develop a theory of the harm from closed source, then consider what ethical rules that theory implies.
I’ve now read Judge Alsup’s ruling in the Oracle vs. Google lawsuit addressing the copyrightability of the Java APIs as a matter of law. This is a bigger win for the good guys than appears at first glance; Alsup has subtly but definitely driven a stake through the heart of API copyrights. The interesting part is how he did it.
To the surprise of nobody who was actually familiar with the underlying law and precedent, the judge in the Oracle-vs.-Google mega-lawsuit ruled today that Oracle’s claim of copyright protection on the Java APIs is contrary to law.
This means Oracle’s claims against Google are toast. Their best case is now that they’ll get $300K in statutory damages for two technical copyright violations, almost noise compared to what Oracle spent in legal fees. The patent claims went just as thoroughly nowhere as I predicted back when the lawsuit was launched.
It’s all over the net today. As I repeatedly predicted, the patent claims in the Oracle-vs.-Java lawsuit over Android have completely fizzled. Oracle’s only shred of hope at this point is that Judge Alsup will rule that APIs can be copyrighted, and given the extent of cluefulness Alsup has displayed (he mentioned in court having done some programming himself) this seems rather unlikely.
First giflib release since I reassumed the lead. Short version: lots of useless old cruft thrown out, everything Coverity-scanned, one minor resource leak found and fixed.
My regular readers will know that (a) I’ve recently been pounding bugs out of GPSD with Coverity, and (b) I hate doing stupid clicky-dances on websites when I think I ought to be able to shove them a programmatically-generated job card that tells them what to do.
So, here’s a side-effect of my recent work with Coverity: coverity-submit. Set up a config file once, and afterwards just run coverity-submit in your project directory and stand back. Supports multiple projects. Because, manularity is evil.
Here’s the HTML documentation.