My title is, of course, a reference to the 1984 paper End-to-End Arguments in System Design by Reed, Saltzer, and Clark. They enunciated what has since become understood as perhaps the single most central and successful principle of the design of the Internet. If you have not read it, do chase the link; it well deserves its status as a classic.
The authors wrote mostly about the design of communications networks. But the title referred not to “network design” but to system design, and some of the early language in the paper hints that the authors thought they had discovered a design rule with implications beyond networking. I shall argue that indeed they had – there is a version of the end-to-end principle that applies productively and forcefully to the design of (non-networked) software. I shall develop that version, illustrating it with a case study from experience.
My regulars will be aware that, since the Mixed Martial Arts program we were in folded up, my wife Cathy and I have been having an interesting learning adventure checking out various schools in our area as possibilities for our next style. We’ve had some more adventures since.
I’ve just shipped irker 1.8, and I think this brings the wild ride I’ve been on for the last eleven days approximately to a close. I consider this release feature-complete; it achieves all the goals I had in mind when the CIA service died and I decided it was up to me to rescue the situation. I expect the development pace to slow down a lot from the almost daily release I’ve been doing.
The last really major feature was irkerhook support for Mercurial repositories. I’d be mildly interested in a bzr extractor class if anyone wanted to contribute one, and that probably wouldn’t be hard – the git and hg extractors are about 70 lines each. But with git, hg, and Subversion covered it’s good enough.
Uptake of irker continues at a pleasingly rapid pace. There’s now a second symbiote application, a poller daemon that watches the log of a specified Subversion repository and uses irkerd to ship notifications from it. This can be useful if you don’t have write access to the repo hooks and thus cannot install irkerhook.
Time for a pause and some reflection on lessons to be learned.
Considering the extent to which I’m still a public figure, it is perhaps surprising how seldom I get email that deserves a thorough, up-one-side-and-down-the-other flaming. I got one today which I shall reproduce here as a perfect example of how not to engage me.
It was just three days ago that I shipped irker 1.0, but the project is already a huge hit out there in hackerland. It’s clear from traffic on the freenode #commits channel that irker installations are springing up everywhere. There’s already one symbiote, a proxy that takes XML-RPC requests in the CIA format and passes them to an irker instance (you have to supply your own mapping of projects to IRC channels for it to use). And at least one custom hook already written and in production – by the Python development list, as it happens.
I’m a bit boggled, actually. I don’t think I’ve ever had a project go from launch to all over the freakin’ landscape this fast before. Guess that’ll happen when you step up with a clean replacement for a service that lots of people were habituated to and have suddenly lost.
There’s more work to be done, of course. (There’s a public repository, and an #irker IRC channel, for people interested in following development.)
The CIA commit-notification service abruptly died two days ago, a development that surprised nobody who has been paying attention to the recent history of the codebase and its one public server site. A screwup at the cloud service hosting the CIA virtual machine irretrievably destroyed the instance data; please don’t ask me for details, I don’t know how it happened and don’t care. The CIA codebase is so messed up that even reconstituting a virgin instance would be way too much work – and that I will talk about a bit later in this post.
Fortunately, I saw this coming and had started work on a CIA replacement in late August. I had been holding off releasing it because there was some effort going on to salvage the CIA code, but that possibility effectively vanished when the only instance was erased. I shipped my replacement just a few minutes ago, and expect to spend much of the next week helping forge-site operators install it so we can have our notification service back.
The remainder of this post is a finished version of a design analysis of CIA I started a couple of weeks ago when the death of the service was still only a theoretical possibility. Since that theory has become actuality, the message should be heard loudly and clearly: this was a truly classic case of over-engineering, code bloat, excessive centralization, and bad practice. Read on for the cautionary tale.
OK, I’ve been hacking intensely for most of the last 24 hours and here’s the payoff: irker-1.0 is shipped. Code and documentation are at http://www.catb.org/esr/irker/.
Out of the starting box we have a hook script with tested support for git and (rather clumsily) Subversion; hg should be a piece of cake for anyone who wants to step up. Forge-site operators can begin installing the relay daemon and the repo hook immediately, and should do so.
Coming soon: a long essay I’ve been sitting on analyzing the now-dead CIA service as a case study in over-engineering. It’s not really very surprising that it collapsed under its own weight.
Also note that there is an XML-RPC proxy for people who have limited ability to change their hook scripts. I haven’t looked at the code myself but there’s a pointer in the irker README file.
A few hours ago I learned that, due to a virtual-server mishap, the cia.vc notification service is dead. And not coming back.
This was not entirely unexpected. The CIA codebase was a shambles, the service has been flaky and subject to outages, and the server-site operator who inherited it has for some time been muttering darkly that the end was probably nigh.
I’ve been sitting on a lightweight replacement for CIA since late August, holding off shipping until it was clear whether or not a salvage effort on the codebase was going to succeed. That option is off the table now, so I’m going into emergency overdrive to get a release out.
The main thing that still needs to be done is for me to finish and test a hook script for git repos, so that when I ship the admins at places like SourceForge and GitHub will be able to drop in both a server instance and the correct hook code. This script will also be a model for hooks serving other VCSes such as Mercurial, Subversion, and (ugh) CVS.
I’m working on that now and expect to ship within 48 hours. Watch this space.
Inspired by Dave Logan’s keynote on tribal leadership at AgileCultureCon, I did a breakout session and then an open-space followup on “Practical Prophecy 101″.
Recall that in Logan’s terms a “prophet” is a person who moves the behavior of his tribe towards greater cooperation and creativity by (his words) “preaching the inevitability of values-based change”.
Venessa Miemis took notes on my talk. Here’s a lightly edited and expanded version of those notes. In each item I have replayed a quote of mine that she recorded; where appropriate I have expanded a little on the thinking behind it.
I learned a new way of thinking about social behavior at Agile CultureCon last week – Dave Logan’s taxonomy of tribal stages and his interestingly specialized notion of what a “prophet” is. For review, see Logan’s TED talk.
Logan explains the distribution of tribal stages as follows: Stage 1, “Life Sucks”, is the violent and profoundly dysfunctional tribalism of gangs and prisons (approximately 2% of tribes); Stage 2, “My life sucks”, is bureaucracy (about 22% of tribes); Stage 3, “I’m great (but you’re not)!” is most of business and academia (about 48% of tribes); Stage 4: “We’re great!” is where you start to see serious creativity, tribal self-awareness, and collective sense of mission (about 22% of tribes); and Stage 5 “Life’s great!” is high-creative behavior totally driven by values rather than ego or struggle against some adversary (about 2% of tribes).
A “prophet”, in Logan’s model, is somebody who expresses the deepest shared values of a tribe and invites people in it to change stage (and fuse with other tribes at the new stage). Because most people, most of the time, live in tribes with a stage 3 culture, the most common upward transition (and the most common kind of prophet) is from stage 3 to stage 4.
I noted in a previous post that hearing this in a talk made the hair on the backs of my arms stand up. Because I have lived through, and was one of the prophets of, the hacker culture’s transition from largely unconscious mixed stage-3/stage-4 to fully conscious mostly Stage 4 behavior (“We’re great!”) in the 1990s.
But. I am by no means sufficiently ignorant or egotistical to think I was our only prophet. Most obviously there was Richard Stallman a decade before me, issuing a stage 4 call to higher values around “free software”. But because I was a historian before I was a prophet, I can’t really stop there. I find myself asking who the earlier prophets were!
My last four days, at the Agile CultureCon split between Philadelphia and Boston, have thrown more new ideas and techniques at me than I’m used to encountering in a normal four months. Or more. It was very challenging and exciting, the more so because I was immersed in a culture at some distance from those where I usually hang out.
The organizers (Dan Mezick & Andre Dhondt) and various friends (now including me) are launching from agile software development into new ways of organizing work and communication that dynamite a lot of common assumptions about the necessity of power relationships and hierarchies. What makes this really interesting is not the theory but the working examples. They’re not dealing in vague platitudes, but in methods that can be taught and replicated. (And yes, I will describe some of them later in this post.)
Nobody in this crowd thinks politically (or at least if they do, it doesn’t show); it’s all framed as ways to fix corporate cultures to make them more productive and happier. But what this was, underneath occasional freshets of vaguely new-agey language, was a three-day workshop in practical anarchy.
Dave Taht is crashing in my basement again. While he’s here Dave is planning to cut another release of CeroWRT (the third one to issue from this basement, actually), and he has decided it needs a name.
And, well, “the release from ESR’s basement” just lacks a certain…zing.
I just left the followiing comment on a Creatice Commons blog thread debating the NonCommercial and NoDerivatives options:
So, after my post on ground-truth documents, one of my commenters argued eloquently that I ought to clean it up and submit it to a journal read by people who manage programming projects. He suggested Software Practice and Experience.
This seemed like a pretty good idea, until I read SP&E’s submission procedures and was reminded that (like most journals) they want me to assign the copyright of my submission to the publisher.
My instant reaction was this: Fuck. That. Noise. I’m certainly willing to cede publication rights when I want to be published, but copyright assignment ain’t going to happen. Ever. Nobody gets to own my work but me. (Yes, I insist on this with my book publishers too.)
I have a message to all you technical journal publishers out there…
Coverity simplified their remote-submission procedure. Because of this, I have been able to remove the ugliest bits of configuration cruft from coverity-submit; you no longer have to specify either a public drop directory for your results tarball or a URL that advertises it.
Get your remote-static-checking goodness here.
Yesterday I applied for allocation of a new public port number from IANA. It’s 6659. When the allocation is confirmed, I’ll publish the source code for a reference implementation of the server. It’s a bit over 300 lines of Python.
Let the speculation begin. :-)
Sometimes good terminology, by making a distinction that wasn’t easily articulated before, can be very clarifying. I was in an IRC conversation about software engineering with A&D regular HedgeMage earlier today and found myself inventing a term that I think may be useful: the ground-truth document.
A few days ago I posted “A martial-arts trilemma” about Cathy’s and my search for a new school to train with following the demise of our MMA program. We’ve since gotten one nice surprise and struck two alternatives off our list. And thereby hangs a tale.
I suppose it was inevitable, in a good way. Some friends of freedom have begun a project dedicated to developing and sharing open-source designs for firearms that can be manufactured with a 3D printer. Read about it here at Defense Distributed.
I approve, of course. I approve of any development that makes it more difficult for governments and criminals to monopolize the use of force. As 3D printers become less expensive and more ubiquitous, this could be a major step in the right direction.
I shipped an updated version of the ciabot hook scripts for git to the git maintainers this morning.
The cool new thing in this release is that the script no longer needs to be modified for installation as a hook. You can install one copy where any number of git repositories can see it; when it’s run it will collect the information it needs either by autoconfiguring or by looking at variables set in each project’s .git/config file.
UPDATE: Once I started looking at the code…I found a way to make it completely self-configuring in the normal case. So I’ve shipped 3.6. This will be helpful for forge sites like Savannah, because it means they’ll be able to install one standard git hook that Just Works.