The following question appeared in my mailbox today:
If a certain program (that you need) was proprietary, and its open-source counterpart was (currently) 40% slower. Which would you use, the open-source one or the proprietary one?
The answer is: it depends. I’m going to answer this one in public because it’s a useful exercise in thinking about larger tradeoffs.
For those of you who have been following my work on tools to muck out the swamp that is CVS, a new shovel: cvs-fast-export. Note, this is an alpha release for testing purposes; double-check the quality of any conversions you do with it carefully.
The 2.13 release of reposurgeon has in it maybe the coolest new feature I’ve ever implemented in five minutes of programming – graphical visualization of a repository’s commit DAG.
Last month I added CVS-reading support to reposurgeon. The reason I haven’t blogged in ten days is that this pulled me down a rathole out of which I am just now beginning to emerge. And now I have a a request for help – I need to collect some perverse CVS repositories, preferably relatively small ones.
Today’s very special non-world-ending software release, triggered if not originated from here at Eric Conspiracy Secret Laboratories is the amazing Zen simulation, Robot Finds Kitten. I bow in respect before Leonard Richardson and the other giants of kitten-finding history and am humbly proud to be counted among the select few who have contributed to this monumental, er, monument.
Get yer hot fresh tarball right here. It will improve your sex life, clear up your financial problems, cure your acne, and make you as a god among men. Would I lie?
Well, er, no. Actually, it attacks CVS.
Yes, that’s right, the just-shipped reposurgeon 2.11 can now read – though not write – CVS repositories. To get it to do this, I got my lunch-hooks on a relatively old program called cvsps that assembles changesets out of CVS repositories for human inspection. I gave it a –fast-export reporting mode that emits a fast-import stream instead, so now CVS has a universal exporter that will talk to any version-control system that speaks import streams. Oh, yes, and I’m maintaining cvsps now too – applause to David Mansfield, who both did a very good job on that code and sees clearly that its original use case is obsolete and –fast-export is a better way forward.
Two substantial releases of different projects in a day is a fast pace even for me. cvsps-3.0 and reposurgeon-2.11; two great tastes that taste great together.
Fear the reposturgeon!
Well-designed software suites should not only be correct, they should be able to demonstrate their own correctness. This is why the new 2.10 release of reposurgeon features a new tool called ‘repodiffer’. And yes, that is what it sounds like – a diff tool that operates not on files but entire repository histories. You get a report on which revisions are identical, which are different, and in the latter case where the differences are, down to which files don’t match. Commits to be paired are matched by committer and commit date. Like reposurgeon, it will work on any version-control system that can emit a fast-import stream.
I had said I wasn’t going to do it, but…I experimented, and it turned out to be easier than I thought. Release 2.7 of reposurgeon writes (as well as reading) Subversion repositories. With the untested support for darcs, which should work exactly as well as darcs fast-export and fast-import do, this now brings the set of fully-supported version-control systems to git, hg, bzr, svn, and darcs; reposurgeon can be used for repository surgery and interconversion on any of these.
I have submitted an essay to the Stanford Law Review for publication. I didn’t tick the box for “exclusive”, so I think I can blog it as well. It’s a reply to Andrew Tutt’s essay on Software Speech.
I shipped reposurgeon 2.0 a few days ago with the Subversion support feature-complete, and a 2.1 minor bugfix release this morning. My previous release announcement was somewhat rushed, so here is a more detailed one explaining why anybody contemplating moving up from Subversion should care.
To go with this, there is a new version of my DVCS Migration HOWTO.
I’ve just shipped reposurgeon 2.0, a power tool for editing and interconverting version-control repositories. This is a major release, adding the capability to read Subversion dump files directly.
This was a sort of spinoff from irker, though I’ve decided it not to use it there because I want to keep irkerd and irkerhook.py in single self-contained files.
No, I don’t know dak180’s real name. He’s a pretty good collaborator, though.
I nearly issued a forking threat a few minutes ago. Only the second time I’ve felt a need to do that and the first was in 1993, so this is not something I do casually. And I drew back from the brink.
But I may have to if the maintainer I’m dealing with doesn’t clean up his act. His library is critical to one of my projects, but his behavior has been increasingly sloppy and erratic lately. He made a serious design mistake which he’s been trying to paper over with kluges; the kluges have made the code unstable and the latest shipped version is actually broken to the point of unusability without a patch.
My title is, of course, a reference to the 1984 paper End-to-End Arguments in System Design by Reed, Saltzer, and Clark. They enunciated what has since become understood as perhaps the single most central and successful principle of the design of the Internet. If you have not read it, do chase the link; it well deserves its status as a classic.
The authors wrote mostly about the design of communications networks. But the title referred not to “network design” but to system design, and some of the early language in the paper hints that the authors thought they had discovered a design rule with implications beyond networking. I shall argue that indeed they had – there is a version of the end-to-end principle that applies productively and forcefully to the design of (non-networked) software. I shall develop that version, illustrating it with a case study from experience.
I’ve just shipped irker 1.8, and I think this brings the wild ride I’ve been on for the last eleven days approximately to a close. I consider this release feature-complete; it achieves all the goals I had in mind when the CIA service died and I decided it was up to me to rescue the situation. I expect the development pace to slow down a lot from the almost daily release I’ve been doing.
The last really major feature was irkerhook support for Mercurial repositories. I’d be mildly interested in a bzr extractor class if anyone wanted to contribute one, and that probably wouldn’t be hard – the git and hg extractors are about 70 lines each. But with git, hg, and Subversion covered it’s good enough.
Uptake of irker continues at a pleasingly rapid pace. There’s now a second symbiote application, a poller daemon that watches the log of a specified Subversion repository and uses irkerd to ship notifications from it. This can be useful if you don’t have write access to the repo hooks and thus cannot install irkerhook.
Time for a pause and some reflection on lessons to be learned.
It was just three days ago that I shipped irker 1.0, but the project is already a huge hit out there in hackerland. It’s clear from traffic on the freenode #commits channel that irker installations are springing up everywhere. There’s already one symbiote, a proxy that takes XML-RPC requests in the CIA format and passes them to an irker instance (you have to supply your own mapping of projects to IRC channels for it to use). And at least one custom hook already written and in production – by the Python development list, as it happens.
I’m a bit boggled, actually. I don’t think I’ve ever had a project go from launch to all over the freakin’ landscape this fast before. Guess that’ll happen when you step up with a clean replacement for a service that lots of people were habituated to and have suddenly lost.
There’s more work to be done, of course. (There’s a public repository, and an #irker IRC channel, for people interested in following development.)
The CIA commit-notification service abruptly died two days ago, a development that surprised nobody who has been paying attention to the recent history of the codebase and its one public server site. A screwup at the cloud service hosting the CIA virtual machine irretrievably destroyed the instance data; please don’t ask me for details, I don’t know how it happened and don’t care. The CIA codebase is so messed up that even reconstituting a virgin instance would be way too much work – and that I will talk about a bit later in this post.
Fortunately, I saw this coming and had started work on a CIA replacement in late August. I had been holding off releasing it because there was some effort going on to salvage the CIA code, but that possibility effectively vanished when the only instance was erased. I shipped my replacement just a few minutes ago, and expect to spend much of the next week helping forge-site operators install it so we can have our notification service back.
The remainder of this post is a finished version of a design analysis of CIA I started a couple of weeks ago when the death of the service was still only a theoretical possibility. Since that theory has become actuality, the message should be heard loudly and clearly: this was a truly classic case of over-engineering, code bloat, excessive centralization, and bad practice. Read on for the cautionary tale.
OK, I’ve been hacking intensely for most of the last 24 hours and here’s the payoff: irker-1.0 is shipped. Code and documentation are at http://www.catb.org/esr/irker/.
Out of the starting box we have a hook script with tested support for git and (rather clumsily) Subversion; hg should be a piece of cake for anyone who wants to step up. Forge-site operators can begin installing the relay daemon and the repo hook immediately, and should do so.
Coming soon: a long essay I’ve been sitting on analyzing the now-dead CIA service as a case study in over-engineering. It’s not really very surprising that it collapsed under its own weight.
Also note that there is an XML-RPC proxy for people who have limited ability to change their hook scripts. I haven’t looked at the code myself but there’s a pointer in the irker README file.
Coverity simplified their remote-submission procedure. Because of this, I have been able to remove the ugliest bits of configuration cruft from coverity-submit; you no longer have to specify either a public drop directory for your results tarball or a URL that advertises it.
Get your remote-static-checking goodness here.
Sometimes good terminology, by making a distinction that wasn’t easily articulated before, can be very clarifying. I was in an IRC conversation about software engineering with A&D regular HedgeMage earlier today and found myself inventing a term that I think may be useful: the ground-truth document.