If you’re a regular at A&D or on my G+ feed, and even possibly if you aren’t, you’ll have noticed that I ship an awful lot of code. I do get questions about this; between GPSD, reposurgeon, giflib, doclifter, and bimpty-bump other projects it is reasonable that other hackers sometimes wonder how I do it.
Here’s part of my answer: be fanatical about automating away every part of your workflow that you can. Every second you don’t spend on mechanical routines is a second you get to use as creative time.
Soon, after an 11-year alpha period, I’m going to ship version 1.0 of one of my main automation tools. This thing would be my secret weapon if I had secrets. The story of how it came to be, and why it took 11 years to mature, should be interesting to other hackers on several different levels.
I’m the designer or maintainer of around 40 open-source projects. Even allowing for the fact that more than half of those are very stable old code that only needs a release once in a blue moon, the cumulative amount of boring fingerwork involved in keeping these updated is considerable.
When I say “boring fingerwork” I’m not even talking about coding effort, but rather the mundane tasks of uploading tarballs to archive locations, updating web pages, mailing out announcements, sending release notifications to Freecode, broadcasting heads-ups on relevant IRC channels, et cetera.
For older projects this shipping overhead is often more work than applying the small fixes and patches that trigger each release. It’s tedious, fiddly stuff – and irritatingly error-prone if done by hand.
A long time ago, now, I decided to stop doing it by hand. My overall goal was simple: I wanted to be able to type “make release” (or, more recently, “scons release”) in my project directory and have the right things happen, without fail. So I started building a tool to automate away as much tedium and fiddliness as I could. I called it “shipper”, because, well, that’s what it does.
Shipper’s job is to identify deliverables (like, say, tarballs and generated web pages) and push them to appropriate destinations (like, a public FTP directory or a website). It’s also intended to issue release notifications over various channels.
One of the things all these announcements and many of the names of deliverables will have in common is an embedded version number. One of the goals of shipper’s design is to allow you to specify the release’s version number in one place and one place only – because when you repeat a detail like that from memory you will occasionally get it wrong, with embarrassing results.
As for version numbers, so for other pieces of metadata that archive sites and forges and announcement channels commonly want – like a sort description of the project’s purpose, or a home page link, or the name of a project IRC channel. A design goal is that you only need to specify anything like this once per project; shipper will find it and publish it anywhere it needs to go.
To that end, shipper looks in several different places to mine the data it wants. You can specify some things that aren’t project specific, like the Web location of your personal website, in a “.shipper” file in your home directory. If your project has a Debian-style control file, or an RPM specification, it will look in those for things they normally carry, like a homepage location or project description. Finally the project can have its own “.shipper” file to specify other things shipper might need to know.
The third kind of knowledge that shipper has is embodied in code. It knows, for example, that if you specify “sourceforge” as a delivery destination, it needs to compose the name of the download directory to which your tarballs should be copied in a particular way that begins with frs.sourceforge.net and includes your project name. Because it would be silly for each and every one of your Makefiles to include that recipe; you might get it wrong the Nth time you repeat it, and what if sourceforge’s site structure changes?
There are some things shipper doesn’t try to know. Like, how to send release notifications to freecode.com; what it knows is how to call freecode-submit to do that. Actually, shipper doesn’t even know how to copy files across the network; instead, it knows how to generate scp and lftp commands given a source and destination.
I’ve been using versions of shipper on my own projects since 2002. It’s an important enabler of my ability to ship three or four or sometimes even more software releases within the span of a week. But here at Eric Conspiracy Secret Labs, we release no code before its time. And until very recently I was just not happy with shipper’s design.
It was getting the job done, but in a ugly way that required lots of option switches and dropping various kinds of intermediate files in the project directory while it was operating. But then I had a conceptual breakthrough.
Old shipper was complicated and ugly because it had two main modes of operation: one to show you what it was going to do, by listing the commands it would generate – then another to actually do them. The intermediate files it was leaving around during the process were text content for email and freecode.com announcements.
The breakthrough was this: Why not give up on executing commands entirely, and instead generate a shellscript to be piped to sh?
With that design, most of the options go away. If you want to see what shipper will do, you run it and look at the output. The contents of what used to be intermediate files are here-documents in the generated shellscript. The Makefile recipe for releasing shipper itself just looks like this:
VERS=$(shell sed <shipper -n -e '/^shipper_version *= *\(.*\)/s//\1/p') release: shipper-$(VERS).tar.gz shipper-$(VERS).md5 shipper.html shipper version=$(VERS) | sh -x -e
Here, the output of shipper is being piped to sh -e -x; the options make the first error in a generated command fatal and echo commands to standard output just before they’re performed.
Note the trick being played here: VERS, as set in the makefile and passed to shipper, is mined from where the version number is set in the shipper script itself. For a C project, it might make more sense to set the version in the Makefile and pass it into the C compilation with -DVERSION=$(VERS).
The point is, either way, there’s a single point of truth about the version number, and all the email and IRC and other announcements that shipper might generate will reflect it.
Here is shipper’s control file:
# This is not a real Debian control file, though the syntax is compatible. # It's project metadata for the shipper tool Package: shipper Description: Automated shipping of open-source project releases. shipper is a power distribution tool for developers with multiple projects who do frequent releases. It automates the tedious process of shipping a software release and (if desired) templating a project web page. It can deliver releases in correct form to SourceForge, Berlios, and Savannah, and knows how to post a release announcement to freecode.com via freecode-submit. XBS-Destinations: freecode, mailto:email@example.com Homepage: http://www.catb.org/~esr/shipper XBS-HTML-Target: index.html XBS-Gitorious-URL: https://gitorious.org/shipper XBS-IRC-Channel: irc://chat.freenode.net/#shipper XBS-Logo: shipper-logo.png XBS-Freecode-Tags: packaging, distribution XBS-VC-Tag-Template: %(version)s
By now you have enough information to guess what most of this is declaring. XBS-Destinations says that shipper should send a release notification to freecode.com and an email notification to me (as a smoke test).
The XBS-HTML-Target line tells it to template a simple web page and include it in the web deliverables; you can see the result here. XBS-Logo, if present, is used in generating that page. The template used to generate the [page is easily customized.
XBS-VC-Tag-Template tells shipper how to compose a tag to be pushed to the project repo to mark the release. This value simply substitutes in the release version. You might want a prefix, something like like “release-%(version)s”, on yours.
Here’s what the shipper-generated release script for shipper looks like:
cat >index.html < <'INAGADADAVIDA' < !DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'> [bulky stuff omitted here] INAGADADAVIDA scp -p COPYING login.ibiblio.org:/public/html/catb/esr/shipper/COPYING scp -p shipper-0.19.md5 login.ibiblio.org:/public/html/catb/esr/shipper/shipper-0.19.md5 scp -p NEWS login.ibiblio.org:/public/html/catb/esr/shipper/NEWS scp -p TODO login.ibiblio.org:/public/html/catb/esr/shipper/TODO scp -p shipper-0.19.tar.gz login.ibiblio.org:/public/html/catb/esr/shipper/shipper-0.19.tar.gz scp -p README login.ibiblio.org:/public/html/catb/esr/shipper/README scp -p index.html login.ibiblio.org:/public/html/catb/esr/shipper/index.html scp -p shipper-logo.png login.ibiblio.org:/public/html/catb/esr/shipper/shipper-logo.png git tag -a 0.19 -m 'Tagged for external release 0.19' git push; git push --tags freecode-submit < <'INAGADADAVIDA' Project: shipper Version: 0.19 Description: Automated shipping of open-source project releases. shipper is a power distribution tool for developers with multiple projects who do frequent releases. It automates the tedious process of shipping a software release and (if desired) templating a project web page. It can deliver releases in correct form to SourceForge, Berlios, and Savannah, and knows how to post a release announcement to freecode.com via freecode-submit. Project-Tag-List: packaging, distribution Website-URL: http://www.catb.org/~esr/shipper Checksum-URL: http://www.catb.org/~esr/shipper/shipper-0.19.md5 Tar/GZ-URL: http://www.catb.org/~esr/shipper/shipper-0.19.tar.gz Use irkerd's new (release 2.3) immediate mode for IRC notifications. INAGADADAVIDA sendmail firstname.lastname@example.org <<'INAGADADAVIDA' Subject: Announcing release 0.19 of shipper Release 0.19 of shipper is now available at: http://www.catb.org/~esr/shipper Here are the most recent changes: Use irkerd's new (release 2.3) immediate mode for IRC notifications. -- shipper, acting for Eric S. Raymond
INAGADADAVIDA irkerd -i 'irc://chat.freenode.net/#shipper' 'shipper-0.19 has just shipped.' # That's all, folks!
Yes, that last line sends an announcement to the #shipper channel on freenode. Notice how things like the Description section in the freecode.com submission form are copied direct from the control file.
It’s worth re-emphasizing that none of those commands were generated by hand – I’m spared the boring and glitch-prone process of typing them all. I just push the go-button and, boom, a complete and consistent release state gets pushed everywhere it needs to go. Look, ma, no hand-work!
And that’s the point. You set up your per-project metadata once and go. Only the things that must change each release need to be altered – and shipper knows how to extract the most recent changes from your NEWS file. Imagine how much mechanical ritual and distraction from more important things this has saved me since 2002!
At long last, I think shipper is ready for beta, for other people to try using it. I’d love it if people contributed shipping methods for other forges. The documentation needs a critique from someone who doesn’t know the tool intimately. There might be ways I’m not seeing to make the tool simpler and more effective – I’m unhappy that the -w option still exists. There’s still work to be done.
But it’s worth doing. This isn’t just about convenience either, though that matters. By reducing the friction cost of shipping, shipper encourages frequent incremental releases on short cycles. That, in turn, makes open-source development work better and faster, which is a good thing for all of us.