shipper is about to go 1.0 – reviewers requested

If you’re a regular at A&D or on my G+ feed, and even possibly if you aren’t, you’ll have noticed that I ship an awful lot of code. I do get questions about this; between GPSD, reposurgeon, giflib, doclifter, and bimpty-bump other projects it is reasonable that other hackers sometimes wonder how I do it.

Here’s part of my answer: be fanatical about automating away every part of your workflow that you can. Every second you don’t spend on mechanical routines is a second you get to use as creative time.

Soon, after an 11-year alpha period, I’m going to ship version 1.0 of one of my main automation tools. This thing would be my secret weapon if I had secrets. The story of how it came to be, and why it took 11 years to mature, should be interesting to other hackers on several different levels.

The background…

I’m the designer or maintainer of around 40 open-source projects. Even allowing for the fact that more than half of those are very stable old code that only needs a release once in a blue moon, the cumulative amount of boring fingerwork involved in keeping these updated is considerable.

When I say “boring fingerwork” I’m not even talking about coding effort, but rather the mundane tasks of uploading tarballs to archive locations, updating web pages, mailing out announcements, sending release notifications to Freecode, broadcasting heads-ups on relevant IRC channels, et cetera.

For older projects this shipping overhead is often more work than applying the small fixes and patches that trigger each release. It’s tedious, fiddly stuff – and irritatingly error-prone if done by hand.

A long time ago, now, I decided to stop doing it by hand. My overall goal was simple: I wanted to be able to type “make release” (or, more recently, “scons release”) in my project directory and have the right things happen, without fail. So I started building a tool to automate away as much tedium and fiddliness as I could. I called it “shipper”, because, well, that’s what it does.

Shipper’s job is to identify deliverables (like, say, tarballs and generated web pages) and push them to appropriate destinations (like, a public FTP directory or a website). It’s also intended to issue release notifications over various channels.

One of the things all these announcements and many of the names of deliverables will have in common is an embedded version number. One of the goals of shipper’s design is to allow you to specify the release’s version number in one place and one place only – because when you repeat a detail like that from memory you will occasionally get it wrong, with embarrassing results.

As for version numbers, so for other pieces of metadata that archive sites and forges and announcement channels commonly want – like a sort description of the project’s purpose, or a home page link, or the name of a project IRC channel. A design goal is that you only need to specify anything like this once per project; shipper will find it and publish it anywhere it needs to go.

To that end, shipper looks in several different places to mine the data it wants. You can specify some things that aren’t project specific, like the Web location of your personal website, in a “.shipper” file in your home directory. If your project has a Debian-style control file, or an RPM specification, it will look in those for things they normally carry, like a homepage location or project description. Finally the project can have its own “.shipper” file to specify other things shipper might need to know.

The third kind of knowledge that shipper has is embodied in code. It knows, for example, that if you specify “sourceforge” as a delivery destination, it needs to compose the name of the download directory to which your tarballs should be copied in a particular way that begins with frs.sourceforge.net and includes your project name. Because it would be silly for each and every one of your Makefiles to include that recipe; you might get it wrong the Nth time you repeat it, and what if sourceforge’s site structure changes?

There are some things shipper doesn’t try to know. Like, how to send release notifications to freecode.com; what it knows is how to call freecode-submit to do that. Actually, shipper doesn’t even know how to copy files across the network; instead, it knows how to generate scp and lftp commands given a source and destination.

I’ve been using versions of shipper on my own projects since 2002. It’s an important enabler of my ability to ship three or four or sometimes even more software releases within the span of a week. But here at Eric Conspiracy Secret Labs, we release no code before its time. And until very recently I was just not happy with shipper’s design.

It was getting the job done, but in a ugly way that required lots of option switches and dropping various kinds of intermediate files in the project directory while it was operating. But then I had a conceptual breakthrough.

Old shipper was complicated and ugly because it had two main modes of operation: one to show you what it was going to do, by listing the commands it would generate – then another to actually do them. The intermediate files it was leaving around during the process were text content for email and freecode.com announcements.

The breakthrough was this: Why not give up on executing commands entirely, and instead generate a shellscript to be piped to sh?

With that design, most of the options go away. If you want to see what shipper will do, you run it and look at the output. The contents of what used to be intermediate files are here-documents in the generated shellscript. The Makefile recipe for releasing shipper itself just looks like this:

VERS=$(shell sed <shipper -n -e '/^shipper_version *= *\(.*\)/s//\1/p')

release: shipper-$(VERS).tar.gz shipper-$(VERS).md5 shipper.html
	shipper version=$(VERS) | sh -x -e

Here, the output of shipper is being piped to sh -e -x; the options make the first error in a generated command fatal and echo commands to standard output just before they’re performed.

Note the trick being played here: VERS, as set in the makefile and passed to shipper, is mined from where the version number is set in the shipper script itself. For a C project, it might make more sense to set the version in the Makefile and pass it into the C compilation with -DVERSION=$(VERS).

The point is, either way, there’s a single point of truth about the version number, and all the email and IRC and other announcements that shipper might generate will reflect it.

Here is shipper’s control file:

# This is not a real Debian control file, though the syntax is compatible.
# It's project metadata for the shipper tool

Package: shipper

Description: Automated shipping of open-source project releases.
 shipper is a power distribution tool for developers with multiple
 projects who do frequent releases.  It automates the tedious process
 of shipping a software release and (if desired) templating a project
 web page. It can deliver releases in correct form to SourceForge,
 Berlios, and Savannah, and knows how to post a release announcement
 to freecode.com via freecode-submit.

XBS-Destinations: freecode, mailto:esr@thyrsus.com

Homepage: http://www.catb.org/~esr/shipper

XBS-HTML-Target: index.html

XBS-Gitorious-URL: https://gitorious.org/shipper

XBS-IRC-Channel: irc://chat.freenode.net/#shipper

XBS-Logo: shipper-logo.png

XBS-Freecode-Tags: packaging, distribution

XBS-VC-Tag-Template: %(version)s

By now you have enough information to guess what most of this is declaring. XBS-Destinations says that shipper should send a release notification to freecode.com and an email notification to me (as a smoke test).

The XBS-HTML-Target line tells it to template a simple web page and include it in the web deliverables; you can see the result here. XBS-Logo, if present, is used in generating that page. The template used to generate the [page is easily customized.

XBS-VC-Tag-Template tells shipper how to compose a tag to be pushed to the project repo to mark the release. This value simply substitutes in the release version. You might want a prefix, something like like “release-%(version)s”, on yours.

Here’s what the shipper-generated release script for shipper looks like:

cat >index.html < <'INAGADADAVIDA'

< !DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN'
    'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'>


[bulky stuff omitted here]



INAGADADAVIDA

scp -p COPYING login.ibiblio.org:/public/html/catb/esr/shipper/COPYING
scp -p shipper-0.19.md5 login.ibiblio.org:/public/html/catb/esr/shipper/shipper-0.19.md5
scp -p NEWS login.ibiblio.org:/public/html/catb/esr/shipper/NEWS
scp -p TODO login.ibiblio.org:/public/html/catb/esr/shipper/TODO
scp -p shipper-0.19.tar.gz login.ibiblio.org:/public/html/catb/esr/shipper/shipper-0.19.tar.gz
scp -p README login.ibiblio.org:/public/html/catb/esr/shipper/README
scp -p index.html login.ibiblio.org:/public/html/catb/esr/shipper/index.html
scp -p shipper-logo.png login.ibiblio.org:/public/html/catb/esr/shipper/shipper-logo.png
git tag -a 0.19 -m 'Tagged for external release 0.19'
git push; git push --tags
freecode-submit < <'INAGADADAVIDA'
Project: shipper
Version: 0.19
Description: Automated shipping of open-source project releases.
    shipper is a power distribution tool for developers with multiple
    projects who do frequent releases.  It automates the tedious process
    of shipping a software release and (if desired) templating a project
    web page. It can deliver releases in correct form to SourceForge,
    Berlios, and Savannah, and knows how to post a release announcement
    to freecode.com via freecode-submit.
Project-Tag-List: packaging, distribution
Website-URL: http://www.catb.org/~esr/shipper
Checksum-URL: http://www.catb.org/~esr/shipper/shipper-0.19.md5
Tar/GZ-URL: http://www.catb.org/~esr/shipper/shipper-0.19.tar.gz

Use irkerd's new (release 2.3) immediate mode for IRC notifications.
INAGADADAVIDA

sendmail esr@thyrsus.com <<'INAGADADAVIDA'
Subject: Announcing release 0.19 of shipper

Release 0.19 of shipper is now available at:

	http://www.catb.org/~esr/shipper

Here are the most recent changes:

  Use irkerd's new (release 2.3) immediate mode for IRC notifications.

--
                             shipper, acting for Eric S. Raymond 

INAGADADAVIDA

irkerd -i 'irc://chat.freenode.net/#shipper' 'shipper-0.19 has just shipped.'
# That's all, folks!

Yes, that last line sends an announcement to the #shipper channel on freenode. Notice how things like the Description section in the freecode.com submission form are copied direct from the control file.

It’s worth re-emphasizing that none of those commands were generated by hand – I’m spared the boring and glitch-prone process of typing them all. I just push the go-button and, boom, a complete and consistent release state gets pushed everywhere it needs to go. Look, ma, no hand-work!

And that’s the point. You set up your per-project metadata once and go. Only the things that must change each release need to be altered – and shipper knows how to extract the most recent changes from your NEWS file. Imagine how much mechanical ritual and distraction from more important things this has saved me since 2002!

At long last, I think shipper is ready for beta, for other people to try using it. I’d love it if people contributed shipping methods for other forges. The documentation needs a critique from someone who doesn’t know the tool intimately. There might be ways I’m not seeing to make the tool simpler and more effective – I’m unhappy that the -w option still exists. There’s still work to be done.

But it’s worth doing. This isn’t just about convenience either, though that matters. By reducing the friction cost of shipping, shipper encourages frequent incremental releases on short cycles. That, in turn, makes open-source development work better and faster, which is a good thing for all of us.

32 comments

  1. Okay, after reading that sample shell-script output, am I the only one that has an urge to go listen to a certain Iron Butterfly song?

    (What’s even sicker is, I have the song on my phone. The full 17+-minute version. Taken from the CD, which I also own. In fairness, my younger brother gave it to me…)

    1. >I wasn’t at all surprised to see that here-script end tag

      Deliberately misspelled, by the way. Wouldn’t do to have a false match on the content.

  2. Just thought of a way to guarantee no false matches, even for pathological text: find the longest single word in the output, and pad your heredoc terminator with “DADADADA” until it’s longer than that.

  3. Neat, I’m always looking for ways to automate work.

    Is it pluggable? Can I have it tweet? Post on Facebook? G+?

    Upload to PYPI?

    Upload the release that Perforce repo that the boss’s boss wants releases tracked in?

  4. I downloaded shipper 0.20. First impressions: not much there. I view the man page, and it’s rendering is odd. Very odd. I see in the makefile how it’s generated, so I regenerate it (had to delete shipper.1 first) and much better. Yes, there is a difference between the shipper.1 that comes with the tarball, and the shipper.1 that’s generated.

    Second—no example configuration files. I figure I could, if I were motivated enough, to construct a configuration file, but maybe just running the executable will generate one by default. Only it doesn’t run on my system. It fails on line 66. I have Python 2.3.4 and 2.6.2 (old, I know, but I don’t use Python so upgrading isn’t a priority; and no, I don’t needlessly upgrade my OS either—if it ain’t broke, don’t fix it). There’s no indication I can find as to which version of Python I need to run this.

    Third, there is no spoon.

  5. > There’s no indication I can find as to which version of Python I need to run this.

    # Requires Python 2.72 or later.
    line 5 of the shipper’ file

  6. >># Requires Python 2.72 or later.

    Ok, I’m going to have to ask. What little tweak did the python guys do that you depend on for that version?

    Jim

  7. >What little tweak did the python guys do that you depend on for that version?

    set literals would be my guess

    1. >set literals would be my guess

      That’s right. I could back compatibility down to 2.6 fairly easily I think.

  8. > # Requires Python 2.72 or later.
    > line 5 of the shipper’ file

    I didn’t think to look at the actual script—I was more interested in running it. I would think that it should at least check to see if its running the proper version of Python and reporting an error otherwise.

    1. >I would think that it should at least check to see if its running the proper version of Python and reporting an error otherwise.

      I’ve added such a check to the repository head version.

  9. @esr:
    I’m unsure as to why you chose to use a set for ‘multiline’ and ‘hardwired’ rather than a list or a tuple other than the very slight performance difference for the two ‘foo in bar’ checks. It’s not like you’re using any of the boolean set operations such as intersection(), union(), or difference(), or even issuperset() or issubset().

    Care to elaborate?

    1. >Care to elaborate?

      The collection has the semantics of a set. Code is not just instructions to a machine, it’s communication to other programmers. By using a set rather than a tuple, I tell the reader that the collection has set semantics.

  10. Actually my informal testing leads me to believe that the sets are at least an order of magnitude slower than a list or tuple would be, at least on Python 2.7.3 on Linux AMD64:


    >>> from timeit import timeit
    >>> timeit("2 in {1,1,2,3,3}")
    0.23831391334533691
    >>> timeit("2 in {1,1,2,3,3}")
    0.24747991561889648
    >>> timeit("2 in [1,1,2,3,3]")
    0.07680201530456543
    >>> timeit("2 in (1,1,2,3,3)")
    0.08085298538208008

  11. >Actually my informal testing leads me to believe that the sets are at least an order of magnitude slower than a list or tuple would be, at least on Python 2.7.3 on Linux AMD64:

    It’s probably because behind the scenes sets are doing hash-table lookups.

    Personally I would have used tuples in this case, in order to express and enforce the immutability of the sequence.

  12. I wrote some quick code to test this for myself:

    https://gist.github.com/thomas-scrace/7808863

    I was curious whether the length of the sequence would make a difference. Perhaps lists had superior performance for short sequences?

    However, as you will see if you run the code, sets actually seem to be far more efficient across the board.

  13. My results:

    10-long set took 0.0126779079437
    100-long set took 0.0721209049225
    1000-long set took 0.667522907257
    10-long list took 0.0193102359772
    100-long list took 1.023209095
    1000-long list took 98.0308930874
    10-long tuple took 0.0185379981995
    100-long tuple took 1.02909088135
    1000-long tuple took 97.4343318939

  14. @Tom:

    However, as you will see if you run the code, sets actually seem to be far more efficient across the board.

    This probably has to do with the necessity to move a list in memory in order to grow it. Python has some optimizations for this (that add extra space at the end for growing room in some cases), but obviously those didn’t help your testcases that much.

    Personally I would have used tuples in this case, in order to express and enforce the immutability of the sequence.

    Like esr, I tend to use sets for things where the order doesn’t really matter, and the mere fact of using a set is a reminder to me that order doesn’t really matter. However, I sometimes then find that the order _does_ matter, for no other reason than repeatability.

    For example, I often write scripts that report problems with schematic netlists. Within some potential problem categories, there is no natural order for the warnings, and I don’t really care what order they come out in. But I _do_ care about repeatability, so I can easily diff the output from two runs of my checker script. For that reason, even though it’s counterintuitive and might seem a bit ugly, I often use sets, and then use sorted() on them when I am performing actions that result in output.

  15. Forgot to mention that, usually the use of sorted() against my sets happens in a very lazy fashion.

    For some outputs, any errors are so egregious that when they happen I don’t care about the order, and make the schematic entry guy fix the schematic, stat. For other outputs, when I get tired of seeing things move around in the outputs, I will judiciously change “for x in badstuff:” to “for x in sorted(badstuff):” or “for x in sorted(badstuff, key=lambda x:x.bestkeytosorton):”

  16. A LOT of times I will write a program that generates shell commands, look at them, and then pipe them into the shell. Helps create confidence that the program will do the right thing before you set it loose.

Leave a comment

Your email address will not be published. Required fields are marked *