I released shipper 1.14 just now. It takes advantage of the conventional asciidoc extension – .adoc – that GitHub and GitLab have established, to do a useful little step if it can detect that your project README and NEWS files are asciidoc.
And I wondered, as I usually do when I cut a shipper release: am I really the only user this code has? My other small projects (things like SRC and irkerd) tend to attract user communities that stick with them, but I’ve never seen any sign of that with shipper – no bug reports or RFEs coming in over the transom.
This time, it occurred to me that if I am shipper’s only user, then maybe the typical work practices of the open-source community are rather different than I thought they were. That’s a question worth raising in public, so I’m posting it here to attract some comment.
The problem shipper solves for me is this: I run, by exact count that I checked just now, 52 small projects. By ‘small’ I really mean “not NTPsec or GPSD”; some, like reposurgeon, are relatively large as codebases. What really distinguishes GPSD and NTPsec is their need to have custom websites full of resource pages, blogs, and other impedimenta; a small project, on the other hand, can do with one page existing mainly to carry a description, recent project news, and a place to download the latest release tarball.
About a quarter of these projects are pretty dormant. On the other hand, among the dozen most active of them it’s not unusual for me to ship two or three point releases a week. Without tooling that would be a really annoying amount of hand-work – making distribution tarballs, copying them to my personal website or places like SourceForge that carry tarballs, editing web pages to publish the latest project news, emailing distribution packagers to tell them there’s a new release, posting announcements to IRC channels. Inevitably I’d make errors, for example by forgetting to email people who have asked to be notified when I ship Project X.
That’s the problem shipper solves. A shipperized project contains a control file that gathers all the metadata required to ship a release. Then its makefile (or equivalent) can have a “release” production that runs the project’s build and regression test to make sure it’s in a sane state, and if so proceeds to ship tarballs to their public download sites, template a web page with project news and documentation/download links and ship that to where it needs to live, and finally send out notifications on all required email and IRC channels.
Here’s what a typical control file looks like:
# This is not a real Debian control file, though the syntax is compatible. # It's project metadata for the shipper tool Package: shipper Description: Automated shipping of open-source project releases. shipper is a power distribution tool for developers with multiple projects who do frequent releases. It automates the tedious process of shipping a software release and (if desired) templating a project web page. It can deliver releases in correct form to SourceForge, Savannah. Homepage: http://www.catb.org/~esr/shipper XBS-IRC-Channel: irc://chat.freenode.net/shipper XBS-Repository-URL: https://gitlab.com/esr/shipper XBS-Logo: shipper-logo.png #XBS-Project-Tags: packaging, distribution XBS-Validate: make check
The shipper code figures out everything it needs to do from that or from looking at the files in your source tree – for example, it assumes that any file with an “.html” extension that exists in the top level of your project tree at the time it’s run should be shipped to the project website along with the templated index page. About all you need to pass it on each invocation is a release number.
From the user’s point of view, the good thing about a shipperized project is that the templated page puts the project blurb, links to up-to-date documentation, and project news in one place that’s easy to find – no need to go digging for them.
My shipperized projects also have a “refresh” production that just updates the project page. This is useful, for example, when I find a typo in the documentation or blurb and want to correct that without shipping an actual code release. Fix it, run “make refresh”, done.
The only real cost this tool imposes other than making the initial control file is that your NEWS file has to be in a particular format in order for it to extract the latest item: stanzas separated by blank lines, each with a recognizable date on the first line, newest at the top.
Over the years this little tool has saved me from immense amounts of process friction and substantially reduced the defect rate in what I actually ship. I can release more frequently, being more responsive to bugs and RFEs, because doing so isn’t much work. It’s easy to use, it’s well documented, it’s very customizable (you can set up your own web page template, etc). So…why isn’t anybody else using it? Why do I never get bug reports or RFEs the way I so frequently do on my other projects?
The possibility that occurs to me now is that maybe it solves a problem nobody else really has to the same extent. While shipper is useful for eliminating stupid finger errors while shipping even if you have just one project, it really comes into its own if you have a dozen. So maybe people aren’t doing that? Is being a lead on over a dozen projects more of an outlier than I thought?
The obvious alternative hypothesis is that repository-centered forges like GitHub and GitLab have eaten the role small-project pages used to have – your project page is now the README on its forge. I could almost believe that, except that (a) “stable release” is an important distinction for a lot of projects, those tarballs have to live somewhere, (b) forge READMEs also aren’t so good for advertising things like project mailing lists, and (c) because of a and b I see a lot of small-project web pages when I go browsing.
Another hypothesis is that there are lots of people who would find shipper useful, but they don’t know it exists. Also possible – but when I write other things that seem comparably useful, like SRC or irkerd or deheader, they don’t seem to have any trouble being found by their natural constituents. My project issue trackers tell the tale!
Is it really normal for hackers to be working on so few projects at a time, and releasing so infrequently, that they never experience release-process friction as a problem? Perhaps so – but I’m curious what my commenters think.
I had no idea it existed, but now my curiosity is piqued. Maybe it’s terrible? I’ll find out and let you know.
>Maybe it’s terrible?
Probably not, it’s been too useful over too many years. But I will be extremely interested in critique from someone who isn’t me.
I was going to use shipper for a project I’ve been working on for a while. Turned out not to be a great match because this project is closed source, but provided a good inspiration for writing my own shipper-like tool – which fits the project rather tightly, but that’s OK.
It’s possible others have just written a similar tool or similar tools? Also, I could see other setups, like GitHub Actions and equivalents, shrinking the use-case for shipper.
From my time as the tooling guy at company X the shipping system, like build and testing, evolves over time internally. And it happens in the between spaces when you’re sick of being Mr. Strategic Reuse and just need to get an archive on a network share someplace. If that eventually aggregates into something usefully reusable within
$groupBoundarythat also happens internally.Eventually, you build your own light saber.
Your explanation of shipper over the years seems to confirm that’s how it entered the world too.
@esr: Speaking personally, I didn’t know till now that Shipper existed. That wasn’t a great loss to me because I’m not doing the stuff Shipper is intended to make easier.
The problem shipper solves for me is this: I run, by exact count that I checked just now, 52 small projects.
You know, I don’t think I’ve ever seen a list of all the projects you maintain. Can you provide one, and possibly put it out somewhere as the canonical list of what you maintain with links to the projects themselves?
If that already exists and I simply missed it, a pointer would be appreciated.
>>Dennis
I thought it was this:
http://www.catb.org/esr/software.html
Not sure what a nosting site is, but anything related to CVS should have a stake driven through it’s heart and be buried in a DEEP DEEP grave at a crossroads at midnight. Preferable a “crossroads” of two shipping lanes.
>anything related to CVS should have a stake driven through it’s heart and be buried in a DEEP DEEP grave at a crossroads at midnight.
No argument from me. The use this was actually written for is copying CVS repos so cvs-fast-export can lift them to git, so it’s part of the solution to that problem.
Shouldn’t that read “hosting sites” not “nosting sites?”
>Shouldn’t that read “hosting sites” not “nosting sites?”
I’ve fixed it.
Shipper takes the sting out of updating those sites, so they’re no-sting sites.
From what I’ve read here, there’s not much advantage over what GitHub does, and neither I nor my users have to install specialist software (or hosting!) to use GitHub.
In my experience, there is no project so small that it doesn’t need a well-hosted bug tracker.
GitHub has a “download ZIP” button which isn’t a tarball but, meh, close enough.
Today, that sounds like redundant extra work. I’m considering moving all my old projects to GitHub so I can stop maintaining websites on unpaid time.
Redundant hosting could be useful if GitHub goes down one day–and in that event, the rise of a compatible replacement is more or less inevitable. Note that several GitHub replacements exist already, and you can substitute some of them here–the point is that for a small project you don’t really need to be published on more than one host.
The common practice for projects with a history on a non-GitHub site seems to be to post a “this site is old, you should go to GitHub” notice on the old site. A few projects work the other way, e.g. there are a few read-only mirrors of projects on Google git repos on GitHub.
Part of the same git push that updates the curated docs.
They sign up for GitHub accounts and get email notifications for projects they follow.
IRC can reach a few thousand users at most, and it imposes nasty client uptime and connectivity requirements that modern users don’t bother with. You and I and a few other people over 30 run IRC proxies to work around this–we are exceptional now.
Also, you didn’t mention posting updates to the project’s Facebook page and Twitter. Apparently those are things. AFAIK they happen to be things GitHub does not do, though I’m not certain of that because GitHub does a lot.
GitHub solves that problem by forcing all of those to be in the same place (or a mechanical transformation or replication thereof, e.g. github.com/user/project -> user.github.io/project).
In my experience, no mere software can eliminate stupid finger errors while shipping. I work with teams of people whose job is exactly that, and stuff still slips through. Running stuff before pushing (e.g. HTML validators to make sure all your doc links work) can be useful, but ordinary build systems can do that, and so can human QA (especially if the project is small). I don’t see where Shipper wins here.
There are several other advantages and disadvantages of (not) using GitHub, of course, but in the specific, limited case of the small projects that Shipper targets, I don’t know of any that really matter.
>In my experience, there is no project so small that it doesn’t need a well-hosted bug tracker.
True. These projects have bugtrackers, on GitLab because it’s what I use. Each generated web pages already has a link to the project’s GitLab main page; a direct link to the bugtracker should be easy to generate.
>You and I and a few other people over 30 run IRC proxies to work around this–we are exceptional now.
Oh, not at all. IRC is still heavily used in the open-source world, and I don’t see that going away any time soon.
>Also, you didn’t mention posting updates to the project’s Facebook page and Twitter.
I looked into how to do this, but couldn’t figure out how to enable shipper to tweet notifications. Not interested in Facebook.
>In my experience, no mere software can eliminate stupid finger errors while shipping.
In principle, no. Mine comes pretty close to it in practice, though. Simply not having to type URLs gets rid of a major venue of errors.
>There are several other advantages and disadvantages of (not) using GitHub, of course, but in the specific, limited case of the small projects that Shipper targets, I don’t know of any that really matter.
One difference I’m seeing is that GitLab doesn’t give me a place to drop release tarballs for download. So as long as my stuff is hosted there something like shipper will still be useful.
> I looked into how to do this, but couldn’t figure out how to enable shipper to tweet notifications.
I see you have it working!
>I see you have it working!
Yeah, I got annoyed and took another whack at it. Polishing the capability now.
> These projects have bugtrackers, on GitLab because it’s what I use. Each generated web pages already has a link to the project’s GitLab main page…
…so why Shipper? After all, the project is already on Gitlab. A lot of potential shipper users will stop looking for tools once they’ve reached Gitlab. What’s the Shipper advantage for a user who has 50 Gitlab projects?
> One difference I’m seeing is that GitLab doesn’t give me a place to drop release tarballs for download.
Does it not have a ‘generate-a-tarball-from-this-tag’ button either? Even gitweb does that.
Are your tarballs different from ‘export master in tar format with GPG signature on the git tag?’ Mine used to be, then I fixed that.
> Not interested in Facebook.
Some open-source projects use a Facebook page as their primary support contact. Well, OK, electricsheep does. To be fair, electricsheep seems to be popular among people who host raves for a living, so Facebook is probably an unusually good fit for that project.
> Simply not having to type URLs gets rid of a major venue of errors.
On GitHub (and hopefully everywhere else) you only have to type “git remote add github [click to paste URL from web browser]” once. Given a familiar publishing platform, users can take what they need from just the main project page URL.
> IRC is still heavily used in the open-source world
…by people under 30? I worry about the demographic trends.
>What’s the Shipper advantage for a user who has 50 Gitlab projects?
As I keep saying, a place to drop tarballs. GitLab doesn’t have the generate-a-tarball button you keep talking about.
Also, I like automating the shipping of release notifications – mail, IRC, and since yesterday Twitter.
>…by people under 30? I worry about the demographic trends.
It depends on subculture. The under-30s more towards the systems end of things cheerfully use IRC. The gamer types have, it’s true, mostly wandered off to Discord.
For me it tends to be a per-project decision, that does indeed follow those rough lines. For instance, the KSP modding community (or at least the bit of it grown-up enough for me to care about) recently decamped from IRC to Discord (though I was not the only member saddened by this).
But IRC remains fruitful; my latest project, passacaglia, was inspired by an argument on — of all things — a #SpaceX channel.
> GitLab doesn’t have the generate-a-tarball button
…which makes GitLab an outlier among git-based publishing sites, and may explain the difference between the value you put on the feature and the value a potential Shipper user might put on the feature.
> Also, I like automating the shipping of release notifications – mail, IRC, and since yesterday Twitter.
This is something that specialist automation tools for social media publishers can do. Some of them will also talk to Facebook. OK, really, Facebook is an absolute requirement to be attractive to social media publishers at all, but a few of the tools can also talk to IRC.
… and then realized Discord censors opinions they find inappropriate, so the next thing is Riot.im based on Matrix. The idea is simply to have something as much decentralized as possible, users owning their servers, so they become effectively uncensorable. The Matrix project is busy building bridges to everything from IRC to Discord so effectively people can transition without losing anything.
Technological solutions against censorship are important in an era of Big Tech. And no, it is not just about far-right “hate speech”. For example, nudity is often a reason for flagging content to be inappropriate on Facebook, so there go skyclad Wiccans and Swedish women sunbathing topless. Needless to say the proper solution would be having to set a checkmark somewhere “I don’t want nudity” or a parent doing this for a minor. This is only solvable on a distributed basis.
The big question is how deeply one has to rebuild everything for that? You cannot just have a new chat protocol, that is still vulnerable. OTOH Yarvin’s answer, to rebuild absolutely everything down to the OS level, may be too radical. Riot.im and Matrix seem to be going for a compromise between these.
It appears that so long as your release tarball is “contents of repository at a given tag”, GitLab can now do releases, per their documentation. You have to use the API currently to create them, though (there is no method yet in the web interface).
> One difference I’m seeing is that GitLab doesn’t give me a place to drop release tarballs for download.
GitLab Pages is that. Open Adventure uses it!
>GitLab Pages is that.
Aha. This is new – I’ve been wondering when GitLab was going to deploy something like that.
>Open Adventure uses it!
What gave you that idea? I didn’t set it up, and there’s no “public” directory in the source tree.
> What gave you that idea?
The code coverage reports are hosted in Pages, and that’s been in place since June 2, 2017, going by the git history. Pages isn’t new. The “public” directory is created on the fly in the CI pipeline.
There may be an answer here about shipper’s apparent non-use. Things like GitLab CI and Pages work in a similar space, but they don’t quite seem to be on your radar scope. Perhaps other open-source projects are making bigger use of these things than you realize?
>One difference I’m seeing is that GitLab doesn’t give me a place to drop release tarballs for download.
This took a little bit of digging to find, and it turns out it’s only accessible through an API, not a web interface, but there’s this:
https://gitlab.com/help/user/project/releases/index
Give it a commit, tag, or branch and some other information and it’ll create archives that are available for download. There’s also a provision for adding binary installer packages and such as downloadables.
There’s also the download link on every page that shows a list of files that’ll create tarballs on the fly for any commit, tag, or branch.
I maintain several open source projects (small to medium). In the past I had a collection of tools that I used to do “shipping”. However, I haven’t used those in years. My release process basically looks like this (for projects where “shipping” is something I care about):
– Commit messages serve as release notes/changelog. This implies rebasing commits into logical units before pushing and a certain level of quality to the messages (and requiring the same for contributors). The incentives/requirements for commits vs change notes aren’t divergent enough in any projects I maintain for it to be worth while having a separate process.
– Automatic Travis tests with status icon at the top of the README.
– Tag to create a release. In github, at least, tagging creates downloadable sources. If it’s a built project (and builds are worth publishing) these can be uploaded in the github tag web interface as part of the tag/release.
– If separate web site is useful, use gh-pages mechanism (pointing to master unless there is a legit reason for it to live on a separate branch). For “shipping” information the front README is typically sufficient.
In terms of your forge deficiency list: a) github allows your to mark release as stable or not when you tag them. b) For github hosted projects, my expectation (and I’m certain I’m not alone) is that the README is where authors will advertise things like mailinglists, IRC channels, license, authors, etc. For this type of “standardized” marketing material, having a common format (markdown) and an advantage to bespoke web pages for that type of info. For more traditional marketing and demos, etc, a bespoke web page might be needed (but for most projects it’s probably not needed at least in the beginning).
Hmm, no visible way to edit posts/typos. “and an advantage” -> “is an advantage”
This is very much not something to wish for, but if as we are anticipating more and more of us get deplatformed from the “regular” internet, a tool like this will make it easier to cope.
Echoing Zygo, web bug tracking seems to be the big feature it’s missing, and unlike code managed by git, it’s much more sticky when it’s in GitHub, or GitLab, etc. Not sure about good solutions to that, with people understandably not wanting to run their own servers, and the transition issues when those servers get abandoned for one reason or another, including the death of the author. For example, Henry Baker’s site with all sorts of LISP and GC goodness is offline right now, and that’s supposed to happen fairly often.
From someone who spends most of his time on the JVM, Maven already takes care of most of the tasks you mention here; the only one missing is the notifications, and I’d be surprised if there’s not a plugin you can add to do it.
As a datapoint, most of my projects are python modules, so ‘releasing’ means ‘tag, build, push to pypi’, for which I have a (not very complex) Makefile target in my top-level Makefile. Everything else is handled by initial setup of the repo on github and the project on pypi.
I took a quick look at the project website and there’s one very obvious adoption hurdle: no link to any HTML documentation. There’s Docbook XML in the project repo, but it’s not obvious from the name “shipper.xml” that that’s what it is, and if you want to read it in rendered form then you have to download and build the project. That’s too much friction for somebody who doesn’t already know that the tool is going to be useful to them.
>there’s one very obvious adoption hurdle: no link to any HTML documentation.
Ugh. That, as it turns out, is a bug I introduced earlier today. Normally there would be a link to a manual page there, as for example there is on the reposurgeon page.
I was going to do another shipper release tomorrow anyway, because it can tweet notifications now if you consider Twitter app credentials in. I’ll make sure this gets fixed
Personally, when it comes to tools like “shipper”, everyone’s use case is slightly different. I have built my own custom tool that does a similar job. In my case, it’s called “package”, and creates a tarball, updates the webpage (which, because I use a very custom ‘CMS’ is a little tricky), etc.
src is useful though, and I use that.
That’s what I was thinking too. Everyone has a custom requirement or two (or three or four or five) and setting up a program to deal with those custom requirements is part of the overhead and isn’t terribly hard. I’m not sure it’s possible to automate that for someone else.
Thinking a little more about the issue, you probably could write a program which everyone would/could use, but it would have to be really huge to cover all the use-cases, almost to the point of creating a programming language to handle all possible shipping needs, like the fact that Joe finds it really useful to post the Masai-language binaries for his herd-management software on a dial-up BBS in Nairobi, and the shipping announcements are printed in Swahili and Maa in the Masai-Mara Wednesday circular… or something equally weird, and most people would still roll their own.
My own experience in this is pretty limited, but at one point I was going to set up a fairly complicated website with a back-end database,* and every once in awhile I’d blow away the data I’d collected in testing and roll all my tables back to zero for one reason or another… so I wrote a bunch of really simple programs with names like “create_address_table” which would rewrite all the database tables from scratch because it was far easier than me fumbling around with table-names and types and creating lots of bugs with my big clumsy fingers.
My point is that for any serious project a coder will write some kind of supporting infrastructure (I’m guessing Shipper started that way, at least in concept) that does easily-automated back-end stuff like recreating a database, and this finally turns into some kind of shipping/installing/build-automating program that does most of what Shipper would do for a particular person’s use-case. The problem is that by the time a coder is ready to actually think about the issues of shipping, they’ve already written most of the code that supports their use-cases, and it’s probably easier to integrate all that into their own “shipping script” than learn to use Shipper.
So I’m not sure you should spend time on this particular issue except as it affects your personal efforts. Where someone might really shine, however, is to think about the common back-end processes (like rewriting all the database tables when things start getting crufty) and find a way to automate that stuff, with the idea that someone can start a project with something like a fully-mature build environment.
Better yet, assign such a project to your apprentice! (You probably don’t have anything to learn from this project.) What I’m imagining is a window which would fit around VI or EMACS, (or an IDE) and allow a one-click commands to “configure, make, make install,” “rebuild the database tables,” or “ship my amazing creation,” plus whatever else would be useful (for a programmer who works daily.)
Does this make any sense? I’m probably out of my depth here, but I think I’m on the right track.
* I’m one of those people who “codes to scratch an itch” and doesn’t code the rest of the time.
This is part of why I still use Emacs to develop in — despite jeers and funny looks from coworkers who either remember last using it from their days working on DEC iron or wonder what kind of cranky old man I must be to use something written before they were born[1]. Every shop has its own fiddly special snowflake of a process, and it’s just much easier to wire the automatable bits into Emacs than it is a modern IDE or one of the pseudo-IDEs currently in vogue like Visual Studio Code.
[1] Vim gets a pass because Vim is old but cool whereas Emacs is just old. Cf. vinyl vs. 8 tracks.
At least in my case that alternative hypothesis is correct. I used to have project pages on my website, but gradually found that they really aren’t worth the trouble. “Stable release” is only really an important concept when you have enterprise or non-technical end users; the rest of the time, a git tag (and maybe a build tarball uploaded to Github) is good enough*. Project mailing lists aren’t really needed either with a forge-centric workflow; on a project that gets a handful of patches per year, they’re pointless overhead. (At the other end of the scale, of course, trying to run something like the kernel on Github Issues would be an hilarious failure.)
Are the small-project web pages you see examples of young, live projects, or are they mostly dead relics and projects that have been around since SourceForge was cutting edge?
One aspect of release manglement that I’d expect shipper can’t help with, btw, is posting to forums. Quite a few of my projects over the years have been adjuncts to something else (e.g. mods or tools for various games) and, for that or other reasons, the obvious place to announce releases has been on someone else’s web forum. Those are usually hideous piles of PHP with no thought given to automated interaction.
* OTOH, this might be part of why none of my projects have any users ;)
>Are the small-project web pages you see examples of young, live projects, or are they mostly dead relics and projects that have been around since SourceForge was cutting edge?
Both. Because of loccount I’ve looked at the development websites for over 100 computer languages recently; of that sample maybe two or three were bare repository pages. It doesn’t seem to be predictive how new or old the language is.
Off topic, but I see that David Brin, whom you called on his lies about Heinlein, has repeated them and completely ignored your critique. What a dishonest piece of shit.
http://davidbrin.blogspot.com/2015/01/robert-heinlein-and-looking-beyond-this.html
That’s four years after you picked the exact same lies apart.
Sigh. Note the date in the URL you provided. I suspect that’s the same post Eric answered before.
ESR’s post “Brin on Heinlein on guns is dead wrong” is dated 2011-7-12. It’s in response to a Brin post on a now-deleted blog. The date of Brin’s post linked to by Ken is Friday, January 16, 2015
I’ll read that. Thanks.
Read it. Looks like Ken is right. My point above that 2019 – 2015 = 4 is obviously not the end of the story.
Perl’s Dist::Zilla works well for preparing shipments, especially since its plugin architecture gives you great configurability. But, you have to work in Perl…
(I do PHP and Perl these days with occasional Java, hosted on GitHub and private GitHub.)
The obvious alternative hypothesis is that repository-centered forges like GitHub and GitLab have eaten the role small-project pages used to have – your project page is now the README on its forge.
This describes all projects that I do releases for (the only thing I would add here is that BitBucket is another site in the Github/GitLab category–but probably gets less usage now since it’s based on hg and git seems to be gaining traction at the expense of other DVCSs).
I could almost believe that, except that (a) “stable release” is an important distinction for a lot of projects, those tarballs have to live somewhere
All of the projects I do releases for are Python libraries, whose “stable release” location is PyPI, so my “release” process is just “upload the latest release to PyPI”. I pop a link to the PyPI project page in the READMEs that are visible on GitLab, and that’s it.
Like other commenters, I support this alternative hypothesis. GitHub Pages replaces a project website in the cases where the README is not enough.
I want to point out a GitHub feature that I haven’t seen anyone mention: “releases”. GitHub releases cover the need “(a)” you described. Not only can you specify a Git tag to be listed as a release, you can also write a title and description (which can include a changelog), and you can manually add binaries to those releases. See an example releases page or GitHub Help documentation on releases for more details.
You would think that a tool that allows one person to ship and release 50+ projects would be invaluable in an enterprise setting yet nobody I know in DevOps or release engineering uses shipper. Tools of choice are things like GitHub, Jenkins, Artifactory, Travis CI, etc. with bespoke scripts or Makefiles. None of these automate announcing releases within a company, which is a pain point nobody talks about even in a microservice heavy world. It may be that not many people maintain 50+ projects and the ones that do, enterprises, roll their own solutions that are targeted to them.