Dec 28

Announcing cvs-fast-export 1.0

Not long ago I pulled the plug on one of the two CVS export utilities I was maintaining. One consequence of this is that I decided I needed to get the other one out of beta and into a state I would be willing to ship as 1.0.

And lo, it has come to pass. I just shipped cvs-fast-export 1.0. It has been well field-tested; a couple of weeks ago I used it to rescue the history of Gnu Troff.

There are several CVS exporters out there that suck pretty badly. (To be fair, the perversity of CVS is such that doing an even half-decent job of lifting CVS histories into a modern version-control system is quite difficult.) Now that this one is shipped I know of exactly two that don’t suck. The other one is Michael Haggerty’s cvs2git, which I’m working with him on improving.

Tradeoffs: cvs2git is slow and a bit clunky to use (I’m improving the latter but can’t fix the former). cvs-fast-export is blazingly fast (like, 3.7K commits a minute) but has a hard repository-size limit – above it you run out of core and the OS reaps the process in mid-flight. (Very few projects will hit this limit.)

For each tool there are weird CVS edge cases that it gets wrong. The sets of edge cases are different. cvs2git’s may be smaller, but I’m not sure of that; we haven’t set up head-to-head testing yet. Most projects will not trip over either set of problems.

cvs-fast-export is better documented, especially around error conditions.

Help stamp out CVS in our lifetime!

Dec 20

Your new word of the week: explorify?

There are a lot of things people writing software do in the world of bits that don’t have easy analogs in the world of atoms. Sometimes it can be tremendously clarifying when one of those things gets a name, as for example when Martin Fowler invented the term “refactoring” to describe modifying a codebase with the intent to improve its structure or aesthetics without changing its behavior.

There’s a related thing we do a lot when trying to wrap our heads around large, complicated codebases. Often the most fruitful way to explore code to modify it. Because you don’t really know you have understood a piece of code until you can modify it successfully.

Sometimes – often – this can feel like launching an expedition into the untamed jungle of code, from some base camp on the periphery deeper and deeper into trackless wilderness. It is certainly possible to lose your bearings. And large, old codebases can be very jungly, overgrown and organic – full of half-planned and semi-random modifications, dotted with occasional clearings where the light gets in and things locally make sense.

Continue reading

Dec 11

How to demolish your software project with style

I did something unusual today. I pulled the plug on one of my own projects.

In Solving the CVS-lifting problem and Announcing cvs-fast-export I described how I accidentally ended up maintaining two different CVS-to-something-else exporters.

I finally got enough round tuits to put together two-thirds of the head-to-head comparison I’ve been meaning to do – that is, compare the import-stream output of cvs-fast-export to that of cvsps to see how they rate against each other. I wrote both git-stream output stages, so this was really a comparison of the analysis engines.

I wasn’t surprised which program did a better job; I’ve read and modified both pieces of code, after all. Keith Packard’s analysis engine, in cvs-fast-export, is noticeably more elegant and craftsmanlike than the equivalent in cvsps. (Well, duh. Yeah, that Keith Packard, the co-architect of X.)

What did surprise me was the magnitude of the quality difference once I could actually compare them head-to-head. Bletch. Turns out it’s not a case of a good job versus mildly flaky, but of good job versus suckage.

The comparison, and what I discovered when I tried to patch cvsps to behave less badly, was so damning that I did something I don’t remember ever having felt the need to do before. I shot one of my own projects through the head.

Continue reading

Dec 05

Heads up: the reposturgeon is mutating!

A few days ago I released reposurgeon 2.43. Since then I’ve been finishing up yet another conversion of an ancient repository – groff, this time, from CVS to git at the maintainer’s request. In the process, some ugly features and irregularities in the reposurgeon command language annoyed me enough that I began fixing them.

This, then, is a reposurgeon 3.0 release warning. If you’ve been using 2.43 or earlier versions, be aware that there are already significant non-backwards-compatible changes to the language in the repository head version and may be more before I ship. Explanation follows, embedded in more general thoughts about the art of language design.

Continue reading

Dec 02

shipper is about to go 1.0 – reviewers requested

If you’re a regular at A&D or on my G+ feed, and even possibly if you aren’t, you’ll have noticed that I ship an awful lot of code. I do get questions about this; between GPSD, reposurgeon, giflib, doclifter, and bimpty-bump other projects it is reasonable that other hackers sometimes wonder how I do it.

Here’s part of my answer: be fanatical about automating away every part of your workflow that you can. Every second you don’t spend on mechanical routines is a second you get to use as creative time.

Soon, after an 11-year alpha period, I’m going to ship version 1.0 of one of my main automation tools. This thing would be my secret weapon if I had secrets. The story of how it came to be, and why it took 11 years to mature, should be interesting to other hackers on several different levels.

Continue reading

Nov 05

Finally, one-line endianness detection in the C preprocessor

In 30 years of C programming, I thought I’d seen everything. Well, every bizarre trick you could pull with the C preprocessor, anyway. I was wrong. Contemplate this:

#include <stdint .h>

#define IS_BIG_ENDIAN (*(uint16_t *)"\0\xff" < 0x100)

That is magnificently awful. Or awfully magnificent, I'm not sure which. And it pulls off a combination of qualities I've never seen before:

Continue reading

Oct 29

Your money or your spec

reposurgeon has been stable for several months now, since the Subversion dump analyzer got to the point where people stopped appearing in my mailbox with the Pathological Subversion Repository Fuckup Of The Week.

Still, every once in a longer while somebody will materialize telling me they have some situation in a repo conversion that they want me to help them fix. The general form of these requests is like this. “I have {detailed description of a branch/merge topology nightmare that makes Eric’s brain hurt to contemplate}. What do I do to fix it?”

I am now going to announce a policy about this. There are exactly two ways ways you can get me to solve your repository problem.

Continue reading

Sep 25

coverity-submit 1.10 is released

coverity-submit automates the process of running the Coverity static checker’s front-end tools and shipping the results to their public server for analysis.

One bug fix, two minor features. The build-version (-b) and description (-t) options now have sensible defaults. When run from a repository, the default for -b is the commit ID of the head revision. The default for -t is an ISO8601 release timestamp.

Actually, the build-version default presently only works in a git repo. I’ll cheerfully take patches that support other version-control systems.

Code here.

Aug 23

vms-empire 1.10 released

There’s a genre of computer games called 4X (explore/expand/exploit/exterminate). well-known examples of which include the Civilization series and Master of Orion.

Ever wonder what the ur-progenitor of this genre was, the game at the root of 4X in the way Colossal Cave Adventure created the genre of dungeon-crawl games? It was Walter Bright’s game “Empire” from the early 1970s. You can read about it at his page on Classic Empire.

Since 1994 I’ve maintained an early Empire workalike written by Chuck Simmons in 1987 to run under the now-extinct VMS operating system; it was ported to Unix immediately, and remains to my best of knowledge the only open-source version or variant of Empire available.

Walter Bright does not acknowledge this version’s existence on his Empire page, which is fair because he didn’t write it and probably doesn’t consider it to be “Empire” at all. But it is close in gameplay and style to the earliest of Bright’s versions, except for being able to display its crude character-cell maps in color (I added that back when color terminals were cutting-edge technology).

If you love Civ or MOO, try this out for a look at what the computer 4X game was like before pixel graphics. The display and command interface are primitive by today’s standards, but the AI and general gameplay have held up surprisingly well. It’s instructive to see how many of the core tropes of later 4X games are already present in this one.

You can get version 1.10 of VMS-Empire here.

Apr 16

Building a better IRC client

I’ve been thinking about how to build a better IRC client recently.

The proximate cause is that I switched to irssi from chatzilla recently. In most ways it’s better, but it has some annoying UI quirks. Thinking they’d be easy to fix, I dug into the codebase and discovered that it’s a nasty hairball. We’ve seen projects where a great deal of enthusiasm and overengineering resulted in code that is full of locally-clever bits but excessively difficult to modify or maintain as a whole; irssi is one of those. Even its maintainers have mostly abandoned it; there hasn’t neen an actual release since 2010.

This is a shame, because despite its quirks it’s probably the best client going for serious IRC users. I say this because I’ve tried the other major contenders (chatzilla, BitchX, XChat, ircii) in the past. None of them really match irsii’s feature set, which makes it particularly unfortunate that the codebase resembles a rubble pile.

I’m nor capable of stumbling over a situation like this without thinking about how to fix it. And yesterday…I had an insight.

Continue reading

Apr 07

Out on the tiles

I’ve been experimenting with tiling window managers recently. I tried out awesome and xmonad, and read documentation on several others including dwm and wmii. The prompt cause is that I’ve been doing a lot of surgery on large repositories recently, and when you get up to 50K commits that’s enough to create serious memory pressure on my 4G of core (don’t laugh, I tend to drive my old hardware until the bolts fall out). A smaller, lighter window manager can actually make a difference in performance.

More generally, I think the people advocating these have some good UI arguments – OK, maybe only when addressing hard-core hackers, but hey we’re users too. Ditching the overhead of frobbing window sizes and decorations in favor of getting actual work done is a kind of austerity I can get behind. My normal work layout consisted of just three big windows that nearly filled the screen anyway – terminal, Emacs and browser. Why not cut out the surrounding cruft?

I wasn’t able to settle on a tiling wm that really satisfied, though, until my friend HedgeMage pointed me at i3. After a day or so of using it I suspect I’ll be sticking with it. The differences from other tiling wms are not major but it seems just enough better designed and documented to cross a threshold for me, from interesting novelty to useful tool. Along with this change I’m ditching Chatzilla for irsii; my biggest configuration challenge in the new setup, actually, was teaching irssi how to use libnotify so I get visible IRC activity cues even when irsii itself is hidden.

Continue reading

Mar 24

Python speed optimization in the real world

I shipped reposurgeon 2.29 a few minutes ago. The main improvement in this version is speed – it now reads in and analyzes Subversion repositories at a clip of more than 11,000 commits per minute. This, is, in case you are in any doubt, ridiculously fast – faster than the native Subversion tools do it, and for certain far faster than any of the rival conversion utilities can manage. It’s well over an order of magnitude faster than when I began seriously tuning for speed three weeks ago. I’ve learned some interesting lessons along the way.

Continue reading

Feb 26

Mode of the Reposturgeon!

It was inevitable, I suppose; reposurgeon now has its own Emacs mode.

The most laborious task in the reposurgeon conversion of a large CVS or Subversion repository is editing the comment history. You want to do this for two reasons: (1) to massage multiline comments into the summary-line + continuation form that plays well with git log and gitk, and (2) lifting Subversion and CVS commit references from, e.g., ‘2345’ to [[SVN:2345]] so reposurgeon can recognize them unambiguously and turn them into action stamps.

In the new release 2.22, there’s a small Emacs mode with several functions that help semi-automate this process.

Fear the reposturgeon!