Announcing cvssync, with thoughts on “good enough”

There’s an ancient Unix maxim to the effect that a tool that gets 85% of your job done now is preferable to one that gets 100% done never. Sometimes chasing corner cases is more work than the problem really justifies.

In today’s dharma lesson, I shall illustrate this principle with a real-world and useful example.

In my last blog post I explained why I had to shoot cvsps through the head. Some of my regulars regretted the loss of the good feature bolted to its crappy repo-analysis code – it could fetch remote CVS repository metadata for analysis rather than requiring them to have been already mirrored locally.

To fill this functional gap, I needed a tool for mirroring the contents of a remote CVS repository to a local directory. There’s floating folklore to the effect that a tool called “cvssuck” does this job, but when I tried to use it it failed in about the most annoying possible way. It mirrored the directory structure of the remote site without fetching any masters!

Upon investigation I discovered that the cvssuck project site has disappeared and there hasn’t been a release in years. Disgusted, I asked myself how it could possibly have become that broken. Seemed to me the whole thing ought to be a trivial wrapper around rsync.

Or…maybe not. What scanty documentation I found for cvssuck made a big deal out of the fact that it (inefficiently) used CVS itself to fetch masters. This doesn’t make any sense if they were rsync accessible. because then it would be a much faster and more efficient way to do the same job.

But I thought about the sites I generally have to fetch from when I’m grabbing CVS repositories for conversion, as I did most recently for the groff project. SourceForge. Savannah. These sites (and, I suspect, most others that still support CVS) do in fact allow rsync access so that project administrators can use it to do offsite backups.

OK, so suppose I write a little wrapper around rsync to fetch from these sites. It might not do the guaranteed fetch that cvssuck advertises…but on the other hand cvssuck does not seem to actually work, at least not any more. What have I got to lose?

About an hour of experimentation and 78 lines of Python code later, I had learned a few things. First, a stupid-simple wrapper around rsync does in fact work for SourceForge and Savannah. And second, there is a small but significant value the wrapper can add.

The only thing you are pretty much guaranteed to be able to find out about a CVS repository is the CVS command needed to check out a working copy. For example, the groff CVS page gives you this command:

cvs -z3 co <modulename>

You have to figure out for yourself that the <modulename> should also be “groff”, but there are clues to that on the web page. For those of you blessed enough to be unfamiliar with CVS, a single instance can host multiple projects that can be checked out separately; the module name selects one of these.

It isn’t necessarily clear how to get from that cvs invocation to an rsync command. Here’s how you do it. First, lop off the “anonymous@” part; that is a dummy log credential. Treat “/sources/groff” as a file path to the repository directory, then realize that the module is a subdirectory. You and up writing this:

rsync -avz my-local-directory

That’s really simple, but it turns out not to work on SourceForge. Because SourceForge runs an rsync daemon and hides the absolute file path to the repository. The corresponding fetch from SourceForge, if groff existed there, would look like this:

rsync -avz groff

Note the double colon and absence of leading ‘/’ on the repository path.

The value a wrapper script can add is knowing about these details so you don’t have to. Thus, cvssync. You call it with the arguments you would give a CVS checkout command. It pulls those apart, looks at the hostname, figures out how to reassemble the elements into an rsync command, and runs that.

This just shipped with cvs-fast-export release 0.7. At the moment it really only knows two things: A special rule about building rsync commands for SourceForge, and a general rule that happens to work for Savannah and should for most other CVS sites a well. More hosting-site would be easy to add, a line or two at most of Python for each hosting side.

This wrapper doesn’t do the last 15% of the job; it will fail if the CVS host blocks rsync or has an unusual directory structure. But that 85% now is more valuable than 100% never, especially when its capabilities are so easily extended.

And hey, it only took an hour for me to write, test, document, and integrate into the cvs-fast-export distribution. This is the Great Way of Unix; heed the lesson.

26 thoughts on “Announcing cvssync, with thoughts on “good enough”

  1. Also known as Perfect is the Enemy of Good … or … Worse is Better.

  2. Which 85% is important, however. Let’s say I’m writing a network interface driver module for the Linux kernel. If my hypothetical driver works with 85% of the NICs 100% of the time, great! Ship it! But what if it works with 100% NICs its supposed to only 85% of the time? That’s much worse.

  3. It’s called “cvssuck” …I don’t think this should have been very surprising.

  4. Terry, grouchy system names don’t seem to be reliably associated with brokenness. “Roadkill” is apparently a pretty good interactive graphics mesh unwrapping program, for example, even though quick automotive squishing followed by prolonged exposure to sunlight doesn’t have a reputation for reliably good taxidermy.

    (If I ever release my CGAL-mesh to PNG unwrapping hackery maybe I should call it “windfarm”. The logo almost draws itself.:-)

  5. On the good enough thought …
    I maintain a production system for a window manufacturing plant. One of the major major methods I use is that works for the 85% and fails loud for almost all of the other 15% — done today or tomorrow keeps a working, very flexible manufacturing system running for years.

    The last 15% you fix as seems appropriate as you find actual cases — things that only show up once every few months are mostly dealt with by the ‘fails loud’ method.

    I am helped by the fact that my normal user base is one or two.


  6. It’s difficult to surprise me, but with a name like cvssuck, I’m not actually suspicious. This is what would make me suspicious:

    220 RETURN

  7. >2. cvssync doesn’t appear to be actually included in the tarball.

    Fixed in the 0.8 release. Which is also way, way faster; I have been speed tuning.

  8. Because of name conflict, and because it is limited tool (rsync based, not cvs client based), perhaps a better name would be `cvs-rsync`?

  9. A thought actually occurs to me, what if repopuller were made independent from reposurgeon and merged with cvssync? They seem to perform effectively the same task (make CVS or SVN replication easy).

  10. >repopuller were made independent from reposurgeon and merged with cvssync?

    Already working on the merge. Hadn’t thought of spinning it out as a seperate project.

  11. Which one will reach the other side of a river: the one who dreams of a raft, or the one who hitchhikes to the next bridge?

    A lesson that could be well learned by many a Lisp hacker.

  12. “This is the Great Way of Unix; heed the lesson.”

    This is indeed the Unix way. A nice, fragile tool. Manipulating text hidden from the user. Subject to being broken by any change in the way that text is parsed. Requiring constant maintenance. In other words, it’s the future cvssuck.

    I am at heart a PLC and industrial control engineer, and I well understand the maxim “Sometimes you have to shoot the engineers and ship the product”. And yes, this does something useful and therefore why not? The tool is fine, for what it is. Being proud of it, not so much.

  13. Ltw, your criticism suggest that you have misunderstood the circumstances and the lesson.

    >Manipulating text hidden from the user. Subject to being broken by any change in the way that text is parsed.

    Yeah, but so what? From the user’s POV the tool is very simple – it takes the stuff you cut and paste from a CVS co command and dpes the right thing (usually). It’s hard to see why the “hiddenness” is an issue – after all, much the same sorts of things happen every 6ime a URL is parsed into a (service, host, path, port) tuple for pricessing.

    >Requiring constant maintenance.

    You may be under a misapprehension about how rapidly hosting sites change their access methods. It’s a rare thing.

    >In other words, it’s the future cvssuck.

    That is definitely not true, for at least two reasons. (1) It looks like cvssuck died of overcomplexity and over-cleverness, which is not the way programs as simple as cvssync fail. (2) The mappings from CVS spec to rsync source will probably be a lot more long-lived than the CVS repos.

  14. It’s not so much that I’ve misunderstood than I disagree with the entire philosophy. For this, sure, it’s valuable in most cases and the interface is unlikely to change much. It just triggers my visceral dislike of the Unix way. Dumping out a bag of bytes and hoping like hell the program at the other end understands them is not a great design goal. I’ve done worse, I just wouldn’t call it an achievement.

  15. To give you an example, a tunnel lighting system where we had to send the power availability status through from the PLCs controlling the lighting level to the system switching them. High pressure sodium lights have a huge inrush current, so if the power goes off then you switch it back on suddenly all the circuit breakers will blow. We had no compatible comms protocols. So while the managers argued about blame and risk at one end of the table, their chief engineer and I mapped out a protocol using digital inputs and outputs that basically presented a BCD encoded circuit breaker status mediated by a strobe and return signal so they could unlatch the circuits and return them one by one if the power failed. It’s been working for fifteen years, but god help anyone who has to troubleshoot it.

    Fixed the problem – but it was an ugly hack.

  16. It just triggers my visceral dislike of the Unix way. Dumping out a bag of bytes and hoping like hell the program at the other end understands them is not a great design goal.

    Fortunately, most developers these days seem to prefer the Windows way, where programs communicate according to structured, well-defined interfaces. This is true even on Linux, where D-Bus is gradually eating all other forms of IPC, even for system-level control and monitoring (via systemd).

  17. (2) The mappings from CVS spec to rsync source will probably be a lot more long-lived than the CVS repos.

    Ah. Hubris.

  18. @Petro

    (2) The mappings from CVS spec to rsync source will probably be a lot more long-lived than the CVS repos.

    Ah. Hubris.

    One of the three great virtues of a programmer according to Perl author Larry Wall (the other two being laziness and impatience) ;-)))

  19. FYI sourceforge docs for rsync access to CVS is here

    rsync -av rsync://* .
    I thought something similar for Savannah in my wanderings. Unfortunately I’m stuck with cvssuck because the host I’m tying to backup from ( doesn’t expose rsync in any way I’ve been able to discover.

  20. >FYI sourceforge docs for rsync access to CVS is here

    I used those. cvssync plays well with SourceForge.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="" highlight="">