There’s an ancient Unix maxim to the effect that a tool that gets 85% of your job done now is preferable to one that gets 100% done never. Sometimes chasing corner cases is more work than the problem really justifies.
In today’s dharma lesson, I shall illustrate this principle with a real-world and useful example.
In my last blog post I explained why I had to shoot cvsps through the head. Some of my regulars regretted the loss of the good feature bolted to its crappy repo-analysis code – it could fetch remote CVS repository metadata for analysis rather than requiring them to have been already mirrored locally.
To fill this functional gap, I needed a tool for mirroring the contents of a remote CVS repository to a local directory. There’s floating folklore to the effect that a tool called “cvssuck” does this job, but when I tried to use it it failed in about the most annoying possible way. It mirrored the directory structure of the remote site without fetching any masters!
Upon investigation I discovered that the cvssuck project site has disappeared and there hasn’t been a release in years. Disgusted, I asked myself how it could possibly have become that broken. Seemed to me the whole thing ought to be a trivial wrapper around rsync.
Or…maybe not. What scanty documentation I found for cvssuck made a big deal out of the fact that it (inefficiently) used CVS itself to fetch masters. This doesn’t make any sense if they were rsync accessible. because then it would be a much faster and more efficient way to do the same job.
But I thought about the sites I generally have to fetch from when I’m grabbing CVS repositories for conversion, as I did most recently for the groff project. SourceForge. Savannah. These sites (and, I suspect, most others that still support CVS) do in fact allow rsync access so that project administrators can use it to do offsite backups.
OK, so suppose I write a little wrapper around rsync to fetch from these sites. It might not do the guaranteed fetch that cvssuck advertises…but on the other hand cvssuck does not seem to actually work, at least not any more. What have I got to lose?
About an hour of experimentation and 78 lines of Python code later, I had learned a few things. First, a stupid-simple wrapper around rsync does in fact work for SourceForge and Savannah. And second, there is a small but significant value the wrapper can add.
The only thing you are pretty much guaranteed to be able to find out about a CVS repository is the CVS command needed to check out a working copy. For example, the groff CVS page gives you this command:
cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/sources/groff co <modulename>
You have to figure out for yourself that the <modulename> should also be “groff”, but there are clues to that on the web page. For those of you blessed enough to be unfamiliar with CVS, a single instance can host multiple projects that can be checked out separately; the module name selects one of these.
It isn’t necessarily clear how to get from that cvs invocation to an rsync command. Here’s how you do it. First, lop off the “anonymous@” part; that is a dummy log credential. Treat “/sources/groff” as a file path to the repository directory, then realize that the module is a subdirectory. You and up writing this:
rsync -avz cvs.savannah.gnu.org:/sources/groff/groff my-local-directory
That’s really simple, but it turns out not to work on SourceForge. Because SourceForge runs an rsync daemon and hides the absolute file path to the repository. The corresponding fetch from SourceForge, if groff existed there, would look like this:
rsync -avz groff.cvs.sourceforge.net::cvsroot/groff/groff groff
Note the double colon and absence of leading ‘/’ on the repository path.
The value a wrapper script can add is knowing about these details so you don’t have to. Thus, cvssync. You call it with the arguments you would give a CVS checkout command. It pulls those apart, looks at the hostname, figures out how to reassemble the elements into an rsync command, and runs that.
This just shipped with cvs-fast-export release 0.7. At the moment it really only knows two things: A special rule about building rsync commands for SourceForge, and a general rule that happens to work for Savannah and should for most other CVS sites a well. More hosting-site would be easy to add, a line or two at most of Python for each hosting side.
This wrapper doesn’t do the last 15% of the job; it will fail if the CVS host blocks rsync or has an unusual directory structure. But that 85% now is more valuable than 100% never, especially when its capabilities are so easily extended.
And hey, it only took an hour for me to write, test, document, and integrate into the cvs-fast-export distribution. This is the Great Way of Unix; heed the lesson.
By the way, your Eric Raymond’s open-source collection doesn’t list cvs-fast-export (and I think it has invalid last change date of 21 Dec 2012 in heading, as it lists just released for first time shipper).
Also,
Resource page for cvs-fast-export has HTML version of cvs-fast-export(1) manpage linked twice, while missing link to cvssync (cvsrsync?) manpage.
>By the way
Thanks, fixed. That turned up a minor bug in shipper.
Also known as Perfect is the Enemy of Good … or … Worse is Better.
At some point, you just have to say “ship it!”.
Which 85% is important, however. Let’s say I’m writing a network interface driver module for the Linux kernel. If my hypothetical driver works with 85% of the NICs 100% of the time, great! Ship it! But what if it works with 100% NICs its supposed to only 85% of the time? That’s much worse.
s/wose/worse
It’s called “cvssuck” …I don’t think this should have been very surprising.
Terry, grouchy system names don’t seem to be reliably associated with brokenness. “Roadkill” is apparently a pretty good interactive graphics mesh unwrapping program, for example, even though quick automotive squishing followed by prolonged exposure to sunlight doesn’t have a reputation for reliably good taxidermy.
(If I ever release my CGAL-mesh to PNG unwrapping hackery maybe I should call it “windfarm”. The logo almost draws itself.:-)
On the good enough thought …
I maintain a production system for a window manufacturing plant. One of the major major methods I use is that works for the 85% and fails loud for almost all of the other 15% — done today or tomorrow keeps a working, very flexible manufacturing system running for years.
The last 15% you fix as seems appropriate as you find actual cases — things that only show up once every few months are mostly dealt with by the ‘fails loud’ method.
I am helped by the fact that my normal user base is one or two.
Jim
It’s difficult to surprise me, but with a name like cvssuck, I’m not actually suspicious. This is what would make me suspicious:
210 GOTO PERFECTLY_WRITTEN_CODE
220 RETURN
I happen to have two immediate thoughts …
1. cvssync sounds too close to cvsync, it could make things confusing: http://www.openbsd.org/cvsync.html
2. cvssync doesn’t appear to be actually included in the tarball.
>2. cvssync doesn’t appear to be actually included in the tarball.
Fixed in the 0.8 release. Which is also way, way faster; I have been speed tuning.
Because of name conflict, and because it is limited tool (rsync based, not cvs client based), perhaps a better name would be `cvs-rsync`?
A thought actually occurs to me, what if repopuller were made independent from reposurgeon and merged with cvssync? They seem to perform effectively the same task (make CVS or SVN replication easy).
>repopuller were made independent from reposurgeon and merged with cvssync?
Already working on the merge. Hadn’t thought of spinning it out as a seperate project.
Which one will reach the other side of a river: the one who dreams of a raft, or the one who hitchhikes to the next bridge?
A lesson that could be well learned by many a Lisp hacker.
“This is the Great Way of Unix; heed the lesson.”
This is indeed the Unix way. A nice, fragile tool. Manipulating text hidden from the user. Subject to being broken by any change in the way that text is parsed. Requiring constant maintenance. In other words, it’s the future cvssuck.
I am at heart a PLC and industrial control engineer, and I well understand the maxim “Sometimes you have to shoot the engineers and ship the product”. And yes, this does something useful and therefore why not? The tool is fine, for what it is. Being proud of it, not so much.
Ltw, your criticism suggest that you have misunderstood the circumstances and the lesson.
>Manipulating text hidden from the user. Subject to being broken by any change in the way that text is parsed.
Yeah, but so what? From the user’s POV the tool is very simple – it takes the stuff you cut and paste from a CVS co command and dpes the right thing (usually). It’s hard to see why the “hiddenness” is an issue – after all, much the same sorts of things happen every 6ime a URL is parsed into a (service, host, path, port) tuple for pricessing.
>Requiring constant maintenance.
You may be under a misapprehension about how rapidly hosting sites change their access methods. It’s a rare thing.
>In other words, it’s the future cvssuck.
That is definitely not true, for at least two reasons. (1) It looks like cvssuck died of overcomplexity and over-cleverness, which is not the way programs as simple as cvssync fail. (2) The mappings from CVS spec to rsync source will probably be a lot more long-lived than the CVS repos.
It’s not so much that I’ve misunderstood than I disagree with the entire philosophy. For this, sure, it’s valuable in most cases and the interface is unlikely to change much. It just triggers my visceral dislike of the Unix way. Dumping out a bag of bytes and hoping like hell the program at the other end understands them is not a great design goal. I’ve done worse, I just wouldn’t call it an achievement.
To give you an example, a tunnel lighting system where we had to send the power availability status through from the PLCs controlling the lighting level to the system switching them. High pressure sodium lights have a huge inrush current, so if the power goes off then you switch it back on suddenly all the circuit breakers will blow. We had no compatible comms protocols. So while the managers argued about blame and risk at one end of the table, their chief engineer and I mapped out a protocol using digital inputs and outputs that basically presented a BCD encoded circuit breaker status mediated by a strobe and return signal so they could unlatch the circuits and return them one by one if the power failed. It’s been working for fifteen years, but god help anyone who has to troubleshoot it.
Fixed the problem – but it was an ugly hack.
Fortunately, most developers these days seem to prefer the Windows way, where programs communicate according to structured, well-defined interfaces. This is true even on Linux, where D-Bus is gradually eating all other forms of IPC, even for system-level control and monitoring (via systemd).
> By the way, your Eric Raymond’s open-source collection doesn’t list
On the esr open-source collection page cvssync currently links to http://www.catb.org/~esr/rcs-fast-export which does not exist. I believe http://www.catb.org/~esr/cvs-fast-export is the intended target.
Ah. Hubris.
One of the three great virtues of a programmer according to Perl author Larry Wall (the other two being laziness and impatience) ;-)))
FYI sourceforge docs for rsync access to CVS is here http://sourceforge.net/apps/trac/sourceforge/wiki/CVS#Backups
“`
rsync -av rsync://PROJECTNAME.cvs.sourceforge.net/cvsroot/PROJECTNAME/* .
“`
I thought something similar for Savannah in my wanderings. Unfortunately I’m stuck with cvssuck because the host I’m tying to backup from (tigris.org) doesn’t expose rsync in any way I’ve been able to discover.
>FYI sourceforge docs for rsync access to CVS is here
I used those. cvssync plays well with SourceForge.
The joke’s on you, ESR. Savannah has recently upgraded their CVS server, and rsyncing via the rsync://vcs.savannah.gnu.org/* URLs no longer works; the hostname has to be changed to cvs.savannah.gnu.org.
By the way, Savannah also provides CVS versioning for web sites, within the /web/ subdirectory, but your script actively breaks it; it sticks /sources/ in front of the path if it’s not there already. It does similar massaging for other hosting sites, which I’d expect to break in similar ways.
And finally: why did you have to invent a whole new ‘cvs:’ URI scheme? Couldn’t it just take a plain CVSROOT string, the same one CVS itself uses? People could then just copy-paste it from their hosting service’s documentation pages instead of being forced to do error-prone cutting and reassembling themselves.
Is that how you put into practice your advice on ‘not trying to be too clever’? I’m not impressed.
>the hostname has to be changed to cvs.savannah.gnu.org
Thanks, I’ve corrected that.
>And finally: why did you have to invent a whole new ‘cvs:’ URI scheme?
For the same reasons synthetic URI schemes have been found good elsewhere – a single address for a single retrieval unit is a useful thing to have. My direct precedent was the svn URLs recognized by newer Subversion tools.
>Couldn’t it just take a plain CVSROOT string, the same one CVS itself uses?
If you’d been less busy polishing your snark, you might have noticed that cvssync actually supports that kind of split specification.
> For the same reasons synthetic URI schemes have been found good elsewhere – a single address for a single retrieval unit is a useful thing to have. My direct precedent was the svn URLs recognized by newer Subversion tools.
An URI scheme may be a ‘useful thing to have’, but making that the only justification for a feature is a symptom of cargo cult design. Your ad-hoc scheme isn’t sufficiently expressive to differentiate between ‘server’, ‘pserver’, ‘kserver’ and other connection methods, which is important for choosing the correct authentication protocol when connecting. Someone else who cares about these distinctions would have to invent another URI scheme, which would probably be incompatible with yours. And inventing a new URI scheme doesn’t make anything easier on your side anyway: you still have to cut the damn string apart and reassemble it. You’re badly solving a problem you don’t even have.
Of course, given that this is just one silly script that probably nobody will ever need to worry about interoperating with, and that by now CVS is a dead project everyone is moving away from, it makes this analysis quite academic. But the general point stands: if there’s anyone who should design a URI scheme for a given protocol, it should be the people who invented the protocol. Either that, or it should be something proposed in a formal specification submitted to a standards body. Leaving implementers to invent their own ad-hoc, sparsely-documented schemes creates a risk of coordination failures, incompatibilities of various degrees of subtlety, and sets you up for the situation from the xkcd ‘Standards’ comic. (Notice how the precedent you cite differs on this matter.)
> If you’d been less busy polishing your snark, you might have noticed that cvssync actually supports that kind of split specification.
Does it?
$ cvssync -v :pserver:cvs.savannah.nongnu.org:/sources/cvs ccvs
Traceback (most recent call last):
File “/usr/bin/cvssync”, line 101, in <module>
(host, path) = host.split(“:”)
ValueError: too many values to unpack (expected 2)
Yes, if I don’t leave out the credential part then it ‘works’, but it’s only by accident.
>An URI scheme may be a ‘useful thing to have’, but making that the only justification for a feature is a symptom of cargo cult design.
All features are justified by being useful. At least useful relative to the technical debt they incur. And as you yourself pointed out, the downside risks are low here.
Get over yourself. You sound inexperienced, arrogant, dogmatic, and frankly rather stupid. I don’t think I have anything to learn on this topic from anyone who flames the way you do.