Mar 20

cvs-fast-export 1.43 is released

Maintaining cvs-fast-export is, frankly, a pain in the ass. Parts of the code I inherited are head-achingly opaque. CVS repositories are chronically prone to malformations that look like bugs in the tool and/or can’t be adapted to in any safe way. Its actual bugs are obscure and often difficult to fix – the experience is not unlike groping for razor-blades in the dark. But people expect cvs-fast-export to “just work” anyway and don’t know enough about what a Zeno’s tarpit the domain problem is to be properly grateful when it does.

Still I persevere. Somebody has to; the thought of vital code being trapped in CVS is pretty nervous-making if you know everything that can go wrong with it.

This release fixes a bug introduced by an incorrect optimization hack in 2014. It should only have affected you if you tried to use the -c option.

If you use this at a place that pays developers, please have your organization contribute to my Patreon feed. Some of my projects are a pleasure to do for free; this one is grubby, hard work.

Feb 15

Brute force beats premature optimization

I made a really common and insidious programming mistake recently. I’m going to explain it in detail because every programmer in the world needs the reminder not to do this, and I hope confessing that even “ESR” falls into such a trap will make the less experienced properly wary of it.

Our sutra for today expounds on the sayings of the masters Donald Knuth and Ken Thompson, who in their wisdom have observed “Premature optimization is the root of all evil” and “When in doubt, use brute force.”

Continue reading

Feb 10

SRC users: check in, please?

I just released version 1.7 of SRC, Simple Revision Control.

For those of you late to the party, SRC is a simple version control system for directories full of small standalone files like FAQs, scripts in your ~/bin, dotfiles, and so forth – cases where you don’t want multi-file changesets. It’s actually a Python wrapper around RCS (or, optionally, SCCS) but gives you integer sequential version numbers, lockless operation, and a modern low-friction UI modeled on Subversion’s.

With 1.7, I think it’s finished – the last two user-visible features I had planned were SCCS support and DOT visualization, and those are done now.

I believe SRC is now feature-complete for its functional niche. Am I mistaken? Is anything missing? Did I do anything that seems wrong?

I know SRC has had real users since about 0.3. If you are an SRC user, please check in in the comments. Most importantly, tell me if you need any feature it doesn’t have. I’m also curious if the actual use cases are any different than I expected, and I am all agog to know if anyone actually has a use for the SCCS support.

Feb 05


I needed a break from serious work yesterday, so SRC now speaks SCCS as well as RCS. This wasn’t difficult, I had SRC carefully factored in anticipation from when I originally wrote it.

I can’t say I think this feature will be actually useful; SCCS is pretty primitive, and the SRC support has some annoying limitations as a result. But some hacks you do just because you can, and this is one of them.

Continue reading

Jan 26

SRC 1.0 is released

If you were reading A&D a year ago, you may recall that I invented a new version-control system to occupy an odd little niche that none of the exiting ones serve very well.

Well, actually, it’s a shell around a very old version-control system that makes a reasonable fast version-storage manager but has a crappy UI. Thus, SRC – RCS reloaded, with a mission to serve cases where you don’t want per-directory changesets but prefer each file to have its own separate change history. Like a directory full of separate FAQs, or your ~/bin full of little scripts.

SRC gives you a modern UI in the svn/hg/git style (but much, much simpler than git’s) and lockless operation. It has full embedded documentation and an Emacs VC backend. If your little project goes multi-file, you can instantly fast-export to git.

Today I shipped Version 1.0. This could have happened sooner, but I’ve been focusing on NTPsec pretty hard in the last year. There was one odd bug in the behavior of multi-file commands that I just hadn’t got around to fixing. (Yes, you can do multi-file commands, but the files still have separate histories.)

The whole thing is just 2KLOC of Python, and that’s with the rather extensive embedded documentation. The sort of person who frequents this blog might find the FAQ entertaining.

Jun 23

How to spot a high-quality repository conversion

In my last post, I inveighed against using git-svn to do whole-repository conversions from Subversion to git (as opposed to its intended use, which is working a Subversion repository live through a git remote).

Now comes the word that hundreds of projects a week seem to be fleeing SourceForge because of their evil we’ll-hijack-your-repo-and-crapwarify-your installer policy. And they’re moving to GitHub via its automatic importer. Which, sigh, uses git-svn.

I wouldn’t trust that automatic importer (or any other conversion path that uses git-svn) with anything I write, so I don’t know how badly it messes things up.

But as a public service, I follow with a description of how a really well-done repository conversion – the kind I would deliver using reposurgeon – differs from a crappy one.

Continue reading

Dec 13

Progress towards the extinction of CVS

The Great Beast, designed for converting large CVS repos, is now in full production. It hasn’t killed off any specimens in the wild yet (and I’ll explain why in a bit), but it’s doing spectacularly well on our test repositories.

As a representative large example, the entire Emacs CVS history, 1985-2009, 113309 CVS commits, lifts clean in 37 seconds at a sustained rate of 3K CVS commits a second. Yes, three thousand.

The biggest beast known to us, the NetBSD src repository, converts in 22 minutes. To give some idea of what a speedup this is, the first time I ran a lift on it – on one of Wendell’s Xeon machines – it took a bit under six hours. That’s about a factor of seventeen, there.

Judging by performance on the other project devs’ machines the Beast is good for a 2x to 3x speedup over a conventionally-balanced PC design (that is, one with worse RAM latency, narrower caches, more cores but somewhat lower single-thread speed). That’s a big enough advantage to validate the design and be practically significant on large repositories.

Continue reading

Nov 07

I wrote a version-control system today

I wrote a version-control system today. Yes, an entire VCS. Took me 14 hours.

Yeah, you’re looking at me like I’m crazy. “Why,” you ask, quite reasonably, “would you want to do a thing like that? We’re not short of powerful VCSes these days.

That is true. But I got to thinking, early this morning, about the fact that I haven’t been able to settle on just one VCS. I use git for most things, but there’s a use case git doesn’t cover. I have some document directories in which I have piles of things like HOWTOs which have separate histories from each other. Changes in them are not correlated, and I want to be able to move them around because I sometimes do that to reorganize them.

What have I been using for this? Why, RCS. The ancient Revision Control System, second oldest VCS in existence and clinging tenaciously to this particular niche. It does single-file change histories pretty well, but its UI is horrible. Worse than git’s, which is a pretty damning comparison.

Then I got to thinking. If I were going to design a VCS to do this particular single-file, single-user job, what would it look like? Hm. Sequential integer revision numbers, like Subversion and Mercurial used locally. Lockless operation. Modern CLI design. Built-in command help. Interchange with other VCSes via git import streams. This sounds like it could be nice

Then, the idea that made it inevitable. “I bet.” I thought, “I could write this thing as a Python wrapper around RCS tools. Use them for delta storage but hide all the ugly parts.”

Thus, SRC. Simple Revision Control, v0.1.

Continue reading

Nov 05

Chipping away at CVS

I’ve just shipped a new version of cvs-fast-export, 1.26. It speeds the tool up more, more, more – cranking through 25 years and 113300 commits of Emacs CVS history, for example in 2:48. That’s 672 commits a second, for those of you in the cheap seats.

But the real news this time is a Python wrapper called ‘cvsconvert’ that takes a CVS repository, runs a conversion to Git using cvs-fast-export, and then – using CVS for checkouts – examines the CVS and git repositories side by side looking for translation glitches. It checks every branch tip and every tag.

Running this on several of my test repos I’ve discovered some interesting things. One such discovery is of a bug in CVS. (Yeah, I know, what a shock…)

Continue reading

Oct 29

When hackers grow old

Lately I’ve been wrestling with various members of an ancient and venerable open-source development group which I am not going to name, though people who regularly follow my adventures will probably guess which one it is by the time I’m done venting.

Why it so freaking hard to drag some people into the 21st century? Sigh…

I’m almost 56, an age at which a lot of younger people expect me to issue semi-regular salvos of get-off-my-lawn ranting at them. But no – I find, that, especially in technical contexts, I am far more likely to become impatient with my age peers.

A lot of them really have become grouchy, hidebound old farts. And, alas, it not infrequently falls to me to be the person who barges in and points out that practices well-adapted for 1995 (or, in the particular case I’m thinking of, 1985) are … not good things to hold on to decades later.

Why me? Because the kids have little or no cred with a lot of my age peers. If anyone’s going to get them to change, it has to be someone who is their peer in their own perception. Even so, I spend a lot more time than seems just or right fighting inertia.

Young people can be forgiven for lacking a clue. They’re young. Young means little experience, which often leads to unsound judgment. It’s more difficult for me to forgive people who have been around the track often enough that they should have a clue, but are so attached to The Way It’s Always Been Done that they can’t see what is in front of their freaking noses.

Continue reading

Sep 02

Adverse selection and old technology

Yesterday I shipped cvs-fast-export 1.15, with a significant performance improvement produced by replacing a naive O(n**3) sort with a properly tuned O(n log n) version.

In ensuing discussion on G+, one of my followers there asked if I thought this was likely to produce a real performance improvement, as in small inputs the constant setup time of a cleverly tuned algorithm often dominates the nominal savings.

This is one of those cases where an intelligent question elicits knowledge you didn’t know you had. I discovered that I do believe strongly that cvs-fast-export’s workload is dominated by large repositories. The reason is a kind of adverse selection phenomenon that I think is very general to old technologies with high exit costs.

The rest of this blog post will use CVS as an example of the phenomenon, and may thus be of interest even to people who don’t specifically care about high version control systems.

Continue reading