Dec 16

You’re gonna need a bigger Beast

I’m taking a management-approved break from NTPsec to do a repository conversion that dwarfs any I’ve ever seen before. Yep, more history than Emacsmuch much more. More backtrail than entire BSD distributions, in fact about an order of magnitude larger than any repo I’ve previously encountered.

Over 255000 commits dating back to 1989 – now in Subversion, formerly in CVS and (I suspect) RCS. It’s the history of GCC, the Gnu Compiler Collection.

For comparison, the entire history of NTP, including all the years before I started working on it, is 14K commits going back to 1999. That’s a long history compared to most projects, but you’d have to lay 18 NTPsec histories end to end to even approximate the length of GCC’s.

Continue reading

Mar 14

Semantic locality and the Way of Unix

An important part of the Way of Unix is to try to tackle large problems with small, composable tools. This goes with a tradition of using line-oriented textual streams to represent data. But…you can’t always do either. Some kinds of data don’t serialize to text streams well (example: databases). Some problems are only tractable to large, relatively monolithic tools (example: compiling or interpreting a programming language).

Can we say anything generatively useful about where the boundary is? Anything that helps us do the Way of Unix better, or at least help us know when we have no recourse but to write something large?

Continue reading

Mar 06

Reposturgeon recruits the CryptBitKeeper!

I haven’t announced a reposurgeon release on the blog in some time because recent releases have mostly been routine stuff and bugfixes. But today we have a feature that many will find interesting: reposurgeon can now read BitKeeper repositories. This is its first new version-control system since Monotone was added in mid-2015.

Continue reading

Feb 17

Automatons, judgment amplifiers, and DSLs

Do we make too many of our software tools automatons when they should be judgment amplifiers? And why don’t we write more DSLs?

Back in the Renaissance there was a literary tradition of explaining natural philosophy via conversations among imaginary characters. I’m going to revive that this evening because I had an IRC conversation this afternoon, about the design insights behind reposurgeon, that pretty much begs to be presented this way.

The person of “Simplicio” was Galileo’s invention in his Dialogue Concerning the Two Chief World Systems. Here he represents four different people, but almost everything he says is something one of them in fact said or very plausibly might have. I’ve cleaned it up, edited, and amplified only a little.

For those of you coming in late, reposurgeon is a tool I wrote for editing version-control histories. It has many applications, including highest-quality repository conversions. Simplicio needed to excise some security-sensitive credentials from a DHS code repository – not just from the tip version but from the entire history. Reposurgeon is pretty much the only practical way to do this.

So, without further ado…

Continue reading

Jan 10

Cometary Contributors

I released reposurgeon 3.30 today. It has been five years and a month since the first public release.

In those five years, the design concept seems to have proved out very well, finding use in many repository conversions. But the project exhibits an unusual sociology; I don’t get lots of casual contributors, only a few exceptional ones.

Continue reading

May 28

Don’t do svn-to-git repository conversions with git-svn!

This is a public-service warning.

It has come to my attention that some help pages on the web are still recommending git-svn as a conversion tool for migrating Subversion repositories to git. DO NOT DO THIS. You may damage your history badly if you do.

Reminder: I am speaking as an expert, having done numerous large and messy repository conversions. I’ve probably done more Subversion-to-git lifts than anybody else, I’ve torture-tested all the major tools for this job, and I know their failure modes intimately. Rather more intimately than I want to…

Continue reading

Aug 12

Ignoring: complex cases

I shipped point releases of cvs-fast-export and reposurgeon today. Both of them are intended to fix some issues around the translation of ignore patterns in CVS and Subversion repositories. Both releases illustrate, I think, a general point about software engineering: sometimes, it’s better to punt tricky edge cases to a human than to write code that is doomed to be messy, over-complex, and a defect attractor.

Continue reading

Mar 29

Ugliest…repository…conversion…ever

Blogging has been light lately because I’ve been up to my ears in reposurgeon’s most serious challenge ever. Read on for a description of the ugliest heap of version-control rubble you are ever likely to encounter, what I’m doing to fix it, and why you do in fact care – because I’m rescuing the history of one of the defining artifacts of the hacker culture.

Continue reading

Dec 05

Heads up: the reposturgeon is mutating!

A few days ago I released reposurgeon 2.43. Since then I’ve been finishing up yet another conversion of an ancient repository – groff, this time, from CVS to git at the maintainer’s request. In the process, some ugly features and irregularities in the reposurgeon command language annoyed me enough that I began fixing them.

This, then, is a reposurgeon 3.0 release warning. If you’ve been using 2.43 or earlier versions, be aware that there are already significant non-backwards-compatible changes to the language in the repository head version and may be more before I ship. Explanation follows, embedded in more general thoughts about the art of language design.

Continue reading

Oct 29

Your money or your spec

reposurgeon has been stable for several months now, since the Subversion dump analyzer got to the point where people stopped appearing in my mailbox with the Pathological Subversion Repository Fuckup Of The Week.

Still, every once in a longer while somebody will materialize telling me they have some situation in a repo conversion that they want me to help them fix. The general form of these requests is like this. “I have {detailed description of a branch/merge topology nightmare that makes Eric’s brain hurt to contemplate}. What do I do to fix it?”

I am now going to announce a policy about this. There are exactly two ways ways you can get me to solve your repository problem.

Continue reading

Mar 24

Python speed optimization in the real world

I shipped reposurgeon 2.29 a few minutes ago. The main improvement in this version is speed – it now reads in and analyzes Subversion repositories at a clip of more than 11,000 commits per minute. This, is, in case you are in any doubt, ridiculously fast – faster than the native Subversion tools do it, and for certain far faster than any of the rival conversion utilities can manage. It’s well over an order of magnitude faster than when I began seriously tuning for speed three weeks ago. I’ve learned some interesting lessons along the way.

Continue reading

Feb 26

Mode of the Reposturgeon!

It was inevitable, I suppose; reposurgeon now has its own Emacs mode.

The most laborious task in the reposurgeon conversion of a large CVS or Subversion repository is editing the comment history. You want to do this for two reasons: (1) to massage multiline comments into the summary-line + continuation form that plays well with git log and gitk, and (2) lifting Subversion and CVS commit references from, e.g., ‘2345’ to [[SVN:2345]] so reposurgeon can recognize them unambiguously and turn them into action stamps.

In the new release 2.22, there’s a small Emacs mode with several functions that help semi-automate this process.

Fear the reposturgeon!

Dec 02

Beware! The Reposturgeon!

I had said I wasn’t going to do it, but…I experimented, and it turned out to be easier than I thought. Release 2.7 of reposurgeon writes (as well as reading) Subversion repositories. With the untested support for darcs, which should work exactly as well as darcs fast-export and fast-import do, this now brings the set of fully-supported version-control systems to git, hg, bzr, svn, and darcs; reposurgeon can be used for repository surgery and interconversion on any of these.

Continue reading

Nov 04

reposurgeon 2.0 announcement – the full-orchestra version

I shipped reposurgeon 2.0 a few days ago with the Subversion support feature-complete, and a 2.1 minor bugfix release this morning. My previous release announcement was somewhat rushed, so here is a more detailed one explaining why anybody contemplating moving up from Subversion should care.

To go with this, there is a new version of my DVCS Migration HOWTO.

Continue reading

Jan 23

Another bite of the reposturgeon

Five weeks ago I wrote that direct Subversion support in reposurgeon is coming soon. I’m waiting on one final acceptance test before I ship an official 2.0; in the meantime, for those of you kinky enough to find the details exciting, description follows of why this feature has required such a protracted and epic struggle. With (perhaps entertaining) rambling through the ontology of version control systems, and at least one lesson about good software engineering practice.

Continue reading

Dec 12

Coming soon: reposurgeon does Subversion

For those of you who have been following the development of reposurgeon, a pre-announcement: the next version, probably to be numbered 2.0, will directly read Subversion dumpfiles and repositories.

I’ve got this feature working now – it’s why my blogging has been scant recently – but I intend to have a really good regression-test suite in place and at least one large repo conversion done before I ship it for general use.

Note an important limitation: it will not write Subversion repos. So it will be useful as a conversion tool but not directly as an editor.

Fear the reposturgeon!