Moving the NetBSD repository

Some people on the NetBSD tech-repository list have wondered why I’ve been working on a full NetBSD repository conversion without a formal request from NetBSD’s maintainers that I do so.

It’s a fair question. An answer to it involves both historical contingency and some general issues about moving and mirroring large repositories. Because of the accident that a lot of people have recently dropped money on me in part to support an attack on this problem, I’m going to explain both in public.

First, the historically contingent part:

1. Alan Barrett tried to run a full conversion of NetBSD using cvs-fast-export last December and failed (OOM). He then engaged me and we spent significant effort trying to reduce the program’s working set, but could not prevent OOM on either of the machines we were using. Because Alan was willing to work on this at some length, I formed the idea that there was real demand for a full NetBSD conversion.

2. The NetBSD repo is large and old. I wanted a worst-possible-case (or near worst-possible-case) to test the correctness of the tool on. I knew there might be larger repositories out there (and now it appears that Gentoo’s is one such) but for obvious historical reasons I thought NetBSD would be an exemplary near-worst case. Thus, it would be a worthy test even if the politics to get the result deployed didn’t pan out.

I have since been told that NetBSD actually has a git mirror of its CVS repository produced with a two-step conversion: CVS -> Fossil -> git.

This makes me nervous about the quality of the result. Repo conversions produce artifacts due to ontological mismatches between the source and target systems; a two-stage process will compound the problems. Which in turn gives rise exactly the kinds of landmines one least wants – not obvious on first inspection but chronically friction-causing down the road.

I’m not speaking theoretically about this; I’m currently dealing with a major case of landmine-itis in the Emacs repository, which has (coincidentally) just been scheduled for a full switch to git on Nov 11. I’ve been working on that conversion for most of a year.

For a really high-quality conversion even a clean single-stage move needs human attention and polishing. This is why reposurgeon is designed to amplify the judgment of a human operator rather than attempt to fully mechanize the conversion.

I understand there is internal controversy within NetBSD over a full switch to git. I don’t really want to get entangled in the political part of the discussion. However, as a technical expert on repository conversions and their problems, I urge the NetBSD team to move the base repository to something with real changesets as soon as possible.

It doesn’t have to be git. Mercurial would do; even Subversion would do, though I don’t recommend it. I’m not grinding an axe for git here, I’m telling you that the most serious, crazy-making traps for the unwary lie in the move from a version-control system without full coherent changesets to a VCS with one. Once you have that conversion done and clean, moving the repository content to any other such system is relatively easy.

(Again, I’m not speaking theoretically – reposurgeon is the exact tool you want for such cross-conversions.)

This is my offer: I have the tools and the experience to get you to the changeset-oriented VCS of your choice. I can do a really good job, better than you’ll ever get from mechanical mirroring or a batch converter, because I know all about common conversion artifacts and how to do things like lifting old version references and ignore-pattern files.

It looks like my tools are git-oriented because they rely on git fast-import streams as an interchange format, but I’m not advocating git per se – I’m urging you to move somewhere with changesets. It’s a messy job and it wants an expert like me on it, but it only has to be done once. Afterwards, the quality of your developer experience and your future technical options with regard to what VCS you actually want to use will both greatly improve.

Related technical point: the architectural insight behind my tools is that the git folks created something more generally useful than they understood when they defined import streams. Having an editable transfer format that can be used to move content and metadata relatively seamlessly between VCSes is as important in the long term as the invention of the DVCS – possibly more so.

cvs-fast-export emits a fast-import stream not because I’m a git partisan (I actually rather wish hg had won the mindshare war) but because that’s how you get to a sufficiently expressive interchange format.

I’ll mail this to tech-repository once I can find out how to sign up.

34 comments

  1. This is the first time I’ve heard of someone using Fossil in the wild, neat. I’ve not used it, but the authors of SQLite have a lot of credibility with me. I’ve been surprised at how little it gets talked about.

    I don’t know how good the tools to go from CVS to Fossil are, but Fossil is certainly a peer to Git and Hg in its DVCS creds.

    1. >Fossil is certainly a peer to Git and Hg in its DVCS creds.

      Note to self: Add full Fossil support to reposurgeon. Since it has both a native importer and exporter for fast-import streams, this should not be at all difficult.

      1. >Note to self: Add full Fossil support to reposurgeon.

        Well, that was easy. Implemented, tested, documented, and pushed.

  2. From what I remember from history, the fast-import format was created to help transition of Mozilla code from CVS to Git, if it were chosen. Mozilla in the end have chosen to use Mercurial (hg). But fast-import stream format remained with us. It is very helpful that if you provide fast-import import and export for your version control system, then you can import and export to (almost) any other VCS. That’s why e.g. Veracity SCM implemented it – to avoid lock-in accusations, and to make it easier to import history from other VCS.

    @esr, why do you say “I actually rather wish hg had won the mindshare war”?

    1. >@esr, why do you say “I actually rather wish hg had won the mindshare war”?

      The UI is much nicer than git’s, and the implementation is in a language that’s easier to hack than C. In addition, it may be that name tracking is the right thing, though I’m not sure of that.

  3. This is the first I’ve heard of Fossil, period. At this point, it’s not much more than an item of intellectual curiosity; the project I work on most heavily uses Mercurial, and as Eric says, git won the mindshare war. That it was developed by Linus is probably why.

  4. > 1. Alan Barrett tried to run a full conversion of NetBSD using cvs-fast-export last December and failed (OOM).

    What if you just gave it a ludicrous amount of swap space on a 64-bit machine?

    1. >What if you just gave it a ludicrous amount of swap space on a 64-bit machine?

      I dunno. In theory it might work. It’d probably be painfully slow, though. With all structures in RAM the conversion takes many hours; swap slowdown would make it worse.

  5. > I actually rather wish hg had won the mindshare war

    Why didn’t it? Why couldn’t the “mindshare war” be restarted, or the two coexist peacefully indefinitely?

    What are the technical obstacles to letting everyone use their own tools (including being able to clone a repository from one format to the other, push/pull changes , etc)?

  6. > The UI [of Mercurial (hg)] is much nicer than git’s,

    This might be caused by the fact that while Mercurial development was API-driven and top-down, Git development was format-driven and bottom-up, using the “worse is better” Unix-way approach. Which while can give better results (e.g. branches and remote-tracking branches vs 3 attempts to get branches right in Mercurial: clone to branch, named branches, bookmarks extension), it can lead to inconsistent UI (e.g. subcommands vs options, naming options, etc.).

    > and the implementation is in a language that’s easier to hack than C.

    Well, usually implementation starts as shell script or Perl script, only later when it turns out to be good idea (and UI stabilizes) it gets translated into C (in git development jargon it is called builtin-ification). Some time in the past, when recursive merge strategy was being development, it even used Python.

    BTW. to better handle development in C language, git has created a few interesting micro-libraries, e.g. parseopt (for handling options and subcommands, in C and in shell), strbuf (string handling), argv and run-command etc. (see Documentation/technical). Though it would be nice if those micro-libraries were exported as separate projects to use by other applications (like perf events UI in Linux kernel, using parseopt).

    > In addition, it may be that name tracking is the right thing, though I’m not sure of that.

    Both “rename tracking” and “rename detection” have their advantages and disadvantages…

  7. > >What if you just gave it a ludicrous amount of swap space on a 64-bit machine?

    > I dunno. In theory it might work. It’d probably be painfully slow, though. With all structures in RAM the conversion takes many hours; swap slowdown would make it worse.

    I’ve seen this happen in 32-bit land – when the in-core requirements of the process get up to a critically large fraction of the address space (or at least that part not occupied by the un-swappable part of the OS), the machine spends f o r e v e r swapping. Not just painfully slow, infinitely slow; no real progress *ever* gets made.

    I assume that it would also work this way in a larger address space, proportionately scaled up.

  8. Well, yeah, obviously it sounds like a really bad idea – I was just wondering if it could work depending on the access patterns of the data it keeps in memory.

  9. > This is the first I’ve heard of Fossil, period. At this point, it’s not much more than an item of intellectual curiosity

    For you maybe.
    For others it has a more acceptable license and has almost zero external depencies* for a full install – the fossil tarball ships with pretty much everything it needs.

    * according to pkgsrc it needs readline, openssl and zlib which are mostly OS supplied anyway.

  10. >@esr, why do you say “I actually rather wish hg had won the mindshare war”?

    The UI is much nicer than git’s

    I realize I might be a small minority in saying this, but I actually disagree. It might be because I started using Git just after 1.5 was released, which had rather notably improved the UI drastically, or implemented an actual VCS, depending on perspective (Linus’s view on the original design was that it was a raw distributed file store that provided the capabilities to build a VCS from, rather than being a VCS all on its own, thus why Cogito and various other UIs were made…). I just never fully wrapped my head around Mercurial, it seemed to try too hard to emulate the CVS/SVN command set in ways that barely made sense in a DVCS, whereas Git didn’t care about such legacy and modelled its command set after the functions of DVCS. I’m probably also influenced heavily by the fact that I’ve never used Mercurial for any large project, at least not directly (within the past year or so, I have git-remote-hg so I can just clone hg repos with Git and use the in a fully bi-directional ways, it only has a couple caveats for features Git supports (like octopus merging) that hg doesn’t.)

    Also, as it was mentioned, Mercurial has taken a long time to “get branches right” and from what I’ve seen, it still doesn’t. In Mercurial they seem to live as some kind of immortal object that cannot be removed once you’re done with it… in Git, it’s much more flexible, and I can push a feature branch to an upstream repo so people can test it out as much as they wish, that I can delete later on after it gets merged (TBH, deleting a branch on a remote is a weak spot in Git UI, it requires git push remote :branch which can look like weird syntax for those that don’t understand what the colon means). This is probably a minor nitpick that they can resolve in the future, if my perspective isn’t already outdated.

    Fossil was mentioned in the thread, and I’ve always kind of like the idea of it, in particular the integrated wiki and issue tracker, but I’ve also found the UI to be lacking. It all starts when you’re expected to begin or clone a Fossil repository to a file, and then you’re expected to make working trees manually, and then using it… I actually didn’t figure out how to teach Fossil my proper name/email, which is a trivial task for both hg and git. The working trees being more separated out from the repo might make it more CVS/SVN-like to have multiple branches at one time, but I can’t honestly say that it’s been a drawback in Git (it is possible to have multiple working directories with a single Git repo too, it’s just difficult). I know there’s also issue trackers/wikis for integration with Git repos around, I just haven’t personally tried any. It kind of ties into Linus’s idea that Git itself doesn’t represent the end-all-be-all of Git as a VCS platform. One thing that does really appeal to be me with Fossil, is that they have a system in-place to shun objects from being cloned around; I work on a project that has had a few copyright violations discovered in the tree and required rewriting the entire Git repository a couple times, which doesn’t really play nicely with downstream repos (the easy, but not only, solution is “reclone the repo”), Fossil could have been beneficial here (reposurgeon could have been benficial too, but it didn’t exist yet…).

  11. @Mike:
    > TBH, deleting a branch on a remote is a weak spot in Git UI, it requires `git push remote :branch` which can look like weird syntax for those that don’t understand what the colon means

    With modern Git you can use `git push remote –delete branch`. The `:branch` syntax means: push empty to branch, i.e. delete.

    > it is possible to have multiple working directories with a single Git repo too, it’s just difficult

    There is work in progress to make it better (than git-new-worktree script in contrib/ that uses symlinks extensively).

  12. Another fan of Fossil here. I came across it after using the CVSTrac tool for many years (both the CVS and SVN variant builds, it also has a Git build I have never tried) which provides a similar Wiki/Ticket front end for those SCM tools. But in Fossil it is all united into a single system. Since it’s from the same folks, it uses SQLite files for the replicated repo and workarea metadata.

    A shout out to D. Richard Hipp for making my life easier.

  13. Eric, while you’re busting out some deeply wizardly (have I got that right?) repo-fu on NetBSD, any thoughts about doing or offering same for FreeBSD and OpenBSD?

    1. >Eric, while you’re busting out some deeply wizardly (have I got that right?) repo-fu on NetBSD, any thoughts about doing or offering same for FreeBSD and OpenBSD?

      Be happy to. Are you affiliated with either effort? Can you help deal with the political end?

  14. I think in the case of OpenBSD a big objection would be git’s license. The team has a strong preference to avoiding GPLd code in base (except as a last resort), and they would certainly want to have the project VCS in base.

  15. > I think in the case of OpenBSD a big objection would be git’s license. The team has a strong preference to avoiding GPLd code in base (except as a last resort), and they would certainly want to have the project VCS in base.

    Git is GPLv2/LGPLv2.1
    JGit (the Java implementation) is EDL (new-style BSD)

    Mercurial is GPLv2+
    Fossil is 2-clause BSD
    Veracity is Apache License v2

  16. Off topic:

    I’ve been having a weird problem with ssh that I want to solicit advice on.

    I’m running sshd on my desktop, and occasionally login from other machines on my local network. The router for the network is an AT&T U-verse access point. The machines’ hostnames are whitelisted in /etc/hosts.deny with an ALL EXCEPT directive of the form:

    sshd: ALL-EXCEPT foo bar baz client-hostname

    If I try connecting to the desktop with:

    ssh desktop-hostname

    The connection is rejected, and auth.log contains:

    sshd[$PID]: warning: /etc/hosts.deny, line 18: host name/name mismatch: client-hostname != client-hostname.attlocal.net
    sshd[$PID]: refused connect from [client’s local IP address]

    The same thing happens if I add client-hostname.attlocal.net to the ALL EXCEPT directive.

    However, if I add my public IP address to the ALL EXCEPT directive and ssh to my public IP address the connection is port-forwarded to the desktop and I am able to log in successfully.

    Can anybody provide any insight on what’s going on here?

  17. A bit off-topic, but does reposurgeon support perforce? And/or what would it take to make reposurgeon support it? In my opinion, perforce needs extermination every bit as much as CVS does.

    1. >And/or what would it take to make reposurgeon support [Perforce]?

      The git suite has a Perforce importer. Import the repository. Calling reposurgeon on the import should work fine.

  18. I’m running sshd on my desktop, and occasionally login from other machines on my local network. The router for the network is an AT&T U-verse access point. The machines’ hostnames are whitelisted in /etc/hosts.deny with an ALL EXCEPT directive of the form:

    sshd: ALL-EXCEPT foo bar baz client-hostname

    why not just put the hosts you care about in /etc/hosts.allow and then have
    sshd: ALL
    in /etc/hosts.deny ?

  19. Err… exactly how clean is the git import stream support between all of these DVCSes?

    If you reconceive of a given repository manipulation as “a command that produces append commands to an import stream”, you could start moving in a direction where the tools really would be fungible. And perhaps start seeing the platonic ideal beneath all of these currently-separate manifestations? And perhaps implement the platonic ideal as another set of commands that can interact via the import streams directly?

    I’m vigorously handwaving here on the grounds that if the stream support is too idiosyncratic it isn’t worth pursuing this line of reasoning.

  20. >>Eric, while you’re busting out some deeply wizardly (have I got that right?) repo-fu on NetBSD, any thoughts about doing or offering same for FreeBSD and OpenBSD?

    >Be happy to. Are you affiliated with either effort? Can you help deal with the political end?

    Sorry, no to both – was just wondering out loud. I apologise if I inadvertently led you up a garden path.

  21. @FrancisT:
    >why not just put the hosts you care about in /etc/hosts.allow and then have
    sshd: ALL
    in /etc/hosts.deny ?

    Because that’s how I learned to do it and it allows me to do a whitelist for a given program with one line in one file.

    Anyways, either way has the same trouble with connections from anywhere on my LAN.

  22. John Brase:

    Try “UseDNS no” in your sshd.conf file. It will stop sshd from checking that both forward and reverse hostname/IP match. THIS DOES LEAVE YOU VULNERABLE TO SPOOF ATTACKS, obviously, so make sure you’ve got mitigating controls (iptables, portknocking, etc.) or that you don’t mind if foo.badguys.net pretends to be you when they try to brute force you port 22.

    Sorry for continuing the off-topic thread, but I got excited that there was something in this discussion I could actually help with ;^).

  23. Thanks for the Fossil support. I use it for all my projects (internal) and while I have no intention to move, it’s nice to have the option.

    FYI for anyone working with Fossil or thinking about it, there’s a nice public Fossil host at http://chiselapp.com/ . Kind of github-y, I guess, but seems to be run as a donation to the Fossil community. (I’d happily donate some money in turn but there’s no ask nor mechanism.)

  24. Fossil is used for the Tcl/Tk project, and related libraries; these were ported from CVS. The fossil repository is at http://core.tcl.tk/

    I have found it very easy to deploy Fossil, and I plan to use it for my own open-source projects.

Leave a comment

Your email address will not be published. Required fields are marked *