I’ve been wanting to migrate the GPSD codebase off Subversion to a distributed version control system for many months now. GPSD has a particular reason for DVCS; our developers often have to test GPSD sensors outdoors and aren’t necessarily in range of WiFi when they do it.
GPSD also needs to change hosting sites, for reliability reasons I’ve written about before. Though I’m a fan of Mercurial, I determined that moving to git would give us a wider range of hosting options. Also, git and hg are similar enough to make intermigration really easy – from SVN to either is 90% of the way to the other.
This blog entry records two problems I ran into, and solutions for them. One is that the standard way of converting repos does unfortunate things with tags directories. The second is that the CIA hook scripts for git are stale and rather broken.
GPSD uses tags directories strictly for archival purposes. When we cut a public release XX, we make a tag with the name “release-${XX}” and never modify that tree copy afterwards. We don’t use tags as branches. So, when I migrated, I wanted the release tags to be mapped into git tag objects rather than branches.
Unfortunately, the git-svn extension can’t do this; it will turn your tags into git branches. I’m told svn2git has the same behavior. Here’s what I ended up doing:
git svn clone –stdlayout –no-metadata file://${PWD}/stage2-repo
The –stdlayout tells git-svn that the project has a stock SVN layout with trunk, tags and branches. It will tell the fetch operation to turn both tags and branches into git branches, then strip those three prefixes out of the repo paths. Then I did this:
git svn –ignore-paths=”tags” fetch
This prevented the tags directories from being turned into branches. But it meant I had to make the git
symbols by hand. I wrote a script to extract the rev levels that looked like this:
#!/bin/sh
#
# Get a table of tag releases and dates from a checkout directory
dir=$1/tags
for x in $dir/*
do
base=`basename $x`
info=`svn info $x | grep Last`
rev=`echo $info | sed -n '/.*Last Changed Rev: \([0-9]*\).*/s//\1/p'`
date=`echo $info | sed -n '/.*Last Changed Date: \(...................\).*/s//\1/p'`
echo "$base\t$rev\t$date"
done
Running it gave me a table that looked like this:
release-2.21 1566 2005-04-12 20:10:40 release-2.22 1592 2005-04-25 17:01:53 release-2.23 1637 2005-05-04 14:07:39 release-2.24 1688 2005-05-17 12:48:47 release-2.25 1737 2005-05-21 00:19:51
The columns are tag name, Subversion revision level, and date-time stamp. I then went through the SVN and git versions of the logs and added git IDs as a fourth field. that gave me a file that looked (in part) like this:
release-2.21 1566 2005-04-12 20:10:40 1dd11f752275842a220ce5b2b93da2e2fa31a53c release-2.22 1592 2005-04-25 17:01:53 d11c967125b8432e7d906fba18d67b3b2e7feaad release-2.23 1637 2005-05-04 14:07:39 70e3d9e0ed7e2676554735ccfce8a4dd46b8bd9c release-2.24 1688 2005-05-17 12:48:47 4fcd4e7bebfbf587c2889657ba94b79f6ace2859e release-2.25 1737 2005-05-21 00:19:51 e2816c19964d124381a90d9338530a17ce47d43
Note: I’d have written a tool to generate the entire thing, but I estimated that for only about 45 tags it would take less time to hand-hack the list. Then I applied the following script:
#!/usr/bin/env python
#
# Apply a table of tag releases and debates from a checkout directory
#
import sys, os
for line in file(sys.argv[1]):
(release, rev, date, time) = line.split()
os.foo('GIT_COMMITTER_DATE="%s %s" git tag -m "Tag for public release." %s' % (date, time, release))
The “foo” in there should actually be the word “system”, but if written that way WordPress thinks it’s an attempt at malicious code injection and barfs.
I think this was more work than I should have had to do. When stdlayout is enabled, the conversion tools should know that SVN tags have different semantics than SVN branches and automatically lift to tag symbols if the tag tree has not been modified.
The second problem I ran into is that the git hook scripts CIA.vc supplies for git are badly out of date. Modern git installations don’t put all the helper commands in the normal $PATH; I had to add these lines to the shell hook script to make it work.
export PATH
PATH=”$PATH:`git –exec-path`”
The Perl script has a similar problem. Investigating further I found that the CIA local copies of these scripts are very stale; they need to refresh from the upstream maintainers.
These were just speedbumps; the git repo is working fine now, and I’ve shut down SVN. I hope this note will be good googlebait for anyone who trips over the same problems.
UPDATE: The author of the gitorious project svn2git alleges that his tool can do this. But you have to write a rules file, and he admits the tool is “not well documented”. Do not confuse this with the tool of the same name on github, which is but a thin wrapper around git-svn…
UPDATE2: I fixed the CIA git scripts. They live in the official git repo now.
UPDATE3: This comment is correct. Because I understood git internals poorly at the time, I missed a simpler way to do this job. Allow git-svn to do the conversion of tags into branches and then just move the tag files!
Since you’re on the topic, how about a little thought about the ‘best’ way to handle a scenario. I work for a company that has an open source project (Foo), plus proprietary and ITAR-controlled software that works with/on said project. Not all of the developers on the Foo project work for the company, so they don’t have access to any servers inside the company firewall. No one is familiar with git or Mercurial. There is more familiarity with Subversion. We want to have an external read-only repository for community access; we must have separately controlled repositories for the internal open source, proprietary and ITAR-controlled work. Currently we have four Subversion repositories (1 outside, 3 inside), but mirroring between the inside and the outside repository is … awkward/difficult. Developers work on Mac OS X, Linux, and Windows, primarily building software that targets embedded Linux (PPC) and FPGAs.
So, with all that in mind, what part, if any, do you think git (or Mercurial) could play in improving our SCM? Using git (or something like it) would make sense for allowing external developers to develop and merge changes, but there isn’t a lot of support for any SCM, much less distributed SCM, and whatever solution we choose, we have to be able to demonstrate to our primary customer that no external changes get into the controlled code base without having been manually vetted.
Thoughts?
>Since you’re on the topic, how about a little thought about the ‘best’ way to handle a scenario.
First: Cross-platform tilts you towards Mercurial. Windows git support ain’t so great, is what I hear.
Second: This kind of mirror-and-controls setup is one of the best arguments for moving off Subversion. DVCSes handle such situations naturally, it’s a lot like the loosely-coupled-workgroup setups they were designed for. Subversion can’t handle this gracefully.
Third: reading over your requirements a third time, I think what you should go for is to replace those Subversion repos with Mercurial repos, then pay careful attention to the precommit hook on each one. That’s where your security policy goes; get the code right and you’ll have your guarantee.
I think this more of a historical issue. I’ve used git personally on both windows and linux for about 2 years now and the only real problem i’ve come across is the linefeed issue. Basically you have to make sure that anything that gets checked in uses linefeeds. Files that use CRLF break the diff. Github suggests that everyone uses autocrlf just to be on the safe side. Having said that the windows version IS a second class citizen so its something to keep in mind.
Mercurial may also make more sense if resistance to learning a new system is an issue.
@JonB: EOL issues aren’t a huge problem if one uses text editors and text processing tools that support auto-conversion between LF and CRLF EOLs. Emacs, VIM, Eclipse, and Notepad++ are all examples of editors that run on Windows that do.
My experience of late is that git on Windows is not something to rely on. Random errors and problems seem to be the order of the day. IMHO, the fact that it relies on Mingw/MSYS is a big liability. Mercurial, on the other hand, seems to be firmly in “just works” territory.
@Michael Hipp: I’ve not used git on Windows (I run Ubuntu on all my home office machines), but I’ve heard that there are “Windows API issues” and lack of hackers willing to work on them.
@morgan: I’ve not dug into it in any depth, but I suspect you’re right, especially on the last point. There doesn’t seem to be much community around it.
And as I’ve learned from ESR, software is a service, not a product. So my main criteria for using a given piece of oss is what I can learn of the depth/strength of the community around it. The fact that I have full access to the source is (almost, not quite) irrelevant if there is no community to fall back on.
Hi Eric, have you found any benefits of using a versioning system for a simple, single-user project?
I am creating a website management front-end in Python, and I’m using git right now, but I’m doing nothing more than making changes and committing them into the repository.
I know it sounds dumb, but I’ve not really been fully acquainted with versioning systems and beyond the basic usage, I’ve not found any use for it so far.
git has always interested me because of its proclaimed benefits over SVN… how do I get more comfortable and productive with version control systems in general? I’ve done programming for years and always felt shy of using these tools.
>Hi Eric, have you found any benefits of using a versioning system for a simple, single-user project?
Yes. I use version control for everything, so I can revert to a previous state in case of editing errors if nothing else.
(I’m not Eric.) Of course there are benefits to using version control. Simply tracking your changes for debugging purposes is important on any piece of code of any of significant size. I work mostly alone, using Git, and I create branches for experimentation all the time. I find it much better than working on a separate copy.
There’s a ton of Git documentation, tutorials, screencasts and whatnot these days. I suppose you get more comfortable by practice. There’s certainly no shortage of material on Git.
Thanks. Actually, one of the issues I face is that I don’t commit often enough when I am in the middle of a coding frenzy. I lose the finer-grained tracking of source control. I am working on it; I think that because I’ve never worked on a collaborative project (I’m more of a reclusive hobbyist) I have not got into the groove of using a VCS.
Another thing: I found subversion too bulky for home projects, so I preferred to use git. Surprisingly I find git a lot more understandable and usable while most people on mailing lists seem to indicate that SVN is the easiest versioning system around.
Another big benefit to using SCM on single-developer projects is that it keeps you in practice and disciplined. It’s too easy, when you’re working alone, to fall into bad habits; using a SCM means that you’re coding for anyone who can view the SCM logs, or checkout a version. Add a build orchestration server (eg: Hudson or CruiseControl) and unit / regression tests and you know that your tests will be run every time you commit a change … suddenly it makes sense to write more useful tests.
Thanks. Yes, that is probably a good enough reason to use a VCS. I think it probably doesn’t matter whether I use a DVCS or a traditional one. My brother swears by perforce and now svn though.
@hari: There’s nothing wrong with Subversion for small projects. It’s only once your project tries to scale up to more developers, more branches, etc., that you start to run into the limitations of a VCS like Subversion or CVS. Even GPSD could probably go on using Subversion for the forseeable future if it weren’t for the fact that the devs just wanted be able to work offline. OTOH, As you can see by this post, the difficulty of migrating between version control systems is a consideration if you think your project could one day become much more involved than a single-developer project.
Sorry if this is getting further OT, but along the lines of hari’s topic (I’m currently learning to work with a VCS, git in particular, as well)- is it unusual or bad form to commit code changes that are incomplete? You know, you’re quietly coding along and then one of the kids projectile vomits in your general direction, causing a minor disturbance in the force…
@Gerry: I think the answer to that question strongly depends on the standards of the community/project you are collaborating on. Obviously if you have no co-developers, committing broken code rather than risking losing changes is trivial. On a project with many developers and after-commit hooks that run builds, unit tests and regression tests automatically, it is probably better to avoid making commits outside a branch until they are fully ready locally.
Utterly OT and made me laugh:
Found on http://twitter.com/perlbuzz
RT @artmonstergirl I like my regular expressions same as my men, powerful, yet confusing, and not really quite what I intended.
Yours,
Tom
This was my experience, too. I never liked CVS or SVN and Git very much seemed like ‘this is the way it was always supposed to be’ when I first used it. Then again, I was never a heavy SVN user. I suppose it’s just the way my brain happens to be. I work with some people who write much more code than I do and deal with large code bases and multiple developers. They didn’t seem too impressed by Git and they haven’t switched from SVN despite having no bureaucratic restrictions keeping them on SVN.
You don’t have to merge your whole messy local history to the shared tree. This is the whole rebase vs. merge debate. Linus himself has given guidelines for the kernel about this at length. Git does give you the freedom to get as messy as you want in your local repository and then only merge a clean, edited history to the shared branch.
I just wanted to thank everybody here for their inputs. I’ve now got bold enough to create an account with gitorious and share my latest project there.
I have numerous small projects that I work on from different places – my home desktop machine, my office desktop and my laptop. In addition, some of the projects are deployed on a stationary server. I use mercurial as my DVCS for all of these (git and bzr would work just as well, but I found mercurial to be marginally closer to my tastes). This setup provides me with a really simple way of moving the work to my workplace for the day and a simple way to deploy new versions of the projects.
SVN would not be very good in my environment. It requires a central repository which is slightly messy to set up. It also requires that I am in contact with the central repository whenever I want to move my work from one machine to another, or if I want a fresh copy of the repository for some experiment. Another great advantage is that I can give away a copy of my repository (with all its history) to somebody else, and any further development in this repo will be interoperable with my own repo. If you give someone a copy of your central SVN repo, you are limited to communicating through patches between the repositories. There is no way you can automatically sync them.
in order to turn tag branches into tags i simply moved the tag refs in .git from the directory where git-svn put them (refs/remotes/tags usually) to refs/tags.
greetings, eMBee.