Jan 23

Another bite of the reposturgeon

Five weeks ago I wrote that direct Subversion support in reposurgeon is coming soon. I’m waiting on one final acceptance test before I ship an official 2.0; in the meantime, for those of you kinky enough to find the details exciting, description follows of why this feature has required such a protracted and epic struggle. With (perhaps entertaining) rambling through the ontology of version control systems, and at least one lesson about good software engineering practice.

Continue reading

Dec 21

Are threads still a menace?

An interesting question showed up in my mailbox today. So interesting that I think it’s worth a public answer and discussion:

In chapter 7 of The Art of Unix Programming, you classified threads under the section “Problems and Methods to Avoid”. You also wrote that with the increased emphasis on thread-local storage, threads are looking more like a controlled use of shared memory. This trend has certainly continued; recent programming languages like D, Scala, and Go encourage the use of threads as mostly isolated lightweight processes with message passing. Observing this trend, I have often wondered, why not go all the way and use multiple OS processes? I can think of two reasons to use threads in this newer, controlled way rather than using full processes:

1. Portability to Windows, which doesn’t have an equivalent of fork(2)

2. Performance, particularly because message passing between real processes requires serialization and deserialization, whereas message passing within a process can be done with shared memory and (maybe) locks

So what do you think? Are threads still a menace to be avoided in favor of full OS processes? Or has the situation improved since 2003?

I think it has, and I think you’ve very nearly answered your own question as to why. Bare threads were dangerously prone to deadlocks, livelocks, context-trashing, and various other sorts of synchronization screwups – so language designers set out to encapsulate them in ways that gave better invariants and locality guarantees without sacrificing their performance advantages. I think Scala’s transactional memory stands out as a particularly elegant stab at the problem.

I don’t develop for Windows or communicate much with people who do, so I’m not equipped to judge how important Windows portability is in motivating these features. But the performance issue you called out is real and quite alive on Unix systems.

UPDATE: Matt Campbell, who has materialized in the comments here, send the original question and has given me permission to cite him. Thanks for a good question!

Dec 12

Coming soon: reposurgeon does Subversion

For those of you who have been following the development of reposurgeon, a pre-announcement: the next version, probably to be numbered 2.0, will directly read Subversion dumpfiles and repositories.

I’ve got this feature working now – it’s why my blogging has been scant recently – but I intend to have a really good regression-test suite in place and at least one large repo conversion done before I ship it for general use.

Note an important limitation: it will not write Subversion repos. So it will be useful as a conversion tool but not directly as an editor.

Fear the reposturgeon!

Nov 16

Against decentralized bugtracking

I’ve spent a lot of time and bandwidth on this blog thinking out loud about version-control systems and software forges. In my last post, I announced that I was going to try to sneak up on the problem of designing a better software forge by enhancing Roundup.

Over the last three years I’ve gotten a couple different versions of the following response to my thinking-out-loud: “Centralized forges and bugtracking are old-school thinking, as hoary as centralized VCSes. Why shouldn’t all that metadata live in the project repo and be peer-merged on demand the way code is?”

This is a good question, but I think the people advocating systems like Bugs Everywhere, scmbug, and ticgit have invested a lot of cleverness in the wrong answer.

Continue reading

Nov 14

Sneaking up on the forge problem

I’ve written before about the problems with today’s software-forge sites – how they’re craptacular piles of PHP driving direct SQL queries with almost zero scriptability that become data jails for open-source projects. I’ve hinted that I think there’s a potential solution based on Roundup, a brilliantly simple and powerful message queue manager disguised as a mere issue tracker.

Continue reading

Nov 14

Attack of the 50-foot reposturgeon

Well, I thought I was done hacking on this for a while. Then one of the projects I did a conversion for disclosed the existence of a second repo for their website, which I had to merge into the code repo. As a subdirectory. Which meant pushing all the file paths into a subdirectory. Which meant the new “paths sub” command; I wrote “paths sup” as its natural dual.

Also in this release: automatic preservation of untracked files under git and hg.

Fear the reposturgeon!

Nov 10

Reposturgeon from the Black Lagoon!

reposurgeon 1.8 is out, and with this release it has all the conversion features I’ve been able to think up while doing the last couple of conversions. This version creates real tags from the lightweight tags generated by git-svn, and also consolidates matched D/A pairs from Subversion into renames.

An “edit multiline” variant of the “edit” command zeroes in on commit comments that need to be tweaked into the approved form for hg and git (summary line, plus optional blank line, plus optional details).

The selection-set syntax has a new element: =H selects tip (or H for head) commits.

A new ‘sort’ command can make the DAG after a graft or merge display better in tools such as gitk.

With this release, I think I’m done for a while – barring bug reports, of course. I’m shipped a new version of my DVCS Migration Guide to go with it.

Fear the reposturgeon!

Nov 03

Names and consequences

freshmeat.net abruptly changed its name to freecode.com a couple of days ago. As a consequence, the little program I wrote to submit release announcements to it is now renamed freecode-submit.

People who ship releases frequently enough to find freecode-submit essential might also want to look at shipper, which I wrote to automate other aspects of release shipping as well.

shipper is how, when I want to ship a release of one of my projects, I can normally just type “make release” and the right things will happen – webpage updates, freecode release notification, SourceForge release, and release-tagging in the project repository.

Oct 30

RFC: Action stamps

This is a request for comment on a convention for uniquely identifying user actions on the Internet. The motivating context was identifying commit changesets in version-control systems in a way independent of the specific VCS. It is anticipated that this format will have uses in recording many other similar sorts of transactions, including actions on web interfaces, where we want a simple cookie identifying “who did this and when”.

Continue reading

Oct 29

Repositories in Translation

I’ve been doing a lot of repository conversions recently, lifting ancient project histories from Subversion or even CVS into modern distributed version control systems. I’ve written about the technical problems with these conversions elsewhere but they also raise issues that are almost philosophical – and not unlike, actually, the challenges natural-language translators face moving a literary work between human languages.

Continue reading

Oct 25

A DVCS migration howto

This is a consequence of my recent adventures in repository conversion – a detailed discussion of how to do a high-quality lift of a CVS or Subversion repo to DVCS-land, how to make both git and hg users happy, and what sorts of good practices to teach to keep things tidy.

Oct 23

Looking for reposurgeon test cases

I just released reposurgeon 1.2 and am continuing to develop the tool. In order to test some of the newer features, I’m looking for repository conversions to do. If you run an open-source project that is still using CVS or Subversion, or some odd non-distributed VCS, I may be willing to lift it to git for you (and from git to any other DVCS you might prefer is a pretty small step). Details of this offer follow; limited time only, first come, first served.

(Why have me do it? Well…especially for older projects with a complex revision history, it’s a messy and daunting job. The tools are somewhat flaky, the difference between a sloppy conversion and a good one is significant, and good conversions require experience and judgment.)

Continue reading

Oct 15

Ubuntu and GNOME jump the shark

I upgraded to Ubuntu 11.04 a week or so back in order to get a more recent version of SCons. 11.04 dropped me into the new “Unity” GNOME interface. There may be people in the world for whom Unity is a good idea, but none of them are me. The look is garish and ugly, and it takes twice as many clicks as it did before to get to an application through their supposedly “friendly” interface as it did in GNOME Classic. No, dammit, I do not want to text-search my applications to call one up!

But the real crash landing was when I found out that the Unity dock won’t let you manage two instances of the terminal emulator separately. Oh, you can click the terminal icon twice and get two instances, and even minimize them separately, but they’re tied to the same dock icon when minimized. If you click it to unminimize, both pop back up. That did it; clearly Unity is a toy, not intended for anybody doing serious work.

I was miserable until I found out how to fall back to GNOME Classic. But then a few days later I upgraded to 11.10 and my real troubles began.

Continue reading

Oct 13

You can tank me later

I have interesting friends. Two of them, who shall remain nameless because it is possible they have let slip to me information that is technically classified, recently told me the best GPSD deployment story since the robot submarine.

So, Friend A says “Hey, Eric, did you know GPSD is used in the on-board nav system of the Abrams tank?” Friend A is in a position to know, because Friend A has done troubleshooting of that nav system – once, over the phone with a tank actually in combat in Iraq. It seems GPSD is used as part of IFF (identification friend or foe) and without that module they are at unpleasant risk of heaving a shell at a friendly. (And no, I am assured the bug was not in GPSD itself.)

Continue reading