May 13

Engineering zero-defect software

I’ve been pounding on GPSD with the Coverity static analyzer’s self-build procedure for several days. It is my great pleasure to report that we have just reached zero defect reports in 72.8KLOC. Coverity says this code is clean. And because I think this should be an example unto others, I shall explain how I think others can do likewise.

Continue reading

May 06

Falling through the cracks: why GPSD sometimes bogarts non-GPS devices

In a recent Google+ comment, H. Peter Anvin grumped about GPSD using “braindead heuristics” to determine which USB devices it should sniff as possible GPses when it gets a hotplug notification saying that one has connected. I was going to reply in a comment there, but the explanation ran too long for that.

Short version: yes, GPSD will very occasionally sniff at a device that is none of its business. We’re stuck in a bad place because of deficiencies in the USB standard, But it doesn’t happen often, and all the alternative behaviors I’ve been able to imagine would be worse in very obvious ways. Detailed explanation follows.

Continue reading

Apr 25

The long past of C

Hacking on the C code of giflib after an absence of nearly two decades has been an interesting experience, a little like doing an archeological dig. And not one that could readily be had elsewhere; nowhere other than under Unix is code that old still genuinely useful under any but carefully sandboxed conditions. Our reward for getting enough things about API design right the first time, one might rightly say. But what’s most interesting is what has changed, and giflib provides almost ideal conditions for noticing the changes in practice that have become second nature to me while that code has stood still.

Continue reading

Apr 21

giflib: everything old is new again

In 1994 I handed off the maintainership of giflib, the open-source library used by pretty much everything in the universe that displays images for the single most widely used icon and image format on the World Wide Web, because patent issues made it unwise for the project to be run by someone in the U.S. Now, eighteen years later, Toshio Kuratomi (the hacker who took it over then) has asked me to resume the lead. I have accepted his request.

Continue reading

Feb 16

Super Star Trek 2.1

I guess it’s paleo-game theme week. For your retrocomputing pleasure, here’s my Python forward-port of the 1973 University of Texas FORTRAN Trek game: Super Star Trek.

Anybody old enough to remember TTYs probably played this on one. While it has accreted some features over time, it’s still functionally pretty close to the original FORTRAN Star Trek. You kids should get off my lawn try it, too – it retains considerable play value despite the primitive interface.

Jan 23

Another bite of the reposturgeon

Five weeks ago I wrote that direct Subversion support in reposurgeon is coming soon. I’m waiting on one final acceptance test before I ship an official 2.0; in the meantime, for those of you kinky enough to find the details exciting, description follows of why this feature has required such a protracted and epic struggle. With (perhaps entertaining) rambling through the ontology of version control systems, and at least one lesson about good software engineering practice.

Continue reading

Dec 21

Are threads still a menace?

An interesting question showed up in my mailbox today. So interesting that I think it’s worth a public answer and discussion:

In chapter 7 of The Art of Unix Programming, you classified threads under the section “Problems and Methods to Avoid”. You also wrote that with the increased emphasis on thread-local storage, threads are looking more like a controlled use of shared memory. This trend has certainly continued; recent programming languages like D, Scala, and Go encourage the use of threads as mostly isolated lightweight processes with message passing. Observing this trend, I have often wondered, why not go all the way and use multiple OS processes? I can think of two reasons to use threads in this newer, controlled way rather than using full processes:

1. Portability to Windows, which doesn’t have an equivalent of fork(2)

2. Performance, particularly because message passing between real processes requires serialization and deserialization, whereas message passing within a process can be done with shared memory and (maybe) locks

So what do you think? Are threads still a menace to be avoided in favor of full OS processes? Or has the situation improved since 2003?

I think it has, and I think you’ve very nearly answered your own question as to why. Bare threads were dangerously prone to deadlocks, livelocks, context-trashing, and various other sorts of synchronization screwups – so language designers set out to encapsulate them in ways that gave better invariants and locality guarantees without sacrificing their performance advantages. I think Scala’s transactional memory stands out as a particularly elegant stab at the problem.

I don’t develop for Windows or communicate much with people who do, so I’m not equipped to judge how important Windows portability is in motivating these features. But the performance issue you called out is real and quite alive on Unix systems.

UPDATE: Matt Campbell, who has materialized in the comments here, send the original question and has given me permission to cite him. Thanks for a good question!

Dec 12

Coming soon: reposurgeon does Subversion

For those of you who have been following the development of reposurgeon, a pre-announcement: the next version, probably to be numbered 2.0, will directly read Subversion dumpfiles and repositories.

I’ve got this feature working now – it’s why my blogging has been scant recently – but I intend to have a really good regression-test suite in place and at least one large repo conversion done before I ship it for general use.

Note an important limitation: it will not write Subversion repos. So it will be useful as a conversion tool but not directly as an editor.

Fear the reposturgeon!

Nov 16

Against decentralized bugtracking

I’ve spent a lot of time and bandwidth on this blog thinking out loud about version-control systems and software forges. In my last post, I announced that I was going to try to sneak up on the problem of designing a better software forge by enhancing Roundup.

Over the last three years I’ve gotten a couple different versions of the following response to my thinking-out-loud: “Centralized forges and bugtracking are old-school thinking, as hoary as centralized VCSes. Why shouldn’t all that metadata live in the project repo and be peer-merged on demand the way code is?”

This is a good question, but I think the people advocating systems like Bugs Everywhere, scmbug, and ticgit have invested a lot of cleverness in the wrong answer.

Continue reading

Nov 14

Sneaking up on the forge problem

I’ve written before about the problems with today’s software-forge sites – how they’re craptacular piles of PHP driving direct SQL queries with almost zero scriptability that become data jails for open-source projects. I’ve hinted that I think there’s a potential solution based on Roundup, a brilliantly simple and powerful message queue manager disguised as a mere issue tracker.

Continue reading

Nov 14

Attack of the 50-foot reposturgeon

Well, I thought I was done hacking on this for a while. Then one of the projects I did a conversion for disclosed the existence of a second repo for their website, which I had to merge into the code repo. As a subdirectory. Which meant pushing all the file paths into a subdirectory. Which meant the new “paths sub” command; I wrote “paths sup” as its natural dual.

Also in this release: automatic preservation of untracked files under git and hg.

Fear the reposturgeon!

Nov 10

Reposturgeon from the Black Lagoon!

reposurgeon 1.8 is out, and with this release it has all the conversion features I’ve been able to think up while doing the last couple of conversions. This version creates real tags from the lightweight tags generated by git-svn, and also consolidates matched D/A pairs from Subversion into renames.

An “edit multiline” variant of the “edit” command zeroes in on commit comments that need to be tweaked into the approved form for hg and git (summary line, plus optional blank line, plus optional details).

The selection-set syntax has a new element: =H selects tip (or H for head) commits.

A new ‘sort’ command can make the DAG after a graft or merge display better in tools such as gitk.

With this release, I think I’m done for a while – barring bug reports, of course. I’m shipped a new version of my DVCS Migration Guide to go with it.

Fear the reposturgeon!

Nov 03

Names and consequences abruptly changed its name to a couple of days ago. As a consequence, the little program I wrote to submit release announcements to it is now renamed freecode-submit.

People who ship releases frequently enough to find freecode-submit essential might also want to look at shipper, which I wrote to automate other aspects of release shipping as well.

shipper is how, when I want to ship a release of one of my projects, I can normally just type “make release” and the right things will happen – webpage updates, freecode release notification, SourceForge release, and release-tagging in the project repository.

Oct 30

RFC: Action stamps

This is a request for comment on a convention for uniquely identifying user actions on the Internet. The motivating context was identifying commit changesets in version-control systems in a way independent of the specific VCS. It is anticipated that this format will have uses in recording many other similar sorts of transactions, including actions on web interfaces, where we want a simple cookie identifying “who did this and when”.

Continue reading

Oct 29

Repositories in Translation

I’ve been doing a lot of repository conversions recently, lifting ancient project histories from Subversion or even CVS into modern distributed version control systems. I’ve written about the technical problems with these conversions elsewhere but they also raise issues that are almost philosophical – and not unlike, actually, the challenges natural-language translators face moving a literary work between human languages.

Continue reading