In my previous two posts I have diagnosed a significant weakness in the open-source infrastructure. The architecture of the code behind the major SourceForge-descended hosting sites is rotten, with all kinds of nasty consequences — data seriously jailed, poor or completely absent capabilities near scripting and project migration. I said I was going to do something about it, and I’m working the problem now — actually writing code.
The rest of this post is not an announcement, because it will be mostly about things I’ have figured out I should not try to do. Yet. But it is a teaser. I see a path forward, and shortly I expect to have some working code to exhibit that shows the way. Actually, I have working code that attacks the problem in an interesting way now, but I’m still adding capabilities to make it a more impressive demonstration.
Here are some approaches I’ve considered, or had suggested to me by others, and rejected:
1. Write a new forge system, focused on import/export and scriptability, from scratch. Tempting, but no. That would divert my energy for many months while the problem that originally exercised me — the data-jail effect — went unsolved. The first priority has to be jailbreaking the data in existing systems,
2. Rebuild Savane from the inside. Also tempting, and theoretically possible; I have developer privileges on that project, and it’s moribund – no commits in like two years. If I wanted to take it over, I probably could. Between gna.org and Savanna it has a pretty large userbase, enough to give a functional rewrite serious cred. But, again, it would divert me from my original gripe, which was the data-jail problem. Also, Savane’s architecture inherits the curse of SourceForge; trying to fix it while preserving its exact appearance and functionality would be painful in the extreme.
3. Finish the SOAP API in FusionForge, the most widely deployed ‘modern’ descendant of SourceForge. I’m a now project member there, though they haven’t given me commit privileges yet. I could fix their SOAP services API. The trouble is, their code is a disaster area worse than Savane’s — layers upon layers of cruft, so poorly integrated and maintained that their source tree doesn’t even have a working “make install”! I was told with a straight face that the preferred way to set up a running instance from source is to build a Debian binary package file from it and install the package. Oh, and just to put the cherry on top, something is broken in their repository — I couldn’t check out a complete source copy without running into some bizarre permissions-related error that hung my Subversion client, eating 100% of my processor. These failures cause me to doubt that the project is sufficiently well run to be a good investment of my time.
4. Write a data-interchange standard, then jawbone existing forges into implementing exporters that speak it. This is what the crowd of research types around COCLICO in France wants to do. I think it’s a doomed effort; if the ‘existing systems’ had a strong enough architecture to support export capabilities that don’t suck we probably wouldn’t have this problem in the first place. On top of that there’s the problems that writing an exporter for an existing forge requires intimate knowledge of the festering crap behind the web interfaces, and deploying it would require install privileges on the forge site. Ain’t going to happen — the site admins will, quite justifiably, wonder what the point of disturbing their running installations and accepting the inevitable security risks is when nothing yet exists that can read the exports! COCLICO’s is a typical over-ambitious academic approach in which you have to solve everything before it solves anything….
If these approaches won’t work, what will?
It’s too soon to try to prescribe a standard data-interchange format for forges, but it’s not too soon to write tools that jailbreak the data out of forges and dump it in formats that are forge-type-specific but human-readable — self-describing JSON or XML dumps, rather than binary blobs.
Also, an ugly fact has to be faced — because of the PHP+SQL mess inside these things, the only viable approach to extracting the data out of them is to use the same web interfaces humans do. Once you’ve faced that fact, though, and realized you’re going to have to build a very smart robotic web scraper, the approach solves a significant set of problems. Most notably, it requires neither cooperation nor competence from any forge site administrators, anywhere.
That’s enough hints. You’ll hear from me on this next when I announce some working tools.