How Not To Tackle the Mess around Forges

In my previous two posts I have diagnosed a significant weakness in the open-source infrastructure. The architecture of the code behind the major SourceForge-descended hosting sites is rotten, with all kinds of nasty consequences — data seriously jailed, poor or completely absent capabilities near scripting and project migration. I said I was going to do something about it, and I’m working the problem now — actually writing code.

The rest of this post is not an announcement, because it will be mostly about things I’ have figured out I should not try to do. Yet. But it is a teaser. I see a path forward, and shortly I expect to have some working code to exhibit that shows the way. Actually, I have working code that attacks the problem in an interesting way now, but I’m still adding capabilities to make it a more impressive demonstration.

Here are some approaches I’ve considered, or had suggested to me by others, and rejected:

1. Write a new forge system, focused on import/export and scriptability, from scratch. Tempting, but no. That would divert my energy for many months while the problem that originally exercised me — the data-jail effect — went unsolved. The first priority has to be jailbreaking the data in existing systems,

2. Rebuild Savane from the inside. Also tempting, and theoretically possible; I have developer privileges on that project, and it’s moribund – no commits in like two years. If I wanted to take it over, I probably could. Between gna.org and Savanna it has a pretty large userbase, enough to give a functional rewrite serious cred. But, again, it would divert me from my original gripe, which was the data-jail problem. Also, Savane’s architecture inherits the curse of SourceForge; trying to fix it while preserving its exact appearance and functionality would be painful in the extreme.

3. Finish the SOAP API in FusionForge, the most widely deployed ‘modern’ descendant of SourceForge. I’m a now project member there, though they haven’t given me commit privileges yet. I could fix their SOAP services API. The trouble is, their code is a disaster area worse than Savane’s — layers upon layers of cruft, so poorly integrated and maintained that their source tree doesn’t even have a working “make install”! I was told with a straight face that the preferred way to set up a running instance from source is to build a Debian binary package file from it and install the package. Oh, and just to put the cherry on top, something is broken in their repository — I couldn’t check out a complete source copy without running into some bizarre permissions-related error that hung my Subversion client, eating 100% of my processor. These failures cause me to doubt that the project is sufficiently well run to be a good investment of my time.

4. Write a data-interchange standard, then jawbone existing forges into implementing exporters that speak it. This is what the crowd of research types around COCLICO in France wants to do. I think it’s a doomed effort; if the ‘existing systems’ had a strong enough architecture to support export capabilities that don’t suck we probably wouldn’t have this problem in the first place. On top of that there’s the problems that writing an exporter for an existing forge requires intimate knowledge of the festering crap behind the web interfaces, and deploying it would require install privileges on the forge site. Ain’t going to happen — the site admins will, quite justifiably, wonder what the point of disturbing their running installations and accepting the inevitable security risks is when nothing yet exists that can read the exports! COCLICO’s is a typical over-ambitious academic approach in which you have to solve everything before it solves anything….

If these approaches won’t work, what will?

It’s too soon to try to prescribe a standard data-interchange format for forges, but it’s not too soon to write tools that jailbreak the data out of forges and dump it in formats that are forge-type-specific but human-readable — self-describing JSON or XML dumps, rather than binary blobs.

Also, an ugly fact has to be faced — because of the PHP+SQL mess inside these things, the only viable approach to extracting the data out of them is to use the same web interfaces humans do. Once you’ve faced that fact, though, and realized you’re going to have to build a very smart robotic web scraper, the approach solves a significant set of problems. Most notably, it requires neither cooperation nor competence from any forge site administrators, anywhere.

That’s enough hints. You’ll hear from me on this next when I announce some working tools.

31 thoughts on “How Not To Tackle the Mess around Forges

  1. Have you perhaps taken a glance at fossil (another commenter in the previous post had mentioned it)?

    It’s basically a distributed version control with built in wiki and bug tracking. Similar to how it’s near impossible with the current generation SCMs (git, bazaar, mercurial, etc) to keep source code history locked in (since every clone (similar to “checkout” in previous-gen ones like CVS or Subversion) is a complete copy of all the history), this expands it further to distribute the entire project management. It’s useful for both keeping your data in your control and working offline (squash known bugs without having online access to the bug-tracker!)

    It’s not something I’m sure many projects would really appreciate, being that you have you bring your own web server. Despite that hosting (including unique domain name) is dirt-cheap… there’s plenty of people that still wouldn’t settle for anything more costly than free.

    ESR says: Fossil is on my list of forge systems to take a serious look at. It’s been described to me as technically brilliant but barely usable. Also, it’s written in C, which at this point in the game is a serious drawback for this class of program – gets a performance gain you’ll never really notice while playing hell with maintainability and hackability.

  2. Doesn’t this work..? Obviously, this leaves room for many of the improvements you suggest, but if the page is accurate it’s hardly fair to say that SourceForge is a ‘data jail’.

  3. Hi, Roland from the FusionForge project here. I’m a little surprised at this blog entry, and I’d like to correct a few inaccuracies.

    1. FusionForge doesn’t have a “make install”, indeed. It does have an installer script, though, as well as packaging for Debian-based and RPM-based distros. I’m not sure writing the installation script as a Makefile would make sense, since the actual deployment of files is a *tiny* part of the installation process. The hard stuff is setting up and upgrading the database, configuring the Apache server, insinuating the forge into the MTA and the authentication system, and so on.

    2. As for the broken repository, it’s the first time I’ve heard of it. Please come (back) to IRC and we’ll see to debugging that, just complaining on a blog isn’t helpful.

    3. COCLICO isn’t an academic project, thank you very much. We do have a university and a research centre in it, but we also have two large companies and a handful of smaller ones (including my own), with a firm intention of getting concrete results. Including importers and exporters for the major existing forges, which will be included in a future release of their code. This means that the site admins will get them for free on upgrade.

    4. Also, to reiterate what’s already been told on IRC, the web interface thingy is not the only way to get at the data, and I resent your calling an API “festering crap” before having used it. You may not like PHP (can’t say I’m too fond of it myself), but FusionForge does have a proper object-oriented API. And some SOAP using it.

    I’m sure we’d welcome your help, though, both in FusionForge and COCLICO.

  4. Your last post inspired me to poke around a bit. I found Tiddlyforge. Doesn’t look like it’s usable for anything but small teams, but might be a promising way to go.

  5. Thank you for looking at fossil.

    I wrote fossil for my own needs, which it serves admirably. If others find it useful too, then so much the better. If you find it useful, I’m happy for you to use it. If you prefer a different approach, you will not hurt my feelings. I do enourage you, however, to at least give some attention to the ideas behind fossil, and even if you don’t like fossil itself, consider reimplementing some of those ideas in other systems (git, mercurial, something new). I consider the ideas behind fossil to be much more important than the implementation.

    Some corrections: Fossil does *not* require a server. If you want to publish your project to the world you will certainly need a server of some kind to publish on, but that is true of anything, right? If, however, you just want to manage your own personal projects, no server is required. Simply type “fossil ui” to bring up the web interface directly on your desktop.

    Regarding the choice of C: Early prototypes of fossil were done in scripting languages. That experience taught me that C really is the best language choice for something like fossil. Most of the work involves low-level byte manipulations (SHA1 hashes, delta encodings, etc.) which are much easier to do in C than in “higher level” languages. Furthermore, the use of C allows fossil to be a self-contained, stand-alone binary without a dependency of a particular script interpreter and its associated supporting libraries. C was not chosen for performance. In fact, fossil makes no effort to be fast. (It is “fast enough”, however.) Fossil values correctness over performance, since I see no benefit in getting the wrong answer quickly.

    Thanks again for looking at fossil. Please consider contributing if you are interested. If not, please at least consider taking some of the ideas expressed in fossil and reimplementing them in other systems.

  6. >Doesn’t this work..? Obviously, this leaves room for many of the improvements you suggest, but if the page is accurate it’s hardly fair to say that SourceForge is a ‘data jail’.

    Nice table. Now show me how to get all of the project data out in a form I can read. Say I want to look at the members table for my project? The only way to do that would be to take the database dump, load it into a running MySQL instance, then hand-craft the right SQL query and write a custom report generator.

    It’s nice that they do periodic dumps of pieces of the structure, but how do I get all the pieces made at the same time?

    That list of facilities is a nice gesture, but nothing more

  7. “It’s too soon to try to prescribe a standard data-interchange format for forges, but it’s not too soon to write tools that jailbreak the data out of forges and dump it in formats that are forge-type-specific but human-readable — self-describing JSON or XML dumps, rather than binary blobs.”

    Planning something like fetchmail –configdump?

    ESR says: Bwahaha. I’m already well beyond the “planning” stage.

  8. >1. FusionForge doesn’t have a “make install”, indeed. It does have an installer script, though, as well as packaging for Debian-based and RPM-based distros. I’m not sure writing the installation script as a Makefile would make sense, since the actual deployment of files is a *tiny* part of the installation process. The hard stuff is setting up and upgrading the database, configuring the Apache server, insinuating the forge into the MTA and the authentication system, and so on.

    Indeed. Now, where are the installation instructions? I couldn’t find any. And I was told on IRC that the best way to do it is to build a deb and install the deb. FAIL.

    >2. As for the broken repository, it’s the first time I’ve heard of it. Please come (back) to IRC and we’ll see to debugging that, just complaining on a blog isn’t helpful.

    You’re right. I went back to the IRC channel a few days ago and complained there. Nobody paid any attention, and I didn’t get a solution. FAIL

    >3. COCLICO isn’t an academic project, thank you very much. We do have a university and a research centre in it, but we also have two large companies and a handful of smaller ones (including my own), with a firm intention of getting concrete results. Including importers and exporters for the major existing forges, which will be included in a future release of their code. This means that the site admins will get them for free on upgrade

    Good luck. It will be wonderful if you succeed. Pardon me if I don’t hold my breath.

    >4. Also, to reiterate what’s already been told on IRC, the web interface thingy is not the only way to get at the data, and I resent your calling an API “festering crap” before having used it. You may not like PHP (can’t say I’m too fond of it myself), but FusionForge does have a proper object-oriented API. And some SOAP using it.

    Yes, I’m aware of the SOAP API. I’d be more impressed if (a) I could check out the code, (b) I could build the code. What I’ve seen so far does not fill me with confidence in what I will find when I examine the SOAP API in detail.

    >I’m sure we’d welcome your help, though, both in FusionForge and COCLICO.

    You might get some help from me yet. But you’ve got a lot to do to clean up your act, first. Right now, from the outside, FusionForge looks desperately badly run and COCLICO looks like a grant-extraction machine destined to get lost in a cloud of AI-complete problems.

  9. 1. Installation instructions are on the wiki, as mentioned on IRC.

    2. IRC is a synchronous communication medium. Not everyone is in your timezone (most of the FF developers are in Europe), and not everyone spends their whole weekends on IRC.

    3. COCLICO has already succeeded in at least one point: Bull’s document manager has been contributed, and it’ll probably be merged into FusionForge soonish. Convergence is a big part of the project. Data migration is another.

    4. You’re aware that the code is PHP and there’s nothing to “build”, right? You’re aware of the Bazaar and Git gateways that allow you to check out the code (including a Bazaar mirror at Launchpad and a Git one at Github), of course? You’re aware that prebuilt packages are available for several versions of Debian and Ubuntu, including in these distributions’ official repositories? All that is documented on the wiki. It’s not very big yet, it’s linked from the homepage, and it’s online even when we lazy Europeans get some rest.

    I’m not commenting on the web scraping idea. Just trying to get you to evaluate a project by spending some time on it before dissing it out. What you do with your time is of course entirely your choice, but please show some respect to how we spend ours.

  10. Also, it’s written in C, which at this point in the game is a serious drawback for this class of program – gets a performance gain you’ll never really notice while playing hell with maintainability and hackability.

    An attitude like that will get you where OLPC is with Sugar. Much of the underlying software is written in Python and it’s insufferably slow. Also, Yum (also written in Python) falls over repeatedly and easily when trying to install anything more complicated than GNU Hello World (now with readline support).

    You’d have a point if the language under discussion were C++. But well-written C is very readable and should be within the capability of any decent programmer to maintain and modify. And yes, performance is important, especially today when we’re trying to reduce our carbon footprints.

  11. Also, Yum (also written in Python) falls over repeatedly and easily when trying to install anything more complicated [...]

    Exactly how is that the fault of the code being written in Python? Do you mean that since Python is easier to write, it attracted bad code? “Well written Python is very readable and should be within the capability of any decent programmer to maintain and modify”.

    I agree on the performance issue, though I’d still like to actually profile the stuff before blanket-stating that Python is the issue. “Code the performance-challenged stuff in C, and use Python as glue”, and all that.

  12. It’s too soon to try to prescribe a standard data-interchange format for forges, but it’s not too soon to write tools that jailbreak the data out of forges and dump it in formats that are forge-type-specific but human-readable — self-describing JSON or XML dumps, rather than binary blobs.

    Yes… but unless you also have a (necessarily forge-specific?) importer for this data as well, how is this significantly better than the sort of dumps you can get out of sourceforge? Clearly it’s an enabler for that, but am I missing something else here?

    I’ll also add that IMAO the Fossil project, whatever its implementation issues, has the right idea. Given modern bandwidth and storage costs, there’s little reason every developer shouldn’t keep a full copy of the project data, and (in the real sign of a correct architecture) it solves most of ESR’s original forge complaints almost as a side-effect.

  13. ESR> Nice table. Now show me how to get all of the project data out in a form I can read. Say I want to look at the members table for my project? The only way to do that would be to take the database dump, load it into a running MySQL instance, then hand-craft the right SQL query and write a custom report generator.

    ESR> It’s nice that they do periodic dumps of pieces of the structure, but how do I get all the pieces made at the same time?

    ESR> That list of facilities is a nice gesture, but nothing more

    I already acknowledged that there’s room for improvement. However, you can still hardly call SourceForge a ‘data jail’ when they have export facilities for all of the most important stuff. Recreating the member list is probably going to be the least of your worries if SourceForge dies. Your complaint that SF doesn’t cook your data enough is irrelevant to its status as a data jail. If you can get your data out periodically, it is not a jail; broken maybe, but not a jail.

  14. Scrape?

    If you have access to the db schema, why not just bypass the web and get the data directly? With that, html is merely a view on the underlying model.

    Now on to read the rest of this thread–its good fun.

    Cordially,

    __t_i_m_o_t_h_y__m_u_r_r_ay__

  15. >Yes… but unless you also have a (necessarily forge-specific?) importer for this data as well, how is this significantly better than the sort of dumps you can get out of sourceforge?

    Er. have you ever tried mining data out of an SQL dump written in an unknown schema?

    The other advantage is the increased evolutionary pressure it puts forges under. It means any forge system that writes an importer is suddenly at a huge competitive advantage against sites without importers, even if it can handle only a few site-specific dump types. I have some longer ends in view here.

  16. Exactly how is that the fault of the code being written in Python?

    If it were just Yum there’d be no problems. When you have an entire software stack written in Python, except for the lowest level bits like gtk and dbus, and it uses a Python app framework which gets loaded into the memory spaces of several processes (modules written in Python can’t leverage C’s shared-library semantics, only the module bits compiled down to .so’s can), on a 256 MB machine with no swap, consumption adds up fast. Yum just happens to be the steamer trunk that broke the camel’s back.

    I think the designers of Sugar made a mistake in implementing stuff in Python at too low a level.

  17. >I think the designers of Sugar made a mistake in implementing stuff in Python at too low a level.

    So do I. As a general rule, anything that spends a lot of time talking over the net (including version-control systems and forges) might as well be written in Python because its time inefficiencies relative to C will be swamped by network latency. Sugar is a textbook example of the opposite case where local performance really is important. Notwithstanding dumb fashionable pieties about carbon footprints.

  18. While I’m not suggesting a tool to scrape data from Forges is a bad idea; quite the contrary, in fact. However, selling it as anything but a last-ditch solution is absurd; you are coupling your backup facilities (fairly critical infrastructure for a project) to the GUI of some web application. There is absolutely no reason backups cannot be produced from the database schema. Any resulting duplication of code is going to be more sensible than duplication at the GUI level.

  19. >There is absolutely no reason backups cannot be produced from the database schema.

    If you have access to the schema. And you know how the schema tables are supposed to map to your application objects. In the world of forges, you cannot really count on either premise being true in a practical sense, though both may be theoretically.

  20. >>There is absolutely no reason backups cannot be produced from the database schema.

    > If you have access to the schema. And you know how the schema tables are supposed to map to your application objects. In the world of forges, you cannot really count on either premise being true in a practical sense, though both may be theoretically.

    Wouln’t it be better solution to ask / browbeat forge authors to provide database schema in some widely understood format? Or failing that try to reverse engineer database schema, instead of relying on slow HTML scrapping?

  21. I’m giddy with anticipation! Really. I am. This could be such a big deal!

    Investigative work:

    Point 1) Not building a forge. So, not an online thing. So something on your machine.

    Point 2) No data jail. So data must easily live in more places than one. That includes your local machine, among others.

    Point 3) No SOAP. Mmmm. maybe interop with xml-rpc^wjson? Look at jsonpickle maybe. But wait, interop between what? Local box and remote repo, remote repo being, gateway between local installs?

    Point 4) Yep, can’t make them forges do anything. Don’t rely on them doing anything. Do your own thing. Scrape the data out if you must.

    But into what? not another forge. In your local box in json text, human readable AND machine usable. By small, simple programs that do one thing well and handle text streams, the universal interface? How very Unixy :)

    Like I said, I can’t wait!

  22. >Wouln’t it be better solution to ask / browbeat forge authors to provide database schema in some widely understood format? Or failing that try to reverse engineer database schema, instead of relying on slow HTML scrapping?

    What, and wait until they get around to delivering – assuming they even want to? No. This would be a recipe for infinite delay.

  23. I have some longer ends in view here.

    Yes, I think I see at least a couple of the angles. I wonder if the short-term you may underestimate the percieved barrier-to-entry that JSON-formatted data poses to an average developer (who isn’t familiar with it). That barrier may be less than they perceive because they’re used to awful data formats, and it may be more than you realize because you’re good at that sort of data-mangling…

    I could be wrong about that; all I’m advising is throwing in a tool to dump the most common JSON-outputs to some kind of appropriate less powerful but simpler data format (even at the cost of some minor data richness loss), be that mbox or tab-delimited text rows or whatever.

  24. JSON is a Python hash. It’s quite structured and dirt-simple to import into Python.

    Yes, my first instinct is always to use mbox, because 1) you can put any metadata you want, and 2) you can put any data you want into the file past the first double-newline.

  25. Jeff: I’m trying to increase my carbon footprint. First, to thumb my nose at Carbon Warriors (just say no to the War on Carbon), and second, because my life is generally made better when I can get robots to do my work for me.

  26. Okay, here we go. The solution is obvious and simple:

    you need some kind of Model/View/Controller development framework. (I like Django, but YMMV.)
    Next, you need something along the lines of XML-RPC or JSON, as previously mentioned. (Cue the everything-should-be-binary-dweeb who comes jumping out from behind a bush to launch into a tirade and voice his support for Protocol Buffers! :)
    Finally, you need some sort of local system to store everything in.

    That last point is what everyone seems to be leaving out. JSON is okay, but editing tables by hand in a text editor is a pain. I suppose you could always run some instance of this “MVC/JSON/OpenForge” locally, but I recommend some set of command line tools and possibly a GUI frontend to be available as an alternative.

    One more small point: there are already projects doing Forge scraping. Check out some of the Eclipse projects, some of which feature integration with SourceForge, for example.

  27. And in other news, A Pennsylvania man was confined to a psychiatric facility last night after being found by police screaming “BRAIN FUCK … MY GOD. BRAIN FUCK. THEY DID IT ALL IN BRAIN FUCK. THIS IS PURE GENIUS” in the middle of his drive way. Apparently, the man had been given access to the source code of popular open source project hosting services by an anonymous whistle blower.

    During a press conference, the man’s wife revealed that her husband was convinced that the popular hosting services were written in a language called PHP. Doctors report that the man is currently too heavily sedated to provide comment.

    In all seriousness, please write a post if/when a project page for this goes up. I’m really interested in something better than what exists (even my own concoctions). Its a big problem that is worthy of attention.

  28. Nice post on web scraping, simple and too the point :), I use python for simple html web scraping, but for larger projects i have used extractingdata.com web scraping which builds custom web scraper and data extracting programs simple and fast

  29. Any updates about the great plan and the working tools? Has the software side been taken over by the political side?

    ESR says: Nope. Progress is being made. Watch for my next post.

  30. I’m devastated at the damage done to the integrity of scientific research. Like Caesar’s wife, researchers need to be above suspicion, and have their methodology, data, and conclusions available and subject to review and analysis.

    Some egregious recent departures from acceptable conduct have involved independent individual researchers. This situation of a group acting in concert with other “scientists” to not only distort information but to pressure peer review publications to suppress contrary views, is inexcusable. And the political alliance in furtherance of reciprocal positions is damnable.

    The effect of this putative misconduct on the billions of persons who are the world’s population, in terms of diversion of resources to remedy a possibly non-existent problem, to necessarily institute greater government interference and controls, to mandate a transfer of resources from one group of nations to another, and to enrich particular positioned businesses and entrepreneurs in manufacture of materials required, and in the parasitic endeavor of “carbon trading” is quite probably a criminal enterprise with the personal incentives of the corrupt “scientists” also an issue to be investigated.

    This situation makes me want to shrink and hide as if the charges are sustained, it is indefensible conduct. It also is difficult to understand how people trained as “scientists” could engage in such compromising behavior. I cringe.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="" highlight="">