Sneaking up on the forge problem

I’ve written before about the problems with today’s software-forge sites – how they’re craptacular piles of PHP driving direct SQL queries with almost zero scriptability that become data jails for open-source projects. I’ve hinted that I think there’s a potential solution based on Roundup, a brilliantly simple and powerful message queue manager disguised as a mere issue tracker.

Two years ago I started an effort to address the data jail called “forgeplucker”. This project has since stalled, partly due to problems (irony alert!) at the hosting site I put it on, and partly because there was a basic problem with it that I acknowledged at the time: it doesn’t do a lot of good to have working project state extractors if none of the forges are writing injectors. Nobody has stepped up to do that, leaving forgeplucker a useful learning experience but something of a dead end in itself.

One of my regulars recently predicted that I would do something dramatic about the forge problem within two years. That prediction was correct; in fact, I’ve already begun the preparation work.

However, I’m not announcing a new forge. Yet. The path I’m pursuing is sneakier and more interesting than that. It relies on the fact that there is a mature, stable, and field-tested implementation of Roundup already in existence in fact, the Python development group uses it as their issue tracker.

Roundup has a number of interesting properties. One is that every capability in it can be scripted via a mail robot or an XML-RPC interface. Another is that the message queues are accessible through either a web-forum-like interface or email – content is separated from presentation. Most importantly, it already works – it’s got ten years of history, tested code, happy users, and a development community.

Some weeks ago I joined the Roundup development list, explaining my interest in using it as a platform for a future forge system. As a first contribution I explained about reposurgeon and offered to lift the project repo from Subversion to a DVCS; the project founder accepted this offer, and I have done the job. As a nice side effect, reposurgeon grew a nice list of new capabilities in the process, essentially everything in releases 1.3 to 1.10.

So now I’m a dev on the Roundup project, in fact an administrator (which was needed so I could do the repo manipulations). My plan is to add certain new primitive capabilities to Roundup. Each of these capabilities is orthogonal to the others and can be functionally justified on its own terms. Here are the big ones:

Extract/inject: Extract will dump the state of a Roundup instance as a big hunk of (yes) JSON; inject will take that JSON and use it to populate a live Roundup instance. The standalone justification for this is that it can be used for tracker state backups.

Namespaces: Presently, each Roundup instance hosts just one tracker (message queue). It should be possible for a Roundup instance to host multiple named message queues. The standalone justification for this is…itself.

Network-wide identities: Presently, Roundup identities are login-name/password pairs. It should move to being based on email addresses coupled with ssh and PGP keys – network-wide credentials the way DVCSes handle identity. The standalone justification for this is to reduce the number of credentials Roundup users need to manage.

A repository-management class. Roundup has a role-based permissions system. A primitive object type could be added to Roundup that would use that permissions system to control access to a DVCS repository.

There’s no mystery intended about where this is going. If all these features were successfully added to Roundup, it would have all the capabilities of a forge. The message queues would span the functions of issue trackers and mailing lists. The “forge” part would be some management logic for associating a group of message-queue namespaces and repository objects with a project name and giving them all one set of access permissions.

But I don’t want or need to promise a forge yet. The plan is to add these primitive orthogonal features to Roundup. At each stage they’ll get actual testing in real deployments. Rather than trying to design a forge now, what I aim to do is push these capabilities into Roundup until one of two things happens:

1. I hit a wall. The underlying architecture isn’t robust enough to carry these features. (I regard this as possible but very unlikely.) If this happens, I have to think up another approach.

2. All these features are successfully integrated. Then a forge could be written as a thin layer over the enhanced Roundup – and, functionally speaking, it would nuke every existing forge from orbit. Just the complete XML-RPC scriptability alone would make it a category-killer.

I have not started coding yet, mainly because I have a writing project I’ve promised I’d finish before I dive into this. But the existence of Roundup will save immense amounts of coding and community-building. The project has a ten-year history; I estimate it would take four to five person-years to duplicate the capabilities it already has in a stable and tested codebase.

With Roundup to start from, on the other hand, I think we could get to a world-beating platform for forge-building in nine months to a year. I’m going to do extract/inject as my first bit, building on experience with ForgePlucker. One of my blog regulars, Susan Stewart aka HedgeMage, has already joined the Roundup project with the intention of doing the network-identity piece. We’d love to have somebody working on implementing namespaces; all these things should be doable in parallel.

If you want to help, please join the roundup-devel mailing list, introduce yourself, and explain which part you want to work on.

28 thoughts on “Sneaking up on the forge problem

  1. Oh kewl, it’s originally from the fevered braiiiins of Ka-Ping Yee. I met him at the Hackers Conference last weekend. It was an amazing piece of cool. He introduced himself as “Ping”. Then I fanboied him later when I saw his name badge. “Oh you’re Ka-Ping Yee!! You should have SAID so!”

  2. Okay, Roundup looks interesting, but … it looks like it has weak handling of bouncing emails. I’ll have to look at it more closely.

  3. >it looks like it has weak handling of bouncing emails.

    You have Python chops. Join the project and fix it.

  4. Will ForgePlucker be restarted if/after you finish your work on Roundup, now that there’s a viable “destination” for the data it produces?

  5. >Will ForgePlucker be restarted if/after you finish your work on Roundup, now that there’s a viable “destination” for the data it produces?

    Quite possibly. But there isn’t a viable destination yet, so…no promises.

  6. As is typical of open source documentation, you have left out the global context required for the uninitiated to make sense of what you are saying:

    The global context being that that if one’s source is in a distributed version control system, one’s source is not imprisoned in a data jail, but one’s project mailing lists and issue tracker is in a data jail. You propose to extend roundup so that it provides mailing lists and issue tracking that is not in a data jail and can be tightly integrated with a distributed version control system, so that it functions as a place to store liberated project mailing lists and liberated issue tracking databases.

    It may seem that I am always giving you, and open source, a hard time, but this is only because I respect you so very much. Those I despise, I ridicule. Those I respect, I criticize.

  7. Unseriously:

    esr> So now I’m a dev on the Roundup project, in fact an administrator (which was needed so I could do the repo manipulations).

    esr might be a hacker rather than a sysop, but that doesn’t mean he can avoid admin duties.

    Seriously:

    Nathan>>Why not just use github?

    esr>Proprietary. Weak services.

    Github have been a real friend to FOSS, and “weak services” actually means that migrating away from the superbly scriptable Github is easy should that friendship weaken, so I don’t think this is quite fair. I’d say the problem is more along the lines that a purely individualist, git-centric “devforge” is not going to have much harmony between its repository and its issue/messaging system: you need per-project infrastructure, not just per-fork infrastructure. Which, I guess, is really the whole point of esr’s plan.

  8. Wonderfully roundabout way to kick the forge in the balls.Well, best o’ luck with that.

    I will take a look at Roundup, it sounds robust. I’d been using OTRS as my de-facto ticket system/issue tracker. It’s so ugly I made a better looking, if simpler, ticket system (too simple to share, useful enough to… you know, use). Let’s see what happens.

  9. Mmm…I think the solution to the forge problem isn’t a better monolithic forge with scriptability but the ability to instantiate a project server VM with all the dev tools configured, integrated and ready to go.

    Folks here have built a system where you can instance a new project server VM with gerrit, redmine, jenkins, git, etc all ready to go and integrated with eclipse. We have many, if not all, the tools for fairly sophisticated end-to-end open source workflow. But it’s a real PITA to stand one up for a new project even if you’ve done it before.

    As a project lead I just go to the website, put in my new project name and description, fill out a form and I get a server VM and a dev VM more or less ready to go with a default workflow that covers scm, bug tracking, code reviews and a continuous integration build server.

    Kind of like Atlassian On-Demand ($10/month for 10 devs) and I presume other cloud based dev environments. The difference is that I have control over my own VM that I can host anywhere…so there’s no data jail. At least not one I can’t fix since I have root and can modify the core tools if I have to do an export.

  10. @Nigel:

    You can do the same thing no matter what the solution by putting the dev server in a VM. Export the thing to an OVF and, in theory, you should be able to import that OVF into any virtualization system that supports OVF

  11. >Mmm…I think the solution to the forge problem isn’t a better monolithic forge with scriptability but the ability to instantiate a project server VM with all the dev tools configured, integrated and ready to go.

    I see what you’re driving at, but the direction I want to go with Roundup will serve either end. Something has to handle message queues and issue tracking and access control for repos, and whether you think of the level under it as a server VM (hosting just one project) or a conventional forge site (hosting many) is orthogonal to most of the interesting design issues.

    >But it’s a real PITA to stand one up for a new project even if you’ve done it before.

    Well, yeah. The repo and communication tools need a management interface. That’s what a forge really is.

  12. Is it possible to have all the messaging data relevant to a repository actually stored in a folder in the repository itself?

    Here’s a fantasy: for a non-bare repository, the working copy will have a directory called, say, roundup-forge, whose contents are all the project-specific data that roundup-forge needs to build its various interfaces, and which it checks in as part of its operation? Then forking the project becomes just a matter of forking the repo, and pointing an instance of roundup-forge at the project, perhaps by moving a directory and sending a HUP to the correct process? Every checked out repo will then have a view of all of the messaging data, and if branches start to become unmergeable, it becomes possible to simply fork the roundup interface.

    I would call that pretty close to frictionless. It doesn’t sound particularly close to what esr describes, but it’s maybe useful to populate the landscape of alternative ways of doing things.

  13. >It doesn’t sound particularly close to what esr describes, but it’s maybe useful to populate the landscape of alternative ways of doing things.

    It’s an interesting idea, but runs into some data-management and security problems. Here’s one: Exposure of passwords in the git checkout. This is a bad idea event if all we’re exposing is password hashes – see https://github.com/juuso/BozoCrack

    Here’s another: supposing a project has multiple repositories, which one gets to own the metadata? And a third: access control on metadata changes would get complicated. These are both symptoms of an underlying problem, which is that the metadata really needs to live in a per-project store with serialized access.

  14. > Two years ago I started an effort to address the data jail called “forgeplucker”. This project has since stalled, partly due to problems (irony alert!) at the hosting site I put it on, and partly because there was a basic problem with it that I acknowledged at the time: it doesn’t do a lot of good to have working project state extractors if none of the forges are writing injectors. Nobody has stepped up to do that, leaving forgeplucker a useful learning experience but something of a dead end in itself.

    Let me say that this is untrue : look at https://mail.gna.org/public/forgeplucker-dev/ and you’ll see that we have tried and continue work on ForgePlucker which was published at http://gitorious.org/forgeplucker.

    On importers of forgeplucker extracts, we have worked on implementing some for FusionForge, Redmine and Trac. It may be ongoing works, but still things have progressed. And any effort from interested parties is welcome… even though it looks like Gna may not be the best place to host development.

  15. >On importers of forgeplucker extracts, we have worked on implementing some for FusionForge, Redmine and Trac.

    Are any of these importers actually working?

  16. >>On importers of forgeplucker extracts, we have worked on implementing some for FusionForge, Redmine and Trac.

    >Are any of these importers actually working?

    AFAIK, they’re already used in production on the INRIA forge which runs fusionforge. It allows the archiving of old projects, while being able to restore them later in case of need. It’s a different branch of forgeplucker that is used, as it is meant to be run serverside, but the export format is common.

    The Redmine and Trac importers are supposed to be working, but I haven’t seen much. Probably not handling all the project artifacts.

    So forgeplucker proves useful, now that people have bits of importers and not only extractors.

    We now have to try and find the way to have a manageable communication channel, and some leadership to merge contributions. For instance, there’s 3 versions of formats for exports at the moment in the various bits we coded in COCLICO, more or less semantic, which would need to be versioned, so as to make sure that there’s a format that can be standardized (based on https://forge.projet-coclico.org/docman/view.php/8/86/ontology.html, for the underlying data model), improved, enriched, and control which level of support there is in various tools.

  17. >already used in production on the INRIA forge

    Huh. That is far more progress than I knew anyone had made. I guess my architecture was pretty sound!

  18. @esr:

    Here’s another: supposing a project has multiple repositories, which one gets to own the metadata? And a third: access control on metadata changes would get complicated. These are both symptoms of an underlying problem, which is that the metadata really needs to live in a per-project store with serialized access.

    I only work on corporate projects, or really small public projects that don’t require too much communication and haven’t had the need/bothered to go to DVCS, so I haven’t really given it that much thought.

    Nonetheless, this sounds suspiciously like a last-decade argument against DVCS, just moved one peg up the abstraction layer. What am I missing?

  19. >Nonetheless, this sounds suspiciously like a last-decade argument against DVCS, just moved one peg up the abstraction layer. What am I missing?

    Actually, very little. There’s a stupid version of this argument that is identical to a last-decade argument against DVCS and just as flawed.

    I’m not going to lay out the smart version here. It’s worthy of a blog post, probably my next one.

  20. @esr:

    It’s an interesting idea, but runs into some data-management and security problems. Here’s one: Exposure of passwords in the git checkout. This is a bad idea event if all we’re exposing is password hashes – see https://github.com/juuso/BozoCrack

    That’s why you need to use _salted_ hashes, not raw MD5 sums.

  21. I wonder if Roundup supports export of project description and metadata via DOAP RDF-based format (like Freshmeat / Freecode)…

  22. The Trac import/export plugin at [1] (developped by the Coclico project) will handle import of data generated by Forgeplucker (using the OSLCM/JSON based format). It’s pretty rough and not tested at large, but it’s trying to cover most data. It’s not handling merges, but is quite usable to import a whole project from scratch.

    [1] http://trac-hacks.org/wiki/PlanetForgeImportExportPlugin

  23. Not sure how to put this, but, why do all sorts of projects insist on reinventing the “logon” wheel? Even things like openid (and *yuck* facebook *yuck* enroaching in the login-on-a-website space, which is a very special kind of problem all on its own, and out of scope here) are reinventions in a framework fashion; I happen to dislike “frameworks” for entirely different reasons. But enough about my dislikes.

    How about using something like PAM as a lever, and connect tracker and versioning tools through that? I can see a couple problems with it, but I’ll leave them as an excercise. Sometimes you want different spaces, sometimes combined ones, but why is it so freaking hard to connect things together and why are people overlooking obvious and pre-existing and pre-deployed solutions (like, again, PAM, or even just a shared table in the database you’re running as a back-end to multiple systems already)?

    Like I said I’ll leave most of the discussion as an excercise but I’ll highlight one thing: Control. Managing privileges (which is nearly not as connected to “managing identity” as conventional thinking would have you believe) is usually externally federated giving away control and/or leaking information in the process. Why go that route at all then? Keeping it all in house doesn’t do that and shouldn’t be hard to do.

  24. Pingback: “Scribo Ergo Sum” » Blog Archive » Teaching myself Python in N easy lessons…. - Part 1

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="" highlight="">