The future of software forges

I’m still not going to talk about my attack on the forge infrastructure problems quite yet; the software is coming along nicely, but I intend to announce only after it handles its fourth forge type (yes, that was a tease). But I will say this: I now think I know what the future of forges looks like. It’s called Roundup, and it is astonishingly elegant and potentially more powerful than anything out there. Anything, not excluding the clever decentralized systems like Fossil or Bugs Everywhere.

Here are the big wins:

1. Mailing lists, issue trackers, and online forums unify into *one* message queue that can be filtered in various ways.

2. Scriptable via XML-RPC or an email responder ‘bot.

3. Small base system with good extensibility – just three base classes (User. Msg, File) and the ability to define new classes. ‘Issue’ is a class built on top of these.

4. Arbitrary attributes per issue is basically free, with baked-in support for defining controlled vocabularies.

5. There’s a uniform way, called “designators”, for messages and other objects to refer to each other in text.

6. Small, clean implementation written in Python.

There are some things it needs, though… (Read the Roundup design document before continuing.)

First, namespaces. A namespace is a set of User, Msg, and File objects that belong together. Each namespace gets its own hyperdatabase blob. Each namespace numbers objects separately. Each namespace has a unique name used as a prefix in designators. Thus, if I say [gpsd/msg23], I’m asking for the 23rd msg object in namespace gpsd. Everyone talking to a system system has, at all times, a current namespace. If I just say [msg23], I am designating the 23rd message object in my current namespace.

Why the slash? because, in the most general case, designators will be URIs – as, for example, [https://developer.berlios.de/gpsd/msg23]. Presto! There is now an internet-wide namespace of Roundup objects all of which can refer to each other!

No prize for guessing that a namespace is a project. I’ve actually done two things above – I’ve defined all the required semantics for multiple projects to coexist in a forgelike system, and I’ve created a way for projects to refer to each others’ data even if they’re not on the same machine. Both features will be sources of power.

The subtler thing about namespaces is that they are perfectly orthogonal to the system’s other primitive concepts. This will pay off huge in some ways I can already foresee and many that I cannot yet.

I think the design also needs a primitive object class to encapsulate repositories. But I may be wrong about that; I’m still considering it.

12 comments

    1. >I find it rather amusing that Roundup’s URL is roundup.sourceforge.net.

      Roundup isn’t a forge., it’s a discussion-moderation system. But it would be a helluva platform to build a forge on.

  1. Having lived in deeply rural parts of the country, the fact that “Roundup” is the most commonly applied general use herbicide is bemusing to me.

    (You know you’re in farm country when nearly every local television ad from 6 PM to 9 PM is for Roundup and its competitors from roughly April through September.)

  2. > Roundup isn’t a forge., it’s a discussion-moderation system. But it would be a helluva platform to build a forge on.

    So would Google Wave…

  3. I think the design also needs a primitive object class to encapsulate repositories. But I may be wrong about that; I’m still considering it.

    Why? You already have a primitive object of ‘File’. You could have multiple instances of File-derived objects, each with a version attribute and other metadata such as owner, check-in timestamps, etc. The implementation could be version-system agnostic, with additional derived objects with attributes and methods necessary to support different version systems. Maybe it could even have its own version system backend, but I’m guessing that would probably be out of scope.

    Roundup isn’t a forge., it’s a discussion-moderation system. But it would be a helluva platform to build a forge on.

    On that we are in 100% agreement. I’m very impressed with what I’ve seen so far. It’s a very clean object-oriented designing and infinitely extensible. There seems to be adequate separation between UIs and implementation, something which I think is key to any project like this.

  4. From what I’ve seen so far, looks very nice. I like the datastore backend system, it seems that it wouldn’t be very hard to write new Roundup backends. It means that Roundup’s hyperdb can potentially run on top of <a href="http://code.google.com/p/redis/"redis or MongoDB. This would meet all potential scaling requirements of a large forge system with no unneeded SQL overhead.

  5. Looks good. The one thing it seems to cry out for is primitive support for versioning and history tracking, down at the file level. It could also use some more active and durable workflow support than simple reactive detectors. It doesn’t seem to be possible to implement something like “alert me when any issue goes unassigned for two days”.

    (I’m unable to see a good design without commenting on how I’d change it. There should be support group meetings for people like me.)

  6. Interesting.

    The Roundup site has a broken link to the original Ka-Ping Yee project that motivated everything. You can find the original project details here

  7. Thanks ESR for this whole illuminating thread on forges.
    To me, it looks as though there are two possible ways to go:
    1. Extend Roundup to become a fully-fledged forge – this is what you seem to be thinking of. Due to its flexibility and the interfaces, this is not so much of a data jail. Still, it would be necessary to actually fetch the data of the central site.
    2. Take the concepts of Fossil, a scripting language (probably Python) to make it better maintainable, and one or all of the existing DVCS (git, mercurial, bzr, whatever) as underlying backend. The advantage of this approach is that all data of the project(s) is distributed right from the start. No data jail here.

    Now, just as an idea, wouldn’t it be possible to even combine both of these approaches? Assume, Roundup would store its data not in a database, but in a DVCS repository instead.
    On the main server of a project, the web interface can be used to write issues, etc.
    Its repo can be cloned easily by a developer, who can then work on his clone – modify code, write issues, etc., ideally via a Roundup CLI. Given that he has the full repo, he can of course also easily set up his own local Roundup web server, and do it via web interface, if he feels like it.
    When he pushes his changes back to the main server, a detector on the origin site will note that some things have been changed, so corresponding Emails can be sent out, etc.
    Doesn’t that sound nice? Or am I missing something?

    1. >Assume, Roundup would store its data not in a database, but in a DVCS repository instead.

      Nearly two years later: I don’t think this is quite the right thing, though you are correct that the forge itself should be DVCS-like. I may blog about this shortly.

Leave a comment

Your email address will not be published. Required fields are marked *