Looking Deeper into Forges, And Not Liking What I See

In my previous post, Three Systemic Problems With Open-Source Hosting Sites I identified some missing features that create serious brittleness in or project-hosting infrastructure. The question naturally arises, why don’t existing hosting systems already have these facilities? I have looked into this question, actually examining the codebases of Savane and GForge/FusionForge, and the answer appears to go back to the original SourceForge. It offered such exciting, cutting-edge capabilities that nobody noticed its internal architecture was a tar-pit full of nasty kluges. The descendants — Savane, GForge, and FusionForge — inherited that bad architecture.

The central problem is implied by the implementation. It’s PHP pages doing SQL queries to a MySQL database. The query logic is inextricably tangled up with the UI. There is no separation of function! Now I understand why, back when I was a director at VA Linux, the original SourceForge team promised me a scriptable release process but never delivered. They couldn’t have done it without either (a) duplicating a significant number of SQL queries in some kind of ad-hoc tool (begging for maintenance problems as the SQL schema changed) or (b) prying the SQL queries loose from the GUI and isolating them in some kind of service broker, either an Apache plugin or a service daemon, that both the web interface and a scripting tool could call on.

Approach (b) would have been the right thing, but would have required re-architecting the entire system. It never happened. When, in my problem statement, I complained of forge systems being excessively tied to Web interfaces, I did not yet know how horribly true that was.

The rottenness of the architecture also accounts, indirectly, for some other features of these systems that have long puzzled me – like the reliance on cron jobs to do things like actually creating new project instances. The cron jobs are a half-assed substitute for a real service broker.

I conclude that the SourceForge/GForge/FusionForge architecture, as it is now, is an evolutionary dead end — overspecialized for webbiness. To tackle challenges like fixing the data-jail problem, scripting, and seamless project migration, one of these systems will need to be rebuilt from the inside out. The surface appearance of the GUI might survive at one end, and the SQL schema at the other, but everything — everything — between them needs to change.

As one of my regulars pointed out in commenting on the previous post, what we see here is a massive failure to apply the Unix philosophy. It needs to be fixed. I’m not sure what the best way to go about it is, though. I’d love to start with a clean sheet of paper; our tools are ten years better now, and with Python and Django I could probably have a proof-of-concept within weeks and deliver a forge system functionally superior to anything now deployed in a couple of months tops. The problem is, if I do that I have a serious political barrier to climb getting it deployed. The alternative would be for me to rebuild an existing forge system from the inside – much more difficult, because I’d be dragging around a lot of legacy, but I’d get to keep the userbase and the brand.

One of these things looks likely to happen. I’m not sure which, yet.

47 comments

  1. There are metaphorical tears in my eyes at the profound truthiness of all this.

    I’m on board…and my coding voodoo is top notch.

  2. PHP gets a bad rap because of bad PHP coders. Using a modern MVC framework like Zend or Cake or CodeIgniter (my favorite), you can build first-class, high-performance, manageable web apps.

    Python is nice, but I really don’t like it for web front-ends; it doesn’t mix well inside of HTML content nearly as cleanly as something like PHP.

    Regardless of the tools you use, I’m potentially interested. I think everyone has a vested interest in this.

    Just curious — how are you going to approach the data jail problem? Are you just going to allow for easy migration, or are you going to build for distribution/replication/synchronization of data and services?

  3. Conflation is the enemy of long-life, i.e. it decreases Entropy and thus is destined to peak and decay. Conflation is only justified when it is the most expediate near-term way to exponential growth (to maximize the local cluster of order), but it never survives the long-term trend to disorder and maximum distribution of the free market (independent actors).

  4. “The problem is, if I do that I have a serious political barrier to climb getting it deployed.”

    What gatekeepers do you think stand in your way? Getting something deployed these days is as easy as renting a Linode instance for $20/month. If you build a forge that’s really that much better than what exists now, I don’t see how you could fail to make $20/month in donations and/or advertising.

  5. I am not familiar with the source code of these systems, but if possible it might help migration if your replacement could be compatible to existing databases.

  6. I have to agree with Bill. The barrier to entry for deploying a new forge system is not high. There’s no chicken-and-egg problem: a forge is a useful service to your project even if nobody else uses it. It’s not like the bad old days before Google where you needed to have your project on a widely-used forge in order for people to come across it.

  7. I’ve missed SourceForge’s compile farm ever since they discontinued the service several years ago. It was a great resource for making sure your build is portable.

  8. There seem to be so many poorly designed PHP code bases. Is PHP the VB of the web?

  9. There is a third way. Modify the outermost php to have two modes: human readable and machine readable. The scripting interface is then the same HTTP POSTs a human would use with one added flag, and responses that are simple xml generated by the same code as the gui.

    Designing from scratch, it’s not ideal. But if you want to avoid the untangling problem, this is the easiest way.

  10. I’m inclined to suspect (though I can think of no concrete reason to believe this) that the most effective approach may be putting up the codebase and a proof-of-concept site, then persuading an existing forge brand to adopt it.

    To this end, the proof-of-concept service should have as ugly-looking a GUI as possible while still being easily usable. Think mid-1990s web design aesthetic: gray backgrounds, square-cornered boxes, no images.

    It might also be a worthwhile feature for the service to be able to automatically synchronize with other instances of itself, so that ExampleDemo doesn’t have to compete with FamousForge for userbase.

  11. Do it right–it might not make you popular, but you will sleep well and have fun.

    As for disentangling the existing web (views) from the embedded SQL, lets look at what they (always) entail.
    (taking the retrieve and display case here; there is a similar process for the collect, update/insert case)

    1. set a db connection
    2. build a sql string
    3. execute the sql (handling errors)
    4. get the result set in a variable
    5. iterate through a result set.
    6. put the values on the page.

    If steps 1 through 4 can be cleanly separated from the bulk of the existing code then its very doable as the bulk of code is on steps 5 and 6 and need not be replaced.

    For the html side of things, the project becomes….

    1. scratch your itch, get something good going and sleep well. (:
    2. provide a roadmap and examples for replacing steps 1 through 4
    3. steps 5 and 6 are already done.
    3. create a project on the forge sites to refactor the forge sites.

    This model holds for all the client side work I have done (Powerbuilder, java, c) but I cannot vouch for Lisp (Xemacs), Bash, etc.

    Cordially,

    t.

  12. Nitpick, but…
    > In my previous port
    Typo. But I agree with everything you’ve said here, whether that be in a post or in a port. ;)

  13. >Python is nice, but I really don’t like it for web front-ends; it doesn’t mix well inside of HTML content nearly as cleanly as something like PHP.

    You talk about mixing markup and executable code like it’s a feature. I think it’s a bug – it’s how we got into this mess.

    >Just curious — how are you going to approach the data jail problem? Are you just going to allow for easy migration, or are you going to build for distribution/replication/synchronization of data and services?

    Presently my goals are seamless migration and scriptability. I think automatic syncing between live instances can be built on top of that later, as long as we identity management is handled gracefully (all namespaces need to be net-wide rather than local).

  14. >What gatekeepers do you think stand in your way?

    It’s not gatekeepers I’m concerned about, it’s inertia and the tendency to stick with known brands.

  15. Sourceforge recently redeplyed its frontend in turbogears. See http://compoundthinking.com/blog/index.php/2009/07/16/turbogears-on-sourceforge/.

    Turbogears is somewhat similar to Django, and I think it would be easier to work with that codebase. Also, the article mentions moving away from mySQL (don’t know to what extent).

    It’s worth a look I suppose.

    Trac, being written in python, probably can be fixed up to do what you envision. I’m pretty sure they would accept patches.

  16. There are dozens of new forges that already have good properties, so better not reimplement one more, and instead look at one with modern architecture (redmine ? codingteam ? etc.) which will help diminish entropy.

    Still the main problem stands : what to do with existing projects trapped in their old PHP *forge.

    And how to help contributors that don’t want to learn a new tool to migrate to better forges ?

    A smooth migration path is needed here… and it’s a real challenge.

    Also a ‘universal’ project export format (and import libraries) needs to be defined and standardized, I think, that all forges (either old and rotten, or new shiny ones) would support.

    We’ll be working on the second part, at least, in COCLICO project that has just started (http://www.coclico-project.org/) and also on trying to diminish fragmentation on maintenance/improvement of the old *forges to make sure there are valid migration paths for existing projects hosted in many gforges, fusionforges, codendis, savanes and such.

    Looking forward to continue the effort together.

    1. >There are dozens of new forges that already have good properties, so better not reimplement one more, and instead look at one with modern architecture (redmine ? codingteam ? etc.) which will help diminish entropy.

      Care to list more than just two, so I can evaluate them? CodingTeam admits to being written in PHP5, which I consider a bad sign. Redmine might be interesting, though.

  17. >It’s worth a look I suppose.

    No. Source Forge turned proprietary years ago. I’m only interested in open-source approaches.

  18. > Care to list more than just two, so I can evaluate them? CodingTeam admits to being written in PHP5, which I consider a bad sign. Redmine might be interesting, though.

    http://wiki.planetforge.org/index.php/ForgesList may be interesting here … there are probably also lists like http://en.wikipedia.org/wiki/Forge_(software)

    http://wiki.PlanetForge.org/ was setup (mainly french content yet, sorry) in an attempt to try and help forge maintainers / users to travel in the vast landscape of forges and collaborate… and I hope we will shape it a bit better in the coming weeks as part of work done on COCLICO

  19. > I’d love to start with a clean sheet of paper; our tools are ten years better now, and with Python and Django I could probably have a proof-of-concept within weeks and deliver a forge system functionally superior to anything now deployed in a couple of months tops.

    IMHO the Trac, integrated into Django, would be a much much better starting point than a clean sheet. The Django project itself uses this approach.

  20. I guess that git hosting solutions such as (listing only those that are open source) Gitorious (in Ruby), InDefero (in PHP, but I think in clean PHP), Girocco + gitweb aka what is powering http://repo.or.cz (Perl + shell scripts) are too limited as forges? From those I think InDefero is closest to full fledged software forge (it started as functional copy of Google Code forge).

    No need to start from scratch. Or from SourceForge codebase (the open-source one).

    Although if you were able to create for example a forge which would seamlessly import from Savane (both Savannah and Gna! versions)…

  21. >No. Source Forge turned proprietary years ago. I’m only interested in open-source approaches.

    Not as a starting point, granted. But maybe to write an interop tool with them. The turbogears guys might work something out. Sourceforge has lots of projects there and that might be worthwhile.

  22. >IMHO the Trac, integrated into Django, would be a much much better starting point than a clean sheet. The Django project itself uses this approach.

    I’ll look into Trac. Unfortunately, the rumor about it isn’t good – some regulars on #gpsd use it and complain that it tries to do too many things and ends up doing none of them particularly well.

  23. These problems are best addressed by building and using distributed infrastructure, rather than redesigning centralised forges. Forge-like systems then become essentially wrappers around and hosters of copies of the needed systems for projects.

    Git, bzr etc does this for source code.
    There are quite a few projects for bug/issue tracking, though none are too mature yet, and I haven’t tried them myself.
    I’m not sure what the best way of dealing with email is… any thoughts? Something like mailman seems fine to me for this part, though with extra bells and whistles to make import/export of lists and archives easy, as you suggest.

    Look forward to hearing your thoughts.

  24. > You talk about mixing markup and executable code like it’s a
    > feature. I think it’s a bug – it’s how we got into this mess.

    Depends on how you use it. In my code, it’s only used for the views. It makes such views very easy to create without having to rely on a hand-coded parser. Yes, most web developers abuse it, like I once did. If you build a framework that has a clear separation of tasks, it’s really fantastic for the view component,

    Again, I’m potentially interested in helping regardless of what tools are used. In some ways, more interested if it uses stuff I’m not familiar with. I want to learn.

  25. In fact, if LDAP integration is built in, as well as support for SVN, there is a chance my company might sponsor this in a small way. We’re currently unhappy with CollabNet EE, which is what we’re currently using. It’s expensive, and crap.

  26. > Unfortunately, the rumor about it isn’t good – some regulars on #gpsd use it and complain
    > that it tries to do too many things and ends up doing none of them particularly well.

    Ditto. Trac is free, and that’s about the only good thing about it.May be worth poring through the source to see if it has any useful stuff, but to use it out of the box would be terrible.

    Frankly, for people to use this, it has to be easy to use, and somewhat attractive.

  27. Eric, You should probably look at Canonical’s Launchpad. It should have at least some of the import/export functionality you’re after, it is in no way a descendant of SF, and it is written in Python.

  28. What about starting from something like Launchpad? See: https://launchpad.net/launchpad

    It is open source (AGPL3), written in Python, and already contains at least some features for importing from or synchronizing with external project trackers. I haven’t looked at the code myself and have no idea if it avoids the same issues you’ve found in *Forges, but it’s worth a look.

  29. This “rant” by the author of psycopg is worth a reference as possible counterbalance to the recommendations for trac:

    http://initd.org/

    I know nothing more than what it says there.

  30. Other than Canonical’s launchpad which allows data export and is written in python, there is Fossil which is distributed version control plus bug-tracking and a wiki which gets rid of the data-jail problem for the non-version control components. Fossil is from the guy who wrote sqlite.

  31. Hmm… LaunchPad looks cool; they key problem, at present, appears to the integration with Bazaar. My company wouldn’t use it, because we’re currently an SVN shop. If it were to support multiple source management backends, that would be great. Otherwise, looks good.

    Haven’t looked at the source yet; don’t know if migration away is an option.

  32. I’d suggest that a forge does two things: 1) provide you with a big pile of infrastructure without needing to install any software, and 2) hosting. Hosting is pretty cheap these days, so, what’s really needed is a data interchange format for all the resources you can put into a forge, then people can use whatever forge they want.

  33. Aaron Traas, “bad PHP coders” are the raison d’être of PHP. The language is actively hostile to good discipline in software construction — like a more pathological Perl. That others have been able to come along and retrofit something not entirely unlike clean design on top of the original crock doesn’t make the crock not-a-crock.

    I’m with Eric. Back when I was in Webdev Hell, Python was the second-least unpleasant language to work in (the least unpleasant being Scheme — yes, I’ve deployed Scheme webapps that did millions of dollars of business). And it’s because the style of Python development enforces clean modularity. This isn’t a feature exclusively of thelanguage itself but also of the culture that accreted around it (consider how things are described as “pythonic” vs. “unpythonic” on mailing lists and such: this is referring to a specific form of what Linus called taste in programming). Mixing logic with presentation is a short trip to Tight Coupling Purgatory. A templating engine such as Cheetah can easily be used to separate these layers with a minimum of fuss.

    However, due to what I call Shenpen’s Principle[0] PHP’s overwhelming popularity persists.

    [0]So called because it was inspired by Shenpen’s comments about Clipper. Say there are two methods to do something, each with tools that support and encourage development within that methodology: one is a method of clean architecture and logical modular decomposition, and the other is a method of quick-and-dirty crufting together, get the minimum amount of work done to get it working now, with tight coupling everywhere and big-ball-of-mud architecture. The business community will overwhelmingly favor the latter approach with its corresponding toolset, mainly because it’s easier to pay the boss’s nephew $15 an hour over the summer to cruft up something that works for sufficiently small values of works, than it is to hire a developer who knows his stuff and will produce a beautifully designed program that superficially differs little from its crufted-up counterpart. The ascendancy and popularity of Visual Basic can be explained this way, and certainly that of PHP. That and the fact that despite its legendary security holes, PHP somehow was considered a safer bet for low-rent virtual-hosting services than something like Python.

  34. “To this end, the proof-of-concept service should have as ugly-looking a GUI as possible while still being easily usable. Think mid-1990s web design aesthetic: gray backgrounds, square-cornered boxes, no images.”

    IMHO Pavitra is talking sense. Keeping the GUI purposefully minimal and unfancy – which, judging from your blog and website, would happen anyway even if you wouldn’t intend it so :-) – would be a good way to convince other Forges not to see the new site as a competitor but as a proof-of-concept for a backend they could adopt and add their GUI on top of it. A way to avoid stepping on toes. Of course, it still could be accidentally successful and become a competitor even when it’s not intended, often minimalism in the interface doesn’t prevent success – I’m thinking about Craigslist and Gumtree and Hacker News.

  35. I’m working with olivier berger since 2002, pushing for some data format to export and import projects from/to forges.
    I think it’s not a matter of creating a new forge. It’s a matter of being able to extract data in one type of forge and import it in an other one.
    You will not easily allow people to move from their old forge to a better newer one if you don’t have a simple way to extract their project data.
    This means, you need an exporter in the old forge.
    If you want to look in libresource, you will see that it already has an export/import format that helps moving a project from one site to another.
    You also will notice that this feature has not helped people moving from older forge to that one.
    Re-structuring old forges is a necessary effort. It will allow people to continue using their installed forge and not oblige them to switch to a new one. It’s also a matter of adoption.

    1. >I’m working with olivier berger since 2002, pushing for some data format to export and import projects from/to forges

      Olivier has explained his preferred approach to me in some detail. I think he means well, but anything that relies on RDF and semantic-web research is a recipe for producing endless white papers without any useful result. Yes, I do have a better idea, and I’m writing actual code now.

      >I think it’s not a matter of creating a new forge. It’s a matter of being able to extract data in one type of forge and import it in an other one.

      Creating a new forge will be necessary to achieve the scriptability goal – as I’ve pointed out repeatedly, all existing designs are far too tied to the Web. But you’re right hat extraction is the first critical-path problem.

      >This means, you need an exporter in the old forge.

      No, it doesn’t. You used the right verb before — “extract”. I am not going to count on the major existing Source-Forge-descended systems to write exporters – I know what’s in those festering piles of kluges now, and their internal architectures are so fucked up that even with 100% cooperation from the designers an exporter would be more trouble to build than it’s actually worth. Extraction tools that use the existing public interfaces are where it’s at.

      >If you want to look in libresource, you will see that it already has an export/import format that helps moving a project from one site to another

      Pointer to the format specification, please? It might save me having to re-invent one,

  36. Have you looked at sharesource.org ? Please keep in mind, there are only 2 – 3 people working on it .. but you are free to download the code that runs the site and improve it if you like.

  37. I’d also like to see what Launchpad can do. It’s base in Bazaar probably means that not just the source code control but the bug tracking and the blueprints and everything else that Launchpad does can also be made available in a distributed fashion. Or maybe CouchDB (which Canonical seems to be quite fascinated with lately), which has some nifty database replication features, could be used to un-jail the auxiliary data for the project.

    I think it’d be cool to someday start your small project on the Launchpad.net host, then at some later date decide to clone your project to a separately hosted instance, without losing any integration with projects and Bazaar branches and user accounts (Launchpad already supports OpenID) on the main host or any other satellite host. Or you could run your own Launchpad without any connection to the main host, your choice.

    That said I don’t know any about how the internals of Launchpad works but I think it’s worth a look at least.

    -Josh

  38. Why don’t put all data into the source control manager itself ? Like wiki pages for google code.
    We can imagine to put bugs, rfe, etc … and manage it with the merge algorithms of the SCM.
    A kind of distributed forge where data could be hosted and visualized by web applications.

    1. >Why don’t put all data into the source control manager itself ?

      It’s an interesting idea, and the direction that systems like Fossil and TiddlyForge are going. I may revisit it if I write my own forge, which is not part of the current plan.

  39. A hge problem in this field is that people generally don’t recognize good if it falls into their laps. At my company we developed a backend that does everything that a persistent storage for a forge service (and a buch of other applications) should provide. Nobody undertands the benefits, ven if you explain them in detail in the most most eloquent way you possibly can. The few people that have bought our products have done so despite our pitch. They happily get along with their business and wonder why their colleagues can’t get their act together (still not understanding why they are doing so well themselves).

    We now have a dumbed down product, which has a slick surface, and it has in just a few weeks gained more interest than our old product managed to collect in several years. The old product is still many times better, but doesn’t look very fancy.

    I think this is akin to why women prefer beauty over intelligence. Men (or in this case people) can see better than they can think.

  40. > They happily get along with their business and wonder why their colleagues can’t get their act together (still not understanding why they are doing so well themselves).

    Hehe. My farmer parents use Linux this way. They never used a computer before (3 years ago), don’t know what a virus is, never had to make an upgrade (i do) and really can’t understand why others speak always of computer problems. From their point of view, it just works and is free, as it should be. Happy parents.

  41. When I originally commented I clicked the -Notify me when new comments are added- checkbox and now each time a comment is added I get four emails with the same comment. Is there any way you can remove me from that service? Thanks!

Leave a Reply to Shenpen Cancel reply

Your email address will not be published. Required fields are marked *