Announcing ForgePlucker

I’ve been strongly hinting in recent blog entries that I planned to do something concrete about the data-jail problems of present open-source hosting sites. Because I believe in underpromising and overperforming, I decided at the outset not to announce a project until I could not only show working code, but code with wide enough coverage to make it crystal-clear that the project goals are achievable with a relatively modest amount of effort.

That time has arrived. I am very pleased to announce ForgePlucker, a project aimed at developing project-state extractor software for backup, offline analysis, and (eventually) re-importation. The proof-of-concept code can extract complete issue-tracker state from Berlios, Gna!, or Savane — and issue trackers are probably the hardest part of the job. I expect extraction of repository histories and developer permissions tables to be easier. Extraction of mailing-list state is probably a bit trickier than either of those, but doable.

The code as it exists now is just 1100-odd lines of pure Python; it can dump tracker state in either JSON or an XML format. Notice that this is already a production-ready tool for, among other things, examining bug lists offline. One of the goals in the project plan is: useful tools at every step of the way. This is not going to be a project where the developers toil in obscurity for years until releasing Epic Code That Changes Everything; the project plan lays out smaller deliverables that can be used to build cooperation with forge-system designers and other people interested in some of the data-transport problems associated with forges.

Accordingly, I’ve actually put as much effort into documentation than I have into code – there’s a project plan, a HOWTO on writing forge handler classes, and even a draft ontology of forge state we can use to translate among the data models of different forges.

Yes, I’m looking for co-developers. What the project especially needs is people interested in taking responsibility for developing and maintaining the handler classes associated with different forge types. Presently, I own the Savane and Berlios handlers; I’d like to give those away so I can concentrate on the framework code, and I hope to recruit owners for SourceForge, GForge/FusionForge, Launchpad, and Trac.

There’s lots of interesting work to be done here. Much of it will be code, some of it will be standards and documentation. Relevant skills and technologies include Python, JSON, RDF, HTTP forms, web-scraping, test-driven development. As always, being highly motivated to address the problem is more important than knowing any specific toolset at the outset. I expect this project to be genuine fun; it breaks naturally into small substeps on which you get rapid feedback at each stage, and testing for correctness will be relatively easy.

Once we field a forgeplucker that can pull complete project state off a suitably wide variety of forges, it will be time to think about tackling the re-import problem. We may solve that by bolting importers onto existing forge systems one by one, or by launching a forge-development project that builds on our extractor tools. Another direction we might go in is supporting scripted interaction with these systems. At every stage, useful tools!

To participate, get an account on Gna! and apply to join ForgePlucker. You can also examine the source-code repository and llist of open tasks. You will probably want to join the mailing list. I’m planning to set up an IRC channel as well.

(Yes, I’d rather have used a distributed VCS; I did my proof-of concept in Mercurial. But Subversion is the best we can do on Gna!, where I have the advantage of being a site administrator.)

30 thoughts on “Announcing ForgePlucker

  1. I am not the forge plucker or the forge plucker’s son, but I’ll pluck forges till the forge plucker comes.

  2. The consonance of the project name is just too good to be coincidence. Will definitely check into the project soon as we have a need for such things against some commercial systems.

  3. You two clowns should go write some code.

    Not one of the clowns, but anyway. can I nitpick your code instead? I see the first thing you do is “canonicalize” (like the term!) the HTML first, by hand, for forge-specific adaptations. The first thing that spring to my mind – without, I must admit, having either tried it out or looked at any of the offending HTML – is: why not just run it through htmltidy?

    Of course htmltidy might not canonicalize specific cases to your satisfaction, in which case you’d be back to writing forge-specific edge cases by hand, though hopefully significantly fewer of them. Or you might not want to have a dependency on a library that isn’t part of standard python. But it’s the first tool I personally would reach for for such a job.

    ESR says: You’ve confused a couple of different functions, I think. canonicalize_date() is forge-type-dependent and can’t be done by htmltidy, which doesn’t know anything about date formats. HTML markup removal is done by dehtmlize(), which is forge-type-independent. There’s no need for what htmltidy actually does at any point.

  4. Thanks! Here’s hoping I’ll soon manage to do something that will make the title “developer” more than strictly theoretical.

  5. Can you please say a bit more about the goals of the project? Pure data extraction, for example, is already going on with flossmole, see

    However, you are talking about re-import, so your efforts are not primarily about project data analysis for scientific/analytical reasons, but you want to make forge data exchangeable, hence make it more easy to move projects between forges?

    Thanks, Dirk

  6. >Can you please say a bit more about the goals of the project?

    See the first item in the FAQ. I’m not interested in just gathering statistics, I want the entire project state – bugtracker items, repositories, and all. Yes, I want to make forge data exchangeable.

    • >I, too, am wondering why you’re trying to parse the HTML by hand instead of just using lxml or BeautifulSoup. Shame on you!

      As I noted there: “It’s a proof-of-concept. Don’t get hung up on the parsing details, I expect I’m going to have to rewrite it at least twice.”

      Hand-parsing this way does have the advantage that there are no external dependencies. But I’m certainly willing to use a more
      systematic technique if it isn’t too heavyweight.

  7. What issues do you see with having external dependencies? I wonder if virtual env and a requirements file might help solve those concerns.

    ESR says: The external dependencies aren’t the biggest issue. See my next post.

    • >Wait, why would that matter? You can at least administer your own project, right?

      Yes. But being an admin of the hosting site is useful when the forge software goes pear-shaped – which it often does, the architectures being as bad as they are. When I launched forgeplucker, I had to go root on the host machine and make my webspace directory by hand because something was broken in the script that should have done it (I’m trying to troubleshoot that problem now). If I’d had to wait for the (other) overworked site admins to get to patching my directory, I might have had to delay the launch for another week.

      Maybe github is better constructed and doesn’t have such glitches, but it would be imprudent for me to count on that.

  8. May I suggest to have a look at the Python code of bts-link that already does a lot of bugtrackers webscraping to monitor bug status on various bugtrackers… maybe there could be some joint effort in building a common Python library for bugtracker scraping.

    More details here :

    Apart from that, glad to see you’re making progress, and will try and provide useful contribution in link with our work in COCLICO (same goals… but less hurry ;)

    My 2 cents,

  9. >May I suggest to have a look at the Python code of bts-link

    Just reading the summary, it doesn’t look like they’re aiming to pull complete tracker state. Still, I’ll look at it – it might help solve the SourceForge login problem, if nothing else.

  10. @esr : probably will not help with the login, but at least already retrieves quite a lot of the existing bug forms for public bugs (then additional scrapping would retrieve the rest of the attributes).

    I’ve been thinking about a python client library for bug trackers for a while, thinking about the various implementations of such scrappers or connectors in many python programs : … but how to achieve some convergence towards common libs is beyond my current expectations… until a standard dump format exists maybe :-/

    • >Je te plumerai la forge, je te plumerai la forge…

      ROTF, LMAO

      OK, that settles it. The official theme song of this project is Alouette.

      Gad. Was it just me, or did every first-year French-as-a-second-language student in the universe have to learn that silly thing?

  11. It’s still day one and I’ve got two dev signups with a third promised. My second dev, who looks like he might turn into my lieutenant, just accepted two task tickets. Seven people on the mailing list already and our first IRC planning session just wrapped up.

    This is coming together so fast it’s almost dizzying. I guess there was a real need…

  12. Seems great. I’ve been reading your blog for a long time now. This first comment from me. I wish i had the time to participate, this project looks promising. Currently almost all time I have goes to “hacking” on various closed source projects. It’s like being in jail. I need the money, but I really need the money. Well time to change things!

  13. > Gad. Was it just me, or did every first-year French-as-a-second-language student in the universe have to learn that silly thing?

    Well, we frenchies have to learn it too. It’s nice to think that we can share a common song during alcoolic international nights. AAAALOUETTE, JE TE PLUMERAIIIII !


  14. Gad. Was it just me, or did every first-year French-as-a-second-language student in the universe have to learn that silly thing?

    I did not as a part of any course of instruction in the French language.

    However, right about the same time I started learning French (sixth grade), we had a sort of multimedia project themed around “Countries Around the World”, staged as a school assembly late in the school year. Each of us would do a brief (2 minute) presentation on a country and then we would all perform a song or dance pertaining to it — usually not the national anthem, but a song strongly associated with it (for example, “Du, Du Liegst Mir im Herzen” for Germany).

    Due to its strong French-Canadian fur trader associations, “Alouette” was the chosen song for Canada.

    Canucks take heart: the one for the USA was even worse, a gut-wrenchingly cheesy affair called “Fifty Nifty United States” (obviously written during or after 1959).

  15. Launchpad web services API? Now with anon access. The Launchpad team are working toward a federated/distributed code hosting network. This is tricky because much of the power and functionality of Launchpad comes precisely because it centralizes some things.

Leave a Reply

Your email address will not be published. Required fields are marked *