Against decentralized bugtracking

I’ve spent a lot of time and bandwidth on this blog thinking out loud about version-control systems and software forges. In my last post, I announced that I was going to try to sneak up on the problem of designing a better software forge by enhancing Roundup.

Over the last three years I’ve gotten a couple different versions of the following response to my thinking-out-loud: “Centralized forges and bugtracking are old-school thinking, as hoary as centralized VCSes. Why shouldn’t all that metadata live in the project repo and be peer-merged on demand the way code is?”

This is a good question, but I think the people advocating systems like Bugs Everywhere, scmbug, and ticgit have invested a lot of cleverness in the wrong answer.

In order to see why, we need to look at a very basic question: Why do DVCSes work? And more to the point, why did it take us so long to figure out that they would work?

The original, oldest-school RCS/SCCS model of source control used locking. When you wanted to modify a file. you checked it out and locked it. Then you modified it, then you checked it back in and released the lock.

The locking model was founded on two assumptions. First, that that modification conflicts would be frequent and severe enough that they could only be prevented by granting programmers temporary keep-out exclusions. Second, that the timescale of whole coherent changes – the interval between “I start working on this thought” and “I’m done” would generally be pretty short.

The people who wrote these early version control systems believed, in other words, that source code has locality and contention patterns a lot like a database. Databases beg to have access to them carefully serialized, with reliable locking, because their usage pattern is one that involves frequent access contention over small pieces of data.

But both database-centric assumptions were incorrect. Actual experience showed that modification conflicts in source code are rare, usually mechanically resolvable, and if not almost always easily resolvable by eyeball and hand. On the other hand, the actual time scale of coherent changes is long enough that locking all the files required for them over the whole span would frequently cause conflicts, even though conflicts over the actual individual small spans of code being modified within them are so rare.

As we gradually figured this out over a span of about twenty years, conflict resolution in VCSes moved from lock-based to merge-based, with DVCSes at the end of that evolution.

DVCSes are based on the assumption that a programmer can clone a repo, disappear into a cave, and spend days or weeks coding in isolation in the sublime confidence that when he/she wants to rejoin the world, peer-merging with other repos will still be pretty easy. And…usually…it is easy. Large projects like the Linux kernel depend especially heavily on this assumption – they’d collapse if it weren’t true.

Code, it turns out, is not like a database. Strictly serializing access to source code isn’t that important, because most changesets are mutually irrelevant and mutually commutative. (Though when you put it that way, a programmer will be apt to gulp and boggle before eventually conceding the point.)

OK, so, the people who advocate decentralized forges and bugtracking have on the face of it a strong historical case. Decentralization worked big-time for managing code changes; isn’t it silly, a repetition of old-school locking-VCS narrow-think, to doubt that it will work just as well for…say…bug-tracking?

Indeed, a stupid person could reject distributed bug-tracking for stupid reasons. But that doesn’t make all reasons for doubt stupid – and the right question to ask is whether bug records (and other pieces of project metadata) have an access pattern more like source code or a database.

I think bug trackers are more like databases than like source code in almost all relevant ways. It’s not necessarily relevant that they’re normally implemented on top of databases; I’m talking about the human workflow around them.

The difference is this: When you check out a revision of software, you need to have a coherent state of it, but it’s not necessary that you have every single bleeding-edge changeset of every developer hacking on it everywhere. If J. Random Neckbeard is off in a cave somewhere refactoring a major subsystem, you don’t even want to see his changes until he decides he’s reached a good point to merge up to the public repository.

The natural cycle on conversations around bug reports is a lot tighter. If you’re part of a issue-tracker thread that is trying to characterize a bug, and someone else posts critical test results, you want to know about that right now.

It’s not that modification conflicts are important in this context; generally they aren’t. No, the issues are (a) timeliness, and (b) having a defined rendezvous point where you can browse fresh metadata, chatter, and attachments related to your bug. All these pull in the direction of centralization, or at the very least a single aggregated event feed at a known location – something more like a blog or social network than a flock of DVCS repos passing around changesets.

Other things pull this way as well. Consider this very apt quote from Jonathan Corbet in 2008:

A bug tracker serves as a sort of to-do list for developers, but there is more to it than that. It is also a focal point for a conversation between developers and users. Most users are unlikely to be impressed by a message like “set up a git repository and run these commands to file or comment on a bug.” There is, in other words, value in a central system with a web interface which makes the issue tracking system accessible to a wider community. Any distributed bug tracking system which does not facilitate this wider conversation will, in the end, not be successful.

He’s got a strong point. The perceived technical elegance of distributed bug-tracking gains us nothing if it locks out people who aren’t developers.

Corbet also reminds us of an interesting fact when he brings up to-do lists. This is that projects normally have several different kinds of to-do-lists that are managed in different ways.

At one extreme, we have roadmap and design documents that change infrequently and have code-like access patterns – that is, modification conflicts are unusual, and having the version of the design document matched to your code is usually much more important than having the latest version.

At the other extreme, we have the implicit to-do queue provided by an issue tracker. Items on this tend to change much more quickly and have shorter lifetimes.

Somewhere in the middle is the traditional TO-DO file, which tends to be a sort of grafitti wall describing medium-scale tasks.

The point I’m driving at here is that the differing ways we manage these to-do lists are a consequence of the workflow around them. To-dos with code-like access patterns want to live in the repository with your code; to-dos with database-like access patterns want to live in a bug tracker or something else like a specialized database engine (blog, wiki, whatever).

There’s a more general point here about software forges. Software forges – centralized rendezvous points where project metadata lives in something that is not your repository – make sense precisely to the extent that some project metadata is not like code.

Bug databases are the most obvious example. Another one is wikis. Also mailing lists (when you’re on a mailing list, you really want the latest state of the conversation, not just the state your repo happened to get on the last pull.)

To sum up: there are natural roles for both the DVCS and the bugtracker/forge, defined by the workflows around them. If we try and force either tool to cover the entire role of the other, the “solution” won’t be comfortable for developers and users, won’t scale well, and just plain won’t fit – no matter how much love and ingenuity we expend on a sweet technical hack.

60 comments

  1. Very sweet, deep thoughts about the right tempos of the various development activities.

    As Strong Bad would say – “My head … asplode!”

  2. There’s a related question that is probably worth asking. Your point about different timescales for different types of to do lists is a good one. That led me to the question of whether the feature road map to do list might be better house in the DVCS. At a more fundamental level, there is a simpler question. Should a bug tracking database be built on top of a DVCS, but with a central web interface to one specific repository that end users can use? Before you discount this, think about the benefit of the DVCS model in handling branching and forking. Imagine, for example, a forkable/branchable/mergable bug tracker for the Linux kernel that could be cloned at a checkpoint for various distros so that they can maintain their own bug trackers, but push and pull issues up and down the food chain. I don’t think there is a value in every developer having a clone of a a bug tracking database. But there might well be value in providing easy forking and merging for distros.

    1. >That led me to the question of whether the feature road map to do list might be better house in the DVCS.

      Well, usually it is. It tends to have a code-like access pattern.

      >Should a bug tracking database be built on top of a DVCS, but with a central web interface to one specific repository that end users can use? Before you discount this, think about the benefit of the DVCS model in handling branching and forking. Imagine, for example, a forkable/branchable/mergable bug tracker for the Linux kernel that could be cloned at a checkpoint for various distros so that they can maintain their own bug trackers, but push and pull issues up and down the food chain.

      This is indeed an interesting possibility, and one that had not escaped my notice. But the difference between this and a flock of forges with cross-references to each other would be more definitional than actual. To see why, flip your view around and think of the bug db in each repo as a submission queue, a feeder for the forge instances.

  3. As a counter counter argument about distributed bug trackers, “merging” tickets is much more mechanically resolvable than source code is. The biggest chunk of changing data in a ticket is the ongoing discussion, but each chunk of discussion is atomic and immutable, and has an author and a timestamp. “Resolution” can be done by merging the two lists and resorting by date, or in a more sophisticated system, discussion threads (and here we have just reinvented USENET, again).

    I can easily see a workable workflow where my distributed ticket system, when I’m connected to the net, does a “merge and push” each time I make a change to a ticket, and when I’m disconnected, still lets me edit tickets, with a bigger merge and push when I reconnect to a peer.

    The only really tight piece of contention is the “resolution” of a ticket, and that turns out to be less important than one would think, and it in itself mostly machine resolvable, with a fallback to a policy, and only occasionally needs a team leader override.

    1. >The only really tight piece of contention is the “resolution” of a ticket, and that turns out to be less important than one would think, and it in itself mostly machine resolvable, with a fallback to a policy, and only occasionally needs a team leader override.

      I agree with everything you wrote. But I nevertheless think you’re making the same mistake I was castigating in the original post – over-focusing on cute tricks to distribute the data representation and ignoring the fact that what would make the whole shmegege actually useful as a communication system is the single point of aggregation all those instances are feeding!

  4. I’m surprised you didn’t mention the distributed ticket system that first introduced me to the idea. SD, written by the same fevered mind that gave us RT in the first place. (And SD interoperates cleanly with RT, so if you are working in an RT shop, and yet are on a semi connected laptop, you can install SD locally, and it will keep in sync with your org’s RT).

  5. @Dale –

    I think you missed Eric’s point about timeliness issues surrounding bug tracking / resolution. You need that current info, and you need to act / react in (nearly) real time with the others who may be providing diagnostics, etc.

    Remember, the Linux kernel is just another “application´´ – that is, it can (and is!) developed in relative isolation to any distro that gets built on top of it. It already has its own buglist, etc. Just because a distro chooses to use a particular release, and then finds bugs, doesn’t mean they should “fork” the bug tracker info, but the distro developers and user community instead should submit what they find back to the kernel developers.

    Just my $0.02.

  6. First para: s/load/loud/

    There may be value in having a bug-tracker that you can check out a read-only copy for when you vanish into the cave. Obviously the value of your local copy diminishes the longer its out of sync, but in some cases its better than not being able to see the bug-tracker at all.

    I see two problems with the approach though. The people who write it, and the people that use it. Cynic? Me?

  7. Strictly serializing access to source code isn’t that important, because most changesets are mutually irrelevant and mutually commutative.

    This partly depends on the quality of the coders and also depends on the language being used. Using subversion with Verilog, for example, I’ve found that best practices definitely involve “svn up”, re-run the regression, fix anything that broke with the merge, do an “svn up” (because the regression took awhile to run) and loop, until your shit and other peoples’ latest shit together don’t smell so bad, and then you commit, and finally you do one last “svn up” to see if anything else changed in the repository around the time you were committing, and loop again if it did…

    The difference is this: When you check out a revision of software, you need to have a coherent state of it, but it’s not necessary that you have every single bleeding-edge changeset of every developer hacking on it everywhere. If J. Random Neckbeard is off in a cave somewhere refactoring a major subsystem, you don’t even want to see his changes until he decides he’s reached a good point to merge up to the public repository.

    OK, but what about when 5 or 6 people clone a repo and spend days or weeks coding together? Won’t they also want a separate version of a bugtracker? They don’t care about bug reports going in on the “current” software, but care deeply about their own reports.

    It’s not that modification conflicts are important in this context; generally they aren’t. No, the issues are (a) timeliness, and (b) having a defined rendezvous point where you can browse fresh metadata, chatter, and attachments related to your bug.

    But when you have a fork in implementation, you also have a fork in the bug reports you care about. If I’m working with a few others on a new implementation of something, I don’t necessarily care about all the bug reports coming in related to the older version.

    All these pull in the direction of centralization, or at the very least a single aggregated event feed at a known location – something more like a blog or social network than a flock of DVCS repos passing around changesets.

    As I said in a comment on your last post, I’ve never used a DVCS because everything I do is small or corporate. But this gets to the crux of the question about DVCSs. The very first bullet point in the wikipedia description of a DVCS is “No canonical, reference copy of the codebase exists by default; only working copies.”

    Now, if there is no default copy of code, then how can there possibly be a default copy of an issue tracker? Each code copy will have its own set of issues.

    Other things pull this way as well. Consider this very apt quote from Jonathan Corbet in 2008:

    Any distributed bug tracking system which does not facilitate this wider conversation will, in the end, not be successful.

    Yeah, but that’s not unique. That’s true of code as well. Public subversion, you pretty much know there’s a web interface and you can browse code easily. Git, sometimes that’s true, but often not implemented very well. If I want to take a look at a project to see if I can figure out a quick patch or maybe use the code, the issues that Jonathan Corbet raised are equally valid for the code, and equally valid for at least this developer.

    To sum up: there are natural roles for both the DVCS and the bugtracker/forge, defined by the workflows around them. If we try and force either tool to cover the entire role of the other, the “solution” won’t be comfortable for developers and users, won’t scale well, and just plain won’t fit – no matter how much love and ingenuity we expend on a sweet technical hack.

    Your description of the role of the DVCS sounds like it’s typically used in a hierarchical fashion. It’s been designed to have no true root node, but in real life, people act as if there is a root node, and that root node (or an associated place) is where you propose to put the serialized metadata.

    Again, I have no real experience in this area, but your description of how DVCSs are being used suggests that the architecture itself has been overengineered. If this is not the case; if there is a compelling reason why code must be allowed to be truly distributed, then it still seems that an architecture for the bug reports and other metadata that isn’t equally distributed might be underengineered.

    1. >This partly depends on the quality of the coders and also depends on the language being used.

      I have little doubt you are correct about the latter. I can imagine that modification conflicts would be more frequent in a language with a missing or only very weak procedural abstraction. On the other hand, I can report that the pattern of mostly commutative changesets is constant across C, C++, Java, Python, Perl, Ruby and every other language in common use in the open-source world. We can tell this by the absence of any correlation between bitching about VCSes and the developer’s source language….

      >OK, but what about when 5 or 6 people clone a repo and spend days or weeks coding together? Won’t they also want a separate version of a bugtracker?

      Possibly. But the best response to that sort of need would be to make spinning up new special-purpose bugtrackers cheap, and than create a centralized one that they can all see.

      >Your description of the role of the DVCS sounds like it’s typically used in a hierarchical fashion. It’s been designed to have no true root node, but in real life, people act as if there is a root node, and that root node (or an associated place) is where you propose to put the serialized metadata.

      That is true – flocks of DVCSes do typically have a distinguished node that is where actual shipping releases get cut. Sometimes they also have distinguished subordinate nodes that act as filters or staging areas for the public repo – the Linux kernel dev group has this kind of organization. Nevertheless….

      >Again, I have no real experience in this area, but your description of how DVCSs are being used suggests that the architecture itself has been overengineered.

      I don’t think that’s true. The fact that there’s no predetermined root gives development groups the freedom to set up many different workflows and to change them fluidly as peoples’ centrality to the dev group changes. Also, the ability to move the root node easily is very valuable; a recent example of that is Linus moving the kernel mainline to github after the kernel.org crack.

  8. Off-topic:

    “Why Most Published Research Findings Are False”

    http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/?tool=pmcentrez

    “There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.”

    Emphasis mine…

  9. I’m not sure timeliness is crucial with regards to the discussion of software (be it bugs, TODOs, etc.). More important is relevance; if I have a copy of the source, I want to see the discussion as it relates to the copy of the that source I possess. That a given bug is fixed *at this very moment* in some other version of the source is not that relevant.

    Assuming that the discussion is updated integrally with the source itself, the importance of relevance argues for the close association of discussion to the source, perhaps facilitated by a DVCS.

    OTOH, part of the reason that the DVCS works so well is that the end-product, the source, has no visible history; it exists in a given state, and how it got to that precise state is primarily of historical concern. The compiler doesn’t care, as long as it accepts what is fed to it! The discussion of software, however, only really makes sense when it is viewed in a correct historical sequence.

    So, suppose a random developer takes a DVCS snapshot, reads a TODO, works on it disconnected and offers the result for merging, including TODO list truncated and features list expanded. If there has been a concurrent discussion of the TODO list in which various parties have modified or retracted the request, the merged result of the discussion is not going to be workable, even if the source merge yields a valid result.

    Perhaps there should be entries for bugs, features, TODOs etc in the DVCS, but all commentary and discussion should occur in something like an external wiki or mailing list.

  10. Also, it is probably worth mentioning that coding practices have been improved over time, specifically to reduce the issues caused by merging. The massive monolithic switch statement, the global container of all state data and the centralised list of claimed numbers were all more prevalent in the past.

    1. >Also, it is probably worth mentioning that coding practices have been improved over time, specifically to reduce the issues caused by merging.

      I’m dubious about the causal claim. I think “reduction in the issues caused by merging” is an effect rather a cause of the trend; the underlying driver is the portfolio-of-options effect of software modularity.

  11. There doesn’t seem to be any reason a distributed bug tracker has to behave much differently to a centralised bug tracker, except when you’re off the network. Many people use Git like this (that is, as a better centralised VCS). If I’m trying to resolve a bug on a plane with no Internet I just do what I do with source code – optimistically make changes in the hope that nothing will change too drastically before getting back on the network. If it does, then I curate my comments/status changes accordingly. I think you’re falling in the same trap as the people who used to dismiss DVCS – prognosticating about the ability of others to make use of something you yourself don’t (yet?) know how to use profitably.

    1. >I think you’re falling in the same trap as the people who used to dismiss DVCS – prognosticating about the ability of others to make use of something you yourself don’t (yet?) know how to use profitably.

      While this is certainly possible, evidence that anyone has learned to use this something “profitably” is not easy to find. Distributed bugtrackers seem mainly to be used by….the developers of distributed bugtrackers. The contrast with DVCSes, which took off like a rocket within about four years of the first barely production-quality DVCS, is acute.

  12. Ultimately everything has a root node when it boils down to the practical side of things. Even in DCVS controlled projects. Otherwise each developer could just as well release their own version independently and end users would end up confused as hell by the different flavours of the same project.

    A centralized bug tracker makes sense because what matters are the bugs that propagate into the merged root node, not the intermediate changes in individual repository clones. Otherwise, as I said, each developer might just as well release their own versions of the same software with their individual flavours.

  13. @esr:

    OK, but what about when 5 or 6 people clone a repo and spend days or weeks coding together? Won’t they also want a separate version of a bugtracker?

    Possibly. But the best response to that sort of need would be to make spinning up new special-purpose bugtrackers cheap, and than create a centralized one that they can all see.

    Fair enough, but wouldn’t it be useful to clone bugs from the main tracker into the new one, then close them, then merge the closed information back into the main tracker when the source is merged back in?

    your description of how DVCSs are being used suggests that the architecture itself has been overengineered.

    I don’t think that’s true. The fact that there’s no predetermined root gives development groups the freedom to set up many different workflows and to change them fluidly as peoples’ centrality to the dev group changes. Also, the ability to move the root node easily is very valuable; a recent example of that is Linus moving the kernel mainline to github after the kernel.org crack.

    If this is not the case; if there is a compelling reason why code must be allowed to be truly distributed, then it still seems that an architecture for the bug reports and other metadata that isn’t equally distributed might be underengineered.

    I didn’t do a very good job of saying “either/or” in the original, but that’s what was meant. For your example of moving the root (or, e.g. cloning a project in its entirety, but with some developers choosing to work on both versions), distributed control of the metadata might be useful. OTOH, a clean metadata architecture can allow ease of some of these things (wholesale move, for example) without being distributed.

    Distributed bugtrackers seem mainly to be used by….the developers of distributed bugtrackers. The contrast with DVCSes, which took off like a rocket within about four years of the first barely production-quality DVCS, is acute.

    I can believe the need is nowhere near as acute. After all, viewed one way, the point of DVCS over VCS is to reduce the amount of required communication, but the entire point of issue tracking is to increase communication capability.

    However, good metadata tracking is at least one rung up the conceptual ladder from good version control, so it wouldn’t surprise me if best practices and usage cases for tracking lag those for version control by a few years. Can anybody here compare and contrast the early history of version control vs. issue trackers?

    1. >Fair enough, but wouldn’t it be useful to clone bugs from the main tracker into the new one, then close them, then merge the closed information back into the main tracker when the source is merged back in?

      Um…if you’re going to do that, why not just use the main tracker in the first place? Seems to me the only point of creating a new one for this subgroup is if you want to treat old bugs as clutter and ignore them.

      >For your example of moving the root (or, e.g. cloning a project in its entirety, but with some developers choosing to work on both versions), distributed control of the metadata might be useful. OTOH, a clean metadata architecture can allow ease of some of these things (wholesale move, for example) without being distributed.

      Right. The latter possibility is the one I’m working.

      >However, good metadata tracking is at least one rung up the conceptual ladder from good version control,

      Why do you believe that?

  14. I never have understood the point of a distributed bug-tracker. My reaction to the idea has always been something like: “Neat idea, but what good is it?”

    I think it took reading this post for me to really figure out the core of what bothered me about them, and it’s this:

    Bugtrackers are not about what’s in your repo, they’re about what’s in your distro. That is, the bugtracker’s function is to record details of and conversation about the behavior of the actual software you’re shipping. This can be either sub-optimal behavior in a current version (bugs/warts) or desired behavior in some future version (feature requests).

    Disconnected operation makes little sense here, to me. If our hypothetical developer goes off to a cave and fixes some bugs, what happens when he comes back and (assuming he’s been gone long enough) someone else has already fixed them? Now you have two sets of code, which is fine, as these are easily merged, and if we’re pre-supposing a DVCS, both trails of development can even be recorded. But you also have two sets of ticket updates…what do you do about them? A ticket is supposed to be a conversation, and is intended for consumption by humans, who typically want to get a _history_ of a particular issue. If you were to merge our hypothetical ticket updates into a linear list, sorted by time, I don’t see how that would be very easy to read, even for the project’s developers, much less user/bug reporters. And if you kept it as a tree structure (it is a DVCS, after all) then how do you go about getting out the history that you went looking for anyway? It seems to me that you just end up with a tree-shaped collection of developer notes, closely correlated with actual code changes. And isn’t that what _commit messages_ are for?

    Anyway, I never saw much point. To me the time to update a ticket as “fixed” is when a bug gets fixed in “trunk” (whatever that may mean for your project, main repo clone, tip, whatever you want to call it). Everything before that is a conversation, and if you’re having a conversation by yourself in a cave, then perhaps you have larger problems than whether your bug-tracker is decentralized or not. :)

    1. >I never have understood the point of a distributed bug-tracker. My reaction to the idea has always been something like: “Neat idea, but what good is it?”

      Yeah. By contrast, it seldom takes any length of time for a programmer to figure out that DVCSes beat the living snot out of centralized VCSes, once they actually understand the feature set – because, in particular, you can emulate a centralized VCS workflow with a DVCS, but the reverse isn’t true.

      Just by coincidence, I was talking with another A&D regular on IRC last night who told me that in her experience developers over the age of about 45 are very resistant to moving to DVCSes. Here’s the interesting part: it’s not because they don’t get that DVCSes are better tools – in fact, one common reason for resistance is “it would make changes by random people too easy”.

      /me shakes head in disbelief

  15. @ Matt:
    “if I have a copy of the source, I want to see the discussion as it relates to the copy of the that source I possess. That a given bug is fixed *at this very moment* in some other version of the source is not that relevant.”

    I completely disagree. If you know a bug is fixed in another version of the source, you won’t waste time trying to trace the bug. Instead, you’ll diff against the version of source with the bug fixed in order to find the code that resolves the issue, then patch it into the earlier version if necessary.

    I did this just recently with FlightGear. The latest builds run too slowly on my old machine, so I’ve reverted to 2.0 (current version is 2.4+). But there was a bug that kept the weather update system from working correctly, and it was trivial to back-patch the fix into my copy of 2.0.

  16. @Cathy

    But practically speaking, in many (most?) cases (for various parties), what you have is not the latest or best version of the source, but a packaged version that corresponds to a particular point in the distributed history of the source. Tracking issues against a particular version is more relevant than against a cloud of unspecified modified sources.

    Your example is also relevant, but it is a minority situation, to my experience.

  17. > But the best response to that sort of need would be to make spinning up new special-purpose bugtrackers cheap, and than create a centralized one that they can all see.

    And then what do we do when that team merges their code changes back into the mainstream. I don’t want their bug tracker data to be lost, but neither do I want a pile of random decaying old bug trackers to have to search in turn. I want that “special-purpose bugtrackers” merged back to the “centralized one”. And once you’ve made that doable and easy (especially in the general case), well, you’ve just build a distributed bug tracker…

    1. >And once you’ve made that doable and easy (especially in the general case), well, you’ve just build a distributed bug tracker…

      Have you? I think you’ve done what I want to do, which is write a tracker with good state export and input facilities.

      The way this differs from a “distributed tracker” is that nobody has to do git pull or equivalent history merge to browse bugs.

    2. >And once you’ve made that doable and easy (especially in the general case), well, you’ve just build a distributed bug tracker

      OK, now I think the term “distributed bug tracker” is being broadened to the point where it’s too fuzzy for good analysis. Time to re-focus the discussion.

      The specific technical approach I was criticizing in the OP as “distributed bug tracking” is the one where you embed bug data in your repo, and in order to get new bug data you have to pull history and merge from another repo. Distributed bug tracking has two goals: (1) unify the decentralized management of bug queues with the decentralized management of code, and (2) abolish the requirement for one unitary bug-report feed at a fixed location.

      You, and Patrick Maupin, are now talking about something different (and, in my opinion, more useful). You’re talking about networks of bugtrackers that can pass reports around among each other. A key difference in this model is that bug storage is not unified with code storage – the bugs don’t live in repos, and the information flow in the bugtracker network is not tied to the information flow between repos.

      This is also why the Linux kernel dev group’s approach of tossing around mail is not really very relevant. They are also not trying to make bugs live in repos and have the same information flow as repos. It is telling, I think, that Linus concedes the value of centralized bugtrackers for small, tightly focused subgroups. In his own way he seems to be moving towards the model you and Patrick are also groping for – multiple bug trackers with careful curation and the ability to pass state to other trackers.

  18. @Mark Atwood:

    And then what do we do when that team merges their code changes back into the mainstream. I don’t want their bug tracker data to be lost, but neither do I want a pile of random decaying old bug trackers to have to search in turn. I want that “special-purpose bugtrackers” merged back to the “centralized one”. And once you’ve made that doable and easy (especially in the general case), well, you’ve just build a distributed bug tracker…

    Exactly what I was getting at.

    @Kevin Horn:

    Bugtrackers are not about what’s in your repo, they’re about what’s in your distro.

    Really? I guess I should tell the people at work that we shouldn’t even bother to spin up a bug-tracker until we ship silicon; that it’s completely useless for all the internal validation and verification work that we do; and that it’s still worthwhile using one, even though it might only have two issues in it instead of the four hundred we found and fixed internally.

    (Mind you, at work we have zero reason to use a distributed bug tracker, and no real urgency to use a DVCS, either.)

    @esr:

    Um…if you’re going to do that, why not just use the main tracker in the first place? Seems to me the only point of creating a new one for this subgroup is if you want to treat old bugs as clutter and ignore them.

    Depending on the state of your surgery, old bugs might be useful, but marking that they’ve been fixed on the main tracker in a branch you haven’t released yet might not go over very well. But even if you decide to dispense with copying old bugs into your branch, what about the metadata for the new bugs on the bugtracker you created for the branch? (See Mark Atwood’s comment above.)

    Why do you believe [that good metadata tracking is at least one rung up the conceptual ladder from good version control]

    Both personal experience and watching other peoples’ learning curves and development habits. Exhibit A is Kevin Horn’s response that effectively says that trackers are only for what you are shipping, but version control is for what you are developing. For his project(s), he’s probably right! And people who think that non-distributed version control systems are adequate for their projects are probably right, as well.

    Sure, trackers are easy to learn and use as a trouble-ticket system from a customer/developer interaction perspective. But thinking of them as an interactive development tool, rather than a simple “to-do” list, is useful when the required communications between project developers hits a certain complexity.

    So, the average developer probably comes to the realization he needs a VCS long before he comes to the realization that an issue tracker can be a bit more than a customer ticket system. The average open source developer working with others probably even wishes for a DVCS before he comes to that realization. The lone wolf programmer certainly will find a use for a VCS before he finds a use for an issue tracker (and might never find a use for a DVCS).

    So maybe “one rung up the conceptual ladder” isn’t quite the correct way to put it. I’m not arguing that the concepts are deeper, just that understanding of the necessity for good version control will come much quicker than the understanding for good issue tracking in a lot of real-world scenarios.

  19. @esr:

    Have you? I think you’ve done what I want to do, which is write a tracker with good state export and input facilities.

    Well, maybe we’re discussing the same thing.

    The way this differs from a “distributed tracker” is that nobody has to do git pull or equivalent history merge to browse bugs.

    But, surely if you spin up a separate bug tracker that can be browsed (that you’re later going to merge into the main tracker), that separate bug tracker is notionally, for its limited lifespan, associated with a separate repository that is shared among the same audience as the tracker. And surely if you have a browseable tracker, it’s not that much more effort to make the code browseable as well?

  20. I do not get the point of this discussion.

    In a source code repository, I am interested in two separate things, the final product without a history, and the history.

    In a bug tracker, I am only interested in one single thing: The complete threaded history.

    A distributed bug-tracker would be nothing but a long list of communications (issues), which could be extended to a directed graph by branching and merging. Merging would only insert new communications into the list at their proper place. The distributed part would only be useful for people who are cut-off from communicating with the rest. If it would be useful at all.

    Such a thing would be rather simple to implement using Eric’s event tagging of a few posts back and some back linking of each comment to the thread.

  21. I think this is a strong argument, but I’m not really sure that it kills the case for forkability of issue tracker databases. I want to think about it more.

    I’m a bit surprised that the point has not yet here been made that the Linux kernel development community, while they have made some use of bug tracking systems, have resisted making use of them in a centralised manner.

    Linus summarised the central objection: “The thing is, bugzilla is totally broken because it’s designed to help track bugs, but it’s *not* designed to actually handle the much harder problem, which is to actually get the *right* developers to be aware of the *right* bugs!” (which is a harder problem for the LK than for most other projects). Also, he *wants* old LK bug reports to be forgotten if they are not acted on, on the grounds that their value declines to the point of negligibility in time. Hence the LKML-centric, “mail, remail, and remail again” approach to bug tracking.

    See http://yarchive.net/comp/linux/bug_tracking.html and see Ted T’so’s comments on the Debian BTS.

  22. What people need first hand is a way to manage bugs/issues offline, without the requirement of using the Web interface.

    So there is a response, which is bugtrackers using mail as an interface, as debbugs (the Debian, and some of the Gnu projects’ bugtracker), for instance.

    Also, having a clone of (a portion of) a bugtracker database locally available (with the commandline or other tools) is also handy, much like you can have a copy of the full code history with a DVCS, which may not be possible if it’s not mail based. Here, a distributed bugtracker can help (SD for instance looks very interesting to me : http://syncwith.us/sd/).

    But I also believe that, as DVCS without a community workflow can lead to a mess (where’s the reference version, a release ?), and similarly, traditional bugtrackers offer a reference point to know whether a bug is open, fixed, or whatever vs a reference version.

    But on the other hand, bugtrackers are built on the assumption that the developers (the one who have the right to modify a bug) are right, and the users / reporters (the one who can’t usually reopen a closed bug) aren’t. But this dyssimetry of privileges is not the way it should be : you should have a mean to manage your own list of issues that you consider open, even if a developer considers it fixed.
    In such a sense, the distributed approach can better model the reality : the discussion thread may be common between different clones of one ticket, whereas metadata aren’t, because they are essentially relative to the one who looks at them.

  23. > OK, now I think the term “distributed bug tracker” is being broadened to the point where it’s too fuzzy for good analysis.

    Maybe. I guess I view it in fairly sweeping terms. If a “distributed VCS” means that you don’t have to do centralized check-ins, surely a “distributed bug tracker” is one way to describe whatever tool is useful to provide a matching granularity to the discussion surrounding the code that is being checked in.

    > The specific technical approach I was criticizing in the OP as “distributed bug tracking” is the one where you embed bug data in your repo,

    Ah, that’s an implementation detail. Might be the easiest way to envision how you distribute bug tracking, but as you point out, there are some issues there.

    > the bugs don’t live in repos, and the information flow in the bugtracker network is not tied to the information flow between repos.

    I think tools to push code and associated issue tracking could be highly useful. Obviously, you don’t want too tight of a coupling, but it might be nice to have the default be to push code and associated metadata.

    In a DVCS, a developer’s own private repo might not have a bug tracker. But there is usually a weak hierarchy of development, and the next level up, where a few developers work together, might want a coupled repo/bugtracker.

  24. I think the real “quantum leap” evolution of version control will not be the ability to clone a repository for a fork in a project. The real quantum leap will be to merge, link and clone bugs across different projects if a bug is caused by project inter-dependencies.

    For example, lets say I write a php application for internal use at my company, and I get a bug report. Then lets assume that I determine that the bug is caused by a bug in php (it actually happened look at php bug50698). The ideal workflow would be to use a workaround in my application, file a bug with PHP, and then remove the workaround when the bug is fixed in PHP. However, that workflow would be made much easier if the closing of a bug on bugs.php.net could trigger a status change in my internal bug database. The ultimate goal would be if a bug being closed in PHP (or even a proposed patch) would lead to my build server building building, the patched version of php and then building a branch of my internal app without the workaround and doing automated unit/functional/smoke/behavioral testing. However, the first step is linking bugs in the bug tracking database. everything else will follow.

  25. @Patrick Maupin

    > Really? I guess I should tell the people at work that we shouldn’t even bother to spin up a bug-tracker until we ship silicon; that it’s completely useless for all the internal validation and verification work that we do; and that it’s still worthwhile using one, even though it might only have two issues in it instead of the four hundred we found and fixed internally.

    When I say “the shipping version” I mean not only a version that has been shipped, but also a version that is going to be shipped. So in SVN terms, the trunk of the repo, or in DVCS terms the tip of the “central” repo clone. Pretty much every project out there has one canonical version that is what either has or will end up in the hands of users. Of course a bug-tracker is useful before you actually ship/release your software. I thought I had made that clear further down in my comment. Apologies for any confusion.

  26. @Kevin Horn:

    > Pretty much every project out there has one canonical version that is what either has or will end up in the hands of users.

    Yes, but the problem is that with a major subsystem re-write, the canonical “is” is not the same as the canonical “will be,” and it often makes sense to not yet put the “will be” on the canonical trunk for awhile, because doing so will cause pain to other subsystems that is not commensurate with the gain of having the new development on the trunk.

    It’s the middle ground between the thing that everybody is working on and the thing that one person is working on that I was trying to address. Assume a major bug in shipping software. Assume a major rewrite happening off-trunk. Now assume somebody puts in a patch to workaround the major bug (but not all the other issues) on-trunk (because the rewrite is going to take a long time, and users need a fix now):

    – The original bug belongs with both the rewrite and the trunk — it needs to get fixed both places.

    – The conversation of how the original bug got fixed in the trunk might (depending on the extent of the rewrite) be useful to the people doing the rewrite; but

    – The fact that the bug was fixed and closed on the trunk should not impact the status of the bug in the rewrite (unless the same fix is applied).

  27. @Justin Dearling:

    > The real quantum leap will be to merge, link and clone bugs across different projects if a bug is caused by project inter-dependencies.

    Agreed. And if/when we have a standard way of doing this, then the Linux kernel itself will most easily be viewed as a collection of much smaller projects.

  28. >Code, it turns out, is not like a database.

    I don’t really agree with this perspective. I think it is more accurate to say that code as we presently quantize it is not like a database. The problem with RCS/SCCS type systems is that you can only lock the file as a whole, not a small part of the file (such as the function you are working on, or the few lines you are working on.) From my point of view merge systems are a solution to this quantization problem rather than the lock-unlock model. DCVS-es are more a solution to a problem more prevalent in bazaar models — namely that ownership of the code is less firmly defined, and task assignment is a buffet rather than a structured menu. You have a bunch of different people working on different agendas without a single centralized goal (get release 7.5.3 out), and the DVCS is a platform for merging these goals into multiple threads.

    I believe that seeing code as a database of pieces is a powerful way to think about the structure of software. It is one of the great features of Smalltalk that kind of got lost in the mix. Putting it together with great tool support would make software development much easier. Most great development tools are kind of ghetto-ized versions of this, where they generate an approximate database structured model of the code out of the text (or in some cases compiled meta data) to provide the analytics that make writing software easier. (Think, for example, Intellisense or Linq and their equivalents.)

    Nonetheless, my comment was more about your bug tracking thing. I think you are right, but I’m not sure it is so much about the workflow as it is about the workforce. Code is worked on by the lone wolf, bugs are worked on by groups of people. A bug has a community, a code change does not. Communities work better when there is a village hall for bake sales and coffee mornings. Lone wolves prefer to sit in the basement in the dark listening to Pink Floyd.

  29. Olivier Berger Says:

    What people need first hand is a way to manage bugs/issues offline, without the requirement of using the Web interface.

    But that is only offline / replicated bug/issue tracker; something you can get for any supported bugtracker using Mylyn ALM when using Eclipse platform.

    What Eric wants is federated set of issue trackers (think IRC or Usenet), not distributed.

    1. >What Eric wants is federated set of issue trackers (think IRC or Usenet), not distributed.

      That is quite right. In fact – and this is something I’ve never revealed anywhere before – my private working name for the forge that may evolve from Roundup is “Federation”.

  30. The Prime Directive of Federation is you do not tell people it’s called “Federation.”

  31. I think that perhaps the even bigger reason why early CVSes used locking is that it’s trivial to implement compared to automated diffing (that isn’t horrible).

  32. @Sigivald:

    > I think that perhaps the even bigger reason why early CVSes used locking is that it’s trivial to implement compared to automated diffing (that isn’t horrible).

    This. Although the switch to concurrent versioning was met with at least as much kicking and screaming as the switch to distributed versioning, because it was different and outside of people’s comfort zones.

  33. Sigivald Says:

    I think that perhaps the even bigger reason why early CVSes used locking is that it’s trivial to implement compared to automated diffing (that isn’t horrible).

    BTW. when was diff3 / rcsmerge (3-way diff) algorithm invented? When was diff and patch?

  34. Something in San Diego goes red at three in the morning and the guy with the pager can go to one place and find everything that was done to that box in the past five years – or in the last five minutes. That’s the beauty of issue tracking systems.

    I worked in one of the first successful ISPs and their goal was to see how far they could push their issue tracking software to solve all of their company’s problems. Pretty damn far, it turns out, as requests, assignments, to do lists, workflow, and history are pretty universal to any group activity. It had a decent API and barely adequate data model, so you could whip up something push data into and out of it with Perl scripts or churn out a report or graph with my favorite toy of the 90s, Powerbuilder, in a couple of weeks.

    So although you would think a big ISP lived and died by its network monitoring software, the issue tracker was far more important to day to day life (although I’m sure the NMS had unused potential). Now if only we had found a good solution for configuration management…

  35. esr:

    While this is certainly possible, evidence that anyone has learned to use this something “profitably” is not easy to find. Distributed bugtrackers seem mainly to be used by….the developers of distributed bugtrackers. The contrast with DVCSes, which took off like a rocket within about four years of the first barely production-quality DVCS, is acute.

    I don’t see your point. So what if few people use them? Relatively few people are using ML-derived programming languages; are they the “wrong answer”? No, they’re just not what most people seem to want, need or be aware of for most projects. I don’t doubt that there’s or will be less interest in distributed bug tracking than in DVCS’, but that is a different claim altogether from them being “the wrong answer”. This is rather like your narrow-minded stance on C++. Distributed bug trackers don’t have to be all things to all people. Your argument-by-time-line demonstrates nothing.

  36. The original patch program was written by Larry Wall (who went on to create the Perl programming language) and posted to mod.sources[1] (which later became comp.sources.unix) in May 1985. – Wikipedia

  37. esr Says:
    > Distributed bugtrackers seem mainly to be used by….the developers of distributed bugtrackers. The contrast with DVCSes, which took off like a rocket within about four years of the first barely production-quality DVCS, is acute.

    This is because actually existed distributed bugtrackers are not production quality. If someone wrote one that did not suck, maybe they would take off like a rocket – or maybe not. The experiment has not yet been done.

    Ideally a bug tracker should report all problems detected on versions ancestral to the version you are working on, and the existence of new versions that purport to fix those problems, but should not automatically report problems reported on code on a fork different from that which you are working on.

  38. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.

    But, but, but… The science is settled!

  39. The problem with RCS/SCCS type systems is that you can only lock the file as a whole, not a small part of the file (such as the function you are working on, or the few lines you are working on.)

    I’ve long wanted a development system where we didn’t keep our source code in flat files at all.

  40. Distributed bugtrackers as embodied solve a few problems that non-distributed ones don’t:
    1) The close association of code and bugs. Keeping the buglist in the same place as the code seems to make sense; moreover, the ability to associate a bug with a particular branch – such that it will then ‘follow’ that branch and all forks of that branch – is an extra level of tracking that non-versioncontrol-integrated bugtrackers can’t really do.
    2) disconnected use – as useful for the bugtracker as it is for the code; fixing code on a plane, in a cave, etc. requires knowing what needs fixing

    There seem to be two main types of distributed bug trackers: those that keep the bugs next to the code (and branch them when the code branches) – this includes bugseverywhere – and those that keep the bugs in what’s essential a branch of their own. The latter lose advantage 1) but still keep advantage 2) and seem to gain something from the closer adherence to the standard mental model of how (centralized) bug trackers work.

  41. @esr
    > I think you’ve done what I want to do, which is write a tracker with good state export and input facilities.

    When the “state input” is the same as “add or modify an entry” then once again, its a distributed bug tracker. Inputting diverged state *is* “merging”, pretty much by definition. And even the simple operation of adding a ticket or a ticket comment is really just a “state input”, where the state that is being input/merged is just a bug database with a single entry.

    > The way this differs from a “distributed tracker” is that nobody has to do git pull or equivalent history merge to browse bugs.

    I can browse code without doing a pull. I just have to browse it remotely on canonical central reposatory (or more likely, over it’s web gui). Likewise, you could browse bugs without doing a pull, by remote connecting to the canonical central repository and reading thru the database there (or again, more likely, over it’s web gui).

    But if you want to read bugs *locally* without a low latency network connection, you have to copy that canonical bug database to your own machine. Even with centralized bug trackers, you can do this, and this is done. I have, more than once, when faced with flacky or high latency network connections, SCPed down the database behind a bugzilla or RT instance, so I could navigate and read them quickly and with agility. Of course, I had to treat it as a read-only copy, which was annoying, because there was no good import/merge feature letting me copy changes back.

    I flat guarantee, if someone builds a working “centeralized” bug tracker that has this lovely “export state / input diverged state” feature you speak of, than I and more than a few other people will promptly start using it as a distributed bug tracker.

  42. Something I did not notice in this discussion is anything about automating the selection of changes to collect into a version release of the software product. I think that this is called software configuration management in some circles. I believe that to do that properly requires keeping track of the connections between issues in the bug tracking system, changesets, and the content list of a release.

    Even assuming that having more automation of this area, I haven’t thought about it enough to say whether it imposes requirements on the (D)VCS and the bug tracker or could be built on top of such systems later, without influencing their designs. I just want to mention the subject in case it ought to influence the current discussion.

  43. This may look like a silly question, but I have never really used either a wiki or a DVCS, and from the discussions I have seen on your blog, and on the web in general, they sound very similar; What are the differences between a wiki and a distributed VCS?

  44. One of the reasons a distributed source versioning system is useful is that you have a full state with you, without needing constant network access. For that purpose, some way of duplicating the entire state of the ticketing system is useful; most of the relevant bits might already be in your mailbox (possibly itself a local cache of an IMAP box, say) but if you need to dig into the history of other tickets that’s different again. Wikis to some extent reinvent the versioning system but then do not provide the replicability, and so lock you into needing network access. In updating patterns they’ll be closer to code than to bug tracking tickets, though. In the locality-for-portability sense, stuffing as much as you can into the same DVCS back-end is fairly natural, excepting of course the above argument. But given that argument, what can we do to enable the (temporary) network non-dependence functionality?

    Oh, and William: A wiki stores versions of “content pages” intended to render to web pages, with a heavy focus on providing an “edit” button so you can change the content from within the browser, whereas a DVCS stores source files (typically plain text of some sort, coming from a programming background) in a distributed fashion and typically has a command-line tool to manipulate it (perhaps in addition to integrating with integrated development environments), so is much more programmer oriented. Both provide versioning, history, metadata tracking. The front-ends and purpose background are different though, and a VCS (not just the D type) typically comes with much more versatile merging, branching, and other concept manipulation tools, and less web frontage. You could write a front that builds a wiki on top of a VCS but the other way around would likely be a lot less convenient–the existence of VCS web front-ends and (ab)use of HTTP as transport notwithstanding.

  45. I just found the mention in Eric’s Understanding Version Control:

    One interesting present-day use of RCS is a revision store for wiki engines. It fits there; filesets and changesets are not necessary (the unit you want to save is always a single page) and speed is paramount.

    There is something weird about that page. It is encoded in UTF-8, and my browser is normally set to UTF-8, but every time I open that page the browser encoding is changed to ISO 8859-1 and I have to change it back to UTF-8 to view the page properly.

  46. (Eight months late to the discussion)

    One of the compelling use cases for decentralized bug tracking is mobile devices. Being able to use a clone/push/pull workflow for a bug tracking database is great for when your network connectivity is flaky or absent.

    Disclosure: I work for SourceGear. We develop Veracity, which has a distributed bug tracker. Veracity core is open source. We also have an iPad app for Veracity, and it supports bug-tracking with full offline operation, but it is NOT open source:

    http://www.ericsink.com/entries/veracity_ipad_app.html

  47. 1. Not all bugs are bugs, they could be tasks, issues etc.
    2. Not only developers (i.e. persons who change the code in the repo) work with “bugs”, what about QA, tech writers, techsupport people?
    3. Good tracker contains workflow (for tasks) and splitting the task into pieces will probably break that workflow. And after sync you can get broken, inconsistent piece of “bugs”.

  48. Hi, thank you for this post I agree with you that modification conflicts would be frequent and severe enough that they could only be prevented by granting programmers temporary keep-out exclusions. very useful information.

Leave a Reply to Charles Stewart Cancel reply

Your email address will not be published. Required fields are marked *