I hate having to be the heavy…

I nearly issued a forking threat a few minutes ago. Only the second time I’ve felt a need to do that and the first was in 1993, so this is not something I do casually. And I drew back from the brink.

But I may have to if the maintainer I’m dealing with doesn’t clean up his act. His library is critical to one of my projects, but his behavior has been increasingly sloppy and erratic lately. He made a serious design mistake which he’s been trying to paper over with kluges; the kluges have made the code unstable and the latest shipped version is actually broken to the point of unusability without a patch.

Some standards have to be maintained, and this guy is breaching most of them. I told him by email “you have set yourself up for serious public embarrassment, which I will (reluctantly) deliver if you don’t resume behaving like a responsible maintainer.”

I hope he gets the message…because I don’t want to threaten him with a hostile fork, but he’s backing me into a position where I think it may be my duty to aim that nuke at him. His library has other users, after all; he’s not just failing me but that whole community.

I’ll do what’s necessary…but I hate having to be the heavy. *Grumble.*

96 thoughts on “I hate having to be the heavy…

  1. >kludge

    Different word. You are excused for not knowing this: they became confused in the early 1960s.

    “Kluge” is originally American, probably from Yiddish/German kluge (clever), and can be backhandedly positive. “Kludge” is originally British, probably from Scots “kludgie” for a toilet, and is always negative. I have a report of at least one U.S. military tech who used both words with distinct meanings, and it appears they became confused by a long-ago Datamation article that imported British “kludge” to the U.S., whereupon it became confounded with native “kluge”.

    However, you are correct in a sense different than you intended. The code I’m thinking of is indeed a kludge – I didn’t misspell “kluge”, but I probably should have used the other word.

  2. I don’t suppose this is the Python IRC library you were talking about in the last post?

  3. >I don’t suppose this is the Python IRC library you were talking about in the last post?

    Since I do not yet wish to embarass the maintainer in question and still hope to avoid doing so, I’m not going to utter any clues at all about this.

  4. I should find a QST article I recall from one April issue in the 1960s or 1970s…the purported author’s callsign was KL0OGE.

  5. I swore up and down I wouldn’t bring the Second Amendment into other discussions on here, but I feel compelled to, because there’s an important similarity to be drawn here.

    Just as the right to keep and bear arms in defense of oneself and one’s country carries with it a duty to think about when one will take up arms, so, too, does the right to fork carry with it a duty to decide when to fork. In both cases, the right is meaningless without the thought process and the willingness and ability to exercise it.

    As with taking up arms in defense of one’s country, the cases where a fork becomes a duty are, or at least should be, very few and far between, the consequences harmful to one or more, and even to oneself, and yet the benefits outweigh the harm.

    It seems you’ve found one such case.

  6. Hmmm. As an admin who is responsible for security, I really would like to know more. As a human being, I applaud your restraint.

  7. Is it really that bad a thing to have your library forked? Not that my own coding skills are such that I’ll be releasing a library any time soon , but I can’t say I would be all that offended. If their fork was better I would probably start using it myself.

  8. Is it really that bad a thing to have your library forked? Not that my own coding skills are such that I’ll be releasing a library any time soon , but I can’t say I would be all that offended. If their fork was better I would probably start using it myself.

    This is not about the Open Source licensing, but about human feelings of possessiveness. Basically it boils down to ego issues. Hostile forks are not productive to Open Source in general as it divides the community and creates ill feelings between developers and users, so I can understand why ESR is hesitant about it.

    Sometimes it is inevitable, but mostly those kinds of forks are taken up from dead or nearly dead projects. Forking an active project is always a sensitive issue.

  9. Is it really that bad a thing to have your library forked? Not that my own coding skills are such that I’ll be releasing a library any time soon , but I can’t say I would be all that offended. If their fork was better I would probably start using it myself.

    Yes, it’s very serious indeed.

    A form of an active project is a serious attack that calls into question the technical competence of the original maintainer, and can do critical damage to the maintainers prestige and reputation/standing in the community. Which is also a crushing ego blow. And when you consider that the heart of open source is talented programmers voluntarily working on projects, with their reward being largely prestige among their peers and ego gratification….

    Performing a hostile fork of someone’s active project could wind up essentially ruining that person for open source development, driving them out of the community. So no, not a step to be taken lightly.

  10. > Different word. You are excused for not knowing this

    Ooooph. While I have an appreciation for both subtlety and precision in language, I have no hope for this distinction returning to general comprehension. Nonetheless, I appreciate being aware of it.

  11. This is not about the Open Source licensing, but about human feelings of possessiveness. Basically it boils down to ego issues. Hostile forks are not productive to Open Source in general as it divides the community and creates ill feelings between developers and users, so I can understand why ESR is hesitant about it.

    Agreed.

    It can also impact the original owner’s ability to find paying work. The way I see it, in an open-source world, your code is your reputation is your livelihood. If I were in ESR’s shoes, I’d be weighing my concern over the stability of my own code against my concern over doing this person real monetary damage. It’s his money, not mine, but part of me also sees benefit in a larger supply of productive coders in the system.

  12. @Paul Brinkley:

    The way I see it, in an open-source world, your code is your reputation is your livelihood.

    Meh. I’ve written some great code, and I’ve written some terrible code. Very little of my open source code is actually in the “great” bucket — most of that is internal proprietary stuff. OTOH, most people perusing my OSS projects would come to the conclusion I can code, and that with a bit of training, I could probably even code to suit them.

    Personally, I prefer Linus’s approach: forks happen.

    In general, “not forking” shouldn’t happen because of a threat. “not forking” happens because of a shared vision, and threats are usually counterproductive to developing such.

    There are at least four good responses to “fix it or I’ll fork it”, the first one being, of course, to fix it, if you have the time and agree it’s a problem and can see a clear way to fix it without causing issues for other library users. But another response is equally valid — “fix it yourself, and when I’m less busy, I’ll look over your changes and decide whether and how to incorporate this functionality.” And of course “I’m busy right now, would you like to maintain the project for awhile” is a valid response as well, but that only works if you’re convinced the potential new maintainer won’t do more damage than good as far as your other users are concerned.

    And, esr, I know that when people are blindly, willfully ignorant of your needs, and release code that breaks things that were previously working, after the third or fourth time of explaining the issues to them, it becomes very difficult to explain the facts in a non-hostile way, but (especially given who you are), I think you should try.

    I really don’t see the message “I appreciate the great codebase you wrote, but the direction you are taking it will not work for me, so I will need to fork it and give it a new name to reduce the likelihood of confusion, and btw, I am sure there are several others out there with the same needs as me, so I will be advertising the project to attempt to attract other users and developers” as a threat.

    Unless, of course, it’s combined with “and I will write mercilessly about how stupid your decisions were on my blog.” Which just isn’t a nice thing to do to somebody who gave you some code that (at some point) was worth using.

    Just my two cents.

  13. It’s a bit saddening that there’s a whole bunch of developers raised on DVCS that do not know the origins and implications of what forking really means.

    I’ve seen several times on mailing lists where people use the word carelessly and attract major flak from the project owner. In their world, a fork is simply a temporary alternate development stream, likely to be merged back through new-fangled pull-requests or some other fancy-pants terminology.

    Back in the days, a fork was a declaration of war, leading to proper hostilities between the two groups.

    ffmpeg vs. libav; NetBSD vs. OpenBSD; X.org vs. whatever-was-before. Those kinds of world-shattering events.

  14. @Lars:

    It’s a bit saddening that there’s a whole bunch of developers raised on DVCS that do not know the origins and implications of what forking really means.

    No, what’s saddening is old developers who haven’t come to terms with the reality that forking no longer has to be all or nothing.

  15. A project’s capital is its developer and user community. A fork that divides these communities is very bad. Due to network effects, the sum of the divided communities is much smaller than the value of the undivided community.

    Linux’ forks do not divide the community. The proposed fork will.

    My personal feeling is that a dividing fork is only required if the current maintainers themselves dammage the developer and user communities.

  16. @Winter:

    Linux’ forks do not divide the community. The proposed fork will.

    Assumes facts (deliberately) not currently in evidence. For example, if the project currently has but a single maintainer, and if esr’s needs are different than those of the majority of the current users, there is not necessarily any reason for a fork to be other than a temporary development branch with a different focus.

  17. A hostile fork is more a glove across the cheek than anything else. It need not necessarily have the destructive effects others have described, though it certainly can. It’s all in the reasons given for why the fork was done.

    In this case, if Eric’s assessment is correct and the library is indeed broken (as defined by the standards it claims to adhere to), then a fork would be damaging, though not insurmountably so, if the original maintainer adopts a pose of “I see your point, here, let me take those fixes” or “I disagree with your assessment, but here’s how we can make it work” or something else reasonable.

    Of course, if he was inclined to be reasonable, there wouldn’t be a need for a fork in the first place…

  18. @Patrick
    I simply got the impression from Eric that more (many) people might be affected by the fork. But if not, then other considerations obviously apply.

  19. I’m curious what the design mistake actually was.

    While it is tempting to take ESR’s refusal to deny as confirmation that this is the IRC/Unicode issue discussed in the preceding journal, I don’t think so. That issue should be quite simple to fix now that it has been identified (especially since irker would be satisfied with a simple discard of unicode-unclean lines). This sounds like something more insidious.

  20. > Back in the days, a fork was a declaration of war, leading to proper hostilities between the two
    > groups.
    >
    > ffmpeg vs. libav; NetBSD vs. OpenBSD; X.org vs. whatever-was-before. Those kinds of
    > world-shattering events.

    gcc vs egcs… sometimes forking either result in merge, or in fork taking over the original project.

  21. Everyone’s remarks here seem pretty consistent with “some forks are amicable and some are casus belli; the difference is intent”.

    I consider it common sense that a fork may cause hard feelings if it came from “fix it or I’ll fork it”; that “don’t fork it; just take it over as I’m too busy” is a fine way to avoid that problem; and that “yes, fork it, as we have different visions on future direction” is also amicable. It’s also apparent that a fork should be a culturally formal thing, like a contract, not to be undertaken lightly. (In other words, it’s definitely possible to be both justified and a jerk about it.)

    I’m of the opinion that ego shouldn’t cause anger over someone needing some improvement to someone else’s code, but that’s a somewhat weak opinion. I wrote a jigsaw puzzle implementation years ago, and I know how it can feel to let that turn over to someone else. (I never did, but someone decided to fix a memory leak and host that version on their own site, and gave credit where due and was overall quite nice about it, and it still feels like I lost some custody rights to a kid.) I imagine this would feel most acute with entertainment software where you’ve created the initial story, but I can see it with strictly utilitarian software, too.

  22. ffmpeg vs. libav; NetBSD vs. OpenBSD; X.org vs. whatever-was-before. Those kinds of world-shattering events.

    gcc vs egcs… sometimes forking either result in merge, or in fork taking over the original project.

    Or not. Emacs vs. XEmacs. We all know who won that one. (*ahem*Vim*ahem*) ;)

  23. @Paul Brinkley:

    I wrote a jigsaw puzzle implementation years ago, and I know how it can feel to let that turn over to someone else. (I never did, but someone decided to fix a memory leak and host that version on their own site, and gave credit where due and was overall quite nice about it, and it still feels like I lost some custody rights to a kid.)

    This seems exactly the sort of situation that github was designed for. It’s hard to imagine that someone who gave credit and was quite nice about it wouldn’t, these days, give you a pull request so that the bugfixes could easily wind up back in your main branch.

    (Having said that, the cost of using github vs. say, googlecode and subversion, seems fairly high for a small project with a single developer. DVCS is certainly more powerful than VCS, but with that power comes additional steps that need to be taken.)

  24. I feel no project should be forked except for major changes*. Minor code changes should be merged with the original via patches.**

    * meaning different features not planned in the original, a major rework/restructure of the codebase etc.
    ** provided of course, the project is alive and the maintainer is willing to accept patches.

  25. However, hari, it would seem from Eric’s original posting that the maintainer is not willing to accept Eric’s fixes…where does that leave the desirability of a fork in your view?

  26. @Morgan Greywolf:

    Or not. Emacs vs. XEmacs. We all know who won that one. (*ahem*Vim*ahem*) ;)

    No wonder you don’t like emacs — you’re not very good at balancing your parentheses, are you?

  27. “However, hari, it would seem from Eric’s original posting that the maintainer is not willing to accept Eric’s fixes.”

    That seems to be implied, but nowhere did Eric state that outright. If that’s the case, I’d like to hear ESR confirm it, and if there was a justification given for rejecting fixes, I’d like to hear that as well.

    The fact that Eric talks of a flawed design decision (he used the phrase “serious design mistake”) implies that there’s more going on here than just buggy code that needs a few patches. Different design decisions implies different visions of where the project is going.

  28. Jay Maynard:

    However, hari, it would seem from Eric’s original posting that the maintainer is not willing to accept Eric’s fixes…where does that leave the desirability of a fork in your view?

    I thought I’d missed that… but reading the post again, Cathy is right. It’s implied, but not specifically mentioned.

  29. I have to disagree with the people claiming that forks are necessarily bad. If a person feels that I’m driving one of my projects in the wrong direction, it is right and proper for him to fork it and remake it in his own image, so to speak. And if it turns out that his version becomes the de-facto standard, I’ll view it as a net win; it means I was probably bottlenecking the user community and it’s an opportunity for me to learn. Should it turn out that my version is still the standard, good. At the very least he got code better suited to his needs.

  30. Without either A) more information about the specific case or B) using this as a jumping-off point for a broader discussion of the ethics of forking and project maintainer responsibilities, I don’t see where this post is going.

  31. There is an important point that I’m a little surprised I didn’t see anyone mention here in the comments — this is a *library*. It’s not end-user code or an app.

    That has several semantic implications, the most important of which has to do with packaging and naming: how do distribution repository managers deal with a fork in a library. It’s somewhere between difficult and impossible to Just Rename the library proper without screwing up a billion and six makefiles,
    so some cooperation from repo maintainers would seem to be essential in making a fork like that without making life difficult for both developers and those packagers themselves.

    Also: which version do you fork?

    I would assume esr would fork an older, cleaner version, and make whatever changes he felt necessary to sort out the mess… but that itself limits who’s old code will link to the library properly.

    No, libraries, like languages (a point I wish language maintainers would learn; how many versions of C have there been since 1979? What’s most of the code (or its interpreters :-) in the world written in?), are *tools for other people to get work done*; a much stricter set of workflow semantics must apply thereto, if you want people to actually, y’know, use them.

  32. @Morgan Greywolf:

    ” Or not. Emacs vs. XEmacs. We all know who won that one. (*ahem*Vim*ahem*) ;) ”

    No wonder you don’t like emacs — you’re not very good at balancing your parentheses, are you?

    Oooooo, that was a telling blow.

  33. Baylink’s point about libraries reminds me of an important actual example of library forking, which was similar except that it was 100% politics and 0% maintainer competence: libpng.

    Mozilla really wanted support for its APNG animated png format, which was formally rejected by the PNG standards authorities. This required some patching of the libpng library. But libpng’s maintainers refused to act as enablers for Mozilla’s fork of the /standard/, and so brickwalled any APNG patches. Refusing to surrender, Mozilla then forked the library as well.

  34. @Baylink:

    There is an important point that I’m a little surprised I didn’t see anyone mention here in the comments — this is a *library*. It’s not end-user code or an app.

    Probably nobody mentioned it because it was a given.

    That has several semantic implications, the most important of which has to do with packaging and naming: how do distribution repository managers deal with a fork in a library.

    I did mention the necessity of a rename on a fork.

    It’s somewhere between difficult and impossible to Just Rename the library proper without screwing up a billion and six makefiles,

    Not necessarily true, e.g. if it’s a pure Python package. Which, for the given problem domain and program, is entirely likely.

    Also: which version do you fork?

    Obviously, that’s up to the forker.

    I would assume esr would fork an older, cleaner version, and make whatever changes he felt necessary to sort out the mess… but that itself limits who’s old code will link to the library properly.

    If I read between the lines, esr is saying (a) the broken changes are _recent_, which means that very few, if any clients would depend on the new behavior, and (b) the new behavior will adversely affect other clients than his (presumably, that’s one of the major reasons for the fork). Assuming those two conditions are true, the correct answer should actually be pretty obvious.

  35. @Patrick Maupin
    “(Having said that, the cost of using github vs. say, googlecode and subversion, seems fairly high for a small project with a single developer. DVCS is certainly more powerful than VCS, but with that power comes additional steps that need to be taken.)”

    I don’t think that the more power of an DVCS translates automatically into more work. I have put some code on GitHub and i can’t imagine it to be any easier with svn.

  36. I find it amusing that this discussion of forking is basically the old “Exit, Voice and Loyalty” discussion in the context of software.

    …which shouldn’t really be amusing, I suppose, because what is the community of software developers other than a community of human beings, with that same old human nature?

    http://en.wikipedia.org/wiki/Exit,_Voice,_and_Loyalty

  37. @Patrick Maupin “Which, for the given problem domain and program, is entirely likely.”

    Do you have information the rest of us do not, or are you jumping to the same conclusion some other people have?

  38. Patrick Maupin said: No wonder you don’t like emacs — you’re not very good at balancing your parentheses, are you?

    How much is Big Parenthesis paying you? HUH?

  39. @Emanuel Rylke:

    I don’t think that the more power of an DVCS translates automatically into more work. I have put some code on GitHub and i can’t imagine it to be any easier with svn.

    Ah, but it is. IMO, of course. (I’ve put some code on github, and I have a few svn projects on googlecode.)

    @Random832 :

    You’re absolutely right — I’m speculating. But I know several of esr’s recent projects involve python, and many of them (I believe) have significant functionality that can be (and probably is) implemented in python. So it wouldn’t _have_ to be the exact same project that others are speculating about…

    @Sigivald:

    How much is Big Parenthesis paying you? HUH?

    Not enough for me to actually endorse emacs, that’s for sure.

  40. @Patrick Maupin:
    > the cost of using github vs. say, googlecode and subversion, seems fairly high for a small project with a single developer.

    Somewhat off topic, but having fairly recently moved a number of personal projects from subversion to mercurial, I would have to disagree. (I have a few small projects on github as well, but I have never used any other VCS for those.) Just the fact that I can branch, merge, commit, etc. without having to talk to a server is enough to make mercurial a big win for me over subversion; not just the fact that with svn you need a connection to the server, but the fact that you have to wait for the server for *every* operation, instead of being able to batch up a bunch of changes into one push. Not to mention the fact that it’s dirt simple to keep clones of my repos in various places so I always have plenty of backups of the project state; setting up mirroring of an svn repository, while certainly doable, is nowhere near as simple as “hg clone”.

  41. > I suppose, because what is the community of software developers other
    > than a community of human beings

    Having dealt with lots of humans, and more than a few software developers, check your assumptions.

  42. I am seeing a lot of pain caused by python libraries, and changes therein. People are getting cross, and struggling with versioning.

    I don’t use python enough to have a competent opinion as to what the problem is, but there is a problem.

  43. @JAD:

    Any examples? Practical interest: I submitted a backwards-compatibility-breaking patch to the python standard library a year or two ago, and while the maintainer told me it was OK I have no idea if it caused anyone else trouble.

  44. “I don’t use python enough to have a competent opinion as to what the problem is, but there is a problem.”

    I sometimes get the impression that some “free software” types (though not the FSF themselves, judging by the addition of symbol versioning in glibc 2) think that maintaining backwards compatibility is a bad thing because it is mainly seen as a tool to help proprietary software [which you can't edit/recompile to conform to the new version of the library] to be on an even playing field with free software. This translates to a more widespread ‘apparently’ less extreme view, that backwards compatibility is _not important_, because breaking changes only “really” affect proprietary software, so if there’s some perceived ugliness in the current API you should just change it and let everyone break.

    If there’s a python thing at all, it’s in the fact that python doesn’t provide as easy a way to bump a major version number and make it _obvious_ that something’s not compatible. But I really don’t see any legitimate reason why we’re on, say, version 3 of GTK (and I don’t know how many incompatible non-major versions there were along the way to 3.6 – I think 1.2 was incompatible with 1.0 [i.e. some programs using 1.0 could not be compiled against 1.2 without modifications]), and it takes effort to make it possible to install two versions of a library (especially with development headers) side by side. And no-one wants to make it _easy_ to use the earlier versions – since in practice that’ll mean that programs that _could_ benefit from newer features (and security!) in the later versions will stay on the earlier version longer.

  45. Random832: The problem with the approaches to backward compatibility you point out is that not all binary software packages are proprietary. Much out there is shipped as proprietary simply because it’s a royal pain in the ass to compile. See, for example, what it takes to build the Firestorm viewer for Second LIfe on Linux, and note that it’s completely open source – 1.13 MLOC of it.

    It’s simply not possible to deliver one package that works across a wide range of distributions and Linux versions. The utter lack of standardization of the API is one major reason that people simply refuse to enter the Linux space.

  46. @Jay: “It’s simply not possible to deliver one package that works across a wide range of distributions and Linux versions. The utter lack of standardization of the API is one major reason that people simply refuse to enter the Linux space.”

    @esr (from “The long past of C”):
    “The autotools suite began life because the combinatorial explosion of feature tests and #ifdefs back in the day was too difficult to manage by hand. Over the years autoconf and friends got hairer and hairier (to the point where I got utterly fed up with it), but at the same time increasingly good standards conformance in C and Unix implementations attacked the problem from the other end. So, during the last couple of days, I’ve found that the sort of platform #ifdefs that used to be autotools’s raison-d’etre can all be yanked out – what’s left is feature switches, and only two of those. That whole standardization thing…actually worked.”

    Interestingly different points of view. Perhaps the next phase for Linux needs to be additional standardization across distributions regarding directory structure, locations where key files (e.g., headers, libraries) are stored, and so forth. Is there really a fundamental reason why distributions need to be so incompatible?

    Main issues I see: (1) rpm vs. deb vs. tarballs; (2) window management (Gnome, KDE) that are more integrated into the distro than they should be; (3) different directory structures.

    (1) doesn’t strike me as likely to change, because too much has been invested by each group. However, if each type of store is sufficiently standardized, it should be possible to write a program to automatically translate one to another.
    (2) is just poor design decisions.
    (3) seems like the easiest one to solve.

    Now if only open source projects would standardize their build process! I tried to build xgammon the other day, as it has some glaring bugs that would probably be pretty easy to fix. But I can’t even get it to build, as it wants to dump me into emacs when I start a build and make some strange path changes. I could probably figure it out, but it’s not worth the effort. If you need to do this sort of thing, ./configure is your friend. If I need to edit a file myself (with my own choice of editor) and it’s sufficiently well documented, I’m not happy about that but I can live with it.

  47. Cathy, you missed 4), and what makes the question relevant to this discussion: library versioning and API incompatibilities. Library maintainers feel no need to use the existing mechanisms to mark their libraries incompatible with earlier versions, and even when they do, distribution maintainers don’t use that to permit both versions to remain on the system. The result is that an application builder can’t depend on writing to one API and actually have it work.

  48. “Is there really a fundamental reason why distributions need to be so incompatible?”

    No-one wants the someone else’s crappy design decisions to become the standard.

  49. I _did_ mean “crappy” as how each person/group/company thinks of their rival’s idea of how to do things, rather than anything objective about how things actually are.

  50. @Jay:
    “Cathy, you missed 4), and what makes the question relevant to this discussion: library versioning and API incompatibilities. Library maintainers feel no need to use the existing mechanisms to mark their libraries incompatible with earlier versions, and even when they do, distribution maintainers don’t use that to permit both versions to remain on the system.”

    Duh, yes, I should have included libraries. Of course, this is nothing unique to Linux; ask any Windows power user or developer about DLL hell.

    Which raises the question of why none of the distros have put in place a formal mechanism to support multiple versions of the same library installed simultaneously via the standard (for each distro) packaging system.

    And related to that, the fact that distro maintainers are slow to get recent library releases into the stable branch (I use Ubuntu, so that’s where my recent experience lies, though at different times I’ve run Red Hat, Caldera, and Slackware).

    For quite some time, I have been building FlightGear, which requires plib 1.8.5. Distros were so slow to move beyond 1.8.4 that even the FlightGear “how to build” page had a note indicating that you’d probably have to compile and install it outside the normal packaging system. Now that I’m running oneiric I actually have 1.8.5 in the standard package system, so at least that’s one less problem to deal with.

  51. @James A. Donald:

    I am seeing a lot of pain caused by python libraries, and changes therein. People are getting cross, and struggling with versioning.

    I would be interested in knowing what context you are discussing.

    One thing happening right now is that version 3 is becoming more popular. It was originally envisioned as a “clean break” but in reality that never happens, and more people are maintaining v2 and v3 compatible stuff from a single codebase now, and recent changes to v3 make it more, not less, compatible with v2. There has been some considerable angst over this.

    One other possibility is that Python makes it _extremely_ easy for rank amateurs to develop code that is useful for other people. Obviously when this happens, it is unsurprising if APIs aren’t stable.

  52. @James A. Donald:

    Also, the recommended installer way (pip vs. easy_install, etc.) is in a state of flux, and not all tools do the right thing with beta vs. released selection. This is somewhat ongoing — what does it mean to get the “latest” version?

  53. Which raises the question of why none of the distros have put in place a formal mechanism to support multiple versions of the same library installed simultaneously via the standard (for each distro) packaging system.

    I can’t speak for the majority of the distributions, but Gentoo has long supported “slots” for side-by-side versioning; Python, for instance, is slotted by minor version (I currently have a 2.7 and a 3.1 installed), while KDE is slotted by major (4) and the kernel sources by micro (3.5.7).

  54. Cathy said: Of course, this is nothing unique to Linux; ask any Windows power user or developer about DLL hell.

    Well, if their memory goes back that long – it’s been (for practical purposes) not a problem since Windows XP let you have local DLLs for each binary.

    (And I say this as a professional Windows developer; I can’t remember the last time we had a DLL problem like the old “DLL hell” days. Ditto as a user.

    Microsoft seems to have pretty much solved that problem. Which is nice.

    Hopefully Linux people will figure out Some Way To Do The Equivalent, but my low-level Unix-fu is not remotely strong enough to even imagine where one would start…)

  55. Duh, yes, I should have included libraries. Of course, this is nothing unique to Linux; ask any Windows power user or developer about DLL hell.

    You’re about 10 years too late. Since Windows XP, all versions of Windows have included SxS, a feature which lets you match DLL versions to specific executables, eliminating the DLL hell problem entirely.

    As usual, Linux has yet to catch up.

  56. I haven’t done professional Windows development since 2000 (and not much then, as I lived mostly in Unixland), so no surprise that I’m out of date.

  57. @Jeff Read:

    > As usual, Linux has yet to catch up.

    WTF? With linux preload, you can override _a few_ functions in a shared library. You don’t even need to replace the whole thing.

    As usual, Microsoft has yet to catch up.

  58. @Patrick Maupin

    That’s not really equivalent. The equivalent would be if there were a copy of /usr/lib for every single executable in /usr/bin, maintained with hardlinks. Which you can _do_, sure but it doesn’t exist yet, nor is it built into the loader.

  59. @Random832:

    The literal question was “ask any Windows power user or developer about DLL hell.”

    So the equivalent would be any Linux power user or developer. Who can easily arrange their .so libraries to suit themselves, and/or their customers.

    People who _really_ remember DLL hell on Windows will, if they clear the cobwebs, realize that the true DLL hell involved not being able to have two simultaneously running programs use two different DLL files that shared the same name. This is not an issue on Linux, and while there are some inconveniences (probably more properly called RPM hell), there is no true current Linux equivalent to what is properly termed DLL hell.

  60. Yes, old DLL hell referred to the fact that you couldn’t map two DLLs with the same name into the global address space that all Win16 programs shared.

    But in later years — like, since 1995 — DLL hell came to refer to the fact that the DLL you linked against when you built the program may not be the same version as what’s on the user’s machine when it runs. Another program may have clobbered the user’s machine with a different version of the same DLL.

    With SxS under Windows, own versions of different DLLs are managed automatically, transparently. No similar automatic and transparent functionality is extant on major Linux distributions.

    The fact that it be automatic, transparent, and EASY is critical. Remember, if technology doesn’t work for people, it doesn’t work.

  61. There is a .so versioning standard in place. It’s just that nobody actually follows it. Library maintainers don’t bump the number when they make incompatible changes, and distribution maintainers don’t look at it when replacing library packages with incompatible versions.

  62. @Jeff Read:

    The fact that it be automatic, transparent, and EASY is critical. Remember, if technology doesn’t work for people, it doesn’t work.

    Yeah, there’s something to be said for fucking up things so badly that you have to go fix them. Microsoft’s good at this.

  63. >There is a .so versioning standard in place. It’s just that nobody actually follows it. Library maintainers don’t bump the number when they make incompatible changes, and distribution maintainers don’t look at it when replacing library packages with incompatible versions.

    Huh? In my experience, neither of these things is true.

    Most Linux distributions can and do carry multiple versions of system libraries, differentiated by x.y.z version specifications. Thus distribution makers don’t have to “replace” libraries at all. Compiled applications (in effect) look for a matching major number among the options.

  64. > Most Linux distributions can and do carry multiple versions of system libraries, differentiated by x.y.z version specifications. Thus distribution makers don’t have to “replace” libraries at all. Compiled applications (in effect) look for a matching major number among the options.

    And this is completly automatic (thanks in part to tools like libtool / libtoolize used during installation time)…

    …with the caveat that: a) library maintainers must use semantic versioning correctly (usually they do); b) program maintainers must link against correct version of library (mostly automatically); c) packages need to have correct version information (though I think it happens mostly automatically)

  65. For example I have on one of Linux systems two versions of readline library and two version of standard C++ library.

    ls /usr/lib/*.so.* | grep “[0-9]” | sed -e ‘s/\.so\..*$/.so/’ | sort | uniq -c | grep -v “2 “

  66. There can still be problems even then. I remember how fast the “standard C++ library” version turnover used to be – download a binary package and odds are the libstdc++ it links against is too old to be packaged in your distro anymore.

  67. I just downloaded a binary blob of Skype. Despite being their “static version” it still depended on libtiff.so.4. I had libtiff.so.5.

    I was able to cruft up a symlink to get it to work, but the average user can’t do that.

    And this is why library management on Linux is an unholy mess. This doesn’t happen on Windows and it doesn’t happen on Mac. It is UNACCEPTABLE for this to happen in any environment.

  68. > And this is why library management on Linux is an unholy mess. This doesn’t
    > happen on Windows and it doesn’t happen on Mac. It is UNACCEPTABLE for
    > this to happen in any environment.

    Doesn’t happen on Mac cause applications package their own libraries, unless they are pre-existing system libraries. There is little stopping Linux packages from doing this as well.

    Windows uses the cruftiest, ad-hoc-iest solution I’ve seen to solve it. How many copies of the same version of DX9 do I have installed on my gaming box? At least 20 or 30 at this point, maybe more. That’s ridiculous.

    Really, you want to bitch about libraries for specific packages on Linux, you should bitch at the developers of the software, not at Linux. Bitch at them for making stupid assumptions that break years-long expectations (like assuming everyone runs crappy Ubuntu), or that every system has 32-bit libraries installed. Bitch at them for calling a dynamic-linked exec a “static” executable when it clearly isn’t, and for not including the requisite libraries in the install package if it actually matters.

    Developers get severely dinged in the Mac world for not following the conventions, instead of Apple getting dinged for having a weird architecture. Why the fuck does that not also apply to Linux? Blame the shitty developers first.

    Hm, this rant gives me an idea for a “power” end-user utility that might be handy. Need to make a note…

  69. “but the average user can’t do that.”

    Good thing the “average user” doesn’t need to. The “average user” installs Skype from a .deb file (either from Skype’s website or their distro’s repository), and things Just Work.

    Yes, it’s possible for someone to call something a “static version” when it really isn’t. It’s also possible for Windows developers to not use SxS.

  70. > I just downloaded a binary blob of Skype. Despite being their “static version” it still depended on libtiff.so.4. I had libtiff.so.5.
    >I was able to cruft up a symlink to get it to work, but the average user can’t do that.

    You have directly downloaded (bypassing the package manager) a blob that is dynamically linked with some_library version x. You have some_library version y != x, but, fortunately, the changes between x and y versions’ ABI do not matter in _this_particular_case_. What should the OS do?

    1) Track your downloads and automagically fetch and install some_library version x when you download a binary that is dependent on it;
    2) As soon as the binary is being run, the OS should assess the degree of compatibility between x and y and decide if that binary can be run with some_library of the existing version;
    3) Just try to run the binary with the existing some_library.
    4) Complain that there is no some_library version x in the system and bail out when the binary is being run.

    There is only one realistic case above. Guess which.

    On the other hand, modern package managers track dependencies not only from packages, but also from specific library versions. For example, if you use rpmbuild to create a package foo that contains /path/to/some_library.so.x, it automatically remarks that this package provides /path/to/some_library.so.x. And, alternatively, if you use rpmbuild to create a package bar that contains a binary that depends on /path/to/some_library.so.x, it will add a corresponding requirement. Therefore, when you will try to install such a package, the package manager (e.g. yum) will automatically consider that package bar requires some package that provides /path/to/some_library.so.x to be installed, for example, package foo.

    The real problems begin when one bypasses the regular mechanisms of package management. But this is certainly the thing that average_users(tm) should not do.

  71. jsk said: Windows uses the cruftiest, ad-hoc-iest solution I’ve seen to solve it. How many copies of the same version of DX9 do I have installed on my gaming box? At least 20 or 30 at this point, maybe more. That’s ridiculous.

    It’s “ridiculous”… but it always works, and the first time*.

    That’s better than any “elegant” competing solution that fails that same test or has any fragility to it.

    Both as an end user and as a (thank god it’s just my one home server) system administrator, “just works” beats “waily, I have 20 copies of some stupid library on my gigantic disk” every day, a million times.

    (* Maybe not literally, but close enough that I can’t remember the last time it didn’t.)

  72. You do realize that the copies of identical files in the WinSXS directory are hardlinks, right?

  73. @Random382:

    Yeah, MS finally added symbolic links to Windows under protest, but they’re still garbage.

  74. *shakes head*

    I have just as much trouble trying to keep my Windows desktop at the office going as I do my home Linux box. The reality is that both can be a real pain. To get FlightGear to work last night I had to create a symlink from libOpenThreads.so.12 to libOpenThreads.so.14, as the program wants 12 and I have 14 installed. It turns out to run fine with 14-masquerading-as-12.

    Then Outlook 2010 refused to let me see my contact list. When I tried to run scanpst on the ost file, Windows complained that another application was using the ost file even when I did this immediately following a reboot. I finally erased the ost file and let Outlook re-create it, which fixed the problem. Of course, this all required fiddling with files in an obscure folder that is declared “hidden” by default on most system.

    No casual user would have been able to fix either one of those problems. The reality is that in 2012, dealing with computers is still hard. You can shift the hard part to someone else (IT support, Geek Squad, techie friend), or learn it yourself, but you cannot make that part just go away.

    Smartphones don’t automatically solve the problem either. I’ve sometimes had to do a reset of all application data, or just reinstall the app, to get an app working again. (This happened today with Yahoo mail.) At least those are steps the average user could perform.

  75. jsk, what is an application developer to do when a library’s behavior changes incompatibly? I’ve personally run into this twice in the past month, once with libjsoncpp (I’m the one who found that bit about the system libraries breaking Firestorm), and once with libunwind on Gentoo for Itanium (and they refused to slot the library version I needed). You either wind up living with breakage on one version of the OS, or else code around the problem once you discover it – which may well be nontrivial – or shipping every library your code depends on. Look at the list of libraries for Firestorm again. There are 40 of them aside from the ones you have to manually build. Is it reasonable to have an application developer ship his own version of libc6?!

  76. @Jay Maynard:

    > Is it reasonable to have an application developer ship his own version of libc6?!

    Depends on the app. When I install one and a half DVDs of Xilinx cruft, I doubt I would notice if they did that. In fact, I’ve had to do the dance that Cathy described on a couple of .so files in the past, so at the point I’m installing a few gig, the pain point shifts — I’m annoyed if I have to do my own symlinks, and I couldn’t possibly notice another few tens of megabytes.

  77. @Jay

    Good gravy that is a lot of deps! I can hardly imagine building ANYTHING that massive without running into snafus, on any system.

    Something like that, I’d imagine it would be difficult for the maintainers to even pick a specific distro version to build against as a baseline. What I would optimally expect in that case is for them to go ahead and pick a reasonable target, declare it as such, and then package any library version differences with the software.

    For the code, I’d list out explicitly every library and version required to build it, and preferably be proactive enough to provide the sources for any versions that are no longer available. Probably, like above, I would also list out what distro versions the code was written against, if it can be narrowed down to a few. It really should be up to the maintainers to ensure their software can be built.

    Of course, none of the above solves the problem of taking a precompiled binary from upstream and shipping it. However, if a user downloads a binary that is qual’d against a specific distro, but the user is running something different, then at that point it does finally fall to the user to resolve the issues. (An aside: this is why having the code available is so important, so that distro folks can do the legwork.) Let’s be clear: if the user is attempting to run a binary outside of the specified requirements as set forth by the packager, it is the user who is responsible. Not the OS, not the distro (unless it’s Ubuntu ; )

    That’s no different than someone running a hacked up version of OSX and finding out that some software doesn’t run properly due to some bizarre difference from stock Apple OSX. You don’t blame OSX at that point.

    Hm. Long post, and your question is pretty can-of-worms, so I think I’m being a little soft. I’ve got my asbestos on; flambe me and I’ll try to shake out something a little clearer from the ashes.

  78. I just downloaded a binary blob of Skype. Despite being their “static version” it still depended on libtiff.so.4. I had libtiff.so.5.

    Ok so i see two points here :-
    * Unless Skype means something different by “Static Version” than i would technically expect, the static version should not be linking to dynamic libraries. So that is their bad.
    * Assuming they did mean to dynamically link a tiff library… why would you automatically assume your version 5 library is compatible with version 4? That seems like a really dangerous assumption.

    The “proper” way to deal with this would be to install the latest “version 4″ of the library, assuming installations work for users (if they don’t you have deeper issues) then either
    a) the user asks for the “version 4″ install
    b) (preferrably) the skype installer should rely on “version 4″ of the library.

    This does require that your package tree isn’t a tangled mess and things that can coexist happily (e.g. tiff libraries) don’t rely on things which can’t coexist happily (e.g. the kernel), which is why i stopped using Debian.

  79. jsk: I’ll look at your post in depth in a bit when I’m not trying to clean up a mess, but I’ll just point out that you don’t get hacked up versions of OS X because Apple maintains them in such a way that things work even with maintenance, something Linux distribution makers don’t understand.

  80. ” I’ll just point out that you don’t get hacked up versions of OS X because…”

    Tell that to the “hackintosh” lion install on my tower that wouldn’t let me link software cause of a kernel version issue. : )

  81. Each app having its own version of libraries mean that to update a library to fix a bug you would need to replace / upgrade said library for each app.

  82. “Each app having its own version of libraries mean that to update a library to fix a bug you would need to replace / upgrade said library for each app.”

    Or just, you know, “upgrade” the apps with the new versions of the shared libs. Yes, this places the burden back on the packager/maintainer/developer. As it should be.

  83. >> “Each app having its own version of libraries mean that to update a library to fix a bug you would need to replace / upgrade said library for each app.”

    > Or just, you know, “upgrade” the apps with the new versions of the shared libs. Yes, this places the burden back on the packager/maintainer/developer. As it should be.

    And delay fixing the vulnerability…

  84. Tell that to the “hackintosh” lion install on my tower that wouldn’t let me link software cause of a kernel version issue. : )

    You have strayed outside the Holy Ecosystem and deserve what you got, heathen!

  85. Ok so i see two points here :-
    * Unless Skype means something different by “Static Version” than i would technically expect, the static version should not be linking to dynamic libraries. So that is their bad.

    It’s statically linked against a specific version of Qt. You’re right though, why the hell not static link all the way down? Fewer headaches…

    * Assuming they did mean to dynamically link a tiff library… why would you automatically assume your version 5 library is compatible with version 4? That seems like a really dangerous assumption.

    I’m assuming that Skype — a bloody chat app — doesn’t actually need libtiff at all, and that is a dependency of whatever Qt they linked against.

  86. I’m assuming that Skype — a bloody chat app — doesn’t actually need libtiff at all, and that is a dependency of whatever Qt they linked against.

    You might be right. However it wouldn’t surprise me if skype sends profile images as tiffs thus explaining the tiff link. Still should have been statically linked in a “static version” but we agree there.

  87. >So, did the Problem Ever Get Resolved?

    Not yet. The showdown may have to wait until I don’t have surviving Hurricane Sandy uppermost in my mind.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>