Ignoring: complex cases

I shipped point releases of cvs-fast-export and reposurgeon today. Both of them are intended to fix some issues around the translation of ignore patterns in CVS and Subversion repositories. Both releases illustrate, I think, a general point about software engineering: sometimes, it’s better to punt tricky edge cases to a human than to write code that is doomed to be messy, over-complex, and a defect attractor.

For those of you new to version-control systems, an ignore pattern tells a VCS about things for the VCS to ignore when warning the user about untracked files. Such patterns often contain wildcards; for example, “*.o” is good for telling almost any VCS that it shouldn’t try to track Unix object files.

In most version control systems ignore patterns are kept in a per-directory dotfile. Thus CVS has .cvsignore, git has .gitignore, etc. Ignore patterns in Subversion are not kept in such a dotfile; instead they are the values of svn:ignore properties attached to directories.

Translating ignore patterns between version-control systems is messy that most conversion tools fluff it. My reposurgeon tool is an exception; it goes to considerable lengths to translate Subversion ignore properties into patterns in whatever kind of dotfile is required on the target system.

Unfortunately, this feature collides with git-svn. People using that tool to interact with a Subversion repository often create .gitignore files in the Subversion repository which are independent of any native svn:ignore properties it might have.

This becomes a problem when you try to convert the repo to git. In that case, .gitignore files created by git-svn users and .gitignore files generated from the native svn:ignore properties can step on each other in odd ways.

I’ve had a bug report about this in my inbox for a couple of months. Submitter innocently asked me to write logic that would automatically do the right thing, merging .gitignore patterns with svn:ignore patterns and throwing out duplicates. And somewhere in the back of my brain, a robot voice called out “WARNING, WILL ROBINSON! DANGER! DANGER!”

One of the senses you develop after writing complex software for a couple of decades is some ability to tell when a feature is going to be a defect attractor – a source of numerous hard-to-characterize bugs and a maintenance nightmare. That alarm rang very loudly on this one. But I was blocked for quite a while on the question of what, if any, simpler alternative to go for.

I resolved my problem when I realized that this challenge – merging the properties – will be both (a) uncommon, and (b) the sort of thing computers find difficult but humans find easy. Typically it would only have to be dealt with once in the aftermath of a repository conversion, rather than frequently as the repo is in use.

My conclusion was that the best behavior is to discard the hand-hacked SVN .gitignores, warning the user this is being done. It’s then up to the reposurgeon user to rescue any patterns that should be moved from the old hand-hacked .gitignores to the new generated ones.

Because, very often, the hand-hacked .gitignores are there just to duplicate way the native svn:ignore properties are doing, the user often won’t have to do any work at all. The unusual cases in which that is false are the same unusual cases that automated merge code could too easily get wrong.

The general point here is that engineering is tradeoffs. Sometimes chasing really recondite edge cases piles up a lot of technical debt for only tiny gains.

The more subtle point is that if you don’t have any way at all to punt weird cases to a human, your software system may be brittle and overengineered – doing sporadic exceptional cases at a high life-cycle cost that a human could do cheaply and at a cumulatively lower defect risk.

This bears emphasizing because hackers have such a horror of manularity, going to extreme lengths to automate instead. Sometimes, doing that gets the tradeoff wrong.

Reposurgeon creates the option get this right because it was designed from the beginning as a tool to amplify human judgment rather than trying to automate it entirely out of the picture. All other repository-conversion tools are indeed brittle in exactly the opposite way by comparison.

A similar issue arose with cvs-fast-export. I got a bug report that revealed a couple of issues in how it translates .cvsignore files to .gitignores in its output stream. Among other things, it writes a representation of CVS’s default ignore patterns into a synthetic .gitignore in the first commit. This is so users browsing the early history in the converted git repo won’t have untracked files jumping out out them that CVS would have kept quiet about.

With the report, I got a request for a switch to suppress this behavior. The right answer, I replied, was not to add that switch and some complexity to cvs-fast-export. Rather, I reminded the requester that he could easily delete that synthetic .gitignore from the history using reposurgeon. Then I added the command to do that to the list of examples on the reposurgeon man page.

The point, again, is that rushing in to code a feature would have been the wrong thing – programmer macho. Alternatively, we could view the cvs-fast-export/reposurgeon combination as an instance of the design pattern alternate hard and soft layers and draw a slightly different lesson; sometimes it’s better to manually exploit a soft layer than add an expensive feature to a hard one.

31 comments

  1. I’ve seen this in other areas. Linux distro repos, where complex conflict resolution is worked over and over, and complex configuration is needed to get it right – when it would be much easier for the update software to say “Hey, got these two packages – which one do you want?”

  2. I first thought this topic was for a story so bad that you didn’t even want to call it a review. Then i started reading and figured out you just added a colon after the ignoring by reflex.

    1. >(And as i read more i think i’m wrong… my punctuation-fu is probably inadequate)

      Read it with the colon. Then without. Notice the effect of setting these readings against one another.

  3. I’ve seen this in other areas too. It tends to take the form of a user asking for an enhancement to a tool that is out of scope. The correct answer will be “That’s not what this tool does. It’s designed to do X. Anything else is something else’s job, and should not be added to this tool.”

    Understanding what the domain boundaries should be and when you have stepped out of the domain is always an issue, and one thing developers need to think through carefully and communicate to the users is just what problem domain their code addresses, to reduce requests for things that *are* something else’s job.

    I think punting edge cases to a human for resolution is the correct behavior in many instances. Even if the tool handles the issue without error, what it does may not be what the user really wanted. It’s best guess at what should be done won’t match the user’s desire, and the user will blame the program when they don’t get what they expect.

    Sometimes you can’t automate everything and shouldn’t try. The trick is knowing when that’s the case.

  4. My conclusion was that the best behavior is to discard the hand-hacked SVN .gitignores, warning the user this is being done.

    Please figure out a way to do this that doesn’t get shoved into a stream of other messages and scrolled off the screen.

    SRSLY, in modern systems a simple operation will often generate 100s of various messages. These will get spewed to the screen and can be permanently lost.

    Because, very often, the hand-hacked .gitignores are there just to duplicate way the native svn:ignore properties are doing, the user often won’t have to do any work at all.

    Wouldn’t the optimal thing be to check for this case and only warn if it’s an issue?

    1. >Please figure out a way to do this that doesn’t get shoved into a stream of other messages and scrolled off the screen.

      Little risk of that. Unless you deliberately jack up the verbosity reposurgeon is pretty quiet.

      >Wouldn’t the optimal thing be to check for this case and only warn if it’s an issue?

      Feature creep! Feature creep! This is a feeping creature.

  5. This may be stating the obvious, but the same basic idea can be used in software development (at least for some R&D). I am writing software that needs an initial State. At some point, this may be partially derived, but initially, I will just specify it in a file.

  6. Sorry – I screwed up the first sentence; it should be…

    This may be stating the obvious, but the same basic idea can be used in highly iterative R&D software development.

  7. >Wouldn’t the optimal thing be to check for this case and only warn if it’s an issue?

    Feature creep! Feature creep! This is a feeping creature.

    Perhaps, but if the number of cases where there are the two ignore mechanisms in use is substantially equal to the number of cases where they’re the same, and it’s easy and inexpensive to check, then this could be a win.

    If either condition is untrue, though, then you’re right to let it drop right where it is.

    1. >If either condition is untrue, though, then you’re right to let it drop right where it is.

      OK, now I’ll have to at least sketch an explanation of why there is a yawning abyss of complexity here.

      At any point where we encounter a .gitignore in the SVN history, we have three sets of patterns to consider:

      A) The default Subversion patterns

      B) Any patterns currently active due to svn:ignores

      C) The pattern set implied by the .gitignore.

      If we warn on A + B != C we’re going to warn every time, because there’s historical stuff in the A set that no git user will ever turn on – like “.nse_depinfo”. Warning on B != C won’t work either – because, as one obvious example, “.o” is in A and likely to be in C but (because it’s in A) is never in B.

      Most programmers, having realized this, will immediately start trying to come up with the “correct” warning-trigger predicate on A, B, and C. This is not substantially less complex than trying to write a correct mapping from (A, B, C) to a result set to be stuffed into the .gitignore. In fact it is arguably the same problem.

      I repeat: this is a feeping creature.

  8. “The more subtle point is that if you don’t have any way at all to punt weird cases to a human, your software system may be brittle and overengineered – doing sporadic exceptional cases at a high life-cycle cost that a human could do cheaply and at a cumulatively lower defect risk.”

    I interpreted this as implying a special case of a general statement: there will always exist demand for human labor despite rampant automation, for such exceptional cases will always exist at the fringes.

    (Implying by way of anthropic principle – if it weren’t that frequent, you wouldn’t have bothered to mention it.)

    1. >I interpreted this as implying a special case of a general statement: there will always exist demand for human labor despite rampant automation, for such exceptional cases will always exist at the fringes.

      Barring AIs that can pass a strong version of the Turing Test, yes.

  9. Original submitter here mentioned in para. 7. :)

    I think a better underlying point behind all of this is that simple human error/ignorance/negligence/oversight creates these kind of problems and trying to automate a solution for all such issues is not going to work. In particular, this was spurred by a git-svn user creating .gitignore files in Subversion, and they weren’t exactly smart ignores (it was for ignoring XCode files, but with hard-coded paths based on his Mac OS user name…). This is despite that Git has a mechanism to add ignore patterns that are not recorded in the repository (.git/info/exclude), despite that git-svn can generate that file from svn:ignore properties, and despite that they should have been added to Subversion anyway. (I’ll leave the project unnamed. That same user re-added stupid ignore patterns to the Git repository later on… I fixed them (again) even though I lack a Mac, because GitHub maintains gitignore patterns I could look up for XCode, which is handy.)

    I would like an automated solution for this as well, but you can never anticipate all forms of abuse that can happen in a repository. Spitting a warning for ones that you can anticipate is a good fallback for when the solution is not obvious nor simple.

  10. Hmmm… I wonder why Git doesn’t have default ignore patterns (well, except for commented out in `.git/info/excludes` template), and whether it is a good thing.

    Should default Subversion ignore patterns be:
    1.) put in top level .gitignore file (inside project tree),
    2.) put in per-repository .git/info/excludes (visible only for specific clone of repository),
    3.) put in per-user ~/.config/git/ignore (visible only for one user, but for all repositories),
    4.) ignored,
    5.) curated according to the type of project and put in 1-3?

    1. >1.) put in top level .gitignore file (inside project tree),

      This is what reposurgeon currently does. The goal behind this choice was to make a repository user’s exposure to messages about untracked files not change in surprising ways. It would be perverse, for example, for *.o files to be unignored after an svn to git repo translation when they were ignored before.

  11. Hmmm… I wonder why Git doesn’t have default ignore patterns (well, except for commented out in `.git/info/excludes` template), and whether it is a good thing.

    I can make no presumption about design decisions behind Git, or whether it had much thought behind it at all, but if I were to design a VCS, this is the behavior I would explicitly go for and I would reject suggestions of any kind of default ignore patterns.

    Any kind of ignore pattern creates an unfair amount of arrogance as to what kind of data your VCS will be used for. For example, just taking the default CVS and SVN ignore of *.o, it already creates an aura of presuming that the VCS is intended for C language programming projects, and if you ever have a reason to store a file with such a name, you will immediately encounter problems. Both in how to commit that file to the repository, and figuring out how to disable that ignore pattern.

    There might be a small benefit too with not wasting cycles checking for object files if you’re making a game in HTML and JavaScript, but you’re probably already not concerned about cycles in such a case. :)

  12. > Any kind of ignore pattern creates an unfair amount of arrogance as to what kind of data your VCS will be used for. For example, just taking the default CVS and SVN ignore of *.o, it already creates an aura of presuming that the VCS is intended for C language programming projects, and if you ever have a reason to store a file with such a name, you will immediately encounter problems.

    *.o isn’t best example; as far as I know it is used _only_ for object files (which is generated file), and there isn’t any source file format with this extension… but *.a _may_ be used for Ada source code, AFAIK.

    > Both in how to commit that file to the repository,

    $ git add -f fo.o

    > and figuring out how to disable that ignore pattern.

    !*.o

  13. One of the caveats of svn:ignore is that they are not recursive and cannot be made recursive — each ignore pattern only applies to the directory it is specified on.

    As a consequence, the set of ignores specified in directory properties will sometimes be incomplete. For example, if you create a new subdirectory under a directory having some ignores, it may not inherit the parent’s ignores until someone comes along and adds them.

    In such cases, explicit .gitignore files would probably be more accurate. I might even say using the existing .gitignore is a way of deferring to the human decision already made.

  14. @esr:

    It’s interesting comparing my experience in the Enterprise Software market with the design decisions you are making. For example:

    > With the report, I got a request for a switch to suppress this behavior. The right answer, I
    > replied, was not to add that switch and some complexity to cvs-fast-export. Rather, I reminded
    > the requester that he could easily delete that synthetic .gitignore from the history using
    > reposurgeon. Then I added the command to do that to the list of examples on the reposurgeon
    > man page.

    What bothers me is cases where we have to add a lot of code to support a use case which shouldn’t be used (security-related, there are much safer/better ways to get the same results with a little more effort), but which, say, $100M in sales are relying on. Imagine if your next 2 years of personal income depended upon you implementing that one switch.

    It’s an interesting problem in economics which I don’t know if anybody’s studied extensively.

  15. Garrett: My call would be – put the switch in, document it for that one customer *ONLY*, and include plenty of extra documentation on how to do it right, why not to use that switch, and desperate pleas to keep our Internet safe.

    And while, I do think sometimes the best decision is to punt to the user, perhaps there’s room for the software to help, by showing the issue, explaining the choices, and making suggestions for resolution.

  16. *.o isn’t best example; as far as I know it is used _only_ for object files (which is generated file), and there isn’t any source file format with this extension… but *.a _may_ be used for Ada source code, AFAIK.
    Perhaps not, but the point wasn’t really in picking an example that would conjure up immediate counter-examples for usefulness, just that it’s a general problem.

    > Both in how to commit that file to the repository,

    $ git add -f fo.o

    > and figuring out how to disable that ignore pattern.

    !*.o

    Again, thanks (though I’m familiar with Git to know this already), but this misses the point. I was speaking in general terms on behalf of a user of any VCS at all. It doesn’t really matter in the how on any one, all that matters is that a user trying to do something that is against the default ignore patterns will be surprised and probably a bit frustrated in figuring out how to make it work for themself.

  17. One of the senses you develop after writing complex software for a couple of decades is some ability to tell when a feature is going to be a defect attractor – a source of numerous hard-to-characterize bugs and a maintenance nightmare.

    Yup.

    Around here we call them “bug pumps”, which is less formal and perhaps more evocative.

    (And just from reading your report/request I had the same feeling you did. There’s No Way That Will End Well.)

  18. @esr, how reposurgeon deals with the fact that explicit svn:ignore patterns are non-recursive (while default Subversion ignore patterns are [equivalent to] recursive), while by default .gitignore patterns are recursive – do you anchor non-default svn:ignore translating them to .gitignore patterns (i.e. “/foo” instead of “foo”)? Do you use one top-level .gitignore, or one per directory following svn:ignores?

    You did remember about escaping special characters and trailing spaces in .gitignore (\#, \!, \ ), didn’t you?

    1. >do you anchor non-default svn:ignore translating them to .gitignore patterns (i.e. “/foo” instead of “foo”

      Oops. Wasn’t catching that detail. Just fixed it.

      >Do you use one top-level .gitignore, or one per directory following svn:ignores?

      One per directory following svn:ignores.

      >You did remember about escaping special characters and trailing spaces in .gitignore (\#, \!, \ ), didn’t you?

      I don’t know of a pattern case that requires this. Feel free to enlighten me if you do.

  19. In .gitignore a line starting with “#” serves as a comment. If the pattern starts with “#” then the first occurence of this character needs to be escaped (e.g. “\#*#” for Emacs temporary files, for example for TRAMP – transparent remote file access).

    In .gitignore pattern can have an optional prefix “!” which negates the pattern (unignoring file if it was ignored by one of earlier patterns / rules). You need to put a backslash (“\”) in front of the first “!” for patterns that begin with a literal “!”, for example, “\!important!.txt”.

    Trailing spaces in .gitignore are ignored unless they are quoted with backlash (“\ “). I cannot think of any sane filename that ends with trailing space ;-P

    BTW. the default Subversion ignore patterns (global-ignores) should be, in my opinion, not anchored (e.g. “*.o” not “/*.o”) and present only in top-level .gitignore.

  20. I’m dealing with this at work right now. I’m writing code to give a certain operation better approximation of atomicity. To achieve that, I need to write code to rollback early parts of the operation if later parts fail. The problem is that the rollback itself might theoretically fail (rollback at all should be very rare, failure of rollback can only happen if the network goes down in that exact quarter second.).

    What I proposed is that the system should scream for help in that case, and let a human repair the situation.

    Other parties insist that complicated retry logic must be written to keep attempting rollback until it succeeds. One proposal would even have reported success to the user, then silently rolled it back minutes later!

    I am going to use this post as ammo for my side.

  21. >This bears emphasizing because hackers have such a horror of manularity, going to extreme lengths to automate instead.

    What we, ERP consuiltants like to do, if an automatization requests sounds fishy, is to prepare a report that offers every kind of information required for doing it manually in a well usable way, and all the user has to do is to use it the simplest way i.e. to e.g. enter the subtotals printed on the the report into an onscreen form and click a button. We hope is that they realize while doing so that wait, this is not always so, it is not that regular, that certain things were either left out from the spec or can’t be decided automatically at all based on the current scope of the data. We are usually right. When we are wrong and it works well enough this way many times, then that report code can be 80% reused for making the actual automatism anyway.

    It is basically a roundabout way of the ubiquitous “Do you really want to do it Yes No?” message box, except that average users never stop and think before clicking Yes (they don’t read it at all, usually), but entering 5 or 10 figures into a form is enough time for their brain to turn on and ask “Wait, what am I even doing, this is not correct?”

  22. I’ve seen this in other areas. Linux distro repos, where complex conflict resolution is worked over and over, and complex configuration is needed to get it right – when it would be much easier for the update software to say “Hey, got these two packages – which one do you want?”

    The longer I Linux, the more I realize there is deep wisdom in Slackware’s decision to punt on package dependency resolution.

Leave a Reply to Jakub Narebski Cancel reply

Your email address will not be published. Required fields are marked *