Thinking like a master programmer, redux

Yes, there was a bug in my vint64 encapsulation commit. I will neither confirm nor deny any conjecture that I left it in there deliberately to see who would be sharp enough to spot it. I will however note that it is a perfect tutorial example for how you should spot bugs, and why revisions with a simple and provable relationship to their ancestors are best

The following call in libntp/ntp_calendar.c is incorrect:

setvint64u(res, vint64s(res)-0x80000000);

Now consider the line this replaced:

res.Q_s -= 0x80000000;

And notice what that expands to, semantically, in C:

res.Q_s = res.Q_s – 0x80000000;

Spotted it yet?

My encapsulation patch is extremely regular in form. One of my blog commenters (the only one to spot the bug, so far) pointed out correctly that an ideal transformation of this kind looks like it was done using a text editor search and replace feature – and, in fact, I did most of it with regexp-replace commands in Emacs.

It’s good when your patches are this regular, because it means that you can spot bugs by looking for irregularities – places where a local change breaks a rule followed in the rest of the patch. Importantly, this way to spot defects works even when you don’t fully understand the code.

This is a major reason the code state after every change should have a single provable relationship to its antecedent – because if it has more than one change in it, telltale irregularities will be harder to see.

OK, here is the corrected form of the call:

setvint64u(res, vint64u(res)-0x80000000);

The one-character difference is that the correct inner call is to vint64u(), not vint64s(). You should have been able to spot this in one of a couple of ways.

One is by noticing that the original expression was doing unsigned arithmetic, so what is that call to get a signed value doing in there?

The even simpler way to spot the irregularity is to have noticed that in the rest of the diff there are no other calls like

setvint64X(res, vint64Y(res) … );

in which X and Y are unequal. There is a purely textual symmetry in the patch that this one statement breaks. Because the author was being careful about simplicity and provable relationships, that in itself should be enough to focus a reviewer’s suspicions even if the reviewer doesn’t know (or has forgotten) the meaning of the s and u suffixes.

I’m writing this as though the author and reviewer are different people, but these techniques for bug spotting – and, more importantly, these techniques for writing patches so bugs are easy to spot – apply even when you are your own reviewer and you are looking at a diff mere moments after you changed the code.

You get fast at coding, and you get good at doing it with a low defect rate, by developing the habits of mind that make self-checking like this easy. The faster you can self-check, the faster you can write while holding expected defect rates constant. The better you can self-check, the lower you can push your defect rate.

“To do large code changes correctly, factor them into a series of smaller steps such that each revision has a well-defined and provable relationship to the last” is good advice exactly because the “well-defined and provable” relationship creates regularities – invariants – that make buggy changes relatively easy to spot before you commit them.

I often go entire months per project without committing a bug to the repository. There have been good stretches on NTPsec in which my error rate was down around one introduced bug per quarter while I was coding at apparently breakneck speed. This is how I do that.

Having good tests – and the habit of adding a unit or regression test on every new feature or bug – helps a lot with that, of course. But prior to testing is good habits of mind. The combination of good habits of mind with good testing is not just additively effective, it’s multiplicatively so.


  1. Speaking of single-character errors, your quote file has Lazarus claiming that a human should be able to conn a shp.

  2. I tend to play with an idea that there is no such thing as a programmer, the same way there is no such thing as a doctor – people who write code are just as specialized as eye surgeons. The eye surgeon is not necessarily very good at setting a broken leg (although better than the average chum – they study that, to a certain extent, too, but have little experience in it).

    The reason people who write code are specialized is that sometime in the late eighties everybody got busy enough not to have time to write specs anymore. For a good generation at least, we cannot afford the kind of “coder” programmer who needs detailed specs with flowcharts and pseudocode written by someone else. “If it is up to spec, it is OK, it is the spec that is wrong” disappeared from our dictionary a generation ago, now it is “either it does the job or fix it”.

    Thus now we expect people who write code to have domain knowledge and more or less able to turn a fairly vague agreed on concept into experimental but largely working code.

    So for a while we are either teaching domain experts to code or programmers sooner or later learn the domain, or, ideally, there are people majoring in X, minoring in programming (because programming is more self-teachable) and then work on automating X.

    Depending on the domain knowledge, we have at least four common subtypes of programmers:

    – Techie programmers. They are roughly equivalent to what the average layperson thinks about computer geeks. They know and like computers, as in, the actual hardware, know and like to do low-level things like C and assembler. They may even have a degree in electrical engineering. They may not always be very good at using a solder iron, but they know why exactly are people using GPU’s for bitcoin mining.

    – Mathy programmers. The theoretical Dijkstra computer science types. Know type theory and all that jazz. They are not weirded out by Haskell. Very good at algorithmical ability and writing provable code.

    I think ESR type master programmers tend to an amalgam of the first two types, because it is the historic meaning of programmer: a half-engineer, half-mathemathician type. Thus they tend to seek out problems which have some mathy aspects and some hardwareish aspects – gpsd being an excellent example.

    A third type is the science programmer, the analyst of experimental data, the modeller. I know little about this field, all I know how surprisingly badly it is often done. Apparently it is a good idea after each reactor experiment to code something up from scratch in C++ to crunch the numbers, instead of using a framework with plugins… at least I keep hearing stories that it is done.

    A fourth type is the business programmer, a distant descendant of the COBOL guys of long past times. He or she is likely more of an SQL guy – because whatever complicated frameworks enterprises use on top of their databases, when you need to know which customers had negative margin right now because the meeting is in 10 minutes, it is typically back to SQL. Their domain knowledge is usually accounting – they have to get good at it, they have to sometimes argue with auditors that yes, the code is up to accepted principles.

    A potential fifth type is the game developer. He might have a specialized job at a large team, looking only after the engine which is math+hardware, but in a pinch he would draw graphics or test play balance. He is versatile.

    A sixth may be the medical programmer. I know little about the field, all I have heard you spend 1% of the effort on doing the job and 99% on not getting sued.

    Anyhow the kind of people ESR calles programmers and master programmers tend to be an amalgam of the first two classes.

    1. >Anyhow the kind of people ESR calles programmers and master programmers tend to be an amalgam of the first two classes.

      I think that is largely true. But I also think my advice about how to code fast and push down defect rates applies to your other types as well.

  3. From experience, I know that there is also a “Designer Programmer”. He specialized in GUIs and data presentation. He knows CSS and JavaScript very well, (or the equivalent UI centric languages for his platforms). He is capable, but probably not expert, in at least one backend language. He needs Good Taste and artsy skills as well as technical ones and probably has some Photoshop/Gimp skills as well.

    I am not one of these, but I work with several of them guys.

  4. This often can be simple, or really hard, but the foundation is the regression test.
    Often I code the inverse function so I can verify X == f**-1(f(X)) and just fuzz it plus the usual suspect edge cases. This is the most basic “f is still the same” test.
    Sometimes you can use asserts, or logs or something else.

    There are TWO different issues here.

    The first is you have an inefficient and/or ugly implementation and need to clean it up. This is simpler in that you can just instrument or create fixtures and see if decrepit-F and clean-F do the same things.

    The second is that decrepit-F is itself untested, broken, or otherwise needs fixing. I would normally defer any attempt and adjust the clean-F (adding in clearly marked tumors) to force errant behavior than can, after review, be removed when the refactoring is finished).

    Sometimes you need to make it do the same thing, not what the intended thing was because the side-effects may be obscure. So the first task is to duplicate an identical part.

  5. > I will neither confirm nor deny any conjecture that I left it in there deliberately to see who would be sharp enough to spot it.

    Thus does Master Foo maintain his reputation. The monk who finds one of these bugs does not think “Master Foo is stupid for letting this bug get into the code”. He thinks “Master Foo is wise to test our skills by leaving a subtle bug for us to find.”

  6. The types can be further generalized. There are programmers, and then there are people who do programming as an adjunct to something else. The first category are types 1, 2, and (1/2) above. The second category is everyone else.

    In practice, I think the transition from my second category into my first is far more common than the reverse. In an inexact analogy, consider Cliff Stoll, who started out using computers for physics, then during the Cuckoo’s Egg era, found himself using computers for computers. (Not programming, of course, and he went back to physics eventually. Why are you complaining? I said it wasn’t an exact analogy.)

    Knuth vs. the Evil Typesetting Industry comes to mind as an example of the reverse, but it still feels less common for some reason.

Leave a comment

Your email address will not be published. Required fields are marked *