Engineering zero-defect software

I’ve been pounding on GPSD with the Coverity static analyzer’s self-build procedure for several days. It is my great pleasure to report that we have just reached zero defect reports in 72.8KLOC. Coverity says this code is clean. And because I think this should be an example unto others, I shall explain how I think others can do likewise.

OK, if you’re scratching your head…Coverity is a code-analysis tool – an extremely good one, probably at this moment the best in the world (though LLVM’s open-source ‘scan-build’ is chasing it and seems likely to pass it sometime down the road), It’s proprietary and normally costs mucho bucks, but as a marketing/goodwill gesture the company allows open source projects to register with them and get remote use of an instance hosted at the company’s data center.

I dislike proprietary tools in general, but I also believe GPSD’s reliability is extremely important. Navigation systems are life-critical – bugs in them can kill people. Therefore I’ll take all the help I can get pushing down our error rate, and to hell with ideological purity if that gets in the way.

Coverity won’t find everything, of course – it’s certainly not going to rescue you from a bad choice of algorithm. But it’s very, very good at finding the sorts of lower-level mistakes that human beings are very bad at spotting – memory allocation errors, resource leaks, null-pointer dereferences and the like. These are what drive bad code to crashes, catatonia, and heisenbugs.

Excluding false positives and places Coverity was being a bit anal-retentive without finding an actual bug, I found 13 real defects on this pass – all on rarely-used code paths, which makes sense for reason I’ll explain shortly. That’s less than 1 defect per 5 KLOC (KLOC = 1000 logical lines of code) which is pretty good considering our last scan was in 2007. Another way to look at that data is that, even while adding large new features like AIS support and NMEA200 and re-engineering the entire reporting protocol, we’ve introduced a bit fewer than three detectable defects per year in the last five years.

Those of you who are experienced software engineers will be picking your jaws up off the floor at that statistic. Those of you aren’t – this is at least two orders of magnitude better than typical. There are probably systems architects at Fortune 500 companies who would kill their own mothers for defect rates that low. Mythically, military avionics software and the stuff they load on the Space Shuttle is supposed to be this good, except I’ve heard from insiders that rather often it isn’t.

So, how did we do it? On no budget and with all of three core developers, only one working anywhere even near full time?

You’ll be expecting me to say the power of open source, and that’s not wrong. Sunlight is the best disinfectant, many eyeballs make bugs shallow, etc. etc. While I agree that’s next to a necessary condition for defect rates this low, it’s not sufficient. There are very specific additional things we did – things I sometimes had to push on my senior devs about because they at times looked like unnecessary overhead or obsessive tailchasing.

Here’s how you engineer software for zero defects:

1. Be open source.

And not just because you get helpful bug reports from strangers, either, all though that does happen and can be very important. Actually, my best bug-finders are semi-regulars who don’t have commit access to the code but keep a close eye on it anyway. Like, there’s this Russian guy who often materializes on IRC late at night and can barely make himself understood in English, but his patches speak clearly and loudly.

But almost as importantly, being open source plugs you into things like the Debian porterboxes. A couple of weeks ago I spent several days chasing down port failures that I thought might indicate fragile or buggy spots in the code. It was hugely helpful that I could ssh into all manner of odd machines running Linux, including a System 390 mainframe, and run my same test suite on all of them to spot problems due to endianness or word-size or signed-char-vs.-unsigned-char differences.

Closed-source shops, in general, don’t have any equivalent of the Debian porterboxes because they can’t afford them – their support coalition isn’t broad enough. When you play with the open-source kids, you’re in the biggest gang with the best toys.

Invest your time heavily in unit tests and regression tests

GPSD has around 90 unit tests and regression tests, including sample device output for almost every sensor type we support. I put a lot of effort into making the tests easy and fast to run so they can be run often – and they are, almost every time executable code is modified. This makes it actively difficult for random code changes to break our device drivers without somebody noticing right quick.

Which isn’t to say those drivers can’t be wrong, just that the ways they can be wrong are constrained to be through either (a) a protocol-spec-level misunderstanding of what the driver is supposed to be doing, or (b) an implementation bug somewhere in the program’s state space that is obscure and difficult to reach. Coverity only turned up two driver bugs – static buffer overruns in methods for changing the device’s reporting protocol and line speed that escaped notice because they can’t be checked in our test harnesses but only on a live device.

This is also why Coverity didn’t find defects on commonly-used code paths. If there’d been any, the regression tests probably would have smashed them out long ago. I put in a great deal of boring, grubby, finicky work getting our test framework in shape, but it has paid off hugely.

Use every fault scanner you can lay your hands on.

Ever since our first Coverity scan in 2007 I’d been trying to get a repeat set up, but Coverity was unresponsive and their internal processes clearly rather a shambles until recently. But there were three other static analyzers I had been applying on a regular basis – splint, cppcheck, and scan-build.

Of these, splint is (a) the oldest, (b) the most effective at turning up bugs, and (c) far and away the biggest pain in the ass to use. My senior devs dislike the cryptic, cluttery magic comments you have to drop all over your source to pass hints to splint and suppress its extremely voluminous and picky output, and with some reason. The thing is, splint checking turns up real bugs at a low but consistent rate – one or two each release cycle.

cppcheck is much newer and much less prone to false positives. Likewise scan-build. But here’s what experience tells me: each of these three tools finds overlapping but different sets of bugs. Coverity is, by reputation at least, capable enough that it might dominate one or more of them – but why take chances? Best to use all four and constrain the population of undiscovered bugs into as small a fraction of the state space as we can.

And you can bet heavily that as new fault scanners for C/C++ code become available I’ll be jumping right on top of them. I like it when programs find low-level bugs for me; that frees me to concentrate on the high-level ones they can’t find.

Be methodical and merciless

I don’t think magic or genius is required to get defect densities as low as GPSD’s. It’s more a matter of sheer bloody-minded persistence – the willingness to do the up-front work required to apply and discipline fault scanners, write test harnesses, and automate your verification process so you can run a truly rigorous validation with the push of a button.

Many more projects could do this than do. And many more projects should.

89 thoughts on “Engineering zero-defect software

  1. >Two real bugs, but in test cases for impossible errors, in 80,000 lines of code:

    Man, that’s good code!

    I think it’s an interesting datum that GPSD and Postscript are within shouting distance of the same size.

  2. Mythically, military avionics software and the stuff they load on the Space Shuttle is supposed to be this good, except I’ve heard from insiders that rather often it isn’t.

    Um, no you’re right. It isn’t. I’ve seen the source code for some declassified military avionics — particularly the navigational component. It runs on MS-DOS computers. The only problem being, it was originally written for an 8088 in MIcrosoft C. I could stop right there, because the bugs in the Microsoft C compiler are legendary, but I’ll take it a step further — if you’ve ever played games on an 8088 and tried to move that game to a faster computer, like a 286 or 386, then you know the faster the computer, the more timing bugs show up because those 8088 machines lacked proper hardware timers. Plus, there was at least one buffer overflow bug.

    They had to have me fix it because I was the only person in the company who actually still knew anything about MS-DOS and Microsoft C, having run a BBS in the 1980s. :)

    The problem is that the code _is_ more or less ‘defect free’ when it is run on the hardware and the in the precise environment in which it was written. Change those variables and the bugs magically appear. I’m sure I don’t need to explain that particular phenomenon to anyone here.

  3. God enchants people to miss mistakes. God controls defects. He can make a defect come-up in a key demo. God says…

    compressed scholars Thine Let brackishness speechless absolute
    consisting Stop displacing servitude marry cogitated waves
    anxious girl

  4. I’m rather surprised you haven’t (or haven’t mentioned that you have) played with Klee.

  5. >I’m rather surprised you haven’t (or haven’t mentioned that you have) played with Klee

    First I’d heard of it. Looks interesting; I’ll read the paper.

  6. 10 PRINT “hello, world”

    Zero-defect software is easy. Zero-defect software that does something useful is hard.

  7. The second hardest part about getting good code is convincing programmers to write regression tests for existing code. The hardest part is convincing managers to budget enough time to write the regression tests.

  8. I have to add another point:

    *** Use a DVCS ***

    If it weren’t so easy to commit my fixes on a private branch, run “scons check” and do a quick “git send-email master..” to send the patches to the mailing list (after configuring a few settings in the Git repo), I would certainly submit less often.

  9. So let me give you my perspective on this from the land of proprietary and in house software. I agree with all of your recommendations for the most part. However, I am sure you will also agree that it is not fair to compare the elite oss projects (like yours) with the average proprietary project. There are lots and lots of crappy, poorly run OSS projects after all. So my comments apply not to the general case but to very well run proprietary projects.

    One of the most important matters to consider is simply this: quality vs cost is a strange curve. (Using cost here not necessarily in terms of $$ but also of sweat equity.)

    For the most part good practice to eliminate and catch bugs early has it such that the cost of doing so is much less than the cost of not doing so. So cost per % quality increase is actually negative for most of the way up to 100%.

    However, when you get toward the top of the scale: eliminating the last few bugs in all those weird never used code paths, and all those rare circumstances, the curve flips dramatically. That is the cost for the last 10% of bugs is generally very large.

    For example one way used in some high grade software is to actually have two separate contractors write the same software from identical specs. Then use a results comparison engine to get a voting decision. Obviously that costs at least twice as much.

    Sorry this is a bit handwavy, but if your read Steve McConnell’s book Code Complete, he has a huge amount of statistical analysis of this stuff.

    So the first question is this: is it really worth the cost to get to 100%? In some cases yes, gpsd could be one, as you say there are life critical things in there. I have a diabetic friend who wears a device that pumps insulin into her blood based on a remote control. Is it worth getting every last bug out of that? Hell yeah.

    But for a tool to post the soccer schedule on the office web? Not so much.

    In my world they say “quality is a feature”, meaning you can trade off the total effort available for quality verses additional features, and that is a good conscious choice to make. Sometimes a slightly crappier program with more features is the right choice, sometimes a less crappy program with less features is the right choice. However, a well run project makes the choice consciously. And like I say the “crappiness” refers to that last 10% of rare bugs, not the 90% of common ones — since the cost of fixing them is negative.

    Second, as you get the higher quality levels, your underlying infrastructure quality looms larger and larger. So, if you are using libraries, frame works their quality grows to be significant.

    OK, I’m going to stop here, and add a few comments about your specific recommendations later.

  10. >*** Use a DVCS ***

    I thought about adding that. But then thought: “It’s 2012. Does anybody not?”

    OK, I’ll concede there’s still a lot of Subversion out there.

  11. >The hardest part is convincing managers to budget enough time to write the regression tests.

    In this case the ‘manager’ was me. So perhaps the actual lesson is that if we want quality, software, projects shouldn’t be managed by people who aren’t software engineers. (Well, duh.)

  12. In my experience, continuously updated code metrics (c0 coverage, cyclomatic complexity, etc.) are invaluable. That, and treating every automated test failure as a blocker.

  13. >God enchants people to miss mistakes. God controls defects. He can make a defect come-up in a key demo. God says…

    I just finished reading the eARC of Ringo’s Queen of Wands, according to it the gods only love gullible idiots, they purposely mislead scientists by manipulating experiments to hide their existence (and that is from a pro-religion author).

  14. > I thought about adding that. But then thought: “It’s 2012. Does anybody not?”

    Yes. My employer still runs off of Perforce *shudder*. Like most everything else, this is due to inertial and tight coupling with other in-house systems which we have.

    > So perhaps the actual lesson is that if we want quality, software, projects shouldn’t be managed
    > by people who aren’t software engineers.

    Yes, though this isn’t optimal for the bottom line. Spending all of the extra effort you did might have been more economic-efficiently allocated to writing a new application, or debugging another one.

    Finally, I’d point out that you don’t have zero-defect code, only zero-known-defect code. :-)

  15. Regarding some of your specific points:

    OSS: the benefits you indicate are the many eyes review, and the availability of many test platforms.

    In regards to the first, well run projects have a strong ethic of peer review, and a process in placed to do it. So it is a concentrated review rather than a broad based review — cathedral not bazaar. There is gigantic variation in the quality of code reviews, and lots of people are very uncomfortable with them because they are often so badly done, but I’ll remind you that the fair comparison is between GPSD and very well run proprietary software, and I have seen this in action.

    BTW one of the main failures of many code review processes is a failure to ask the question “review for what.” When code reviews are focused on verifying certain specific traits then they are more successful, even at identifying flaws outside of those traits. I’m not sure how that works with the many eyes thing.

    In regards to the access to multiple platforms, big shops in the past in fact did have banks and banks of computers in many formats. However, now that isn’t necessary. Most well run shops have large libraries of VMWare images of computers in all the targeted platforms and software is regularly regression tested on them. Usually, in nightly build shops they don’t regression on all platforms, since it is usually too time consuming, and certainly not on CI build shops (where a build and smoke test kicks with every check in.) But well run shops running agile always kick them on all platforms before each iteration cycle completes.

  16. >Finally, I’d point out that you don’t have zero-defect code, only zero-known-defect code. :-)

    I was careful about the difference in the body of the article :-)

  17. Eric, am I right in thinking that the tools you mention only analyse the C portion of your codebase?

    If so, do you have experience with any Python-oriented fault-detection tools?

  18. >Eric, am I right in thinking that the tools you mention only analyse the C portion of your codebase?

    Yes. I also use pychecker and pylint on the Python code.

  19. >I offer these not as recommendations, but as a request for comment in the event that you are familiar with these tools.

    Nope, I’ll look into them.

  20. @ESR – “software projects shouldn’t be managed by people who aren’t software engineers. (Well, duh.)”

    Please excuse this topic tangent, but the above comment brought to mind an inverse scenario from my past.

    A few years ago, a group of “software” engineers from Silicon Valley came to my town and started a company that aimed to build a “green solution to the world’s energy needs.” The process they were intent on building consisted of a sophisticated petrochemical plant. Unlike video games, these plants really can blow up if not designed, operated, and maintained properly. Think Bhopal, if you have any doubts.

    Luckily, they only succeeded in creating a financial disaster, but it could have have been much worse and they were completely oblivious to their own shortcomings.

  21. Yes. My employer still runs off of Perforce *shudder*. Like most everything else, this is due to inertial and tight coupling with other in-house systems which we have.

    A lot of proprietary shops use Perforce, simply because there’s nothing else out there that can wrangle large code and asset bases the way Perforce can. Git doesn’t cut it, I’m afraid; when you’re talking about large code bases with large histories, especially when there’s a lot of binary data involved. It starts slowing way down. But you can check anything into Perforce and it will chug right along. That’s why virtually every AAA game studio uses it. (Games are very asset-heavy.) It’s why Microsoft was using a modified version of it at the same time they were trying to foist VSS on everyone else.

  22. @Jeff Read

    Git doesn’t cut it, I’m afraid; when you’re talking about large code bases with large histories, especially when there’s a lot of binary data involved. It starts slowing way down.

    Slowdowns with big binary data I’ll give you (although there are ways to work around it, e.g. git-annex) but large code-bases and large histories? I don’t think so. Just look at the linux kernel. Millions of lines of code and a multi-decade development history. I think you’d have to have a pretty huge project for git not to be workable. Should be fine for practically anything.

  23. > What about sparse static checker used by Linux kernel?

    To quote the page you linked to: “Sparse provides a set of annotations designed to convey semantic information about types, such as what address space pointers point to, or what locks a function acquires or releases.” It also has annotations that allow to distinguish types for litte/big/host endian values.

    It is designed to find typical errors in kernel code, and requires these annotations for most of its checks because it tries very hard to avoid false positives.

    Many annotations can be hidden in typedefs, but sparse is not very useful for general user space code.

  24. Eric, with regards to point 2, any thoughts on using the test driven development methodology espoused by Kent Beck (any many others)? I’ve been using where I work, and have decent success with it.

  25. Eric: Having read your article, I’m interested in sorting out the effects of good software engineering in general from any GPSD-specific artifacts. If I remember correctly, you once mentioned that GPSD has an exceptionally low rate of memory bugs because of an almost-no-malloc policy you impose on your C code. What defect rate would be realistic in similarly well-engineered code where such a policy is impractical? For example, what would happen if you ran a coverty scan on the fetchmail source tree?

  26. I’m surprised you didn’t mention Valgrind.

    Coupled with Alleyoop, I have always found it to be a mighty powerful tool for revealing all those same low-level defects.

    I’m going to update my GPSD codebase soon and try it ;)

  27. ’10 PRINT “hello, world”’

    Error: No ‘END’ statement

    @Mr. No: Zero-defect software isn’t as easy as you thought….

  28. aha….after grabbing a fresh copy from your git repo, I see reference to a Valgrind “torture test”

    Is that test part of your routine test suite?

  29. >Many annotations can be hidden in typedefs, but sparse is not very useful for general user space code.

    I came to the same conclusion after reading its manual page.

  30. >For example, what would happen if you ran a coverty scan on the fetchmail source tree?

    I can’t know without doing it, and attempted generalizations about this sort of thing are doomed to spectacular failure. I will note that (a) there are some kinds of malloc-related errors that static-checkers do catch (notably, they’re not bad at finding memory leaks) and (b) there are tools specialized for finding malloc-usage problems, such as mudflap and valgrind. A properly-designed test strategy for fetchmail would emphasize these more.

  31. I’d also throw out needless complexity. There are a number off tools that will give you scores for cyclomatic (McCabe) complexity in a codebase. Higher numbers point to code that is harder to comprehend & debug, often obscuring tricky bugs. Refactoring code to eliminate complexity (within reason) leads to code that is much easier to read & reason about and leaves fewer places for bugs to hide.

  32. >Is that [valgrind torture] test part of your routine test suite?

    No. I do tend to run it before release, but because it’s aimed a kind of error our coding standards exclude it’s not part of scons testregress.

  33. True, your ‘no malloc’ policy does make part of Valgrind’s checking redundant…..but if you’re playing with pointers, it’ll catch the errant knaves too ;)

  34. >There is a bug on the following line:

    Yes, there is. I find it curious that none of the static checkers caught this.

  35. >There are a number off tools that will give you scores for cyclomatic (McCabe) complexity in a codebase.

    I downloaded pmccabe and gave it a try. I didn’t find it useful.

    The problem was there were no surprises in the results. I already knew where the complexity is – in, for example, the packet sniffer, which is a rather complex finite-state-automaton (essentially, a lexical analyzer) implemented as the Monster Case Statement From Hell. Or, right, the main sequence of the gpsd daemon; that, with all the socket-foo and session-state management in it, is complex – no shit? And, the thread monitor function for PPS is a serious hairball – what a shock.

    Yes, I could factor these lumps. But doing so would seriously damage the readability of the code. They’re large because they need to be.

    I am left with an unfavorable impression of cyclomatic measurement. If you actually need it to know where your complexity is you are unlikely to benefit from it, because that need means you don’t understand your own code well enough to refactor it.

  36. > Yes, I could factor these lumps. But doing so would seriously
    > damage the readability of the code. They’re large because they
    > need to be.

    Right, hence my “within reason” caveat.

    > I am left with an unfavorable impression of cyclomatic
    > measurement. If you actually need it to know where your
    > complexity is you are unlikely to benefit from it, because that
    > need means you don’t understand your own code well enough
    > to refactor it.

    If you own the code and completely grok it then it would probably not give much useful info. I’ve found it useful when reviewing or inheriting other peoples’ code, especially when trying to help other developers to make their code more modular, rather than single functions that are (needlessly) 5 pages of spaghetti. Trying to find/fix a bug in a 5 page function is much harder than doing so in a 5 liner.

  37. I am left with an unfavorable impression of cyclomatic measurement.

    You have the distinct fortune of spending virtually all of your tube time on open source projects where your work is accountable mainly to other hackers. In plenty of real-world software projects, metrics like cyclomatic complexity are tracked and reported by nonhacker project managers, and the information they convey is useful to them and their superiors, e.g., for formulating estimates or setting maintainability targets and guidelines.

  38. That said, I too have my doubts about cyclomatic complexity, as there is research to show that SLOC is as good a proxy for code complexity as cyclomatic measurement is. Measuring SLOC has the advantage that it can be done with the equivalent of simple hand tools — to wit, the contents of /usr/bin on a standard Unix system.

  39. More from the world of proprietary software:

    Automated testing. Very much in vogue is the idea of test driven development, or at least test centered development. I’m not especially familiar with GPSD but from what I know it is the sort of software that is relatively easy to unit test. That isn’t to say it isn’t a complex piece of software, clearly it is a medium sized, high complexity product.

    However its nature is very much these inputs lead to this output without too much state or too much arbitrariness. The devices produce a data stream in this form, and GPSD produces a data stream in this form.

    This strictly controlled inputs leands outputs is pretty easy to handle in a unit testing scenario. Of course that is a simplification, but you don’t have to deal with some of the complexities many applications do.

    Most of the stuff I write deals with things that are very difficult to unit test — things like databases and GUIs. In one standard model of software, MVC, the model is easy to test, the controller and the view are very difficult to test.

    This is especially so because the doctrine of unit testing is not systemic end to end, but isolated testing of a particular module or function. Isolation means that the inputs and outputs are directly controlled in the testing infrastructure itself. In my world this is implemented via DI and IoC, which not only is challenging, but directly impacts HOW you design the software in the first place. In addition to the qualities normally looked for in software — quality, performance, fidelity to spec, low bug count — a new quality is now added — testability.

    I suspect this is really hard to do in C. Most of the time it depends on introspection of the code and interface based programming (or ABCs in C++.)

    Anyway, my main point is that designs that are testable are crucial to good unit testing in software that uses complex dependencies like databases and GUIs. I agree 100% with Eric that regression testing is crucial to software quality, and a point he did not make, but which I imagine he agrees with, is that unit regression testing is especially important to plasticity in software, when the inevitable change comes along. Refactoring without high coverage unit tests is a contradiction in terms.

    But it is relatively easy to do when you don’t have to deal with DI and IoC complexities. This is an active area of research in my world and the Java world. For the interested you might want to look at Moq or RhinoMoq to find out some of the profound ideas spinning in my world on this stuff.

  40. >That isn’t to say it isn’t a complex piece of software, clearly it is a medium sized, high complexity product.

    Exactly. GPSD is not huge in KLOC terms, but the internals are complex and dense for their size – there’s big honkin’ FSA at the heart of it, a lot of platform-sensitive bit-bashing in the drivers, stuff that involves 3D geometry and matrix arithmetic in the error modeling. The top level doing session-state maintenance and socket-foo is simpler but by no means trivial.

    >Of course that is a simplification, but you don’t have to deal with some of the complexities many applications do.

    That is correct. Databases don’t scare me because queries and updates can be scripted and run on replicas. But GUIs, in particular, are difficult to unit-test. Here is where Unix-foo comes in; what that tells me is that you make your GUI as thin a layer as possible over something that can be scripted – an API manipulating and reporting on a model. Then, while end-to-end testing of the GUI is still difficult, at least you can script the whole chain from client-side model back to DB.

    >This is especially so because the doctrine of unit testing is not systemic end to end, but isolated testing of a particular module or function.

    I don’t think this is sufficient. GPSD has unit tests, to be sure, but our framework emphasizes end-to-end testing – from actual device logs to the same JSON the clients see.

    >a new quality is now added — testability.

    I had some things to say about that in The Art of Unix Programming, yes.

    >I suspect this is really hard to do in C

    It’s not particularly hard. It just takes more forethought and custom framework code than in a more naturally introspective language like (say) Python. See “bloody-minded persistence” in the original post.

    >a point he did not make, but which I imagine he agrees with, is that unit regression testing is especially important to plasticity in software, when the inevitable change comes along

    Absolutely. Regression tests give you freedom to change things without worrying that you’ll cause undetected breakage.

  41. What’s the least amount of experience it takes for a programmer to take testing programs seriously? The typical amount of experience? Is it essential for programmers to see the results of not doing careful testing for themselves, or can the habit of testing be trained in?

  42. @Nancy – my particular set of experiences have led me to believe that it isn’t so much *programming* experience that encourages people to take testing seriously, but rather *artisan* experience.

    It seems to be about ‘pride’ in your code. When I deliver something with a defect, it stings. Other developers rather robotically grind out some code that satisfies some unit testing/spec and feel content that their job is done; when the defects are reported, they just put them on a pile and grind through those….without seeming to care that they have generated workload and frustration for others – adding friction to both the development environment and business itself.

    Pobody’s nerfect, of course, but the ‘experienced’ developers I know build software that they believe they are capable of making confident statements about, regarding its correctness and stability. The more formal the development process, the greater this confidence can be. But when the inevitable happens, and a problem emerges, they know right where to hunt down the problem and they fix it toot sweet. A combination of pride in one’s craft., and a genuine sense of caring towards the larger development/business team.

  43. @esr:
    >I don’t think [unit testing] is sufficient. GPSD has unit tests, to be sure, but our framework emphasizes end-to-end testing – from actual device logs to the same JSON the clients see.

    Totally agree. Unit testing is one piece of the puzzle, but a really important one. Obviously integration or end to end testing is extremely important too, as is user testing. And there are other testing strategies too like load testing to determine the software’s maximum capacity, and stress testing to determine its characteristics during overload that are often less important.

    However, unit testing is all on us programmers, and is heavily neglected. I am a huge advocate because when you have high level code coverage in your unit tests it definitely changes they way you think about your code. It gives you a level of confidence to make solid changes rather than fiddling with your code.

    BTW, you are completely right, when you are trying to test an FSA, especially if it is implemented using an LR table driven algo rather than a recursive descent parser, it is really hard to ensure full coverage of all the states of the parse. That is definitely one nasty wrinkle in your code that most people don’t have to deal with. It is one of those weird corners where code coverage in terms of LoC doesn’t mean as much.

  44. Eric, re: “platform-sensitive bit-bashing”, you can get rid of a lot of that if you really want to. Here’s end_write() from driver_zodiac.c, rewritten so that it doesn’t care about endianness:

    static int end_write(int fd, void *d, int len)
    {
        char buf[BUFSIZ];
        char *p = buf;
        u_int16_t *data = (u_int16_t *)d;
        size_t n = (size_t) len;
    
        while (n > 0) {
          *p++ = (char)((*data >> 8) & 0xFF);
          *p++ = (char)(*data & 0xFF);
          data += 1;
          n -= 2;
        }
        return write(fd, buf, len);
    }
    

    See Rob Pike’s “The byte order fallacy” rant for further inspiration.

  45. >Eric, re: “platform-sensitive bit-bashing”, you can get rid of a lot of that if you really want to.

    Shit. You know, I hadn’t looked at driver_zodiac.c in so long that I had clean forgot it had an #ifdef WORDS_BIGENDIAN in it. I thought the only one in the C files was the only other one, in driver_rtcm2.c. Here is how that ought to look in GPSD’s house style (untested, I have a pre-release feature freeze on code not related to NMEA2000):

    static int end_write(int fd, void *d, int len)
    {
        char buf[BUFSIZ];
        u_int16_t *data = (u_int16_t *)d;
        size_t n = 0;
    
        while (n < len) {
            putbe16(buf, n*2, data[n]);
        }
        return write(fd, buf, len);
    }
    

    This is much cleaner. putbe16() is from bits.h, which is a set of endianness-independent macros written in perfect conformity to Rob Pike’s advice years before he actually issued it :-).

    I’ve actually thought seriously about issuing bits.h/bits.c as a separate mini-project. There are an astonishing number of bad, platform-sensitive implementations of the same idea out there, and they should all die.

  46. People who snarf bytes directly off the wire and stuff them into blocks of memory which are then cast as pointer-to-int or pointer-to-struct are doing it wrong. “But that’s what #pragma pack is for,” you say. Well, yeah. #pragma pack was introduced to be compatible with compilers for Microsoft environments, where this kind of madness is industry best practice. (Now you know why the .doc and .xls formats are so inscrutable; historically they were dumps of structs (perhaps with swizzled pointers) to disk.)

  47. >“platform-sensitive bit-bashing”

    I wasn’t clear enough what I meant by that, I guess.

    Rob Pike is right; It’s not hard to make endianness issues go away, you just have to be thorough about it (and Daniel Franke pointed at a place where I wasn’t quite thorough enough). But there are other platform variances not so easily hidden. One is 32-bit vs. 64-bit words. Another is signed vs. unsigned characters – that one actually bit GPSD in the ass recently, causing a very odd port bug that manifested as miscomputed altitudes (and only that!) on some platforms.

  48. >“But that’s what #pragma pack is for,”

    I’ve been guilty of this one. But only once, and fixing it is on the to-do list for GPSD 3.7. There’s a #pragma pack(1) in my RTCM2 driver, supporting some epic-scale type punning involving overlaying a huge honkin’ struct full of bitfields on a raw-data buffer. Actually, two different big honkin’ structs depending on whether your machine is big-endian or little-endian.

    Gaze upon it in wonder and horror, and marvel that it actually works flawlessly on half-a-dozen processors (i386 and all descendants, multiple ARM variants, MIPS, Itanium, and others I don’t recall). Fails only on System 390 mainframes and SPARC, where – apparently – #pragma pack isn’t implemented.

    What can I say except mea maxima culpa and I hadn’t written bits.c yet? This code probably ought to be shot, stuffed, and mounted in a glass case with a sign that says “NEVER, EVER DO THIS!”

  49. A few years ago, a group of “software” engineers from Silicon Valley came to my town and started a company that aimed to build a “green solution to the world’s energy needs.” The process they were intent on building consisted of a sophisticated petrochemical plant. Unlike video games, these plants really can blow up if not designed, operated, and maintained properly. Think Bhopal, if you have any doubts.

    @TomA – I agree completely. I’ve been in industrial control for 15 years, and the worst thing I ever heard was a software guy saying “why do you muck around with these PLCs? Just put in a Linux box and we’ll program the control sequence.”

    After I mopped up the spilled coffee, I had to explain to him that ladder logic is designed to be extremely high level to minimise race conditions, buffer overflows, etc – and the underlying hardware is required to be very reliable.

    On the other hand, I’ve come across some shocking PLC code in my time.

  50. I’ve actually thought seriously about issuing bits.h/bits.c as a separate mini-project.

    Do it. Anything that can help people write portable code as cleanly as possible is a Good Thing™.

  51. I’ve been in industrial control for 15 years, and the worst thing I ever heard was a software guy saying “why do you muck around with these PLCs? Just put in a Linux box and we’ll program the control sequence.”

    Madness! Hopefully he was a buck n00b? :)

    Maybe it’s because I work in robotics, and there’s just so much data going into and out of various devices that it would be impossible for the main CPU to handle all that I/O and the high-level logic, even assuming a hard-RT OS and 100% software reliability (neither of which, of course, we actually have). But I’ve NEVER heard that from a colleague; and if I did… well, at minimum they’d have to put a dollar in the “Just Jar”. :)

    That said, from time to time I still like to tell the Tale of the Breakfast Food Cooker as prophylaxis against any latent grand architecture-astronautics ideas.

  52. @Jeff Read
    “That said, from time to time I still like to tell the Tale of the Breakfast Food Cooker as prophylaxis against any latent grand architecture-astronautics ideas.”

    Good story. But most readers seem to miss the difference in underlying problem spaces.

    An (electrical) engineer takes raw stuff that does nothing useful, and combines it to do something useful. The less stuff is used, the better.

    A Computer Scientist takes a Universal Turing Complete device, and has to cut it down to make is usable. Because, a raw UT machine can potentially do everything, but is next to useless.

    So, an engineer creates value by adding capabilities. A CS creates value by taking away capabilities. It is logical for an engineer to limit the number of capabilities to add, as it is logical for the CS to limit the number of capabilities to remove.

  53. @Jeff Read

    He was a programmer working on the SCADA database, so the nuances of real-time control weren’t something he was familiar with. I re-educated him quite quickly.

    Robotics? A very unforgiving field. One mistake and bang, there goes the machine, or possibly someone’s head.

    That being said, there is a lot that both engineers and CS types can learn from each other. To start with, being able to write common function blocks that could be instanced many times in a a program revolutionised the Programmable Logic Controller field – in the early 1990s. Before then we just edited each and every instance of the same function after a change approval.

  54. As for testing for code correctness – in the industrial control field, forget it. If the program editor accepts it, that’s it. You have no other tools.Even unit testing is hard, how do you simulate the external inputs? Ok, the language is simpler, but the potential downside is pretty damned bad. Which is why I think engineers veer towards simple or brute force solutions, we’re just too scared of the consequences of elegance.

    That’s not to say we don’t take risks sometimes – I’ve force-commissioned a plastics plant before where we were upgrading from an obsolescent controller to a newer model. There was a conversion/upgrade path for the code, but the new controller didn’t interpret it quite the same and caused *heaps* of problems. I had two choices – rewrite it entirely, and probably introduce just as many bugs, or run the plant over and over, correcting code live when it stalled, and sometimes having to take a two hour break while they cleaned out the mixer after a screwup. We got it done, but my blood pressure was never the same :)

  55. @LS on Monday, May 14 2012 at 10:17 am
    Just ’cause your version of BASIC requires an end statement, doesn’t mean mine does. (I debated whether or not to include it, and decided to leave it off. It makes the code slightly more portable, yes. But, I’m not considering not having the ‘END’ statement a defect.)

  56. Just ’cause your version of BASIC requires an end statement, doesn’t mean mine does.

    To make things worse, I seem to recall that in most BASICs, END had an operational semantics — i.e., stop program run; while in a few, it was used simply as an end of program marker; whereas STOP was used to terminate the program run.

    I could be misremembering things horribly; I left BASIC behind when I was a kid and haven’t looked back.

  57. “…whereas STOP was used to terminate the program run.”

    Actually, in days of yore, you would use STOP for debugging. After STOP, you could examine variables, change them if you wanted to, then type ‘CONTINUE’ and execution would resume.
    The best one I remember was a BASIC for the MC6800 micro, that would let you execute:

    100 NEW

    with suicidal results.

  58. @esr

    But GUIs, in particular, are difficult to unit-test. Here is where Unix-foo comes in; what that tells me is that you make your GUI as thin a layer as possible over something that can be scripted – an API manipulating and reporting on a model. Then, while end-to-end testing of the GUI is still difficult, at least you can script the whole chain from client-side model back to DB.

    While I agree in general with making the GUI a thin layer on top of an API or protocol — for example, like the various frontends to cdrecord or the way the GIMP is very modularized — automated end-to-end testing of GUIs is not impossible and is, in fact, somewhat easy to do given a combination of Xvnc and Xnee. Using Xvnc means your Xnee scripts can run unattended on various pieces of hardware without worrying about variances in screen resolution.

    I have actually created such automated tests for pay.

  59. @Morgan
    > automated end-to-end testing of GUIs is not impossible and is, in fact, somewhat easy to do given a combination of…

    There are few reasons why GUIs are hard to test, notwithstanding ESR’s recommendation to make them a thin layer (though I’m not 100% sure what that means.)

    1. GUIs are difficult to unit test (that is test just that part in total isolation controlling the inputs and outputs directly) because of their nature.

    2. GUI test are very brittle — GUIs change a lot, in simple subtle ways, and so GUI tests are a maintenance headache.

    3. GUI test results are very hard to determine — it is not easy to determine in the input really did change the output in the appropriate way, because the result is often in a convoluted tree of controls, or might even be just bitmapped. Add in different screen configurations and you get a lot of challenges.

    4. Many times the tools suck, though that is less so today (for example Selenium is an excellent tool for web testing.)

    I’m sure there are more problems, but those are the ones that gag my craw. I too have written them for pay. It is easy to do badly, it is tough to do well.

  60. @Jessica Boxer
    >There are few reasons why GUIs are hard to test, notwithstanding ESR’s recommendation to
    > make them a thin layer (though I’m not 100% sure what that means.)
    That means give your program a command-line interactive prompt, then have your GUI emit commands for it. They may or may not be compiled into the same executable.

  61. Wow lots of food for thought here.

    I’m strongly in favor of (usually REST/JSON) API-izing everything possible for a bunchaton of reasons, testing being just one of them. And I certainly take on board the build a test harness then build the code school of thought.

    One thing I note is not discussed in the SCADA/PLC bit of the comment thread (and it applies elsewhere too) is security and access control. Particularly with API code you need to ensure that the thing/person calling your API is allowed to do so to do the task it is asking your API to do. A lot of SCADA code has hard coded passwords and/or well known defaults that no one ever changes. And a lot of that code (and %deity% knows the PLC writers are far from alone – Microsoft only really got this more or less right in Win 7) also tends to give everyone equal superuser access when actually you ought to be more granular and give access to different classes of people/devices only on what they need. So in the SCADA world that means you allow lots of things read access to PLC statistics and very few things (checked by user, password and IP address/subnet) the rights to make changes.

    These days you have to assume that everyone and his smartphone is going to want to access your API from places as diverse as down the hall to half way around the world, and you also have to assume that someone (either deliberately or accidentally) is going to allow a smarthphone in Timbucktu access to your API and whatever it is driving. Just as testing has to be built in before/during development, so does security and you have to do security first because the tests have to include testing the security. Bolting security and access control onto something later is a right mess and leads to more bugs because the security code will change the interactions and those changes tend to cascade through the entire project. (See MS windows development from ~1992-2010 as an object lesson in this).

  62. I normally think of UIs as thin control surfaces that should be independent of the underlying core – typically some combination of streaming/ReSTful interfaces. If you can’t rip a UI off and replace it with another written in a different language, perhaps running on a remote machine, that gives me great concern that “Yore Doin’ It Rong”

  63. Bah, I need an edit button. If the GUI and program are well enough separated, you can regression-test the output of the GUI itself.

  64. Life lesson number one for hackers about software in the real world: from the end user perspective, the interface is the product. I’ve said it before, but I mean it: presentation is everything. The very first task that you should do is interface design: lay out in detail what you expect the end user to achieve with your product and how they may do it. That determines the structure and capability of the back end code that does all the work. Interface drives all other aspects of software development.

    Secondly, unless your UI is presented through an IBM 3270 terminal or its modern equivalent, a static Web page with no dynamic JavaScript whatsoever, then it’s going to have all sorts of weird dependencies where bugs may crop up; for example, you may be entering line items in rows C through E, but they are not immediately added to the total at the bottom. Or they might be in the computer’s _memory_, but this is not visible on the screen until the user pages left, then right. That sort of thing. These bugs have to be tested and corrected, and they may be more visible, hence more annoying to the user and more critical to fix, than logic errors deep in the program.

    So in many, if not most, cases the UI is a large, complex, critical part of a software product, full of corner cases and difficult to test. It is certainly not a “thin layer” over functionality that was predetermined before the GUI was added. Jessica is right.

  65. @Max E.
    > That means give your program a command-line interactive prompt, then have your GUI emit commands for it. They may or may not be compiled into the same executable.

    That’s silly, there is lots of complex functionality in a gui that is different from the command line equivalent. Really, can you fully test a complex web page using Lynx?

  66. @FrancisT
    >I’m strongly in favor of (usually REST/JSON) API-izing everything possible for a bunchaton of reasons,

    I am too, but that doesn’t mean you don’t have to test the javascript that renders the results, and interacts with the user to create the calls, right?

    You are right that it provides a good separation of functionality, and RESTful, Web Services, or whatever are a nice separation for testing, though not unit testing, since it is end to end. But we still have to test the front end sizzle too. And worst of all, in my experience, that sizzle is the most brittle, and most prone to change. God god, on the web, it is in Javascript, no doubt the worst programming language currently in common use.

  67. >from the end user perspective, the interface is the product.

    Mindless, idiotic dogma only credited only by people who have drunk way too much Apple kool-aid.

    Compare: “From a driver’s prespective, the steering wheel and pedals are the car.” “From a homeowner’s perspective, the paint, switchplates, and doorknobs are the house.” “From a carpenter’s perspective, the shape of the handle is the hammer.”

  68. @esr
    >“From a driver’s prespective, the steering wheel and pedals are the car.

    Sorry, that is not a fair comparison. All the other parts of the car are readily visible, and the driver interacts with them all the time. I think “the user interface is the program” is a little simplistic, but the user interface is the majority of the software from the point of view of the non technical user.

  69. >All the other parts of the car are readily visible

    I defy you to show me a daily interaction with the crankshaft. Critical part, behavior invisible, but if it fails you’re inert.

    If you’re willing to extend the causal chain of “interacts with” far enough to falsify the above, the idiotic dogma becomes true, but only vacuously so because everything is interface.

  70. His point that “It is certainly not a “thin layer” over functionality that was predetermined before the GUI was added.” still stands. If the interface that works well for the user isn’t one that maps easily to how the underlying software actually works, then it needs to be more complicated. And declaring the complicated stuff that maps the thin UI’s semantics to the underlying software’s semantics without any abstraction leakage (yeah, right) to not be part of the UI makes _your_ claim only vacuously true.

  71. In other words, imagine a car that has a clever new efficient steering/engine/brake system that happens to require you turn the column in the opposite direction, and the pedal inputs have opposite semantics (brakes on and throttle open until you push down – oh and the brake is on the right).

    Now your “UI layer” just got fatter because you’ve got to add extra linkages to let the steering wheel turn the normal way and the pedals work the normal way on the correct side. If you don’t do it, if you stick to a “thin” UI layer, nobody will buy your car.

  72. @Jeff Read
    “from the end user perspective, the interface is the product.”

    SMS texting, as well as as teletext (Ceefax) are product with interfaces so horrible, they could make grown people cry. But these interfaces were never ever the product.

    From a writers perspective, LaTeX is the printed article. From a statisticians perspective, R is the values and graphs it delivers.

    As a homeowner, the interface of the paint cans or drill are completely irrelevant. I want a hole in the wall and a cover of paint over it, any way it can be done.

  73. > Compare: “From a driver’s prespective, the steering wheel and pedals are the car.”

    This one’s true. Most people drive their cars with no other knowledge than how to work the steering wheel and pedals. They do care about the look-and-feel of the car too. They manage to transport themselves without even knowing there are such things as combustion, hydraulics, transmission, the ratio of sprung weight to unsprung weight, …

    > “From a homeowner’s perspective, the paint, switchplates, and doorknobs are the house.”

    This one works too.
    People don’t know how doorknobs work, have no idea what the mechanism inside the door looks like, and they only care that the doorknobs open doors, and look good.
    People don’t want to know the chemistry of paint and the techniques involved in painting a wall, they want a nicely painted wall in a color they like.
    They don’t know the first thing about electricity, they just expect the lights to come on when they throw a switch or hit a button.

    > software in the real world: from the end user perspective, the interface is the product
    to wit: I’ve seen helpdesk calls from users saying things like “the internet is broken” when the default home page of their browser returned a 404.

    Also:
    http://www.joelonsoftware.com/articles/fog0000000356.html

  74. @esr
    >I defy you to show me a daily interaction with the crankshaft.

    But that is the whole point Jeff was making, isn’t it? Most (or perhaps I am over confident in my quantities, lets at least say “many”) drivers know very little about the crankshaft.

    I defy you to show me many users of GPSD who know squat about the complex FSA in the heart of it. To them the TCP port, and the USB plug are GPSD. It isn’t correct, but it is the perception, and the plain fact is that if your TCP port output is crappy, then no matter how great your FSA is nobody is going to use it.

    I don’t want to speak on behalf of Jeff, he is plenty loquacious on his own, but I believe he was talking about perception not reality.

    Personally, I think Jeff overstated the case, but I will make the claim that for most people, the quality of a piece of software is mostly judged by the quality of the user interface.

  75. @Winter “I want a hole in the wall[...] any way it can be done.”

    If the interface doesn’t matter, then why isn’t your drill the Hole Hawg? Why do other drills (or drills with other types of interface) even exist?

  76. I don’t want to speak on behalf of Jeff, he is plenty loquacious on his own, but I believe he was talking about perception not reality.

    Oh, you’ve really gotten what I’m talking about, and restated it nicely.

    The problem with the analogies to other objects is, as you said, one of visibility. Cars go, hammers hit nails. Most of what computers do is invisible to end users, except inasmuch as the computer shows the user the results of its actions. When the UI is both a means of control and the sole way to get feedback about what the program is doing, then extensionally it is the program, and its presentation had better make commanding the program and reading its results trivially easy.

    Yes, I know. Some programs create files, etc. Which just means the UI to your filesystem has to be really good.

    Correspondingly, where computers do have some effect external to the machine itself — robots, industrial control, automotive applications, etc. — the need for a shiny Apple-like UI goes down considerably.

    As for Apple koolaid, well, that company is now the biggest in the world, and has been in a growth pattern for more than a decade. They’re doing something right.

  77. > Personally, I think Jeff overstated the case, but I will make the claim that for most people, the quality of a piece of software is mostly judged by the quality of the user interface.

    I myself am probably guilty of this, and I’m the kind person who bought a working engine just so I could take it apart and see with my own eyes how all the internals went together and interacted. That, and I wanted to rebuild it faster, stronger, and better with a bigger turbo, but that’s kinda beside the point. My Linux box at home is probably far more versatile and customizable, but my assigned Windows 7 box at work certainly looks a lot prettier.

    You could offer a couple looking to buy a car an older VW diesel that gets 50mpg on the highway without trying and go 500,000 miles before the engine started to show any wear. They’d still go buy something else because the other car has an automatic that does the thinking of when to change gears for them, dual climate control that lets the driver and passenger respectively crank the A/C and heater without bothering each other, and a cool stereo that plays their MP3s nice and loud. How fuel efficient and well built the engine is would be a secondary consideration, if that. And even in another scenario with the buyer’s primary consideration being the fuel efficient engine, given the option between cars with two cars with the same or similar mileage, he’ll still go for the one with the stereo and dual climate control — the nicer interface.

  78. @Random832
    The Hawg is too expensive and not for sale overhere. But I bought a cheap no nonsense hammer drill instead. It has to do one thing only: make holes in concrete. And that is what it does.

  79. That was so far from being my point that I don’t even know where to begin.

  80. > That was so far from being my point that I don’t even know where to begin.

    Was it? I think Winter agreed with your point. The hammer drill has a very simple, easy to understand interface – no electric plug.

    Or did I misunderstand you?

    Yours,
    Tom

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="" highlight="">