<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Armed and Dangerous</title>
	<atom:link href="http://esr.ibiblio.org/?feed=rss2" rel="self" type="application/rss+xml" />
	<link>http://esr.ibiblio.org</link>
	<description>Sex, software, politics, and firearms. Life&#039;s simple pleasures...</description>
	<lastBuildDate>Sun, 13 May 2012 10:44:05 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.1</generator>
		<item>
		<title>Engineering zero-defect software</title>
		<link>http://esr.ibiblio.org/?p=4340</link>
		<comments>http://esr.ibiblio.org/?p=4340#comments</comments>
		<pubDate>Sun, 13 May 2012 05:19:40 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=4340</guid>
		<description><![CDATA[I&#8217;ve been pounding on GPSD with the Coverity static analyzer&#8217;s self-build procedure for several days. It is my great pleasure to report that we have just reached zero defect reports in 72.8KLOC. Coverity says this code is clean. And because &#8230; <a href="http://esr.ibiblio.org/?p=4340">Continue reading <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve been pounding on GPSD with the Coverity static analyzer&#8217;s self-build procedure for several days.  It is my great pleasure to report that we have just reached zero defect reports in 72.8KLOC.  Coverity says this code is clean. And because I think this should be an example unto others, I shall explain how I think others can do likewise.</p>
<p><span id="more-4340"></span></p>
<p>OK, if you&#8217;re scratching your head&#8230;Coverity is a code-analysis tool &#8211; an extremely good one, probably at this moment the best in the world (though LLVM&#8217;s open-source &#8216;scan-build&#8217; is chasing it and seems likely to pass it sometime down the road), It&#8217;s proprietary and normally costs mucho bucks, but as a marketing/goodwill gesture the company allows open source projects to register with them and get remote use of an instance hosted at the company&#8217;s data center.</p>
<p>I dislike proprietary tools in general, but I also believe GPSD&#8217;s reliability is extremely important. Navigation systems are life-critical &#8211; bugs in them can kill people. Therefore I&#8217;ll take all the help I can get pushing down our error rate, and to hell with ideological purity if that gets in the way.</p>
<p>Coverity won&#8217;t find everything, of course &#8211; it&#8217;s certainly not going to rescue you from a bad choice of algorithm. But it&#8217;s very, very good at finding the sorts of lower-level mistakes that human beings are very <em>bad</em> at spotting &#8211; memory allocation errors, resource leaks, null-pointer dereferences and the like.  These are what drive bad code to crashes, catatonia, and heisenbugs.</p>
<p>Excluding false positives and places Coverity was being a bit anal-retentive without finding an actual bug, I found 13 real defects on this pass &#8211; all on rarely-used code paths, which makes sense for reason I&#8217;ll explain shortly.  That&#8217;s less than 1 defect per 5 KLOC (KLOC = 1000 logical lines of code) which is pretty good considering our last scan was in 2007. Another way to look at that data is that, even while adding large new features like AIS support and NMEA200 and re-engineering the entire reporting protocol, we&#8217;ve introduced a bit fewer than three detectable defects <em>per year</em> in the last five years.</p>
<p>Those of you who are experienced software engineers will be picking your jaws up off the floor at that statistic.  Those of you aren&#8217;t &#8211; this is at least two <em>orders of magnitude</em> better than typical.  There are probably systems architects at Fortune 500 companies who would kill their own mothers for defect rates that low. Mythically, military avionics software and the stuff they load on the Space Shuttle is supposed to be this good, except I&#8217;ve heard from insiders that rather often it isn&#8217;t.</p>
<p>So, how did we do it? On no budget and with all of three core developers, only one working anywhere even near full time?</p>
<p>You&#8217;ll be expecting me to say the power of open source, and that&#8217;s not wrong. Sunlight is the best disinfectant, many eyeballs make bugs shallow, etc. etc.  While I agree that&#8217;s next to a necessary condition for defect rates this low, it&#8217;s not sufficient.  There are very specific additional things we did &#8211; things I sometimes had to push on my senior devs about because they at times looked like unnecessary overhead or obsessive tailchasing.</p>
<p>Here&#8217;s how you engineer software for zero defects:</p>
<h2>1. Be open source.</h2>
<p>And not just because you get helpful bug reports from strangers, either, all though that does happen and can be very important. Actually, my best bug-finders are semi-regulars who don&#8217;t have commit access to the code but keep a close eye on it anyway.  Like, there&#8217;s this Russian guy who often materializes on IRC late at night and can barely make himself understood in English, but his patches speak clearly and loudly.</p>
<p>But almost as importantly, being open source plugs you into things like the Debian porterboxes.  A couple of weeks ago I spent several days chasing down port failures that I thought might indicate fragile or buggy spots in the code.  It was hugely helpful that I could ssh into all manner of odd machines running Linux, including a System 390 mainframe, and run my same test suite on all of them to spot problems due to endianness or word-size or signed-char-vs.-unsigned-char differences.</p>
<p>Closed-source shops, in general, don&#8217;t have any equivalent of the Debian porterboxes because they can&#8217;t afford them &#8211; their support coalition isn&#8217;t broad enough. When you play with the open-source kids, you&#8217;re in the biggest gang with the best toys.</p>
<h2>Invest your time heavily in unit tests and regression tests</h2>
<p>GPSD has around 90 unit tests and regression tests, including sample device output for almost every sensor type we support. I put a lot of effort into making the tests easy and fast to run so they can be run often &#8211; and they are, almost every time executable code is modified.  This makes it actively difficult for random code changes to break our device drivers without somebody noticing right quick.  </p>
<p>Which isn&#8217;t to say those drivers can&#8217;t be wrong, just that the ways they can be wrong are constrained to be through either (a) a protocol-spec-level misunderstanding of what the driver is supposed to be doing, or (b) an implementation bug somewhere in the program&#8217;s state space that is obscure and difficult to reach.  Coverity only turned up two driver bugs &#8211; static buffer overruns in methods for changing the device&#8217;s reporting protocol and line speed that escaped notice because they can&#8217;t be checked in our test harnesses but only on a live device.</p>
<p>This is also why Coverity didn&#8217;t find defects on commonly-used code paths.  If there&#8217;d been any, the regression tests probably would have smashed them out long ago. I put in a great deal of boring, grubby, finicky work getting our test framework in shape, but it has paid off hugely.</p>
<h2>Use every fault scanner you can lay your hands on.</h2>
<p>Ever since our first Coverity scan in 2007 I&#8217;d been trying to get a repeat set up, but Coverity was unresponsive and their internal processes clearly rather a shambles until recently. But there were three other static analyzers I had been applying on a regular basis &#8211; splint, cppcheck, and scan-build.</p>
<p>Of these, splint is (a) the oldest, (b) the most effective at turning up bugs, and (c) far and away the biggest pain in the ass to use. My senior devs dislike the cryptic, cluttery magic comments you have to drop all over your source to pass hints to splint and suppress its extremely voluminous and picky output, and with some reason.  The thing is, splint checking turns up real bugs at a low but consistent rate &#8211; one or two each release cycle. </p>
<p>cppcheck is much newer and much less prone to false positives. Likewise scan-build.  But here&#8217;s what experience tells me: each of these three tools finds overlapping but <em>different</em> sets of bugs.  Coverity is, by reputation at least, capable enough that it might dominate one or more of them &#8211; but why take chances?  Best to use all four and constrain the population of undiscovered bugs into as small a fraction of the state space as we can.</p>
<p>And you can bet heavily that as new fault scanners for C/C++ code become available I&#8217;ll be jumping right on top of them.  I like it when programs find low-level bugs for me; that frees me to concentrate on the high-level ones they can&#8217;t find.</p>
<h2>Be methodical and merciless</h2>
<p>I don&#8217;t think magic or genius is required to get defect densities as low as GPSD&#8217;s.  It&#8217;s more a matter of sheer bloody-minded persistence &#8211; the willingness to do the up-front work required to apply and discipline fault scanners, write test harnesses, and automate your verification process so you can run a truly rigorous validation with the push of a button.</p>
<p>Many more projects could do this than do.  And many more projects should.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&#038;p=4340</wfw:commentRss>
		<slash:comments>59</slash:comments>
		</item>
		<item>
		<title>The &#8220;Plain Jane&#8221; timing GPS is real</title>
		<link>http://esr.ibiblio.org/?p=4335</link>
		<comments>http://esr.ibiblio.org/?p=4335#comments</comments>
		<pubDate>Fri, 11 May 2012 08:01:57 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=4335</guid>
		<description><![CDATA[The GPS with my magic modification that makes it into a 1ms-accurate time source over USB arrived here last week. And&#8230;wow. It works. Not only is it delivering 1PPS where I can see it, it&#8217;s the best GPS I&#8217;ve ever &#8230; <a href="http://esr.ibiblio.org/?p=4335">Continue reading <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>The GPS with my <a href="http://esr.ibiblio.org/?p=4281">magic modification</a> that makes it into a 1ms-accurate time source over USB arrived here last week.  And&#8230;wow.  It works.  Not only is it delivering 1PPS where I can see it, it&#8217;s the best GPS I&#8217;ve ever handled on a couple other axes as well, including superb indoor performance.  Despite the fact that it&#8217;s been sitting on my desk five feet from a window blocked by large trees, it acquired sat lock in seconds and (judging by the steadily blinking LED) doesn&#8217;t appear to have lost it even transiently at any time since.</p>
<p><span id="more-4335"></span></p>
<p>(Fun fact about that blinking LED on your GPS &#8211; that&#8217;s actually being lit up by the 1PPS pulse! Yes, the dumb flashing LED telling you your GPS has a fix is actually marking top-of-second with 50ns atomic-clock accuracy &#8211; kind of like using an F16 to deliver junk mail.)</p>
<p>I&#8217;m kind of boggled, actually. This device, my very first hardware hack, went from from mad gleam in my eye to shipping for production in <em>less than ten weeks</em>.  No, you can&#8217;t easily buy one yet, but that&#8217;ll change within a few weeks when the first U.S. retailer lands a shipment.  </p>
<p>Um, so maybe I really am <a href="http://en.wikipedia.org/wiki/Accelerando_%28novel%29">Manfred Macx</a> after all? I have spent an awful lot of time pulling people into <a href="http://www.openverse.com/~dtinker/agalmics.html">agalmic</a> positive-sum games, and the hypervelocity hack of the market I&#8217;ve just done (make a bunch of other people rich and empowered with a simple idea and some connective juice) is very much the same sort of thing Manfred does all through Charles Stross&#8217;s novel <cite>Accelerando</cite>.  The guys on the thumbgps-devel list think this is hilarious and have talked the Chinese into nicknaming the device the Macx-1. Two of them are now addressing me as &#8216;Manfred&#8217; in a ha-ha-only-serious way; I am not sure I approve of this.</p>
<p>The Chinese we&#8217;re dealing with (the company is <a href="http://www.navisys.com.tw/">Navisys</a>) seem to be enjoying all this. Of course they make agreeable noises at customers as a matter of commercial reflex, and it&#8217;s not easy to be sure through the slightly stiff Chinglish they speak, but&#8230;I think they actually <em>like</em> us.  I think they&#8217;re not used to having customers that are <em>interesting</em> and know their engineering and make jokes at the same time.  It seems to have been a fun ride for all parties involved.</p>
<p>The non-Plain-Jane concept designs that the thumbgps list was kicking around haven&#8217;t completely died as topics of discussion, but the existence of real hardware for cheap does tend to concentrate minds on it. The other company I was talking with, UniTraq, hasn&#8217;t been heard from in a couple of weeks; perhaps they lost interest after we downchecked the CP2101 USB adapter in their prototypes. </p>
<p>Dunno what the quantity-one retail price in the U.S. will be yet, but a little birdie tells me Navisys is quoting less than $30 qty 100, so make your own guess about retailer markup.  No, it&#8217;s not on the Navisys website yet, but they are taking bulk orders. Ask for the Macx-1 by name &#8211; formally it&#8217;s a revision of the GR601W, but they had to shift from a dongle to a mouse enclosure for the prototypes at least and it&#8217;s unknown to me whether the older designation will survive.  I suspect the Chinese are still thinking out how exactly to market this thing.</p>
<p>There&#8217;s an opportunity here for anyone in the retail consumer-electronics biz.  This is a great product &#8211; inexpensive, well designed, almost uniquely capable, My opinion of uBlox (the GPS chip&#8217;s vendor) has gone way, way up; this beats the snot out of the SiRF-II- and SiRf-III-based designs I&#8217;m used to even if you ignore the timing-source use.</p>
<p>It&#8217;s pretty hard to see how this project could gone better, actually.  Now it&#8217;s time for phase II, where we use a hundred or so copies of the Macx-1 to build the Cosmic Background Bufferbloat Detector and <a href="http://esr.ibiblio.org/?p=4171">fix the Internet</a>. </p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&#038;p=4335</wfw:commentRss>
		<slash:comments>58</slash:comments>
		</item>
		<item>
		<title>Some figures that make me happy.</title>
		<link>http://esr.ibiblio.org/?p=4326</link>
		<comments>http://esr.ibiblio.org/?p=4326#comments</comments>
		<pubDate>Tue, 08 May 2012 04:45:31 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=4326</guid>
		<description><![CDATA[I got emailed summaries from a Coverity scan of the repo head version of GPSD today. Analysis summary report: ------------------------ Files analyzed : 86 Total LoC input to cov-analyze : 72824 Functions analyzed : 585 Classes/structs analyzed : 128 Paths &#8230; <a href="http://esr.ibiblio.org/?p=4326">Continue reading <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>I got emailed summaries from a Coverity scan of the repo head version of GPSD today.</p>
<p><span id="more-4326"></span></p>
<pre>
Analysis summary report:
------------------------
Files analyzed                  : 86
Total LoC input to cov-analyze  : 72824
Functions analyzed              : 585
Classes/structs analyzed        : 128
Paths analyzed                  : 125590
Time taken by Coverity analysis : 00:01:22
Defect occurrences found        : 45 Total
                                   2 BAD_SIZEOF
                                   2 CHECKED_RETURN
                                   4 CONSTANT_EXPRESSION_RESULT
                                   3 DEADCODE
                                  12 FORWARD_NULL
                                   1 INFINITE_LOOP
                                   1 NEGATIVE_RETURNS
                                   1 NULL_RETURNS
                                   1 OVERRUN_DYNAMIC
                                   3 OVERRUN_STATIC
                                   6 RESOURCE_LEAK
                                   1 REVERSE_INULL
                                   7 UNINIT
                                   1 UNREACHABLE
</pre>
<p>Those of you who&#8217;ve seen Coverity scans before will know that (1) a defect rate of 1 per 1.68KLOC is pretty damned good to begin with, and (2) some of those reports are probably false positives.</p>
<p>I am <em>extremely</em> pleased, and looking forward to analyzing the detail logs.</p>
<p>UPDATE: Oops. Misplaced a decimal point on first post.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&#038;p=4326</wfw:commentRss>
		<slash:comments>16</slash:comments>
		</item>
		<item>
		<title>Falling through the cracks: why GPSD sometimes bogarts non-GPS devices</title>
		<link>http://esr.ibiblio.org/?p=4319</link>
		<comments>http://esr.ibiblio.org/?p=4319#comments</comments>
		<pubDate>Sun, 06 May 2012 04:42:47 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=4319</guid>
		<description><![CDATA[In a recent Google+ comment, H. Peter Anvin grumped about GPSD using &#8220;braindead heuristics&#8221; to determine which USB devices it should sniff as possible GPses when it gets a hotplug notification saying that one has connected. I was going to &#8230; <a href="http://esr.ibiblio.org/?p=4319">Continue reading <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>In a recent Google+ comment, H. Peter Anvin grumped about GPSD using &#8220;braindead heuristics&#8221; to determine which USB devices it should sniff as possible GPses when it gets a hotplug notification saying that one has connected.  I was going to reply in a comment there, but the explanation ran too long for that.</p>
<p>Short version: yes, GPSD will very occasionally sniff at a device that is none of its business. We&#8217;re stuck in a bad place because of deficiencies in the USB standard, But it doesn&#8217;t happen often, and all the alternative behaviors I&#8217;ve been able to imagine would be worse in very obvious ways. Detailed explanation follows.</p>
<p><span id="more-4319"></span></p>
<p>USB devices are supposed to present a &#8220;device class&#8221; number that clues in the host system what kind of thing it is and how to talk to it.  In an ideal world, there would be a &#8220;GPS&#8221; device class which every GPS would present, and GPSD would cheerfully ignore anything that didn&#8217;t present it, and all would be gladness and joy.</p>
<p>We don&#8217;t live in that world.  Look at the <a href="http://www.usb.org/developers/defined_class">list of USB device classes</a> and notice that none of them is &#8220;GPS&#8221;.  Lacking any defined GPS class, the standard tells vendors that GPSes have to sit in the stupid corners labelled &#8220;00h&#8221; and &#8220;FFh&#8221;.  Along with every other USB devices type lacking an assigned class number, including in particular serial-to-USB adapter chips.</p>
<p>You may recall from some of my previous rants that your basic bog-standard USB GPS consists of a GPS chip shipping data over TTL-level RS232 lines to a serial-to-USB adapter chip. And that&#8217;s the root of GPSD&#8217;s problem right there. GPSD will never mess with your mouse or your mass storage but here&#8217;s no way that a USB GPS is distinguishable from some random other device that happens to be lurking behind a serial-to-USB adapter &#8211; it&#8217;s classes 00h/FFh all the way down.</p>
<p>So GPSD goes to the next level and watches for the specific Vendor-ID/Product IDs of serial-to-USB adapters that usually mean aha that&#8217;s a GPS. The most common VID/PID combination is 067b:2303  which is a Prolific Logic 2303; GPSD also recognizes nearly a dozen other specific VID/PID pairs mostly corresponding to various serial-to-USB adapter chips.</p>
<p>And 99% of the time &#8220;that&#8217;s a GPS&#8221; is exactly what these VID/PID pairs do in fact mean; for end users, anyway.  The exceptions are things like Arduino prototyping boards that will only ever be hooked up by a tiny minority of geeks like&#8230;H. Peter Anvin.  Who will grumble.</p>
<p>This sucks in a minor way, but watcha gonna do?  With this behavior. GPSD has very nice autoconfiguration in almost all cases &#8211; you plug in the GPS, hotplug wakes up the daemon, it autobauds and autoconfigures, and location data magically appears on port 2947 without the user having had to hand-tweak a blessed thing. This is good!</p>
<p>Unfortunately, the only way to prevent GPSD from occasionally sniffing at a device that turns out not to be a GPS would be to disable hotplug startup and require users to manually launch GPSD when they want to run it and know what their device inventory is.  This would end the muttering from people like H. Peter Anvin, but at the cost of requiring hundreds of thousands of bewildered end-users to hand-lunch GPSD and remember details like device names, rather than having things Just Work.</p>
<p>This would not, in my view, be an acceptable tradeoff.  This is one time I&#8217;m afraid I have to tell the hard core to live with the occasional glitch; greatest good for the greatest number and all that.  Even adding a GPS class to the USB standard wouldn&#8217;t solve the problem at this point &#8211; too many GPSes already fielded, and many of the vendors are such sketchy low-margin operations that it&#8217;s not clear they&#8217;d add the production step to do the right thing even if it were available.</p>
<p>UPDATE: Several commenters had the usual Unix-programmer reaction: just add a config switch! They&#8217;re not thinking the problem through; no config switch can solve enough cases to be worth the freight.  </p>
<p>One very common suggestion whenever this comes up is a blacklist option for particular VID/PID pairs, like &#8211;no-probe=067b:2303.   The trouble with this is that there are too many different devices using so few different USB-to-serial adapters, and this option would only be useful when you have multiple class 00/FF devices using <em>different</em> VID/PID pairs. In particular it&#8217;s quite likely that your GPS and your serial-port adapter dongle will <em>both</em> be using a PL2303, so blacklisting by VID/PID will make both invisible.</p>
<p>Another common but futile suggestion is blacklisting by device path.  USB device-path assignments aren&#8217;t stable in the presence of hotplugging, enough said.</p>
<p>Not only isn&#8217;t there enough information for autoconfiguration to work perfectly, the same missing information makes it (so far) impossible to specify a config option that I judge to be than a feel-good gesture or wishful thinking.</p>
<p>Also, people looking at this problem often jump to the conclusion that the failure cases are much more frequent and dangerous than they actually are. GPSD typically is <em>not</em> sending config strings to random devices it hasn&#8217;t identified as GPSes yet; it&#8217;s just sniffing data, and only immediately after a hotplug event.  The worst case is that another class 00/FF loses a second&#8217;s worth of data, not that it gets accidentally told to self-destruct or something.</p>
<p>UPDATE2: Another safeguard, which I had forgotten writing, is that under Linux GPSD uses a sadly non-portable hack which determines if a device has been opened by another process and if so ignores it.  So the only way gpsd even <em>reads</em> from an unknown device is when to other application has claimed it.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&#038;p=4319</wfw:commentRss>
		<slash:comments>105</slash:comments>
		</item>
		<item>
		<title>Penguicon 2012 party location</title>
		<link>http://esr.ibiblio.org/?p=4315</link>
		<comments>http://esr.ibiblio.org/?p=4315#comments</comments>
		<pubDate>Fri, 27 Apr 2012 15:00:36 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=4315</guid>
		<description><![CDATA[As promised, the Penguicon 2012 Friends of Armed &#038; Dangerous party location: room 370. Begins 9PM tonight.]]></description>
			<content:encoded><![CDATA[<p>As promised, the Penguicon 2012 Friends of Armed &#038; Dangerous party location: room 370. Begins 9PM tonight.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&#038;p=4315</wfw:commentRss>
		<slash:comments>8</slash:comments>
		</item>
		<item>
		<title>The long past of C</title>
		<link>http://esr.ibiblio.org/?p=4304</link>
		<comments>http://esr.ibiblio.org/?p=4304#comments</comments>
		<pubDate>Wed, 25 Apr 2012 04:02:06 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=4304</guid>
		<description><![CDATA[Hacking on the C code of giflib after an absence of nearly two decades has been an interesting experience, a little like doing an archeological dig. And not one that could readily be had elsewhere; nowhere other than under Unix &#8230; <a href="http://esr.ibiblio.org/?p=4304">Continue reading <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>Hacking on the C code of giflib after an absence of nearly two decades has been an interesting experience, a little like doing an archeological dig.  And not one that could readily be had elsewhere; nowhere other than under Unix is code that old still genuinely useful under any but carefully sandboxed conditions. Our reward for getting enough things about API design right the first time, one might rightly say.  But what&#8217;s most interesting is what has changed, and giflib provides almost ideal conditions for noticing the changes in practice that have become second nature to me while that code has stood still.</p>
<p><span id="more-4304"></span></p>
<p>I mean, this is code so ancient and unchanged that it still had a macro to hide the difference between C compilers that have void pointers and those that don&#8217;t &#8211; it wasn&#8217;t assuming even C89, let alone C99.  I only resolved the uses of that to &#8220;void *&#8221; last night, and I&#8217;ve left the definition in the header just in case.</p>
<p>But the largest single executable-code change I found myself making was introducing C99 bools where the code had simulated them with old-style TRUE and FALSE macros. Only internally, I might add &#8211; there are a couple of pseudo-bools in the externally visible API that I haven&#8217;t touched so as not to risk breaking source compatibility.  That and counting on stdint.h to supply uint32_t were two of the big three; I&#8217;ll get to the third in a bit.</p>
<p>The fact that editing in now-standard typedefs was the largest change in executable code is a testament to the remarkable stability of C, I think. I can barely think of any other language in which bringing code across an eighteen-year gap would be anywhere near that easy, and you would not want to use any of the handful of partial exceptions for systems programming on a modern machine.</p>
<p>Something else I noticed is that there was no point at which I looked at my old code (or code from Gershon Elber, the original author of the DOS version) and had a &#8220;What the fsck were you <em>thinking</em>?&#8221; moment.  Which is good: I guess I was already a pretty good programmer 20 years ago, and the proof is that this code ended up deployed <em>everywhere</em> that displays pixels, including the phone on your hip, without my ever hearing barely a bug report. On the other hand, it&#8217;s a little disconcerting to think that I might not have learned anything about practical C in two decades&#8230;</p>
<p>There were some moments of pure amusement in the quest as well, such as my rediscovery of gif2epsn &#8211; a utility for dumping GIFs to Epson dot-matrix printers, actually using the individual print head wires as scanning pixels.  Just what the world needs in 2012.  I removed it; it&#8217;s still in the repo history in the unlikely event that anyone ever cares about that code again.</p>
<p>In general a lot of what I&#8217;m going to be doing for the upcoming 4.2 release is removing stuff.  Back then, it made sense for giflib to carry a fairly elaborate set of image viewing, image capturing, and image format conversion tools, because relatively few other codebases spoke GIF.  Of course that&#8217;s all changed now, with multiformat viewers and editors the norm.  I&#8217;m probably going to throw out several more utilities because it doesn&#8217;t make any sense to compete with the likes of ImageMagick and the GIMP in that space.  In a sense, giflib is a victim of its own success at making its format ubiquitous.</p>
<p>I&#8217;ve saved what I think is the most interesting change for last.  When I&#8217;m not looking at code this old I tend to forget what a thicket of #ifdefs we had to manage back in the day.  Cross-platform portability was <em>hard</em>, and it made our code ugly. Just the huge numbers of conditional includes of header files was bad enough; we used to be nibbled to death by a thousand little ducks like the stdargs.h/varargs.h skew. It&#8217;s easy to forget how much cleaner and lighter C code is in the era of full POSIX conformance (well, everywhere except the toxic hell-swamp that is Windows, anyway) and C99 required headers &#8211; that is, until you go looking at code as ancient as giflib and it all comes flooding back.</p>
<p>I had almost forgotten how liberating it felt four or five years back when I made a policy decision about GPSD.  Yes, we&#8217;re going to assume full POSIX/C99/SuSv2 and if that chaps your ass you can damn well <em>fix your broken toolchain</em>!  The giflib code is much older and more encrusted.  Well, it was until about 48 hours ago.  I ripped all that gunge out. The code almost doesn&#8217;t need autotools now, and if I chisel a little harder and drop the X viewer and Utah Raster Toolkit support it won&#8217;t need autotools at all.</p>
<p>That is, in itself, interesting news.  The autotools suite began life because the combinatorial explosion of feature tests and #ifdefs back in the day was too difficult to manage by hand. Over the years autoconf and friends got hairer and hairier (to the point where I got <a href="http://esr.ibiblio.org/?p=1877">utterly fed up with it</a>), but at the same time increasingly good standards conformance in C and Unix implementations attacked the problem from the other end.  So, during the last couple of days, I&#8217;ve found that the sort of platform #ifdefs that used to be autotools&#8217;s raison-d&#8217;etre can all be yanked out &#8211; what&#8217;s left is feature switches, and only two of those.</p>
<p>That whole standardization thing&#8230;actually worked.  Though because it snuck up on us by stages, old farts like me have half- to three-quarters forgotten the problems it was solving, and younger programmers barely ever saw those problems to begin with.  This is called &#8220;progress&#8221;. </p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&#038;p=4304</wfw:commentRss>
		<slash:comments>147</slash:comments>
		</item>
		<item>
		<title>Penguicon party 2012!</title>
		<link>http://esr.ibiblio.org/?p=4306</link>
		<comments>http://esr.ibiblio.org/?p=4306#comments</comments>
		<pubDate>Tue, 24 Apr 2012 23:02:40 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=4306</guid>
		<description><![CDATA[Reminder: The Armed &#038; Dangerous party at Penguicon 2012 is now barely more than 72 hours away. Begins 9:00PM Friday night, room number to be announced on this blog and probably tweeted too. All readers of this blog will be &#8230; <a href="http://esr.ibiblio.org/?p=4306">Continue reading <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>Reminder: The Armed &#038; Dangerous party at <a href="http://www.penguicon.org/CMS/">Penguicon 2012</a> is now barely more than 72 hours away.  Begins 9:00PM Friday night, room number to be announced on this blog and probably tweeted too.</p>
<p>All readers of this blog will be welcome.  See the real faces behind the Gravatar icons!  Pursue discussion threads in person!  Refrain from actual physical violence!  Wheeee&#8230;.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&#038;p=4306</wfw:commentRss>
		<slash:comments>6</slash:comments>
		</item>
		<item>
		<title>giflib: everything old is new again</title>
		<link>http://esr.ibiblio.org/?p=4298</link>
		<comments>http://esr.ibiblio.org/?p=4298#comments</comments>
		<pubDate>Sat, 21 Apr 2012 16:37:23 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=4298</guid>
		<description><![CDATA[In 1994 I handed off the maintainership of giflib, the open-source library used by pretty much everything in the universe that displays images for the single most widely used icon and image format on the World Wide Web, because patent &#8230; <a href="http://esr.ibiblio.org/?p=4298">Continue reading <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>In 1994 I handed off the maintainership of giflib, the open-source library used by pretty much everything in the universe that displays images for the single most widely used icon and image format on the World Wide Web, because patent issues made it unwise for the project to be run by someone in the U.S.  Now, eighteen years later, Toshio Kuratomi (the hacker who took it over then) has asked me to resume the lead. I have accepted his request.</p>
<p><span id="more-4298"></span></p>
<p>I don&#8217;t expect this to involve a lot of work.  The code is very mature and stable; I left it in good shape and it&#8217;s coming back to me in good shape. I asked to rejoin because I thought it would be a good idea to run some auditing tools on the code to check it for correctness, and polish it into more modern C if that&#8217;s needed.  Major changes seem highly unlikely.</p>
<p>Toshio has lost interest in the project, and was looking for someone to hand it to. Eighteen years is a long time, much longer than I myself spent on the project. He&#8217;s owed everyone&#8217;s thanks for his responsible stewardship of the code. </p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&#038;p=4298</wfw:commentRss>
		<slash:comments>54</slash:comments>
		</item>
		<item>
		<title>Open source warfare != open source software</title>
		<link>http://esr.ibiblio.org/?p=4289</link>
		<comments>http://esr.ibiblio.org/?p=4289#comments</comments>
		<pubDate>Sun, 15 Apr 2012 18:22:37 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=4289</guid>
		<description><![CDATA[One of my commenters brought up John Robb, a former SpecOps pilot who has made a name for himself as a counter-terror theorist by writing about &#8220;open-source warfare&#8221;. Mutual acquaintances confirm what Robb&#8217;s own writings suggest, which is that his &#8230; <a href="http://esr.ibiblio.org/?p=4289">Continue reading <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>One of my commenters brought up John Robb, a former SpecOps pilot who has made a name for himself as a counter-terror theorist by writing about &#8220;open-source warfare&#8221;.  Mutual acquaintances confirm what Robb&#8217;s own writings suggest, which is that his notions of open-source warfare are heavily influenced by what I have called the bazaar model of software development.</p>
<p>When I learned this in 2004 I attempted to begin a conversation with John Robb by email.  The remainder of this post is my email to him &#8211; to which, for whatever reason, he never replied. I have edited the one link reference into a live link.</p>
<p><span id="more-4289"></span></p>
<p>A friend pointed me at <a href="http://globalguerrillas.typepad.com/globalguerrillas/2004/09/bazaar_dynamics.html">THE BAZAAR&#8217;S OPEN SOURCE PLATFORM</a>.  I had been previously aware of the application of social network theory to counterinsurgency and half-expecting my work to show up in that conversation at some point, but this is the first time I actually know of it happening.</p>
<p>You analogy between terrorist networks and the open-source community is thought-provoking but, I think, flawed.  The flaws are good news, as they offer guidance towards ways to disrupt and hinder terror networks.</p>
<p>There are at least four traits that make the the network of open-source hackers structurally different from the terror network.</p>
<p>1. Visibility is safe</p>
<p>The first difference is that the hacker network can be entirely visible without risk, but the terror network must remain almost entirely invisible (except at the edges, where it recruits through deniable cut-outs).</p>
<p>Thus, communication between terrorists is much riskier than it is between hackers.  (There have already been some well-publicized incidents in which tapped cell-phone conversations led to portions of the terror network being rolled up.)  This matters, because it means that every attempt at coordination has to be traded off against the probability that it will result in exposure.  </p>
<p>You have already noticed that this tradeoff implies a maximum feasible network size.  It also implies a minimum feasible action-reaction loop; the riskier communication is, the longer coordination at an acceptable risk level will take.  (Thus Al-Quaeda&#8217;s observed pattern of long latency periods between attacks.)</p>
<p>2. Outcomes are easy to measure</p>
<p>Second, success is easier to measure for hackers than for terrorists. A program either runs and gives the expected output, or it doesn&#8217;t. Of course there are important kinds of programs for which you cannot predict the output, but it is usually possible to check that output for correctness by various means and be confident that you know whether or not it meets your objectives.</p>
<p>Terrorists have more difficulty measuring outcomes.  Let&#8217;s take the Chechen separatists as an example: presuming their outcome is to break the Russians&#8217; will to fight in Chechnya, how are they to know whether the massacre at Beslan succeeded or not?  </p>
<p>3. The cost of failure is low</p>
<p>There is very little downside risk in what hackers do.  If aparticular way of writing a program fails, you throw it away and write a new one.  Failure can be sad for individuals or project groups but does not threaten the network as a whole.  </p>
<p>The terror network, on the other hand, can be badly damaged by the blowback from its actions.  In the aftermath of the 9/11 attack, Al-Qaeda lost its base camps in Afghanistan, and state sponsorship of terrorism became covert rather than overt.</p>
<p>No analogous loss is really imaginable for hackers.  They are protected partly by the fact that there are large demand sinks for what they do in the aboveground economy &#8212; our equivalent of state sponsorship is the Fortune 1000.</p>
<p>4. Attack methods are perfectly transmissible</p>
<p>When hackers successfully attack a problem, they produce an algorithm that can be cheaply replicated anywhere.  Terrorists, on the other hand, rely on skills that are difficult to replicate (such as bomb-making) and materiel that isn&#8217;t easy to get (consider the relative cost of a personal computer versus an RPG).</p>
<p>The bazaar model will only work for terrorists insofar as they can suppress these differencies &#8212; e.g., be safely visible, measure outcomes, control the cost of failure, and transmit attack methods.</p>
<p>It follows that counterterror strategy must be aimed at amplifying  these differences.  Here&#8217;s how we can do that:</p>
<p>1. Make it more dangerous for terrorists to be visible.</p>
<p>Terrorists can afford to be visible only where either (a) no local authority can suppress them, or (b) they are sponsored by the strongest local authority.  (It is immaterial whether the local authority is a nation-state; this analysis applies equally to Iran, pre-liberation Iraq and Somalia.)</p>
<p>Thus, raising the perceived risk from sponsoring terrorism will force terrorists to operate undercover, making their network less like a bazaar in both communication richness and action/reaction time.</p>
<p>2. Make it more difficult for terrorists to measure outcomes</p>
<p>The most effective step we could take towards this is probably for responsible news media to voluntarily stop covering individual terrorist attacks.  Note that this would not be the same as denying or covering up the phenomenon; monthly aggregate statistics on terrorist attacks, for example, would suffice for purposes such as risk evaluation by commercial travelers.</p>
<p>Unfortunately, responsibility by the news media seems rather unlikely.</p>
<p>3. Make the cost of failure high</p>
<p>This can be best achieved by the traditional method of giving no quarter &#8212; killing terrorists swiftly and without mercy whenever they present targets by mounting an operation. </p>
<p>4. Make it more difficult to transmit attack methods.</p>
<p>This implies that disrupting terrorist training facilities should be a priority in counter-terror.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&#038;p=4289</wfw:commentRss>
		<slash:comments>304</slash:comments>
		</item>
		<item>
		<title>Making simple connections</title>
		<link>http://esr.ibiblio.org/?p=4281</link>
		<comments>http://esr.ibiblio.org/?p=4281#comments</comments>
		<pubDate>Thu, 12 Apr 2012 21:52:22 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=4281</guid>
		<description><![CDATA[About six weeks ago I asked How would you like to help fix the Internet? It was an open invitation to help develop a cheap millisecond-precision time source for instrumented routers, so we can do delay tomography on the Internet &#8230; <a href="http://esr.ibiblio.org/?p=4281">Continue reading <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>About six weeks ago I asked <a href="http://esr.ibiblio.org/?p=4171">How would you like to help fix the Internet?</a>  It was an open invitation to help develop a cheap millisecond-precision time source for instrumented routers, so we can do delay tomography on the Internet and measure the <a href="http://www.bufferbloat.net/">bufferbloat</a> problem.</p>
<p>The discussion thread on that post was lively, but eventually moved to a <a href="https://lists.bufferbloat.net/listinfo/thumbgps-devel">thumbgps project mailing list</a> out of public view. I&#8217;m going to summarize what has gone on here because parts of it are very interesting in a &#8220;Wow&#8230;it really is the 21st century!&#8221; sort of way, and illustrate that there can be a lot of power in making simple connections.</p>
<p><span id="more-4281"></span></p>
<p>My call went out on 1 March. Dave T&auml;ht launched the mailing list on 10 March.  Early discussion consisted mainly of  a lot of kicking around of concept designs for time hardware. While this was going on, I was holding on to the idea of a simple mod to one of the existing cheap-ass GPS designs, just wiring the 1PPS pin from the GPS to DCD or RI on the serial-to-USB adapter chip and getting PPS with at worst one USB polling interval&#8217;s worth of jitter.  There&#8217;s one simple connection for you</p>
<p>On 15 Mar I discovered alibaba.com. This is a site that make connections in mainland China through which you can chat in real time with trade reps from Chinese electronics fabricators. Turns out these guys are constantly looking for OEM/ODM business, and I was able to interest two different companies that already make cheap GPSes for export &#8211; UniTraq and NaviSys.  </p>
<p>During the following week I exchanged email with these companies describing in detail the required single-trace hardware mod, explaining exactly how and why I believe it would work, and what the potential market is. I explained all the relevant obscure technicalia &#8211; 1PPS, handshake input pins on USB-to-serial adaptors, USB notification events, effects of polling on latency.  </p>
<p>It helped that on 16 Mar one of the thumbgps guys breadboarded a test setup and reported measured jitter of 300&mu;s with 1PPS from a Garmin 18 feeding a common serial-to-USB chip, the FTDI SIO.  OK, yes, you can in fact crowdsource some of the critical test engineering on a project like this. </p>
<p>And the people in Shenzhen went for it &#8211; started modding prototypes and reporting progress. </p>
<p>Now I want to pause for a moment so you can really take this in.  In effect, I became the lead designer on a new electronics product by email.  Just me.  No corporate-backing, no million-dollar development budget, one guy saying &#8220;Hey, if you connect this to that, cool things will happen!&#8221; &#8211; negotiating directly with people on the other side of the planet who&#8217;ll never meet me face to face.  And it&#8217;s more interesting because I&#8217;m not famous to these guys and not even a hardware geek.</p>
<p>All that SF/future-shock speculation about disaggregated manufacturing, contract and trust networks replacing corporations, Coasian effects disappearing as communications get cheaper? It&#8217;s here.  Now.  I&#8217;m living it. I&#8217;m <em>using</em> it.</p>
<p>So, the current state of play is that both companies are working on minor technical glitches.  One outfit used a USB-to-serial adaptor, the CP2101, that turned out to be unusable for our purposes when I tested the engineering samples.  The chip vendor persistently refused to release enough interfacing info to allow the open-source Linux driver to wait on a change in the state of an emulated handshake line. Now they&#8217;ve effectively EOLed the chip.  We&#8217;re going to have to move to something like a PL2303.</p>
<p>The other design, which does use a PL2303, has been tested with GPSD in China and works &#8211; they emailed me the test logs.  But they did it with a patch wire, and there&#8217;s some internal connector issue they have to work out for the production design.</p>
<p>These are solvable problems.  The important point is that (a) the basic concept of adding one PCB trace to pass out 1PPS over USB has been shown to work to sub-1ms accuracy, and (b) the design and communication process is working.  Nothing but a little elbow grease and routine product engineering stands between where we are now and multiple-source availability of 1ms-accuracy time sources for about $30 each quantity 100.</p>
<p>Connections are powerful things. This is a novel capability, and a dramatic improvement in the price-performance ratio of existing precision-time sources as finished products that can be deployed immediately. From one PCB trace!  Which the GPS vendors could have added years ago if they had known just a little more about their chipsets.</p>
<p>Sometimes the most important connection of all is just knowing that a given thing is possible.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&#038;p=4281</wfw:commentRss>
		<slash:comments>154</slash:comments>
		</item>
	</channel>
</rss>

