I’ve been radio silent the last couple of weeks mainly because I’ve been concentrating furiously on getting a GPSD release out the door. This one is a little more noteworthy than usual because it may actually have fixed a well-hidden flaw or vulnerability of some significance.
Regular readers may recall from back in 2013 that I published a heads-up titled No, GPSD is not the battery-killer on your Android! addressing a power-drain bug reported from a handful of Android phones.
I believed at the time that the proximate cause of the bug was in the kernel serial device-drivers somewhere specific to particular hardware on those phones. I still believe that, because if it had been a purely GPSD problem the error would likely have been much more widespread and I’d have been flooded with complaints.
However, I’ve been concerned ever since that GPSD might not have been doing everything it could to armor itself against bugginess in the layers below it. And a couple of weeks ago I found a problem…
If you’ve ever wanted a JSON parser that can unpack directly to fixed-extent C storage (look, ma, no malloc!) I’ve got the code for you.
The microjson parser is tiny (less than 700LOC), fast, and very sparing of memory. It is suitable for use in small-memory embedded environments and deployments where malloc() is forbidden in order to prevent leaked-memory issues.
This project is a spin-out of code used heavily in GPSD; thus, the code has been tested on dozens of different platforms in hundreds of millions of deployments.
It has two restrictions relative to standard JSON: the special JSON “null” value is not handled, and object array elements must be homogenous in type.
A programmer’s guide to building parsers with microjson is included in the distribution.
I’ve been blog-silent the last couple of days because I’ve been chasing down the bug I mentioned in Request for help – I need a statistician.
I have since found and fixed it. Thereby hangs a tale, and a cautionary lesson.
I just had a rather hair-raising experience with a phase-of-moon-dependent bug.
I released GPSD 3.11 this last Saturday (three days ago) to meet a deadline for a Debian freeze. Code tested ninety-six different ways, run through four different static analyzers, the whole works. Because it was a hurried release I deliberately deferred a bunch of cleanups and feature additions in my queue. Got it out on time and it’s pretty much all good – we’ve since turned up two minor build failures in two unusual feature-switch cases, and one problem with the NTP interface code that won’t affect reasonable hardware.
I’ve been having an extremely productive time since chewing through all the stuff I had deferred. New features for gpsmon, improvements for GPSes watching GLONASS birds, a nice space optimization for embedded systems, some code to prevent certain false-match cases in structured AIS Type 6 and Type 8 messages, merging some Android port tweaks, a righteous featurectomy or two. Good clean fun – and of course I was running my regression tests frequently and noting when I’d done so in my change comments.
Everything was going swimmingly until about two hours ago. Then, as I was verifying a perfectly innocent-appearing tweak to the SiRF-binary driver, the regression tests went horribly, horribly wrong. Not just the SiRF binary testloads, all of them.
Blogging has been light recently because I’ve been working very hard on a major GPSD release, which I just shipped. This is mostly new features, not bugfixes, and it’s probably the most new code we’ve shipped in one release since about 2009.
I’ve been working on GPSD a lot recently – we’re heading towards a 3.10 release with a lot of new features. As part of this release I’ve decided to ship a HOWTO on setting up a high-quality NTP time server using GPSD. In the course of working on that, I’ve had an idea.
The idea has two antecedents. One is that if you start with any one of several inexpensive GPS modules (my favorite of which is the u-blox 6), and add GPSD to read it and feed an ntpd instance, it’s possible to build an NTP server that meets the usual standard for public Stratum 1 time servers – 10mSec or better accuracy to UTC.
The other is that there is a raft of inexpensive SBCs that run Linux out there – Arduino, Raspberry Pi and the current new hotness BeagleBone. So here’s my thought: why not build a low-power Stratum 1 timeserver on a credit-card-sized SBC?
Today, while doing research to answer some bug mail, I learned that all versions of Android since 4.0 (Ice Cream Sandwich) have used gpsd to read the take from the onboard GPS. Sadly, gpsd is getting blamed in some quarters for excessive battery drain. But it’s not gpsd’s fault! Here is what’s actually going on.
I’ve been pounding on GPSD with the Coverity static analyzer’s self-build procedure for several days. It is my great pleasure to report that we have just reached zero defect reports in 72.8KLOC. Coverity says this code is clean. And because I think this should be an example unto others, I shall explain how I think others can do likewise.
I got emailed summaries from a Coverity scan of the repo head version of GPSD today.
In a recent Google+ comment, H. Peter Anvin grumped about GPSD using “braindead heuristics” to determine which USB devices it should sniff as possible GPses when it gets a hotplug notification saying that one has connected. I was going to reply in a comment there, but the explanation ran too long for that.
Short version: yes, GPSD will very occasionally sniff at a device that is none of its business. We’re stuck in a bad place because of deficiencies in the USB standard, But it doesn’t happen often, and all the alternative behaviors I’ve been able to imagine would be worse in very obvious ways. Detailed explanation follows.
About six weeks ago I asked How would you like to help fix the Internet? It was an open invitation to help develop a cheap millisecond-precision time source for instrumented routers, so we can do delay tomography on the Internet and measure the bufferbloat problem.
The discussion thread on that post was lively, but eventually moved to a thumbgps project mailing list out of public view. I’m going to summarize what has gone on here because parts of it are very interesting in a “Wow…it really is the 21st century!” sort of way, and illustrate that there can be a lot of power in making simple connections.
How would you like to help fix the Internet?
One of the efforts I’ve been contributing to during the last year is the Bufferbloat project, a group of experienced Internet engineers who believe that excessive buffering and poor queue-management strategies may be the real villains behind a lot of network problems commonly attributed to undercapacity.
Before we can solve the problem, we need to measure and map it by collecting a lot of packet-propagation-time statistics. Awkwardly, we suspect that one of the services being screwed up by bufferbloat-induced latency spikes is the Network Time Protocol. So…Dave Täht (aka Dave from my basement) is trying to build a device he calls the Cosmic Background Bufferbloat Detector. The CBBD would be a flock of routers scattered all over the world, watching NTP packet timings using a common timebase independent of NTP, and sending data back to a collection server for analysis and visualization.
That’s where I, as the lead of the GPSD project, come in. GPSes are an obvious candidate for a high-precision NTP-independent time service. But there’s a problem with that…
This is a shout out to all hackerspaces and engineering schools within easy reach of Philadelphia. I’ve got a nice little design-and-build project that would do the world some good, but I don’t have the skills or facilities to do it myself.
The problem: build a ruggedized special-purpose test enclosure to be mounted on a roof or utility pole and host a bunch of GPS sensors. The tricky part is that it needs to be outside and not under top cover (for good skyview) and thus weatherproof, but also transparent to the GPS radio frequencies. Another part of the design problem is getting data and power cabling back to my development computer.
UPDATE: I’m now pursuing a different path – trying to figure out how to build a GPS repeater on the cheap so I can effectively pipe the RF from a roof antenna to be retransmitted in my office. This has the obvious advantage that the GPS test rack will be able to live inside, near my desk, rather than outside in an enclosure that can only be reached with a ladder. So now I’m looking for a hackerspace frequented by radio hams.
I have interesting friends. Two of them, who shall remain nameless because it is possible they have let slip to me information that is technically classified, recently told me the best GPSD deployment story since the robot submarine.
So, Friend A says “Hey, Eric, did you know GPSD is used in the on-board nav system of the Abrams tank?” Friend A is in a position to know, because Friend A has done troubleshooting of that nav system – once, over the phone with a tank actually in combat in Iraq. It seems GPSD is used as part of IFF (identification friend or foe) and without that module they are at unpleasant risk of heaving a shell at a friendly. (And no, I am assured the bug was not in GPSD itself.)
Four days after I got the word that Berlios is dying, I have saved GPSD from being pulled under as it sinks. A couple of observations on the project migration follow.
I just got word that berlios.de, where I have several projects hosted including GPSD, is going to shut down at the end of the year.
This is a huge pain in the ass. It means I’m going to have to bust my hump to get us to new hosting space. Moving the git repo won’t be bad, but moving the mailing list and bugtracker content is going to suck. What’s worse, all the project URLs are going to break.
Back in 2009 I launched a project called forgeplucker to address this sort of migration problem. It stalled due to a flaky hosting site…
Protocol transitions are hard.
Yeah, I know the experienced software developers reading that are thinking “Well, duh!“. Shut up already, I’m venting here. I’ve just spent the better part of two years – actually, if you include design time and false starts it could be closer to five years – designing a new application protocol for my gpsd service daemon, implementing it, getting it deployed, dealing with problems and course-correcting.
For those of you in the cheap seats, an “application protocol” is a kind of language that a program uses to communicate with other programs. Normally humans never see these, but there’s one big example that non-geeks have often seen bits of. HTML, the Web’s markup language, is an application protocol. Designing these is not easy. There are difficult issues and tradeoffs around flexibility, economy, expressiveness, and extensibility for uses we haven’t imagined yet.
When you have a whole bunch of programs written by different people communicating with a particular application protocol, changing that entire ecology to use a different one is not easy. A good analogy in the physical world is the difficulty of changing an entire country’s railroad gauge. The long-term benefits may be huge, but the short-term cost in capital and service disruption is daunting.
Someone on the gpsd-users list asked:
I was just wondering why gpsd doesn’t have a configuration file in /etc/gpsd.conf, like most other Unix/Linux software?
Because configuration files are evil, and not to be countenanced unless they become an absolutely necessary evil. Which in gpsd’s case is not yet, and I sincerely hope not ever.
I’ve gotten used to being cited in computer science and software engineering papers over the last decade, but here’s a new one. Today I read a draft in which I and the GPSD project get cited a bunch of times and it’s – er – not about open source. It’s about Marine AIS in disaster management. Broadening my deepwater horizons, as it were.
This is a followup to The Rollover of Doom: a Trap for Good Programmers. That post ended “This problem is a Chinese finger-trap for careful and conscientious programmers. The better you are, the worse this problem is likely to hurt your brain. Embrace the suck.”
That last phrase is a take on a military objurgation which translates as “The situation is bad. Deal with it.” Well, my friends, I am about to tell you how bad the GPS rollover situation really is.