Freeing technical standards

I grew up on Internet RFCs, so technical standards issued under licenses that forbid free redistribution offend me. Every such document, whether intentionally or not, is a device for hindering open-source software projects and privileging closed-source developers with big budgets and lawyers to hand.

They offend me even more when (as, for example, when the GPS reporting standard NMEA 0183) the proprietary “standard” is so badly designed that a mob of crack-addled rhesus monkeys could have done a better job – and for this they want me to pay? They offend me the absolute most when the “standard” is distribution-restricted, expensive, badly written, and its topic is a safety-critical technology – so that people could actually die because some jerk wants to collect a trivial amount of secrecy rent on a standard that was crappy to begin with.

Fortunately, there is a way to monkeywrench the organizations that perpetrate this sort of thing – and I’ve spent a substantial part of the last couple of weeks doing exactly that. I’m writing about it here to encourage others to do likewise.

In connection with my ongoing work on GPSD I’ve recently been studying AIS, the marine Automatic Identification System. Since about 2004, maritime authorities worldwide have required every passenger vessel and every cargo ship of more than 300 tons displacement to carry a transmitter that periodically squawks navigational information about the vessel. AIS receivers in the vicinity can pick up this information and make it available for useful things like collision-avoidance systems.

A few AIS receivers have integral map displays meant to be read by a human, but these are largely toys for casual boaters. Most behave a lot like shipboard GPS sensors; they ship AIS sentences out a serial or USB port to be read by navigational systems that will do something useful with them…like, say, automated collision avoidance.

Well…you can do something useful with these sentences if you can read them. And why is that a problem? Because distribution of the core standard for the AIS reporting format (ITU-R 1371) is, you guessed it, restricted under terms that are proprietary and evil. Just like NMEA 0183. But what’s even worse, in this case, is that the International Telecommunications Union is charging secrecy rent on a standard that appears to have been designed mostly by the U.S. Coast Guard. So, a organization that is (a) private, and (b) foreign is charging Americans secrecy rent on a design built with U.S. taxpayer dollars.

When I learned this, I decided I was going to (a) do the open-source world a service, and (b) break their secrecy any way I could. Yeah, I know, some of you are going to tell me this sort of thing is pretty normal in every area of technical standardization other than the Internet; I even knew that, having run into it occasionally before. But this time I reached the “mad as hell and not gonna take-it any more” stage. I’m pushing back.

Here’s what I’ve done. Without looking at the two proprietary standards that bear on AIS (NMEA 4.0 and ITU-R 1371), I’ve collected all the public information on AIS messages and reporting formats into a document which essentially blows the lid off their secrecy. The “without looking” is important; at this point, I don’t want to see them so that I can”t be gigged for copyright infringement over my document. Here it is: AIVDM/AIVDO protocol decoding.

And how do I know this describes reality? Because it describes the masses of AIS sentences helpful people have been sending me. I have a working decoder in the GPSD suite now; it’s not integrated into gpsd itself yet, but you can use it to filter logs full of armored/encoded AIS sentences into readable text. Full support, and test clients that do cool stuff, will follow soon.

Actually, this thing wouldn’t have been trivial to put together even without the legal barriers. The public sources, and conversations with people who have seen NMEA 4.00 and ITU1371, reveal that these documents have been through multiple revisions with substantial changes in them – and the latter one is so densely and badly written that I was able to gather much of what I needed from technical corrigenda published by the U.S. Coast Guard. Also, I document common practice that the official standard doesn’t.

OK, so you should now go and do likewise with some other evil, locked-up-behind-a-paywall standard. We need to teach the ITU and other organizations like it that they will get p0wned and scorned every time they try to collect secrecy rent on these things, so they’ll stop trying.

And don’t worry that you’ll stop good standards from issuing; these groups are generally funded by vendor consortia and governments heavily enough that I can’t see the document fees they collect making any actual difference to them. And that, really, is the final turn of the screw – not only are they locking out open-source projects, they’re doing so for an amount of rent that’s tiny in comparison to the rest of their funding base.

How do I know this? Simple – because it has to be; after all, the Internet Engineering Task Force successfully organizes the same sort of standards work without charging one thin dime for viewing RFCs. So screw you, ITU and NMEA both; your paywalls aren’t just greedy, they’re stupid greedy.

(Oh, and anyone who’s got test sets pairing armored AIVDM sentences with text decodings of same should send them to me so I can add them to my regression test. Don’t bother if it’s just messages types 1-5, though, I’ve got plenty of test sets for those.)

18 thoughts on “Freeing technical standards

  1. For those of you not as geeky as Eric and I, maybe you are sitting here wondering, “What the hell does RFC mean?” Well, it means request for comment.

    Sometimes you have to assume there are people reading this who are just getting started, you know?

    But yeah, it is major suckage that anyone would try to charge to read a god damn rfc. What are they thinking?

  2. I’m really not nitpicking but I think you have a typo: “[..] worldwide have required very passenger [..]”. Shouldn’t it be every? Interesting read non the less.

  3. the biggest reason not to expand RFC is that the name is completely misleading at this point. while they may once have actually been drafts, comments on which were invited, that was back in the ’70’s, and they never came up with a name for a “finalized” RFC, so a set-in-stone standard that’s been driving the whole net for decades (e.g. IP) is still called an RFC (791, in the case of IP). that said, a link to wouldn’t go amiss.

  4. another micro-typo–there’s a stray period before “actually” in the second paragraph

  5. Excellent!

    *saves offline copy just in case*

    It’s also too bad that the RF side is so patent encumbered. STDMA would be a really kickass thing to play with on the amateur radio side, I’d hate to write a open source variant and have the patent holder come after me.

  6. Oh, for the good old days when you had to publish to get a copyright, and the definition of “publish” included “make public”– that is, distribute without further restrictions on distribution.

  7. Is there a clearinghouse website out there that lists the standards that need this treatment, with contact info for people working on each alongside? If not, maybe there should be.

  8. Well, no Jeremy, it isn’t that hard to figure out what any of these things mean, what with the Google and all.

    But it disrupts the flow of reading the thing to have to stop every other sentence and Google.

  9. It’s also idiotic to call something a standard if it’s not released in a way that allow everybody to see,use, copy and share it..
    Then it’s proprietary, if enough people use it, it’s still not a standard, then it’s a conspiracy..

  10. I like the clearinghouse idea. Seems like a perfect application for a wiki.

  11. > that the International Telecommunications Union is charging secrecy rent on a standard that appears to
    > have
    > been designed mostly by the U.S. Coast Guard. So, a organization that is (a) private, and (b) foreign is
    > charging Americans secrecy rent on a design built with U.S. taxpayer dollars.

    This is where I get confused. As its not classified, its used by the public and authored (mostly) by the government, isn’t it supposed to be in the public domain to begin with? For instance, why can’t I ask for copies of all RFC’s, look at the thing as a whole, re-write it and publish it? Or at least, RFC’s that originated from the US Coast Guard. Or, was that the information you were able to obtain in order to publish?

    How is this kind of paywall even legal to begin with, why do companies who manufacture such equipment even subscribe to these kinds of squatters? Surely, some engineer who had access to the full standard looked at it and said “ewww!”. So a good question is, how do these leeches manage to latch on in the first place?

    I’m about to buy a Beagle Board and start getting to know how to read a GPS, I’d like to get into RC helicopters. Thank you for all of your work in this field.

  12. “It’s also idiotic to call something a standard if it’s not released in a way that allow everybody to see,use, copy and share it..” Yes, of course. It’s like an unpublished law – that’s not a law.

    “Then it’s proprietary, if enough people use it, it’s still not a standard, then it’s a conspiracy..” Well… no.

    Basically, an unreleased standard IMHO is an internal process document and rulebook. “Dear Employees, this is how we are doing this here.”

  13. Wikipedia is pretty much doing this sort of thing with proprietary chemical databases. They’re not happy about it, and we’re generally responding with the very politest of “fuck you”s. It’s PUBLIC INFORMATION that’s PRINTED on BOTTLES!

  14. >secrecy rent

    oh, superb term. (shades of Esoteric Sect — perhaps something to stick on your anthropological-article (eg CatB) backburner)

    oh and:
    > AIVDM/AIVDO protocol

Leave a Reply

Your email address will not be published. Required fields are marked *