I made a finger error. Full version of A Low Performance Mystery to follow shortly.
In days of yore, Yacc and Lex were two of the most useful tools in a Unix hacker’s kit. The way they interfaced to client code was, however, pretty ugly – global variables and magic macros hanging out all over the place. Their modern descendants, Bison and Flex, have preserved that ugliness in order to be backward-compatible.
That rebarbative old interface generally broke a lot of rules about program structure and information hiding that we now accept as givens (to be fair, most of those had barely been invented at the time it was written in 1970 and were still pretty novel). It becomes a particular problem if you want to run multiple instances of your generated parser (or, heaven forfend, multiple parsers with different grammars) in the same binary without having them interfere with each other.
But it can be done. I’m going to describe how because (a) it’s difficult to extract from the documentation, and (b) right now (that is, using Bison 3.0.2 and Flex 2.5.35) the interface is in fact slightly broken and there’s a workaround you need to know.
A bit late, because I’ve been hammering on some code the last several days. But here it is: Time, Clock, and Calendar Programming In C.
Suggestions for 1.1 revisions and improvements will of course be cheerfully accepted. Comments here or email will be fine.
This landed in my mailbox yesterday. I reproduce it verbatim except for the sender’s name.
> Dear authors of the RFC 3092,
> I am writing this email on behalf of your Request For Comment “Etymology of
> ‘Foo’.” We are currently learning about the internet organizations that set
> the standards of the internet and our teacher tasked us with finding an RFC
> that was humorous. Me and my two friends have found the “Etymology of
> ‘Foo'” and have found it to be almost as ridiculous as the RFC about
> infinite monkeys; however, we then became quite curious as to why you wrote
> this. Obviously, it is wrote for humor as not everything in life can be
> serious, but did your manager task you to write this? Are you a part of an
> organization in charge of writing humorous RFC’s? Are you getting paid to
> write those? If so, where do you work, and how may we apply? Any comments
> on these inquiries would be greatly appreciated and thank you in advance.
> XXXXXXXXXXXXXX, confused Networking student
I felt as though this seriously demanded a ha-ha-only-serious answer – and next thing you know I was channeling Master Po from the old Kung Fu TV series. Reply follows…
In the process of working on my Time, Clock, and Calendar Programming In C document, I have learned something sad but important: the standard Unix calendar API is irremediably broken.
The document list a lot of consequences of the breakage, but here I want to zero in on what I think is the primary causes. That is: the standard struct tm (a) fails to be an unambiguous representation of time, and (b) violates the SPOT (Single Point of Truth) design rule. It has some other more historically contingent problems as well, but these problems (and especially (a)) are the core of its numerous failure modes.
These problems cannot be solved in a backwards-compatible way. I think it’s time for a clean-sheet redesign. In the remainder of this post I’ll develop what I think the premises of the design ought to be, and some consequences.
I’ve been gifted with a lot of help on my draft of Time, Clock, and Calendar Programming In C. I think it’s almost time to ship 1.0, and plan to do so this weekend. Get your last-minute fixes in now!
I will of course continue to accept corrections and additions after 1.0. Thanks to everyone who contributed. My blog and G+ followers were very diligent in spotting typos, helping fill in and correct standards history, and pointing out the more obscure gotchas in the API.
What I’ve discovered is that the Unix calendar-related API is a pretty wretched shambles. Which leads directly to the topic of my next blog entry…
A provocative article at the conservative blog Hot Air comments on a pattern in American coverage of violent interracial crimes. When the perps are white and the victims are black, we can expect the press coverage to be explicit about it, with predictable assumption of racist motivations. On the other hand, when the perps are black and the victims are white, the races of all parties are normally suppressed and no one dares speak the r-word.
If I were a conservative, or a racist, I’d go off on some aggrieved semi-conspiratorial rant here. Instead I’ll observe what Hot Air did not: that the race of violent black criminals is routinely suppressed in news coverage even in the much more common case that their victims are also black. Hot Air is over-focusing here.
That said, Hot Air seems to have a a separate and valid point when it notes that white victims are most likely to have their race suppressed from the reporting when the criminals are black – especially if there was any hint of racist motivation. There is an effective taboo against truthfully reporting incidents in which black criminals yell racial epithets and threats at white victims during the commission of street crimes. If not for webbed security-camera footage we’d have no idea how depressingly common this seems to be – the press certainly won’t cop to it in their print stories.
No conspiracy theory is required to explain the silence here. Reporters and editors are nervous about being thought racist, or (worse) having “anti-racist” pressure groups demonstrating on their doorsteps. The easy route to avoiding this is a bit of suppressio veri – not lying, exactly, but not uttering facts that might be thought racially inflammatory.
The pattern of suppression is neatly explained by the following premises: Any association of black people with criminality is inflammatory. Any suggestion that black criminals are motivated by racism to prey on white victims is super-inflammatory. And above all, we must not inflame. Better to be silent.
I believe this silence is a dangerous mistake with long-term consequences that are bad for everyone, and perhaps worst of all for black people.
Neil Gaiman writes On Terry Pratchett, he is not a jolly old elf at all.. It’s worth reading.
I know that what Neil Gaiman says here is true, because I’ve known Terry, a little. Not as well as Neil does; we’re not that close, though he has been known to answer my email. But I did have one experience back in 2003 that would have forever dispelled any notion of Terry as a mere jolly elf, assuming I’d been foolish enough to entertain it.
I taught Terry Pratchett how to shoot a pistol.
(We were being co-guests of honor at Penguicon I at the time. This was at the first Penguicon Geeks with Guns event, at a shooting range west of Detroit. It was something Terry had wanted to do for a long time, but opportunities in Britain are quite limited.)
This is actually a very revealing thing to do with anyone. You learn a great deal about how the person handles stress and adrenalin. You learn a lot about their ability to concentrate. If the student has fears about violence, or self-doubt, or masculinity/femininity issues, that stuff is going to tend to come out in the student’s reactions in ways that are not difficult to read.
Terry was rock-steady. He was a good shot from the first three minutes. He listened, he followed directions intelligently, he always played safe, and he developed impressive competence at anything he was shown very quickly. To this day he’s one of the three or four best shooting students I’ve ever had.
That is not the profile of anyone you can safely trivialize as a jolly old elf. I wasn’t inclined to do that anyway; I’d known him on and off since 1991, which was long enough that I believe I got a bit of look-in before he fully developed his Famous Author charm defense.
But it was teaching Terry pistol that brought home to me how natively tough-minded he really is. After that, the realism and courage with which he faced his Alzheimer’s diagnosis came as no surprise to me whatsoever.
The C/UNIX library support for time and calendar programming is a nasty mess of historical contingency. I have grown tired of having to re-learn its quirks every time I’ve had to deal with it, so I’m doing something about that.
Announcing Time, Clock, and Calendar Programming In C, a document which attempts to chart the historical clutter (so you can ignore it once you know why it’s there) and explain the mysteries.
What I’ve released is an 0.9 beta version. My hope is that it will rapidly attract some thoroughgoing reviews so I can release a 1.0 in a week or so. More than that, I would welcome a subject matter expert as a collaborator.
In the wake of the Shellshock bug, I guess I need to repeat in public some things I said at the time of the Heartbleed bug.
The first thing to notice here is that these bugs were found – and were findable – because of open-source scrutiny.
There’s a “things seen versus things unseen” fallacy here that gives bugs like Heartbleed and Shellshock false prominence. We don’t know – and can’t know – how many far worse exploits lurk in proprietary code known only to crackers or the NSA.
What we can project based on other measures of differential defect rates suggests that, however imperfect “many eyeballs” scrutiny is, “few eyeballs” or “no eyeballs” is far worse.
I’m not handwaving when I say this; we have statistics from places like Coverity that do defect-rate measurements on both open-source and proprietary closed source products, we have academic research like the UMich fuzz papers, we have CVE lists for Internet-exposed programs, we have multiple lines of evidence.
Everything we know tells us that while open source’s security failures may be conspicuous its successes, though invisible, are far larger.
The patent-troll industry is in full panic over the consequences of the Alice vs. CLS Bank decision. While reading up on the matter, I ran across the following claim by a software patent attorney:
“As Sun Microsystems proved, the quickest way to turn a $5 billion company into a $600 million company is to go open source.”
I’m not going to feed this troll traffic by linking to him, but he’s promulgating a myth that must be dispelled. Trying to go open source didn’t kill Sun; hardware commoditization killed Sun. I know this because I was at ground zero when it killed a company that was aiming to succeed Sun – and, until the dot-com bust, looked about to manage it.
In a recent discussion on G+, a friend of mine made a conservative argument for textual over binary interchange protocols on the grounds that programs always need to be debugged, and thus readability of the protocol streams by humans trumps the minor efficiency gains from binary packing.
I agree with this argument; I’ve made it often enough myself, notably in The Art of Unix Programming. But it was something his opponent said that nudged at me. “Provable programs are the future,” he declaimed, pointing at sel4 and CompCert as recent examples of formal verification of real-world software systems. His implication was clear: we’re soon going to get so much better at turning specifications into provably correct implementations that debuggability will soon cease to be a strong argument for protocols that can be parsed by a Mark I Eyeball.
Oh foolish, foolish child, that wots not of the Rule of Technical Greed.
Last night, my wife Cathy and I passed our level 5 test in kuntao. That’s a halfway point to level 10, which is the first “guro” level, roughly equivalent to black belt in a Japanese or Korean art. Ranks aren’t the big deal in kuntao that they are in most Americanized martial arts, but this is still a good point to pause for reflection.
If you’ve ever wanted a JSON parser that can unpack directly to fixed-extent C storage (look, ma, no malloc!) I’ve got the code for you.
The microjson parser is tiny (less than 700LOC), fast, and very sparing of memory. It is suitable for use in small-memory embedded environments and deployments where malloc() is forbidden in order to prevent leaked-memory issues.
This project is a spin-out of code used heavily in GPSD; thus, the code has been tested on dozens of different platforms in hundreds of millions of deployments.
It has two restrictions relative to standard JSON: the special JSON “null” value is not handled, and object array elements must be homogenous in type.
A programmer’s guide to building parsers with microjson is included in the distribution.
GPSD has a serious bug somewhere in its error modeling. What it effects is position-error estimates GPSD computes for GPSes that don’t compute them internally themselves and report them on the wire. The code produces plausible-looking error estimates, but they lack a symmetry property that they should have to be correct.
I need a couple of hours of help from an applied statistician who can read C and has experience using covariance-matrix methods for error estimation. Direct interest in GPS and geodesy would be a plus.
I don’t think this is a large problem, but it’s just a little beyond my competence. I probably know enough statistics and matrix algebra to understand the fix, but I don’t know enough to find it myself.
Hundreds of millions of Google Maps users might have reason to grateful to anyone who helps out here.
UPDATE: Problem solved, see next post.
In a blog post on Computational Knowledge and the Future of Pure Mathematics Stephen Wolfram lays out a vision that is in many ways exciting and challenging. What if all of mathematics could be expressed in a common formal notation, stored in computers so it is searchable and amenable to computer-assisted discovery and proof of new theorems?
Infinite Science Fiction One (edited by Dany G. Zuwen and Joanna Jacksonl Infinite Acacia) starts out rather oddly, with Zuwen’s introducton in which, though he says he’s not religious, he connects his love of SF with having read the Bible as a child. The leap from faith narratives to a literature that celebrates rational knowability seems jarring and a bit implausible.
That said, the selection of stories here is not bad. Higher-profile editors have done worse, sometimes in anthologies I’ve reviewed.
Collision of Empires (Prit Buttar; Osprey Publishing) is a clear and accessible history that attempts to address a common lack in accounts of the Great War that began a century ago this year: they tend to be centered on the Western Front and the staggering meat-grinder that static trench warfare became as outmoded tactics collided with the reality of machine guns and indirect-fire artillery.
Concentration on the Western Front is understandable in the U.S. and England; the successor states of the Western Front’s victors have maintained good records, and nationals of the English-speaking countries were directly involved there. But in many ways the Eastern Front story is more interesting, especially in the first year that Buttar chooses to cover – less static, and with a sometimes bewilderingly varied cast. And, arguably, larger consequences. The war in the east eventually destroyed three empires and put Lenin’s Communists in power in Russia.
A Call To Duty (David Weber, Timothy Zahn; Baen Books) is a passable extension of Baen Book’s tent-pole Honorverse franchise. Though billed as by David Weber, it resembled almost all of Baen’s double-billed “collaborations” in that most of the actual writing was clearly done by the guy on the second line, with the first line there as a marketing hook.