Jan 13

Rust and the limits of swarm design

In my last blog post I expressed my severe disappointment with the gap between the Rust language in theory (as I had read about it) and the Rust language in practice, as I encountered it when I actually tried to write something in it.

Part of what I hoped for was a constructive response from the Rust community. I think I got that. Amidst the expected volume of flamage from rather clueless Rust fanboys, several people (I’m going to particularly call out Brian Campbell, Eric Kidd, and Scott Lamb) had useful and thoughtful things to say.

I understand now that I tested the language too soon. My use case – foundational network infrastructure with planning horizons on a decadal scale – needs stability guarantees that Rust is not yet equipped to give. But I’m somewhat more optimistic about Rust’s odds of maturing into a fully production-quality tool than I was.

Still, I think I see a problem in the assumptions behind Rust’s development model. The Rust community, as I now understand it, seems to me to be organized on a premise that is false, or at least incomplete. I fear I am partly responsible for that false premise, so I feel a responsibility to address it square on and attempt to correct it.

Continue reading

Jan 12

Rust severely disappoints me

I wanted to like Rust. I really did. I’ve been investigating it for months, from the outside, as a C replacement with stronger correctness guarantees that we could use for NTPsec.

I finally cleared my queue enough that I could spend a week learning Rust. I was evaluating it in contrast with Go, which I learned in order to evaluate as a C replacement a couple of weeks back.

Continue reading

Jan 01

How to educate me about prejudice in the open-source community

Every once in a while I post something just to have it handy as a reference for the next time I have to deal with a galloping case of some particular kind of sloppy thinking. That way I don’t have to generate an individual explanation, but can simply point at my general standards of evidence.

This one is about accusations of sexism, racism, and other kinds of prejudice in the open-source culture.

Continue reading

Dec 14

Hey, Democrats! We need you to get your act together!

It’s now just a bit over a month since Election Day, and I’m starting to be seriously concerned about the possibility that the U.S. might become a one-party democracy.

Therefore this is an open letter to Democrats; the country needs you to get your act together. Yes, ideally I personally would prefer your place in the two-party Duverger equilibrium to be taken by the Libertarian Party, but there are practical reasons this is extremely unlikely to happen. The other minor parties are even more doomed. If the Republicans are going to have a counterpoise, it has to be you Democrats.

Donald Trump’s victory reads to me like a realignment election, a historic break with the way interest and demographic groups have behaved in the U.S. in my lifetime. Yet, Democrats, you so far seem to have learned nothing and forgotten nothing. Indeed, if I were Donald Trump I would be cackling with glee at your post-election behavior, which seems ideally calculated to lock Trump in for a second term before he has been sworn in for the first.

Stop this. Your country needs you. I’m not joking and I’m not concern-trolling. The wailing and the gnashing of teeth and the denial of reality have to end. In the rest of this essay I’m not going to talk about right and wrong and ideology, I’m going to talk about the brutal practical politics of what you have to do to climb out of the hole you are in.

Continue reading

Dec 07

Spelunking the alt-right

Recently, on a mailing list I frequent, one of the regulars uttered the following sentence: “I’m told Breitbart is the preferred news source for the ‘alt-right’ (KKK and neo-nazis)”.

That was a pretty glaring error, there.

I was interviewed on Breitbart Tech once. I visit the site occasionally. I am not affiliated with the alt-right, but I’ve been researching the recent claims about it. So I can supply some observations from the ground.

First, while I’m not entirely sure of everything the alt-right is (it’s a rather amorphous phenomenon) it is not the KKK and neo-Nazis. The most that can truthfully be said is that ‘alt-right’ serves as a recent flag of convenience to which some old-fashioned white supremacists are busily trying to attach themselves.

Also, the alt-right is not Donald Trump and his Trumpkins, either. He’s an equally old-fashioned populist continuous with Willam Jennings Bryan and Huey Long. If you tossed a bunch of alt-right memes at him, I doubt he’d even understand them, let alone agree.

The defining characteristic of the alt-right is, really, corrosive snarkiness. To the extent an origin can be identified, it was as a series of message-board pranks on 4chan. There’s no actual ideological core to it – it’s a kind of oppositional attitude-copping without a program, mordantly nasty but unserious.

There’s also some weird occultism attached – the half-serious cult of KEK, aka Pepe, who may or may not be an ancient Egyptian frog-god who speaks to his followers via numerological coincidences. (Donald Trump really wouldn’t get that part.)

Some elements of the alt-right are in fact racist (and misogynist, and homophobic, and other bad words) a la KKK/Nazi, but that’s not a defining characteristic and it’s anyway difficult to tell the genuine haters from those for whom posing as haters is a form of what 4chan types call “griefing”. That’s social disruption for the hell of it.

It is worth noting that another part of what is going on here is a visceral rejection of politically-correct leftism, one which deliberately inverts its premises. The griefers pose as racists and misogynists because they think it’s the most oppositional stance they can take to bullies and rage-mobbers who position themselves as anti-racists and feminists.

My sense is that the true haters are a tiny minority compared to the griefers and anti-PC rejectionists, but the griefers are entertained by others’ confusion on this score and don’t intend to clear it up.

Whether the alt-right even exists in any meaningful sense is questionable. To my anthropologist’s eye it has the aspect of a hoax (or a linked collection of hoaxes) being worked by 4chan griefers and handful of more visible provocateurs – Milo Yiannopolous, Mike Cernovich, Vox Day – who have noticed how readily the mainstream media buys inflated right-wing-conspiracy narratives and are working this one for the lulz. There’s no actual mass movement behind their posturing, unless you think a thousand or so basement-dwelling otaku are a mass movement.

I know Milo Yannopolous slightly – he is who interviewed me for Beitbart – and we have enough merry-prankster tendency in common that I think I get how his mind works. I’m certain that he, at any rate, is privately laughing his ass off at everyone who is going “alt-right BOOGA BOOGA!”

And there are a lot of such people. What these provocateurs are exploiting is media hysteria – the alt-right looms largest in the minds of self-panickers who project their fears on it. And of course in the minds of Hillary Clinton’s hangers-on, who would rather attribute her loss to a shadowy evil conspiracy than to a weak candidate and a plain-old bungled campaign.

I’m worried, however, that that the alt-right may not remain a loose-knit collection of hoaxes – that the self-panickers are actually creating what they fear.

For there is a deep vein of anti-establishment anger out there (see Donald Trump, election of). The alt-right (to the limited and conditional extent it now exists) could capture that anger, and its provocateurs are doing their best to make you think it already has, but they’re scamming you – they’re fucking with your head. The entire on-line ‘alt-right’ probably musters fewer people than the Trumpster’s last victory rally.

It’s a kind of dark-side Discordian hack in progress, and I’m concerned that it might succeed. Vox Day is trying to ideologize the alt-right, actually assemble something coherent from the hoaxes. He might succeed, or someone else might. Draw some comfort that it won’t be the Neo-Nazis or KKK – they’re real fanatics of the sort the alt-right defines itself by mocking. Mein Kampf and ironic nihilism don’t mix well.

The best way to beat the “alt-right” is not to overestimate it, not to feed it with your fear. If you keep doing that, the vast majority of the rootless and disaffected who have never heard of it might decide there’s a strong horse there and sign on.

Oh, and a coda about Breitbart: anyone who thinks Breitbart is far right needs to get out of their mainstream-media bubble more. Compared to sites like WorldNetDaily or FreeRepublic or TakiMag or even American Thinker, Breitbart is pretty mild stuff.

All those fake-news allegations against Breitbart are pretty rich coming from a media establishment that gave us Rathergate, “Hands up don’t shoot!”, and the “Jackie” false-rape story and was quite recently exchanging coordination emails with the Clinton campaign. Breitbart isn’t any more propagandistic than CBS or Rolling Stone, it’s just differently propagandistic.

Dec 02

Some of my blogging has moved

I’ve been pretty quiet lately, other than short posts on G+, because I’ve been grinding hard on NTPsec. We’re coming up on a 1.0 release and, although things are going very well technically, it’s been a shit-ton of work.

One consequence is the NTPsec Project Blog. My first major post there expands on some of the things I’ve written here about stripping crap out of the NTP codebase.

Expect future posts on spinoff tools, the NTPsec test farm, and the prospects for moving NTPsec out of C, probably about one a week. I have a couple of these in draft already.

Sep 27

Twenty years after

I just shipped what was probably the silliest and most pointless software release of my career. But hey, it’s the reference implementation of a language and I’m funny that way.

Because I write compilers for fun, I have a standing offer out to reimplement any weird old language for which I am sent a sufficiently detailed softcopy spec. (I had to specify softcopy because scanning and typo-correcting hardcopy is too much work.)

In the quarter-century this offer has been active, I have (re) implemented at least the following: INTERCAL, Michigan Algorithmic Decoder, and a pair of obscure 1960s teaching languages called CORC and CUPL, and an obscure computer-aided-instruction language called Pilot.

Pilot…that one was special. Not in a good way, alas. I don’t know where I bumped into a friend of the language’s implementor, but it was in 1991 when he had just succeeded in getting IEEE to issue a standard for it – IEEE Std 1154-1991. He gave me a copy of the standard.

I should have been clued in by the fact that he also gave me an errata sheet not much shorter than the standard. But the full horror did not come home to me until I sat down and had a good look at both documents – and, friends, PILOT’s design was exceeded in awfulness only by the sloppiness and vagueness of its standard. Even after the corrections.

Continue reading

Sep 24

Dilemmatizing the NRA

So, the Washington Post publishes yet another bullshit article on gun policy.

In this one, the NRA is charged with racism because it doesn’t leap to defend the right of black men to bear arms without incurring a lethal level of police suspicion.

In a previous blog post, I considered some relevant numbers. At 12% of the population blacks commit 50% of violent index crimes. If you restrict to males in the age range that concentrates criminal behavior, the numbers work out to a black suspect being a a more likely homicidal threat to cops and public safety by at least 26:1.

Continue reading

Sep 18

Thinking like a master programmer, redux

Yes, there was a bug in my vint64 encapsulation commit. I will neither confirm nor deny any conjecture that I left it in there deliberately to see who would be sharp enough to spot it. I will however note that it is a perfect tutorial example for how you should spot bugs, and why revisions with a simple and provable relationship to their ancestors are best

Continue reading

Sep 17

Thinking like a master programmer

To do large code changes correctly, factor them into a series of smaller steps such that each revision has a well-defined and provable relationship to the last.

(This is the closest I’ve ever come to a 1-sentence answer to the question “How the fsck do you manage to code with such ridiculously high speed and low defect frequency? I was asked this yet again recently, and trying to translate the general principle into actionable advice has been on my mind. I have two particular NTPsec contributors in mind…)

So here’s a case study, and maybe your chance to catch me in a mistake.

Continue reading

Sep 13

Trials of the Beast

This last week has not been kind to the Great Beast of Malvern. Serenity is restored now, but there was drama and (at the last) some rather explosive humor.

For some time the Beast had been having occasional random flakeouts apparently related to the graphics card. My monitors would go black – machine still running but no video. Some consultation with my Beastly brains trust (Wendell Wilson, Phil Salkie, and John D. Bell) turned up a suitable replacement, a Radeon R360 R7 that was interesting because it can drive three displays (I presently drive two and aim to upgrade).

Last Friday I tried to upgrade to the new card. To say it went badly would be to wallow in understatement. While I was first physically plugging it in, I lost one of the big knurled screws that the Beast’s case uses for securing both cards and case, down behind the power supply. Couldn’t get it to come out of there.

Then I realized that the card needed a PCI-Express power tap and oh shit the card vendor hadn’t provided one.

Much frantic running around to local computer stores ensued, because I did not yet know that Wendell had thoughtfully tucked several spares of the right kind of cable behind the disk drive bays when he built the Beast. Which turns out to matter because though the PCI-E end is standardized, the power supply end is not and they have vendor-idiosyncratic plugs.

Eventually I gave up and tried to put the old card back in. And that’s when the real fun began. I broke the retaining toggle on the graphics card’s slot while trying to haggle the new card out. When I tried to boot the machine with the old card plugged back in, my external UPS squealed – and then nothing. No on-board lights, no post beep, no sign of life at all. I knew what that meant; evidently either the internal PSU or the mobo was roached.

Continue reading

Aug 18

Some aphorisms on software development methodology

The net benefit of having anything that can be called a software development methodology is in inverse proportion to the quality of your developers – and possibly inverse-squared.

Any software-development methodology works well on sufficiently small projects, but all scale very badly to large ones. The good ones scale slightly less badly.

One thing all methodologies tagged “agile” have in common is that they push developers away from the median. Competent developers work better; mediocre developers don’t change; incompetent ones get worse.

Software metrics have their uses. Unfortunately, their principal use is to create the illusion that you know what’s going on.

Structured development methodologies have their uses, too. Unfortunately, their principal use is to create an illusion of control.

Trust simple, crude metrics over complex ones because the simple ones are less brittle. KLOC is best, though a poor best.

Agile development is efficient only in environments where the cost of flag days is low. Otherwise, slow down, and take time to think and write about your architecture.

Good programmers are difficult to control; great ones are nearly impossible to control. Different methodologies require different kinds and degrees of control; match them to your developers wisely.

Process is not a good substitute for judgment; when you have to use it as one, your project is probably too large. Sometimes this is unavoidable, but don’t fool yourself about what that will cost you.

The difference between O(n**2) and O(n log n) really matters. It’s why lots of small teams working coordinated small projects works better than one big team building a monolith.

A million dollars is roughly a 55-gallon oil drum full of five-dollar bills. Large-scale software development is such a difficult and lossy process that it’s like setting fire to several of these oil drums and hoping a usable product flutters out of the smoke. If this drives you to despair, find a different line of work.

Jul 22

Big fish, small pond?

As of tonight, I have a new challenge in my life.

Sifu Dale announced tonight that next year he plans to send a team to next year’s Quoshu, a national-level martial-arts competition held every year in northern Maryland in late summer.

I told Sifu I want to go and compete in weapons, at least. He was like, “Well, of course,” as though he’d been expecting it. Which is maybe a little surprising and flattering considering I’ll be pushing 60 then.

It’ll mean serious training for the next year, and maybe a pretty humiliating experience if it turns out I’m too old and slow. But I want to try, because it’s national-level competition against fighters from dozens of different styles, and…frankly, I’m tired of not having any clear idea how good I am. Winning would be nice, but what I really want is to measure myself against a bigger talent pool.children tracker

Continue reading

Jul 03

Count the SKUs

The Washington Post is running a story alleging that surveys show gun ownership in the U.S,. is at a 40-year low. I won’t link to it.

This is at the same time gun sales are at record highs.

The WaPo’s explanation, is, basically, that all these guns are being bought by the same fourteen survivalists in Idaho.

Mine is that the number of gun owners with a justified fear that “surveys” are a data-gathering tool for confiscations is also at a record high, and therefore that the number lying to nosy strangers about having no guns is at a record high. videochat

I think there’s a way to discriminate between these cases on the evidence.

Continue reading

Jun 25

More scenes from the life of a system architect

Haven’t been blogging for a while because I’ve been deep in coding and HOWTO-writing. Follows the (slightly edited) text of an email I wrote to the NTPsec devel list that I I think might be of interest to a lot of my audience.

One of the questions I get a lot is: How do you do it? And what is “it”, anyway? The question seems like an inquiry into the mental stance that a systems architect has to have to do his job.

So, um, this is it. If you read carefully, I think you’ll learn a fair bit even if you haven’t a clue about NTP itself.

Continue reading

May 04

NTPsec dodges 8 of 11 CVEs because we’d pre-hardened the code

While most of the NTPsec team was off at Penguicon, the NTP Classic people shipped a release patched for eleven security vulnerabilities in their code. Which might have been pretty embarrassing, if those vulnerabilities were in our code, too. People would be right to wonder, given NTPsec’s security focus, why we didn’t catch all these sooner.

In fact, we actually did pre-empt most of them. The attack surface that eight of these eleven security bugs penetrate isn’t present at all in NTPsec. The vulnerabilities were in bloat and obsolete features we’ve long since removed, like the Mode 7 control channel.

I’m making a big deal about this because it illustrates a general point. One of the most effective ways to harden your code against attack – perhaps the most effective – is to reduce its attack surface.

Thus, NTPsec’s strategy all along has centered on aggressive cruft removal. This strategy has been working extremely well. Back in January our 0.1 release dodged two CVEs because of code we had already removed. This time it was eight foreclosed – and I’m pretty sure it won’t be the last time, either. If only because I ripped out Autokey on Sunday, a notorious nest of bugs.

Simplify, cut, discard. It’s often better hardening than anything else you can do. The percentage of NTP Classic code removed from NTPsec is up to 58% now, and could easily hit 2/3rds before we’re done,

May 03

With a little help from my friends

I had a great time at Penguicon 2016, including face time with a lot of the people who help out on my various projects. There are a couple of thoughts that kept coming back to me during these conversations. One is “It is good, having so many impressively competent friends.”

The other is that without me consciously working at it, an amazing support network has sort of materialized around me – people who believe in the various things I’m trying to do and encourage them by throwing hardware and money and the occasional supportive cheer at me.

Because I didn’t consciously try to recruit these people, it’s easy for me to miss how collectively remarkable they are and how much they contribute until several of them concentrate in one place as happened at Penguicon.

Where I thought: “I’ve been taking these people a bit for granted. I should do better.”

So here, in no particular order, is a (partial) list of people who are really helping. It focuses on those who were at Penguicon and are A&D regulars, so I may have left off some people that would belong on a more complete list.

Continue reading

Apr 27

Where your donations go (#1)

Because people do in fact drop money in my PayPal and Patreon accounts, I think a a decent respect to the opinions of mankind requires that I occasionally update everyone on where the money goes. First in an occasional series,

Recently I’ve been buying Raspberry Pi GPS HATs (daughterboards with a GPS and real-time clock) to go with the Raspberry PI 3 Dave Taht dropped on me. Yesterday morning a thing called an Uputronics GPS Extension Board arrived from England. A few hours ago I ordered a cheap Chinese thing obviously intended to compete with the Adafruit GPS HAT I bought last week.

The reason is that I’m working up a very comprehensive HOWTO on how to build a Stratum 1 timeserver in a box. Not content to merely build one, I’m writing a sheaf of recipes that includes all three HATs I’ve found and (at least) two revisions of the Pi.

What makes this HOWTO different from various build pages on this topic scattered around the Web? In general, the ones I’ve found are well-intended but poorly written. They make too many assumptions, they’re tied to very specific hardware types, they skip “obvious” steps, they leave out diagnostic details about how to tell things are going right and what to do when things go wrong.

My goal is to write a HOWTO that can be used by people who are not Linux and NTP experts – basically, my audience is anyone who could walk into a hackerspace and not feel utterly lost.

Also, my hope is that by not being tightly tied to one parts list this HOWTO will help people develop more of a generative understanding of how you compose a build recipe, and develop their own variations.

I cover everything, clear down to how to buy a case that will fit a HAT. And this work has already had some functional improvements to GPSD as a side effect.

I expect it might produce some improvements in NTPsec as well – our program manager, A&D regular Mark Atwood, has been smiling benignly on this project. Mark’s plan is to broadcast this thing to a hundred hackerspaces and recruit the next generation of time-service experts that way.

Three drafts have already circulated to topic experts. Progress will be interrupted for a bit while I’m off at Penguicon, but 1.0 is likely to ship within two weeks or so.

And it will ship with the recipe variations tested. Because that’s what I do with your donations. If this post stimulates a few more, I’ll add an Odroid C2 (Raspberry Pi workalike with beefier hardware) to the coverage; call it a stretch goal.