Feb 23

Announcing loccount 2.0 – now up to 74 languages

I just released the 2.0 version of loccount.

This is a major release with many new features and upgrades. It’s gone well beyond just being a faster, cleaner, bug-fixed port of David A. Wheeler’s sloccount. The count of supported languages is now up to 74 from sloccount’s 30. But the bigger change is that for 33 of those languages the tool can now deliver a statement count (LLOC = Logical Lines Of Code) as well as opposed to a line count (SLOC = Source Lines of Code, ignoring whitespace and comments)

To go with this, the tool can now perform COCOMO II cost and schedule estimation based on LLOC as well as COCOMO I based on SLOC.

The manual page includes the following cautions:

Continue reading

Jan 29

RISC-V is doing disruption right

I’ve recently become aware of RISC-V.

Verry innterresting.

Technical introduction here (somewhat out of date; hardware support is broader and deeper now, and I have seen video of a full Linux port running Doom), but the technicalia is not mostly where I’m going with this post.

I’m seeing a setup for a potentially classic disruption from below here. And not mainly because this instruction set is well designed, though it is. Simple, clean, orthogonal – it makes my compiler-jock heart happy; writing a code generator for it would be fun. If I needed to, but there’s already an LLVM back end for it.

And that points at what’s really interesting about RISC-V. whoever is running their product strategy has absorbed the lessons of previous technology disruptions and is running this one like a boss.

Continue reading

Jan 13

A martial artist looks at swordfighting in the movies

I was reminded, earlier today, that one of the interesting side effects of knowing something about hand-to-hand and contact-weapons-based martial arts makes a big difference in how you see movies.

Most people don’t have that knowledge. So today I’m going to write about the quality of sword choreography in movies, and how that has changed over time, from the point of view of someone who is an experienced multi-style martial artist in both sword and empty hand. I think this illuminates a larger story about the place of martial arts in popular Western culture.

Continue reading

Dec 24

Pessimism about parallelism

Massive concurrency and hardware parallelism are sexy topics in the 21st century. There are a couple of good reasons for this and one rather unfortunate one.

Two good reasons are the combination of eye-catching uses of Graphics Processing Units (GPUs) in games and their unexpected secondary uses in deep-learning AI – these exploit massive hardware parallelism internally. The unfortunate reason is that single-processor execution speeds hit a physics wall in about 2006. Current leakage and thermal runaway issues now sharply limit increases in clock frequency, and the classic way out of that bind – lowering voltage – is now bumping up against serious quantum-noise issues.

Hardware manufacturers competing for attention have elected to do it by putting ever more processing cores in each chip they ship and touting the theoretical total throughput of the device. But there have also been rapidly increasing amounts of effort put into pipelining and speculative execution techniques that use concurrency under the hood in attempts to make the serial single processors that programmers can see crank instructions more rapidly.

The awkward truth is that many of our less glamorous computing job loads just can’t use visible concurrency very well. There are different reasons for this that have differing consequences for the working programmer, and a lot of confusion abroad among those reasons. In this episode I’m going to draw some distinctions that I hope will help all of us think more clearly.

First, we need to be clear about where harnessing hardware parallelism is easy and why that seems to be the case. We look at computing for graphics, neural nets, signal processing, and Bitcoin mining, and we see a pattern: parallelizing algorithms work best on hardware that is (a) specifically designed to execute them, and (b) can’t do anything else!

We also see that the inputs to the most successful parallel algorithms (sorting, string matching, fast-Fourier transform, matrix operations, image reverse quantization, and the like) all look rather alike. They tend to have a metric structure and an implied distinction between “near” and “far” in the data that allows it to be carved into patches such that coupling between elements far from each other is negligible.

In the terms of an earlier post on semantic locality, parallel methods seem to be applicable mainly when the data has good locality. And they run best on hardware which – like like the systolic-array processors at the heart of GPUs – is designed to support only “near” communication, between close-by elements.

By contrast, writing software that does effective divide-and-conquer for input with bad locality on a collection of general-purpose (Von Neumann architecture) computers is notoriously difficult.

We can sum this up with a heuristic: Your odds of being able to apply parallel-computing techniques to a problem are inversely proportional to the degree of irreducible semantic nonlocality in your input data.

Another limit on parallel computing is that some important algorithms can’t be parallelized at all – provably so. In the blog post where I first explored this territory I coined the term “SICK algorithm”, with the SICK expanded to “Serial, Intrinscally – Cope, Kiddo!” Important examples include but are not limited to:¬†Dijkstra’s n-least-paths algorithm; cycle detection in directed graphs (with implications for 3-SAT solvers); depth first search; computing the nth term in a cryptographic hash chain; network-flow optimization.

Bad locality in the input data is implicated here, too, especially in graph- and tree-structure contexts. Cryptographic hash chains can’t be parallelized because their entries have to be computed in strict time order – a strictness which is actually important for validating the chain against tampering.

There’s a blocking rule here: You can’t parallelize if a SICK algorithm is in the way.

We’re not done. There are at least two other classes of blocker that you will frequently hit.

One is not having the right tools. Most languages don’t support anything but mutex-and-mailbox, which has the advantage that the primitives are easy to implement but the disadvantage that it induces horrible complexity explosions and is nigh-impossible to model accurately in your head at scales over about four interacting locks.

If you are lucky you may get some use out of a more tractable primitive set like Go channels (aka Communicating Sequential Processes) or the ownership/send/sync system in Rust. But the truth is, we don’t really know what the “right” language primitives are for parallelism on von-Neuman-architecture computers. And there may not even be one right set of primitives; there might be two, three, or more different sets of primitive appropriate for different problem domains but as incommensurable as one and the square root of two. At the present state of the art in 2018 nobody actually knows.

Last but not least, the limitations of human wetware. Even given a tractable algorithm, a data representation with good locality, and sharp tools, parallel programming seems to be just plain difficult for human beings even when algorithm being applied is quite simple. Our brains are not all that good at modelling the simpler state spaces of purely serial programs, and much less so at parallel ones.

We know this because there is plenty of real-world evidence that debugging implementations of parallelizing code is worse than merely _difficult_ for humans. Race conditions, deadlocks, livelocks, and insidious data corruption due to subtly unsafe orders of operation plague all such attempts.

Having a grasp on these limits has, I think, has been growing steadily more important since the collapse of Dennard scaling. Due to all of these bottlenecks in the supply of code that can use multiple cores effectively, some percentage of the multicore hardware out there must be running software that will never saturate its cores; or, to look at it from the other end, the hardware is overbuilt for its job load. How much money and effort are we wasting this way?

Processor vendors would love you to overestimate the functional gain from snazzy new silicon with ever larger multi-core counts; however else will they extract enough of your money to cover the eye-watering cost of their chip fabs and still make a profit? So there’s a lot of marketing push out there that aims to distract capacity planners from ever wondering when those gains are real.

And, to be fair, some places they are. The kind of servers that live in rack mounts and handle hundreds of thousands of concurrent transactions per second probably have their core count matched to their job load fairly well. Smartphones or embedded systems, too – in both these extreme cases a lot of effort goes into minimizing build costs and power budgets, and that’s going to exert selective pressure against overprovisioning.

But for typical desktop and laptop users? I have dark suspicions. It’s hard to know, because we’ve been collecting real performance gains due to other technology changes like the shift from spinning-rust to solid-state mass storage. Gains like that are easy to mistake for an effect of more CPU throughput unless you’re profiling carefully.

But here’s the shape of my suspicion:

1. For most desktop/laptop users the only seriously parallel computing that ever takes place on their computers is in their graphics chips.

2. More than two processor cores is usually just wasteful hotrodding. Operating systems may be able to parcel out applications between them, but the general run of application software is unable to exploit parallelism and it is rare for most users to run enough different processor-hungry applications simultaneously to saturate their hardware that way.

3. Consequently, most of the processing units now deployed in 4-core-and-up machines are doing nothing most of the time but generating waste heat.

My regulars include a lot of people who are likely to be able to comment intelligently on this suspicion. It will be interesting to see what they have to say.

UPDATE: A commenter on G+ points out that one interesting use case for multicores is compiling code really quickly. Source for a language like C has good locality – it can be compiled in well-separated units (source files) into object files that are later joined by a linker.

Dec 16

The blues about the blues

Some kinds of music travel well – they propagate out of their native cultures very readily. American rock music and European classical music are obvious examples; they have huge followings and expert practitioners pretty much everywhere on earth that’s in contact with civilization.

Some…don’t travel well at all. Attempts to imitate them by people who aren’t native to their home culture seldom succeed – they fall afoul of subtleties that a home-country connoisseur can hear but not explain well, or at all. The attempts may be earnestly polished and well meant, but in some ineffable way they lack soul. American blues music and to a lesser but significant extent jazz are like this, which is all the more interesting because they’re close historical and genetic kin to rock.

Why am I thinking about this? Because one of the things that YouTube’s recommender algorithms make easy (and almost inevitable) is listening to strings of musical pieces that fit within what the algorithms recognize as a genre. I’ve noticed that the places where its genre recognition is most likely to break down are correlated with whether the genre travels well. So whatever I’m noticing about that distinction is not just difficult for humans but for machine learning as well, at least at current state of the art.

Most attempts at blues by non-Americans are laughable – unintentional parodies by people trying for the real thing. Not all; there was an older generation of British and Irish musicians who immersed in the form in the early Sixties and grokked it well enough to bring it back to the U.S., completely transforming American rock in the process. There are, for some reason, a small handful of decent blues players in Holland. But elsewhere, generative understanding of the heart of the blues is so rare that I was utterly gobsmacked when I found it in Greece.

I don’t know for sure, not being a home-country connoisseur, but I strongly suspect that Portuguese fado is like this. I have a pretty good ear and readily synchronize myself to different musical styles; I can even handle exotica like Indian microtones decently. But I wouldn’t go near fado, I sense a grave risk that if I tried any actual Portuguese fado fan would be politely suppressing a head-shaking he-really-don’t-get-it reaction the same way I usually have to when I listen to Eurojazz.

And Eurojazz players have a better frequency of not ludicrously failing than Euro blues players! Why? I don’t know. I can only guess that the recognition features of “real” jazz are less subtle than for “real” blues, and imitators are thus less likely to slide into unintentional parody. But since I can’t enumerate those recognition features this remains a guess. I do know timing is part of it, and there are uses of silence that are important. Eurojazz tends to be too busy, too slick.

If it’s any consolation to my non-American readers, Americans don’t automatically get it either. My own beloved wife, despite being musically talented, doesn’t have the ear – blues doesn’t speak to her, and if she were unwise enough to try to imitate it she would doubtless fail badly.

One reason I’m posting this is that I hope my commenters might be able to identify other musical genres that travel very poorly – I want to look for patterns. Are there foreign genres that Americans try to imitate and don’t know they’re botching?

And now a different kind of blues about the blues…

There’s an unacknowledged and rather painful truth about the blues, which is that that the primitive Delta versions blues fans are expected to revere are in many ways not as interesting as what came later, out of Chicago in particular. Monotonous, repetitive lyrics, primitive arrangements…but there’s a taboo against noticing this so strong that it took me over forty years to even notice it was there, and I might still not have if I hadn’t spent two days immersed in the rootsiest examples I could find on YouTube.

I found that roots blues is surrounded by a haze of retrospective glorification that (to my own shock!) it too often fails to deserve. And of course the obvious question is “Why?”. I think I’ve figured it out, and the answer is deeply sad.

It’s because, if you notice that later, more evolved and syncretized versions of the blues tend to be more interesting, and you say so, you risk making comparisons that will be interpreted as “white people do it better than its black originators”. And nobody wants that risk.

This came to me as I was listening to a collection of blues solos by Gary Moore, a now-deceased Irishman who played blues with both real heart and a pyrotechnic brilliance you won’t find in Robert Johnson or (one of my own roots favorites) John Lee Hooker. And found myself flinching from the comparison; took me an act of will to name those names just now, even after I’d been steeling myself to it.

Of course this is not a white > black thing; it’s an early vs. late thing. Recent blues players (more likely to be white) have the history of the genre itself to draw on. They have better instruments – Gary Moore’s playing wouldn’t be possible without Gary Moore’s instrument, you can get more tone colors and dynamic range out of a modern electric guitar than you could out of a wooden flattop with no pickups. Gary Moore grew up listening to a range of musical styles not accessible to an illiterate black sharecropper in 1930 and that enriched his playing.

But white blues players may be at an unfair disadvantage in the reputational sweepstakes forever simply because nobody wants to takes the blues away from black people. That would be a particularly cruel and wrong thing to do given that the blues originated as a black response to poverty and oppression largely (though not entirely) perpetrated by white people.

Yes, the blues belongs to all of us now – it’s become not just black roots music but American roots music; I’ve jammed onstage with black bluesmen and nobody thought that was odd. Still, the shadow of race distorts our perceptions of it, and perhaps always will.

Dec 04

The curious case of the missing accents

I have long been a fan of Mark Twain. One of the characteristics of his writing is the use of “eye dialect” – spellings and punctuation intended to phoneticize the speech of his characters. Many years ago I noticed a curious thing about Twain’s eye dialect – that is, he rendered few or no speech differences between Northern and Southern characters. His Northerners all sounded a bit Southern by modern standards, and his Southerners didn’t sound very Southern.

The most obvious possible reason for this could have been that Twain, born and raised in Missouri before the Civil War, projected his own border-state dialect on all his characters. Against this theory I could set the observation that Twain was otherwise a meticulously careful writer with an excellent ear for language, making that an unlikely sort of mistake for him. My verdict was: insufficient data. And I didn’t think the question would ever be resolvable, Twain having died when sound recording was in its infancy.

Then I stumbled over some fascinating recordings of Civil War veterans on YouTube. There’s Confederate “General” Julius Howell Recalls the 1860s from 1947. And 1928-1934: Recollections of the US Civil War. And here’s what jumped out at me…

Continue reading

Nov 27

SRC, four years later

Four years ago, I wrote an entire version-control system in a 14-hour burst of inspiration. It’s a small, lightweight tool designed for solo single-file projects that allows several histories to coexist in a single directory – good for /etc files, HOWTOs, or that script collection in your ~/bin directory.

I wasn’t certain, at the time, that the concept would prove out as a production tool for anyone but me. But it did. Here are some statistics: Over 4 years, 21 point releases, 644 commits, 11 committers. Six issues filed by five different users, 20 merge requests. I know of about half a dozen users who’ve raised their hands on IRC or in blog comments. Code has about quintupled in size from the first alpha release (0.1, 513 lines) to 2757 lines today.

That is the statistical profile of a modest success – in fact the developer roster is larger than I realized before I went back through the logs. The main thing looking at the history reveals is that there’s a user community out there that has been sending a steady trickle of minor bug reports and enhancement requests over the whole life of the project. This is a lot more encouraging than dead air would be.

Of course I don’t now how many total users SRC has. But we can base a guess on fanout patterns observed when other projects (usually much larger ones) have done polls to try to measure userbase size. A sound extrapolation would be somewhere between one and two orders of magnitude more than have made themselves visible – so, somewhere between about 200 and 2000.

(There seems to be something like an exponential scaling law at work here. For random open source project X old enough to have passed the sudden-infant-death filter, if there’s an identifiable core dev group in the single-digit range you can generally expect the casual contributors to be about 10x more and the userbase to be at least 100x more.)

SRC has held up pretty well as a design exercise, too. I’ve had complaints about minor bugs in the UI, but nobody bitching about the UI itself. Credit to the Subversion developers I swiped most of the UI design from; their data model may be obsolete, but nobody in VCS-land has done better at UI and I was at least smart enough not to try.

2.7KLOC is nicely compact for an entire version-control system supporting both RCS and SCCS back ends. I don’t expect it to get much larger; there are only two minor items left on the to-do list, neither of which should add significant lines of code.

Today I’m shipping 1.21. With gratitude to everyone that helped improve it.

Nov 22

Contemplating the cute brick

Some years ago I predicted that eventually the core of your desktop PC would morph into a physically tiny compute engine that would merge with your smartphone, talking through standard ports and cables to full-sized peripherals like a keyboard and (a too large to be portable) flatscreen.

More recently I examined the way that compute bricks – small-form-factor fanless PCs running low-power chips – have been encroaching on the territory of traditional tower PCs. Players in this space include Jetway, Logic Supply, Partaker, and Shuttle. Poke a search engine with “fanless PC” to get good hits.

I have a Jetway running production in my basement; it’s my Internet-facing mail- and web-server. There’s a second one I have set up with Devuan that I haven’t assigned a role to yet; I may use it as a backup host.

These compute bricks are a station on the way to my original prediction, because they get consumers used to thinking of their utility machines as small compute nodes attached to human-sized peripheral hardware that may have a longer lifetime than the compute node itself.

At the lowest end of the compute-brick class are little engines like the Raspberry Pi. And right above it is something slightly different – bricks with a fan, active cooling enabling them to run the same chips used in tower PCs.

Of course the first machine in this class was the Apple Mac Mini, but it dead-ended years ago for reasons that aren’t Apple’s fault. It was designed before SSDs were really a thing and has spinning-rust-centric design assumptions in its DNA; thus, it’s larger, louder, noisier and waaay more expensive than a Jetway-class brick. Apple must never have sold very many of them; we can tell this by the fact that the product went four years between refreshes.

On the other hand, a couple days ago I dropped in a replacement for my wife’s aging tower PC. It’s an Intel NUC, a brick-with-fan, but unlike the Mac Mini it seems to have been designed from the start around the assumption that its mass storage would be SSD. As such, it achieves what the Mac Mini didn’t quite; it opens a new front in the ephemeralization wars.

Continue reading

Nov 18

Stop whining and get the job done

I’ve been meaning to do something systematic about losing my overweight for some time. last Thursday I started the process by seeing an endocrinologist who specializes in weight management.

After some discussion, we developed a treatment plan that surprised me not at all. I’m having my TSH levels checked to see if the hypothyroidism I was diagnosed with about a year ago is undertreated. It is quite possible that increasing my levothyroxin dose will correct my basal metabolic rate to something closer to the burn-food-like-a-plasma-torch level it had when I was younger, and I’ll shed pounds that way.

The other part is going on a low-starch, high protein calorie-reduction diet, aiming for intake of less than 1500 calories a day. Been doing that for nine days now. Have lost, according to my bathroom scale, about ten pounds.

I’d have done this sooner if I knew it was so easy. And that’s what I’m here to blog about today.

Continue reading

Oct 27

On the Squirrel Hill shooting

To my Jewish friends and followers:

I’m grieving with you today. I know the neighborhood where Tree of Life synagogue sits – it’s a quiet, well-off, slightly Bohemian ‘burb with a lot of techies living in it.

I’m not Jewish myself, but I figured out a long time ago that any society which abuses its Jews – or tolerates abuse of them – is in the process of flushing itself down the crapper. The Jews are almost always the first targets of the enemies of civilization, but never the last.

But I’m not posting to reply only with words.

Any Jew who can get close enough to me in realspace for it to be practical and asks can have from me free instruction in basic self-defense with firearms and anti-active-shooter tactics. May no incident like this ever occur again – but if it does, I would be very proud if one of my students took down the evildoer before it reached bloodbath stage.

Oct 22

How to write narrative documentation

The following is a very lightly edited version of email I wrote to my apprentice Ian Bruene after he wrote documentation for his new Kommandant project that was, alas, as awful as I generally expect from programmers. I’m not training Ian for mere coding competence; he’s too talented for that and anyway I have higher standards. This is my way of insisting that he do documentation well – and it was he who suggested it would make a good blog post.

Continue reading

Oct 08

Reposurgeon’s Excellent Journey and the Waning of Python

Time to make it public and official. The entire reposurgeon suite (not just repocutter and repomapper, which have already been ported) is changing implementation languages from Python to Go. Reposurgeon itself is about 50% translated, with pretty good unit-test coverage. Three of my collaborators on the project (Daniel Brooks, Eric Sunshine, and Edward Cree) have stepped up to help with code and reviews.

I’m posting about this because the pressures driving this move are by no means unique to the reposurgeon suite. Python, my favorite working language for twenty years, can no longer cut it at the scale I now need to operate – it can’t handle large enough working sets, and it’s crippled in a world of multi-CPU computers. I’m certain I’m not alone in seeing these problems; if I were, Google, which used to invest heavily in Python (they had Guido on staff there for a while) wouldn’t have funded Go.

Some of Python’s issues can be fixed. Some may be unfixable. I love Guido and the gang and I am vastly grateful for all the use and pleasure I have gotten out of Python, but, guys, this is a wake-up call. I don’t think you have a lot of time to get it together before Python gets left behind.

I’ll first describe the specific context of this port, then I’ll delve into the larger issues about Python, how it seems to be falling behind, and what can be done to remedy the situation.

Continue reading

Oct 02

Rule-swarm attacks can outdo deep reasoning

It not news to readers of this blog that I like to find common tactics and traps in programming that don’t have names and name them. I don’t only do this because it’s fun. When you have named a thing you give your brain permission to reason about it as a conceptual unit. Bad jargon obfuscates, map hiding territory; good jargon reveals, aiding reflection on and and improvement of your practice.

In my last post I coined “shtoopid problem”. It went viral; every programmer has hit this, and it’s useful to have the term because you can attach to it recognition rules and tactics for escaping such traps. (And not only in programming; consider kafkatrapping).

Today’s invention is the term “rule-swarm attack”. It’s derived from the military term “swarm attack” and opposed to “deep reasoning”, “structural analysis” and “generative rules”. I’ll explain it and provide some case studies.

Continue reading

Sep 27

Solving shtoopid problems

There is a kind of programming trap I occasionally fall into that is so damn irritating that it needs a name.

The task is easy to specify and apparently easy to write tests for. The code can be instrumented so that you can see exactly what is going on during every run. You think you have a complete grasp on the theory. It’s the kind of thing you think you’re normally good at, and ought to be able to polish off in 20 LOC and 45 minutes.

And yet, success eludes you for an insanely long time. Edge cases spring up out of nowhere to mug you. Every fix you try drags you further off into the weeds. You stare at dumps from the instrumentation until you’re dizzy and numb, and no enlightenment occurs. Even as you are bashing your head against a wall of incomprehension, consciousness grows that when you find the solution, it will be damningly simple and you will feel utterly moronic, like you should have gotten there days ago.

Welcome to programmer hell. This is your shtoopid problem.

Continue reading

Sep 23

On holy wars, and a plea for peace

I just posted the following to the Linux kernel mailing list.

Most of you know that I have spent more than a quarter century analyzing the folkways of the hacker culture as a historian, ethnographer, and game theorist. That analysis has had large consequences, including a degree of business and mainstream acceptance of the open source way that was difficult to even imagine when I first presented “The Cathedral and the Bazaar” back in 1997.

I’m writing now, from all of that experience and with all that perspective, about the recent flap over the new CoC and the attempt to organize a mass withdrawal of creator permissions from the kernel.

I’m going to try to keep my personal feelings about this dispute off the table, not because I don’t have any but because I think I serve us all better by speaking as neutrally as I can.

First, let me confirm that this threat has teeth. I researched the relevant law when I was founding the Open Source Initiative. In the U.S. there is case law confirming that reputational losses relating to conversion of the rights of a contributor to a GPLed project are judicable in law. I do not know the case law outside the U.S., but in countries observing the Berne Convention without the U.S.’s opt-out of the “moral rights” clause, that clause probably gives the objectors an even stronger case.

I urge that we all step back from the edge of this cliff, and I weant to suggest a basis of principle on which settlement can be negotiated.

Before I go further, let me say that I unequivocally support Linus’s decision to step aside and work on cleaning up his part of the process. If for no other reason than that the man has earned a rest.

But this leaves us with a governance crisis on top of a conflict of principles. That is a difficult combination. Fortunately, there is lots of precedent about how to solve such problems in human history. We can look back on both tragic failures and epic successes and take lessons from them that apply here.

To explain those lessons, I’m going to invite everybody to think like a game theorist for a bit.

Every group of humans trying to sustain cooperation develops an ethos, set of norms. It may be written down. More usually it is a web of agreements that one has to learn by observing the behavior of others. The norms may not even be conscious; there’s a famous result from experimental psychology that young children can play cooperative games without being able to articulate what their rules are…

Every group of cooperating humans has a telos, a mutually understood purpose towards which they are working (or playing). Again, this purpose may be unwritten and is not necessarily even conscious. But one thing is always true: the ethos derives from the telos, not the other way around. The goal precedes the instrument.

It is normal for the group ethos to evolve. It will get pulled in one direction or another as the goals of individuals and coalitions inside the group shift. In a well-functioning group the ethos tends to evolve to reward behaviors that achieve the telos more efficiently, and punish behaviors that retard progess towards it.

It is not normal for the group’s telos – which holds the whole cooperation together and underpins the ethos – to change in a significant way. Attempts to change the telos tend to be profoundly disruptive to the group, often terminally so.

Now I want you to imagine that the group can adopt any of a set of ethoi ranked by normativeness – how much behavior they require and prohibit. If the normativeness slider is set low, the group as a whole will tolerate behavior that some people in it will consider negative and offensive. If the normativeness level is set high, many effects are less visible; contributors who chafe under restriction will defect (usually quietly) and potential contributors will be deterred from joining.

If the normativeness slider starts low and is pushed high, the consequences are much more visible; you can get internal revolt against the change from people who consider the ethos to no longer serve their interests. This is especially likely if, bundled with a change in rules of procedure, there seems to be an attempt to change the telos of the group.

What can we say about where to set the slider? In general, the most successful – most inclusive – cooperations have a minimal ethos. That is, they are just as normative as they must be to achieve the telos, *and no more so*. It’s easy to see why this is. Pushing the slider too high risks internal factional strife over value conflicts. This is worse than having it set too low, where consensus is easier to maintain but you get too little control of conflict between *individuals*.

None of this is breaking news. We cooperate best when we live and let live, respecting that others may make different choices and invoking the group against bad behavior only when it disrupts cooperative success. Inclusiveness demands tolerance.

Strict ethoi are typically functional glue only for small groups at the margins of society; minority regious groups are the best-studied case. The larger and more varied your group is, the more penalty there is for trying to be too normative.

What we have now is a situation in which a subgroup within the Linux kernel’s subculture threatens destructive revolt because not only do they think the slider been pushed too high in a normative direction, but because they think the CoC is an attempt to change the group’s telos.

The first important thing to get is that this revolt is not really about any of the surface issues the CoC was written to address. It would be maximally unhelpful to accuse the anti-CoC people of being pro-sexism, or anti-minority, or whatever. Doing that can only inflame their sense that the group telos is being hijacked. They make it clear; they signed on to participate in a meritocracy with reputation rewards, and they think that is being taken way from them.

One way to process this complaint is to assert that the CoC’s new concerns are so important that the anti-CoC faction can be and should be fought to the point where they withdraw or surrender. The trouble with this way of responding is that it *is* in fact a hijacking of the group’s telos – an assertion that we ought to have new terminal values replacing old ones that the objectors think they’re defending.

So a really major question here is: what is the telos of this subculture? Does the new CoC express it? Have the objectors expressed it?

The question *not* to get hung up on is what any individual’s choice in this matter says about their attitude towards, say, historically underepresented minorities. It is perfectly consistent to be pro-tolerance and pro-inclusion while believing *this* subculture ought to be all about producing good code without regard to who is offended by the process. Not every kind of good work has to be done everywhere. Nobody demands that social-justice causes demonstrate their ability to write C.

That last paragraph may sound like I have strayed from neutrality into making a value claim, but not really. It’s just another way of saying that different groups have different teloi, and different ethoi proceeding from them. Generally speaking (that is, unless it commits actual crimes) you can only judge a group by how it fulfills its own telos, not those of others.

So we come back to two questions:

1. What is our telos?

2. Given our telos, do we have the most inclusive (least normative) ethos possible to achieve it?

When you have an answer to that question, you will know what we need to do about the CoC and the “killswitch” revolt.

Email archive thread at: https://lkml.org/lkml/2018/9/23/212

Sep 13

Hacker culture and the politics of process defense

In my last two blog posts, on the attempted hijack of the Lerna license and speech suppression in the Python documentation, I have both urged the hacker culture to stay out of political issues and urged what some people will interpret as “political” stance with regard to political correctness and “diversity”-driven speech demands.

The expected “gotcha!” comments that “ESR is saying hacker projects should stay clear of politics while arguing politics” have duly followed. While the way this sort of objection is usually posed barely rises above the level of a stupid rhetorical trick, there is an actual issue of principle here that deserves exploring.

Continue reading

Sep 12

Slaves to speech suppression are masters of nothing

Comes the news that the Python project has merged a request to partially eliminate the terms “master” and “slave” from its documentation on “diversity” grounds. Sensibly, Guido van Rossum at least refused to sever the project from uses of those terms in documentation of the underlying Unix APIs.

I wish Guido had gone further and correctly shitcanned the change request as political bullshit up with which he will not put. I will certainly do that if a similar issue is ever raised in one of my projects.

The problem here is not with the object-level issue of whether the terms “master” and “slave” might be offensive to some people. It’s with the meta-level of all such demands. Which the great comedian George Carlin once summed up neatly as follows: “Political correctness is fascism pretending to be manners.”

That is, the demand for suppression of “politically” offensive terms is never entirely or usually even mostly about reducing imputed harms. That is invariably a pretense covering a desire to make speech and thought malleable to political control. Which is why the first and every subsequent attempt at this kind of entryism needs to be kicked in the teeth, hard.

Continue reading

Aug 29

Non-discrimination is a core value of open source

Today I learned that something called the Lerna project has added a codicil to its MIT license denying the use of its software to a long list of organizations because it disagrees with a political choice those organizations have made.

Speaking as one of the original co-authors of the Open Source Definition, I state a fact. As amended, the Lerna license is no longer conformant with the OSD. It has specifically broken compliance with clause 5 (“No Discrimination Against Persons or Groups”).

Accordingly, Lerna has defected from the open-source community and should be shunned by anyone who values the health of that community. I will not contribute to their project, and will urge others not to, until and unless this change is rescinded.

We wrote Clause 5 into the OSD for a good reason. Exclusions and carve-outs like Lerna’s, if they became common, would create tremendous uncertainty about the ethics and even the legality of code re-use. Suppose I were to take a snippet from Lerna code and re-use it in a project that (possibly without my knowledge) was deployed by one of the proscribed organizations; what would my ethical and legal exposure be?

It gets worse. Suppose I write code that happened to be identical, or very similar to, portions of Lerna? Could anyone make a case that I was in violation of their license? It is definitely unsafe when a question like that turns on facts of knowledge and intent no one outside a putative violator’s skull can know for certain.

The Lerna project’s choice is, moreover, destructive of one of the deep norms that keeps the open-source community functional – keeping politics separated from our work. If we do not maintain that norm, we risk fractionating into a collection of squabbling tribes arguing particularisms and unable to sustain really large-scale cooperation.

I would consider such a disintegration not merely unpleasant but actually dangerous to civilization, which relies on us for an increasing portion of its critical infrastructure. Accordingly, we need to cooperate more, not less.

That, in turn, means that, even as we may hold strong individual opinions about issues like those motivating Lerna’s proscription list, we need to be more neutral and non-discriminatory in our collective behavior about such issues, not less.

Updare: Less than 24 hours after I posted this, the license change was revoked and its committer expelled from the project. This is how sanity wins – one fightback at a time.

Aug 22

Unix != open source

Yesterday a well-meaning hacker sent me a newly-recovered koan of Master Foo in which an angry antagonist berated Master Foo for promoting an ethic of open-source software at the expense of programmers’ livelihoods.

Alas, I knew at once that he had been misled by a forgery, or perhaps some dreadful chain of copying errors, at whatever venerable monastic library had been the site of his research. Not because the economics was wrong – Master Foo persuades the antagonist that his assumption is in error – but because the koan conflates two things that were not the same. Actually, at least three things that are not the same.

Eighteen years into the third millennium, long after the formative events of Master Foo’s time, many people fail to understand how complex and contingent the relationship between the Unix tradition and the open-source ethos actually was in the old days. Too readily we project today’s conditions backwards in a way that impedes understanding of history.

Here’s how it was…

Continue reading