Jul 29

Right back at ya, Captain

Last Saturday morning in San Diego I had breakfast with Steven den
Beste, the redoubtable captain of U.S.S. Clueless. One of the
side-effects of that meeting was a long
of open-source development. Herewith my response.

Steve and I agree on the scaling problem that has pushed software
development efforts to the ragged edge of what is sustainable even by
corporations with lots of money. Software project sizes are roughly
doubling every eighteen months, and for reasons Steve alluded to the
expected bug count per thousand lines is actually rising.

My assertion is that software development has reached a scale at
which (a) even large corporations can often no longer afford to field
enough developers to be effective at today’s project scales, and (b)
traditional methods of software quality assurance (ranging from formal
methods to internal walkthroughs) are no longer effective. The only
development organizations that seem to thrive on today’s complexity
regime are open-source teams.

Note that I am not claiming that open source is a silver bullet for
the software-complexity problem. There are no silver bullets, no
permanent solutions. What I am claiming is that at the
leading edge of large-scale software, closed-source development
doesn’t work any more. The future belongs to open source plus
whatever other practices and technologies we learn to use with
it to develop at ever-higher scales of complexity.

Steve’s analysis of the open-source phenomenon is very intelligent,
but doesn’t quite understand either the mode of organization, the
associated technology, or the factional politics within the movement.
Diagnostic of the slight disconnect is when he writes “For [the
zealots], the only true “Open Source” is governed by the strong form
of the GPL, and all other forms and licenses are harmful dilution of
the concept.” In fact, the people he’s talking about reject the term
“open source” entirely and insist on the ideologically-loaded term
“free software”.

A more serious error is when he writes “It is plausible that an OSS
project would require each participant to sign an NDA before being
given access to the source.” It is not plausible. The licenses
and community values of the open-source community would not permit this.
His two bullet points characterizing open source are missing its most
important characteristic: the entire practice is designed to facilitate
scrutiny by people with no institutional or contract relationship to the core
development team. The astringent effect of peer review by people who
have nothing to lose by reporting bugs is precisely the
point of the whole game.

Steve doesn’t undertand the importance or the power of this effect. This
slightly skews his whole essay; much of it is talking past what open-source
people do, rather than addressing us. He’s also unaware of a lot of the
real-world evidence for the success of the method. Some of the things he
thinks are technologically or economically impossible are actually being
done, routinely.

He’s correct when he says that most contributors are self-selected and
self-motivated. He overestimates the cost of training newbies, though. They
self-train; normally, the first time a core developer hears from a newbie
is typically when the newbie sends a patch — self-evidence that the newbie
has already acquired a critical level of knowledge about the
software. The “sink or swim” method turns out to work, and work well.

It’s incorrect to imply, as he does, that open-source development
is unsustainable because the people doing it are flaky amateurs.
Steve hasn’t absorbed the implications of the Boston Consulting
Group study that shows that about 40% of contributors to core projects
are professionals getting paid for working on open source by patrons
who need to use the results. In fact, what the open-source community
is evolving into is something very like a huge machine for bringing
newbies into apprenticeship contact with experienced developers and
professionalizing both groups.

He also writes “OSS by its nature tends to be reactive rather than
predictive. It doesn’t look into the future, try to predict a problem
which doesn’t exist now but will exist then, and be ready with a
solution. Rather, it tends to see problems that exist now and work on
solutions for them.” This is false — or, at any rate, no more true
than it is for closed-source development.

The open-source community built the Web and the Internet before it
had acquired a name for itself and full consciousness of its own
practices. Today, the cutting-edge work in operating systems
languages, desktop user interfaces, relational databases and many
other areas is being done either within the open-source community or
in cooperation with it by academics. These prodigious efforts of
imagination dwarf any “prediction” produced by closed-source software
development in the last two decades.

Steve’s “open source is reactive” claim strikes me as ironically
funny, because I can remember when the standard knock on my crowd was
that we’re great at innovation but can’t actually field product. How
quickly they forget…

He’s right enough about the difficulty of planning and high cost
of face-to-face meetings, though. These are real problems. It’s
a testimony to the power of our practices that we manage to ship large
volumes of high-quality software despite these obstacles.

What Steve called “player-killer” tactics have been tried — there
was a famous incident a few years back in which a TCP-wrappers
distribution was Trojaned. The crack was detected and the community
warned within hours. The black hats don’t seem to bother trying this
any more; our reaction time is too fast for that game to be very
rewarding. The technical design of Linux helps here in ways that
I won’t go into here — suffice it to say that it’s intrinsically
much harder to get a Trojan to do anything interesting than it
is under Windows or other single-user operating systems.

So far, the supply of open-source developers seems to be pretty
elastic — we’re not limited much by lacking bodies. Other factors
loom much larger; patents, the DMCA, intrinsically hard technical
problems. I don’t understand why this is as well as I’d like to, but
the facts are undeniable; the community is ten times the size my
wildest high-end scenarios predicted a decade ago and seems to be
growing faster as it gets larger.

Steve’s whole argument that open-source can’t win in embedded
systems is very curious, since it predicts exactly the opposite of
what is actually happening out there. Linux is taking over in
embedded systems — in fact, many observers would say it has already
won that space. If Steve had worked in the field within the last
three years he would probably know this.

Here are some data about the demand; the only non-general-purpose
open-source software magazine in existence is the Linux Embedded
Systems Journal. Open-source embedded developers like Monta Vista
Software are bucking the recession by growing like crazy. The first
cell-phone prototype running entirely open-source software just
entered beta testing.

I was in California to meet Steve partly because Real Networks
wanted me to be on stage when they announced the open-sourcing of
their RTSP engine. Their CEO, Rob Glaser, was quite frank about the
immediate business reasons: they needed to get ports to forty
different Nokia cellphones and just couldn’t figure out how to muster
the resources for that short of inviting every stakeholder on the
planet to hack the problem. Scaling bites. Hard.

In fact, some of the very characteristics that Steve thinks make
embedded systems like cellphones safe for closed development seems to
be the factors that are driving increased open-sourcing. The close
tie to hardware actually decreases the value of secrecy,
because it means the software is typically not easily re-usable by
hardware competitors. Thus open sourcing is often a great way to
recruit help from customer engineers without a real downside risk of

In fact, it’s an open secret in the industry that the most
important reason most closed-source embedded and driver software
remains closed is not nerves about plagiarism but fear of patent
audits on the source code. Graphics-card manufacturers, in
particular, routinely swipe patented techniques from their competitors
and bury them in binaries. (This is generally believed to be the
reason nVidia’s drivers aren’t open.)

Another trend that’s driving Linux and open-sourcing in embedded
stuff is the shift from specialty embedded 8-bit processors to 32-bit
chips with general-purpose architectures. Turns out the development
costs for getting stuff to run on the 8-bit chips are sickeningly high
and rising — partly because the few wizards who can do good work on
that hardware are expensive. The incremental cost for
smarter hardware has dropped a lot; it’s now cheaper to embed
general-purpose chips running Linux because it means you have a
larger, less expensive talent pool that can program them. Also,
when your developers aren’t fighting hardware limits as hard,
you get better time to market (which, as Steve observes, is

Steve is right about the comparative difficulty of applying
open-source methods to vertical applications. But the difficulty is
only comparative; it’s happening anyway. The metalab archive carries
a point-of-sale system for pizza parlors. I know of another case in
which a Canadian auto dealership built specialized accounting software
for their business and open-sourced it. The reasons? Same as usual;
they wanted to lay off as much as possible of the development and
maintainance cost on their competitors.

This is the same co-opetition logic that makes the Apache Software
Foundation work — it’s just as powerful for vertical apps, though
less obviously so. Each sponsoring company sees a higher payoff from
having the software at a small fraction of the manpower cost for a
complete in-house development. The method spreads risk in a way
beneficial to all parties, too, because the ability of separate
companies to sustain development tends to be uncorrelated — unless
they all sink, the project endures.

The way to solve the problem of not exposing your business logic to
competitors is to separate your app into an open-source engine and a
bunch of declarative business-rule schemas that you keep secret.
Databases work this way, and websites (the web pages and CGIs are the
schema). Many vertical apps can be partitioned this way too — in
fact, for things like tax-preparation software they almost have to be,
because the complexity overhead of hacking executable code every time
the rules change is too high.

Steve thinks the differences between Apache and Mozilla are bigger
than they are. In fact, the core groups of both projects are
full-time pros being funded by large users of the software.

So, let’s address Steve’s objections point by point:

For embedded software, OSS has the following problems:

  • It can’t be scheduled; timely delivery can’t be relied

    Timely delivery can’t be relied on for any software; see
    De Marco and Lister’s excellent book Peopleware: Productive
    Projects and Teams
    on the delusion of deadlines, especially
    the empirical evidence that the “wake me up when it’s done” strategy
    of not setting them actually gets your project done faster (also the
    implication of a recent Harvard Business School study of software
    project outcomes).

    Open source is at least not noticeably worse than closed-source on this
    axis. Arguably it’s better, because the rapid release cycles allow users
    to pick up on project results as soon as they’re good enough.

  • Debugging requires access to custom hardware which usually
    can’t easily be accessed across the net.

    There aren’t good solutions to this problem yet, but the increasing
    use of “overpowered” 32-bit processors using standard busses is
    tending to reduce it in scope. The development tools and interface
    hardware used in embedded stuff are rapidly getting more generic and closer
    to what’s used in general-purpose computers.

  • Active participation even for junior people requires substantial
    amounts of project-specific knowledge which isn’t easily acquired,
    especially remotely.

    This one puzzles me, because I think Steve ought to be right about
    it — but I’m not hearing the kinds of noises that I’d hear if it were
    slowing down the move to Linux and open source significantly.

    At least part of the answer is that embedded-systems work is
    getting de-skilled in a particular sense — more of it’s being done by
    application specialists who are training up to the required level of
    programming, rather than programmers who have acquired expensive
    application-specific knowledge.

  • A great deal of proprietary information is usually involved in
    the process, and if that’s released the company can be seriously

    It’s a question of tradeoffs. As RealNetworks found out when
    costing its Nokia contract, the choice is increasingly between giving
    up control of some of your proprietary IP and being too resource-bound
    to ship at all.

    There is no market for secrecy. There’s a market for product. If
    you can’t ship product, or your customers aren’t confident that you
    can maintain it after shipping, all that proprietary IP amounts to is
    a millstone around your neck.

    There will be more stories like RTSP in the future. Count on it.
    In fact, the day will come when most of your contract partners simply
    won’t accept the business risks of having someone else hold
    proprietary rights on the embedded software they use.

  • It’s nearly impossible to do embedded software without
    common impromptu face-to-face meetings with co-workers, either to ask
    questions or to brainstorm. Doing this electronically is sufficiently
    different as to not be practical.

    Yeah. They used to think that about operating systems, too. Obviously
    the Linux kernel is impossible, and therefore doesn’t exist.

    (At which point Oolon Colluphid disappeared in a puff of logic.)

For vertical apps, the objections are:

  • Security, security, security. You want me to trust my
    billing system to code written by anyone who happens to come along and
    volunteer to work on it, without any kind of check of credentials or
    checks on trustworthiness?

    One of the lessons the business world has been absorbing is that
    open-source projects are dramatically more secure than their
    closed-source competition — anybody who compares the Bugtraq records
    on Apache vs. ISS defacements, or Linux vs. Windows remote exploits,
    will notice that real fast.

    It’s not hard to understand why this is — I’ve found that even
    corporate executives grok the theory pretty quickly. I won’t do the whole
    argument here, but this article on Kerckhoff’s
    holds the crucial clue. When you rely on the obscurity of source
    code for security, it means that the bad guys find the bugs faster than
    you can plug them — there are more of them, and they have entropy on
    their side. Open source evens the odds for the good guys.

  • Recruitment: for most of the kind of people involved in
    OSS, vertical apps are boring. (Unless they want to figure out how to
    steal from it.)

    This remains a problem. On the other hand, open source makes it
    easier to train domain specialists to be good enough programmers to
    get the job done. It’s easier for physicists to learn to hack than
    it is for hackers to learn physics.

  • It takes a lot of knowledge of the specific aspects of the
    problem to make a significant contribution, which means things like
    observing the actual process of guests checking in at the front desk
    of the hotel.

    This just reinforces the tendency for vertical-app developers to be
    obsessives about something else who learn to program, rather than obsessives
    about programming who learn something else.

    Professional programmers tend to bridle at this thought. Well, better
    learn to live with it. As software becomes more pervasive, the amount
    of it done by application-specialist “amateurs” is going to increase.

  • The industry is full of horror stories of vertical apps
    which ran badly over budget and over schedule; the idea scares the
    hell out of business people. They’re unlikely to be very enthused by
    the use of a process which by its nature *cannot* be reliably
    scheduled. (Remember that Mozilla ran two years long.)

    Schedules — and the belief that deadlines make software happen
    faster — are a delusion in the mind of management, one not supported
    by the actual evidence about project outcomes. This delusion is
    so entrenched that managers fail to interpret the 70% rate of
    project failures correctly. It’s as if people were so determined
    to believe the Earth is flat that they ignore what their eyes tell
    them when ships sink over the horizon.

    No software larger than toy programs can be scheduled.
    Tactics aimed at doing so normally have the actual effect of
    increasing the time to market. `Aggressive’ schedules
    effectively guarantee failure. The sooner we learn these objective
    truths, and that the illusion of control that schedules give is not
    worth the real costs, the sooner rates of outright project failure
    will dip below 70%.

    Go read Peopleware. Now.

For short life apps:

  • Schedule is everything. If you’re six months late, you’re dead.

    See above. There are reasons open sourcing is less applicable to short-life
    applications, but this turns out not to be one of them.

  • Secrecy is everything else. If you’re on time but your
    competitor knows what you’re doing a year ahead, he’ll wipe you

    This argument has more force for short-life apps than for Steve’s other
    categories, but remember that increasingly the alternative to open source
    is not being able to ship at all. Your competitor is in the same boat
    you are.

  • How do you make money selling what anyone can get for free
    from any developer? If your product was developed out in the open, who
    exactly buys it afterwards?

    Steve has a stronger point here. It’s one that people used to
    think applied to almost all software, but which turns out to be mainly
    a problem for short-life apps. Actually the distinguishing
    characteristic isn’t expected lifetime per se, but something
    correlated with it — whether the product needs continued downstream
    work (maintainance and upgrades) or not.

    Long-life, high-maintainance apps create niches for service businesses.
    That’s the main way you make money in an open-source world. It’s
    harder to make that work with a short-life app. Sometimes it’s
    impossible. Life is hard.

For long life apps:

  • Will the participants be willing to work on what our
    marketing analysis says we need, or will they insist that they know
    what is required and try to add that instead? We don’t need feature
    creep, or people trying to change the direction we’re moving.

    In open-source projects, the function of “marketing analysis” tends to
    be taken be direct interaction with the user community. We find we
    do better work without a bunch of marketroids getting between us and
    our customers.

  • There is major learning curve involved in making a
    reasonable contribution to these kinds of programs; you don’t learn
    how a circuit board router works in a few days of study. In most cases
    you have to be conversant with the way that the package’s customers do
    what they do, and most programmers don’t know these things and can’t
    easily learn them.

    See my previous remarks about application specialists and the
    democratization of programming. And every time you’re tempted to
    say “But they couldn’t possibly get away with that in application
    area X” remember that they once said that about all the areas where
    open source now dominates.

    It’s just not smart to bet against the hackers. Not smart at all.
    We generally end up having the last laugh on the naysayers. As recently
    as 1990, “serious analysts” laughed at the idea of ubiquitous Internet.
    As late as 1996, they said Unix was dead. We showed them — and there
    are more of us now, with better tools, than ever.

Steve is right that one of the most effective ways to head off bugs
is to have a core group of professional engineers do a clean design.
Where he’s mistaken is in believing this truth has anything to tell
us about open vs. closed development. Us open-source guys, it turns
out, are really good at clean design.

This something to do with the fact that, as individuals, we tend to
be exceptionally capable and self-motivated — an elite selected by
dedication to the art of programming. It has more to do with not
having managers and marketroids pissing in the soup constantly,
telling us what tools to use, imposing insane deadlines, demanding
endless checklist features that don’t actually benefit anyone.

But mostly it has to do with the ruthless, invaluable pressure of
peer review — the knowledge that every design decision we make will
be examined by thousands of people who may well be smarter than we
are, and if we fail the test our effort will be pitilessly
discarded. In that kind of environment, you get good or you get

Blogspot comments

Jul 21

Run Silent, Go Feep

Warning: The following blog entry provides way more than the
recommended daily allowance of geeking. If you don’t have a serious
propeller-head streak, surf outta here now before it’s too

I’m mainly a software guy, but occasionally I build PCs for fun.
Design them, rather; the further away I stay from actual hardware the
happier it usually is for everybody. Last year, I designed an Ultimate
Linux Box
; the good folks at Los Alamos Computers built it and
will cheerfully sell you one. It was a successful design in most
respects, but unpleasantly noisy. This year, as we do the 2002
refresh, I’m going to be working hard at getting the most noise
reduction I can without sacrificing performance. I’m experimenting
now with ways and means.

So I spent a couple of hours today disassembling the case of my
wife Cathy’s machine (minx.thyrsus.com) and lining three sides of it
with Dynamat, a kind of stick-on
rubber acoustic insulation often used in car-stereo installations.
The malevolent god that normally attends me when I futz with hardware
must have been off tormenting some other hapless ex-mathematician; no
hardware was destroyed, no blood was shed, and I’m typing this on the
selfsame reassembled machine.

Minx is a pretty generic mid-tower system made with cheap Taiwanese
parts in mid-2002 by my local hole-in-the-wall computer shop: I
spent only $150 to have it built, recycling a few parts from an only
slightly older machine. It has a 300W power supply, Athlon 950 mobo
with stock CPU cooler fan, one 80mm case fan, 7200RPM ATA drive. I
succeeded in lining both 14″-square side panels and the case top; this
used up the 4’sq piece I bought so efficiently that there was only
about 10″sq in two small piece left over. I used those to cover the
only exposed solid section of the back panel.

If you want try this yourself, the tools I found useful were a
utility knife and a metal footrule, the latter useful both for
measuring to fit and as a cutting guide.

I took before and after measurements with the db meter. dbA scale,
measurements made with the probe one inch above the center-rear edge
of the case.

Machine off: 44dbA
Machine on, before: 63dbA
Machine on, after: 61dbA

In other words, only a 2dbA drop — marginal when you consider
that the meter is only rated 1.5dB accurate! but it’s worth bearing in
mind that the scale is logarithmic; 2dbA is more than it looks like.

I have studio-engineer ears and sensitive musician fingers. I took
before-and-after measurements with those, too, listening to the sound
tambre and feeling for case resonance.

My ears tell me that the box is only slightly quieter, but the noise
spectrum has changed. The proportion of high-frequency noise has
dropped; more of what I’m hearing is white noise due to turbulant
airflow, less is bearing noise. This is a good change even if total
emission hasn’t dropped much.

My fingers tell me that the amount of case resonance has dropped quite
dramatically, especially on the side panels.

Was it worth doing? I am not sure. There would probably be more
benefit on a system emitting more bearing noise from 10K or 15Krpm
drives. On this one, I think the power supply is emitting most of
the noise, and acoustic lining can’t do much against that.

In fact, my clearest take-away from this is that the big gains in
noise reduction on conventional PCs are likely to come from
obsessing about power-supply engineering — including details like
whether the fan blows through a slotted grille or a cutout with a
wire-basket finger guard (the latter will generate less turbulence

I’d like to retrofit minx with a Papst 12dbA muffin fan and see if
that makes a measurable difference. But the best change would
probably be one of the Enhance
300W PSUs that are supposed to only emit 26dbA. I’ll bet that would
win big.

Blogspot comments

Jul 18

The Non-Portability of Barbecue

(Originally titled: Travelling in Texas)

I was on the road in Texas last week, addressing Linux user groups in Dallas and Austin. I always enjoy visiting Texas. It’s a big, wide-open place full of generous people who cultivate a proper appreciation of some of my favorite things in life — firearms, blues guitar, and pepper sauces.

And, of course, one of the biggest things Texas has going for it is barbecue. And not the pallid imitation served up by us pasty-faced Yankees here where I live (near Philadelphia, PA) but the real thing. Barbecue, dammit. Red meat with enough fat on it to panic a health-foodist right out of his pantywaist, slow-cooked in a marinade sweeter than a mother’s kiss and eaten with sauces hot enough to peel paint. Garnish with a few extra jalapenos and coleslaw and wash it down with cheap soda, lemonade, or beer. Food of the gods.

I swear your testosterone level goes up just smelling this stuff. After a few mouthfuls of Rudy’s carnivoral bliss you’ll be hankerin’ to cultivate a drawl, wear a Stetson and drive a pickup truck with a gun rack. (I draw the line at country music, though. A man’s got to have some standards.)

At a real barbecue joint like Rudy’s (“Worst barbecue in Texas!”) they serve you piles of beef, pork and chicken wrapped in butcher paper in a plastic basket. No plates, just more butcher paper and bread. And, unfortunately, the bread is where this gustatory Nirvana nearly crashes back to earth. Because the bread at real barbecue places is invariably utter crap — spongy sliced white with all the taste of building insulation.

Here in Philadelphia we can’t make barbecue worth a damn, but we know better than to put a hot sandwich on American bread. One of our regional-food glories is the Philly steak sandwich, fried beef and onions and mushrooms (and usually cheese, but I don’t eat cheese) nestled in a foot-long Italian roll. The bread is important. It’s tasty, it’s chewy, it’s got a crust on it. It’s worthy of respect. One of the reasons you can’t get a decent steak sandwich more than fifty miles from Billy Penn’s hat is that bread. It depends on an Italian baking tradition that just doesn’t exist outside the mid-Atlantic metroplex, and is found in its highest form only in Philly and South Jersey. Philadelphians laugh at the pathetic imitations of “Philly steaks” offered elsewhere for the same reason Texans laugh at barbecue made north of the Mason-Dixon line. And both groups are right to laugh. It just ain’t the same.

Every time I order up a mess of barbecue at a place like Rudy’s or County Line or Dick’s Last Resort I think to myself “Someday, one of these barbecue outfits has got to start offering decent bread. Their sales would go through the roof.” I’ve been waiting for the market to correct this problem for more than twenty years now — and it hasn’t happened. And thereby hangs a mystery.

The mystery is the curious persistence of regional food differences in a country with cheap transport and the best communications network in the world. There are places in the U.S. where you can reliably get really good bread — mostly the coastal metroplexes. There are places you can get real barbecue, in the heartland South and Southwest. And these zones just don’t overlap. (Yes, they have a gourmet-bread bakery in Austin. I suspect, if I went there, I’d find it a lot like the Chinese food in Ann Arbor — impressive to the locals, maybe, but only because their standards are so low.)

I could multiply examples. Sourdough bread — I’ve had it everywhere you can get it and it just doesn’t taste right outside of San Francisco. The East Coast versions are competent, but lack some subtle tang. Yeast strain? Something in the water? Who knows?

Cheesecake. There’s a good one. Anybody who has lived in New York won’t touch most cheesecake made elsewhere at gunpoint, and with good reason. Next to a traditional New-York-style baked cheesecake (the kind you can stand a fork in because it has the approximate density of neutronium) all others are a sort of pathetic, tasteless cheese gelatin. In this case the recipe is clearly what matters.

Or deep-dish pizza. Try to get that done right anywhere but Chicago. Good luck. Actually, the Philly/South Jersey area may be the only other part of the U.S.that can almost make this nut, and our thin-crust pizza is better. But why? Why don’t the good techniques go national and drive out the weaker competition?

The obvious answer would be that nationwide, tastes differ too much for one regional variant to dominate. But many cases there isn’t even any dispute about where the best variant comes from; the superiority of “New York style” cheesecake. for example, is so universally understood that restaurants elsewhere often bill their cheesecake that way even when it’s actually half-composed of “lite” garbage like ricotta or cottage cheese. Nobody who has ever tasted one doubts that Philly steaks are the acme of the art. And nobody — but nobody — who can get both passes up Texas barbecue for what they make in New Haven or Walla Walla.

So you’d think that the market would have propagated Texas slow-cooking, San Francisco yeast starters and the Philly steak roll all over the country by now. But some food technologies travel better than others, and some seem curiously unable to thrive outside their native climes. Cheesecake recipes may survive transmission relatively well, but the mysteries of good barbecue are subtle and deep. Pizzas rely on elaborate oven and dough-mix technology that probably tends to conserve regional variations simply because it’s too capital-intensive to mess with casually.

I’ve meditated on the matter and still can’t decide whether I think that’s a good thing or not. The approved thing for travel writers to do is wax lyrical about the wonderfulness of regional variety, as if it would somehow fail to be an improvement in the world if I could get decent bread with my barbecue. The hell with that kind of sentimentality; I’d rather have a better meal.

But there’s a point buried there somewhere — something that isn’t about the bread or the barbecue, but about what it feels like to sit in a dusty roadside joint like Rudy’s,surrounded by cases of Red Pop and overweight rednecks in tractor caps and checked shirts, with the food of the gods melting in your mouth, and thinking “Damn, this place is tacky, but I hope it lives forever.”

And you know what? I suspect that kind of barbecue joint will live forever, or as close to forever as humans manage, anyway. They’ve probably existed since the first proto-hominids roasted mammoth haunch over a slow fire, washing it down with some badly-made tuber-beer equivalent of Red Pop. And their equivalents will probably persist in the zero-gee arcologies and Dyson spheres of the year 3000. Even if they get hip about the good bread, somewhere in the universe there will always be a Texas. And that’s a good thing.

UPDATE: Some respondents have reminded me of the Piedmont (and specially North Carolina) tradition of pulled-pork barbecue. Let me state for the record that I find it equally delicious. Both the Texas and Piedmont versions are so damn good that there is no call for petty disputation about which is superior. But for those of you who know what I am talking about, I am quite partial to burnt

UPDATE: Jane Galt has commented in her usual witty and illuminating fashion.

UPDATE: The mystery of San Francisco sourdough, was, as it turns out, solved in 1970. You can buy a starter with the proper symbiosis of bacteria and yeast — and, contrary to myth, local bacteria won’t overwhelm it. Of course this makes it harder to understand why the stuff isn’t everywhere…

Blogspot comments

Jul 17

Diet Considered as a Bad Religion

A current New York Times news story, What If It’s All Been A Big Fat Lie, entertainingly chronicles the discovery that low-fat diets are bad for people. More specifically, that the substitution of carbohydrates like bread and pasta and potatoes for meat that we’ve all had urged on us since the early 1980s is probably the cause of the modern epidemic of obesity and the sharp rise in diabetes incidence.

I have long believed that most of the healthy-eating advice we get is stone crazy, and the story does tend to confirm it. One of my reasons for believing this is touched on in the article; what we’re told is good for us doesn’t match what humans “in the wild” (during the 99% of our species history that predated agriculture) ate. The diet our bodies evolved to process doesn’t include things like large amounts of milled grain or other starches. Our hunter-gatherer ancestors ate wild vegetables (especially tubers) and meat whenever they could get it.

I’ve always had to suppress a tendency to laugh rudely when vegeterians touted their diet as “natural”. Vegetarianism is deeply unnatural for human beings; it’s marginally possible in warm climates only (there are no vegetarians in Tibet because the climate kills them), and only possible even there because we’re at the near end of 4,000 years of breeding for high-caloric-value staple crops.

So what’s the natural diet for human beings? Our dentition (both slashing and grinding teeth) and the structure of our digestive system (short colon, no rumen) is intermediate between that of herbivores like cows and obligate carnivores like cats; both systems resemble those of non-specialized omnivores like bears. Actually, the earlier hominids in the human ancestral line were designed for a more vegetarian diet than we; they had large flat molars and powerful jaws designed for grinding seed-cases. The increase in brain size in the hominid line correlates neatly with a shift to a more carnivorous dentitition and skull structure.

Physical anthropologists will tell you that the shift from hunter-gatherer existence to sedentary agriculture enabled human beings to live at higher population densities, but at the cost of a marked deterioration in the health of the average person. The skeletons of agricultural populations are shorter, less robust, and show much more evidence of nutritional diseases relative to their hunter-gatherer ancestors.

For twenty years I’ve consciously been trying to eat what I think of as a caveman diet — heavy on the meat and raw vegetables, very little sugar, light on the starches. I’m a bit overweight now, not seriously so for a 44-year-old man, but enough to notice; what this NYT article tells me is that I didn’t follow my own prescription strictly enough and ate too much bread and potatoes.

But the evolutionary analysis only tells us what we probably should be eating. It doesn’t explain how the modern diet has come to be as severly messed up as it is — nor why the advice we’ve been getting on healthy eating over the last twenty years has been not merely bad but perversely wrong.

The answer is, I think, implicit in the fact that “health food” has a strong tendency to be bland, fibrous, and nasty — a kind of filboid studge that we have to work at convincing ourselves we like rather than actually liking. Which is, if you think about it, nuts. Human food tropisms represent two million years of selective knowledge about what’s good for our bodies. Eating a lot of what we don’t like is far more likely to be a mistake than eating things we do like, even to excess.

Why do we tend to treat our natural cravings for red meat and fat as sins, then? Notice the similarity between the rhetoric of diet books and religious evangelism and you have your answer. Dietary mortification of the flesh has become a kind of secular asceticism, a way for wealthy white people with guilt feelings about their affluence to demonstrate virtue and expiate their imagined trangressions.

Once you realize that dieting is a religion, the irrationality and mutual contradictions become easier to understand. It’s not about what’s actually good for you, it’s about suffering and self-denial and the state of your soul. People who constantly break and re-adopt diets are experiencing exactly the same cycle of secondary rewards as the sinner who repeatedly backslides and reforms.

This model explains the social fact that the modern flavor of “health”-based dietary piety is most likely to be found in people who don’t have the same psychological needs satisfied by an actual religion. Quick now: who’s more likely to be a vegetarian or profess a horror of “junk food” — a conservative Christian heartlander or a secular politically-correct leftist from the urban coasts?

The NYT article tells us that the dominant dietary religion of the last twenty years is cracking — that the weight of evidence against the fat-is-evil/carbs-are-good theory is no longer supportable. Well and good — but it won’t necessarily do us a lot of good to discard this religion only to get stuck with another one.

I say it’s time to give all bossy nutritionists, health-food evangelists and dietary busybodies the heave-ho out of our lives — tell the sorry bitches and bastards to get over themselves and go back to eating stuff that tastes good and satiates. And enjoy the outraged squawking from the dietarily correct — that, my friends, is the music of health and freedom.

Blogspot comment

Jul 03

Winning the War Against Terror

(Final essay of the series.)

In previous essays in this series, I have described
as a warlike and bloody religion subject to periodic fits of
violent fundamentalist revival. I have analyzed the roots of Islamic
terror in the Koranic duty of jihad, and elucidated Osama bin Laden’s
goal as nothing less than the destruction of the West and the establishment
of a global
Islamic theocracy. I have analyzed the reason
Americans have trouble comprehending
the scope of the threat
, and I have explained why Western-style
diplomacy is next
to useless
in this situation. In this final essay I’ll suggest
paths towards a solution.

In order to win, we must begin with realism about the scope of the
war and the objectives of the enemy. We must realize that although in
theory and theology al-Qaeda is making war on the entire infidel West,
in practice they are only interested in attacking the U.S., the
`hyperpower’ that leads it.

There is no possible gain for al-Qaeda in attacking Europe and
risking a change in the pro-Arab, pro-Palestinian tilt of the EU
(which has just resumed support payments to the Palestinian Authority
despite conclusive evidence that the money is diverted to pay for
massacres of Israeli children). Nor can al-Qaeda gain any leverage by
attacks on the remainder of the world. The theaters of the war will
include the U.S. and terrorist base areas in the Islamic arc
stretching from Morocco through the Maghreb through the Middle East to
Pakistan, and perhaps in Indonesia and the Phillipines as well.

To people who view the entire world through the lens of the Western
tradition, the strategy I will outline is doubtless going to sound
bellicose and regressive. It is not; it is founded on a cold-blooded
realization that Arab cultures (and the Arabized cultures of the rest
of the Islamic world) regard victory in war as a sign of Allah’s favor
and regard compromise and concession as a sign of weakness.

The war against Islamic terror must be fought on three levels:
homeland defense, military power projection, and cultural subversion.
We must foil terrorist acts; we must imprison or kill the terrorists
who plan and execute them; and we must dry up the pool of potential
recruits before they become terrorists who can only be stopped by
being imprisoned or killed.

Homeland defense includes all those measures designed to make the
attacks on U.S. civilians less likely to succeed. These will include
conventional police and security measures. It must also include a
revival of the role of the unincorporated militia and the armed
citizen. Al-Qaeda has limited resources, but the advantage of
choosing where they will strike; since the police and military cannot
be everywhere, civilians (like the passengers of flight 93) must take
anti-terrorist defense into their own hands.

Military power projection includes direct military action against
terrorist bases and havens. As an anarchist, I would prefer a world
in which private security agencies under contract to insurance
companies pursued al-Qaeda; persons of some other political persuasions
might propose supranational agencies such as the U.N. Unfortunately,
under the current world system there is no alternative to governments
to do this work. The U.S. has begun it in Afghanistan; the war must
continue in Iraq, and it is likely to encompass Pakistan, Iran, and Saudi
Arabia as well.

The goal of military power projection must be twofold: physical and
psychological. The physical goal must be to destroy the physical
infrastructure of terrorism — the headquarters, bases and training
camps. While this is important, the psychological goal of humiliating
and crushing jihadists is even more important.

Islamic armies and resistance movements are fanatical in attack but
brittle on the defense. When motivated by the conviction that Allah
guides their arm, suicidal bravery is routine.
On the other hand, when the fortunes of a cause decline past a
certain point, Arabs tend to consider the will of Allah to be manifest
and abruptly abandon it. These tendencies form part of the cultural
background that includes even secularized terrorist movements
(such as Yasser Arafat’s al-Fatah) in the Islamic world.

The U.S. was able to exploit this brittleness effectively in
Afghanistan. By moving in overwhelming force when it moved at all,
the U.S. was able to intimidate many warlords affiliated with the
Taliban into switching sides — an important reason the campaign
involved so little actual fighting.

We must repeat this maneuver on a larger scale. We must teach the
Dar-al-Islam to respect and fear the power of the West. We
must not negotiate or offer concessions until it is clear from the
behavior of governments, the umma, and the “Arab street” that the
public will to support jihad has been broken.

Our most important long-term weapon against Islamic terrorism,
however, will be cultural subversion. That is, to break the hold of
the Islamist/jihadist idea on the minds of Muslims. To do this, it
may be necessary to discredit the entirety of Islam; the question
depends on whether any Islamic figure will be clever enough to
construct an interpretation of Islam purged of jihadist tendencies,
and whether that version can propagate and displace the
Sunni-fundamentalist varieties now dominant in the Islamic world.

I can do no better than to quote Michelle Efird, the woman who
inspired my essay
We Are All Jews Now. In private mail afterwards
(quoted with permission) she wrote:

I don’t want to appease them, I don’t want to understand them, I
don’t want to let them reap the benefits of our liberalism while
plotting our destruction. Like most Americans, I would have been more
than happy to let them pretend the last 400 years of progress never
happened, as long as they didn’t force their warped-vision goggles on
anyone else. But since they brought the war to us, let’s pave the
middle east with outlet malls, fast food franchises, and Disney
Mecca. Let’s infect their entire population with personal liberty and
dissension and critical thinking. And if that doesn’t work, let’s
flood them with porn spam.

Osama bin Laden may, in the end, have materialized his own worst
fears. The ideology of jihad has created its mirror and opposite; the
dawning sense that we in the West have the right, the power, and the
duty to wipe bin Laden’s brand of religion from the face of
the earth before it destroys us all.

UPDATE: N.Z. Bear has written an

excellent essay
on memes and cultural subversion.

Blogspot comments