Right back at ya, Captain

Last Saturday morning in San Diego I had breakfast with Steven den
Beste, the redoubtable captain of U.S.S. Clueless. One of the
side-effects of that meeting was a long
of open-source development. Herewith my response.

Steve and I agree on the scaling problem that has pushed software
development efforts to the ragged edge of what is sustainable even by
corporations with lots of money. Software project sizes are roughly
doubling every eighteen months, and for reasons Steve alluded to the
expected bug count per thousand lines is actually rising.

My assertion is that software development has reached a scale at
which (a) even large corporations can often no longer afford to field
enough developers to be effective at today’s project scales, and (b)
traditional methods of software quality assurance (ranging from formal
methods to internal walkthroughs) are no longer effective. The only
development organizations that seem to thrive on today’s complexity
regime are open-source teams.

Note that I am not claiming that open source is a silver bullet for
the software-complexity problem. There are no silver bullets, no
permanent solutions. What I am claiming is that at the
leading edge of large-scale software, closed-source development
doesn’t work any more. The future belongs to open source plus
whatever other practices and technologies we learn to use with
it to develop at ever-higher scales of complexity.

Steve’s analysis of the open-source phenomenon is very intelligent,
but doesn’t quite understand either the mode of organization, the
associated technology, or the factional politics within the movement.
Diagnostic of the slight disconnect is when he writes “For [the
zealots], the only true “Open Source” is governed by the strong form
of the GPL, and all other forms and licenses are harmful dilution of
the concept.” In fact, the people he’s talking about reject the term
“open source” entirely and insist on the ideologically-loaded term
“free software”.

A more serious error is when he writes “It is plausible that an OSS
project would require each participant to sign an NDA before being
given access to the source.” It is not plausible. The licenses
and community values of the open-source community would not permit this.
His two bullet points characterizing open source are missing its most
important characteristic: the entire practice is designed to facilitate
scrutiny by people with no institutional or contract relationship to the core
development team. The astringent effect of peer review by people who
have nothing to lose by reporting bugs is precisely the
point of the whole game.

Steve doesn’t undertand the importance or the power of this effect. This
slightly skews his whole essay; much of it is talking past what open-source
people do, rather than addressing us. He’s also unaware of a lot of the
real-world evidence for the success of the method. Some of the things he
thinks are technologically or economically impossible are actually being
done, routinely.

He’s correct when he says that most contributors are self-selected and
self-motivated. He overestimates the cost of training newbies, though. They
self-train; normally, the first time a core developer hears from a newbie
is typically when the newbie sends a patch — self-evidence that the newbie
has already acquired a critical level of knowledge about the
software. The “sink or swim” method turns out to work, and work well.

It’s incorrect to imply, as he does, that open-source development
is unsustainable because the people doing it are flaky amateurs.
Steve hasn’t absorbed the implications of the Boston Consulting
Group study that shows that about 40% of contributors to core projects
are professionals getting paid for working on open source by patrons
who need to use the results. In fact, what the open-source community
is evolving into is something very like a huge machine for bringing
newbies into apprenticeship contact with experienced developers and
professionalizing both groups.

He also writes “OSS by its nature tends to be reactive rather than
predictive. It doesn’t look into the future, try to predict a problem
which doesn’t exist now but will exist then, and be ready with a
solution. Rather, it tends to see problems that exist now and work on
solutions for them.” This is false — or, at any rate, no more true
than it is for closed-source development.

The open-source community built the Web and the Internet before it
had acquired a name for itself and full consciousness of its own
practices. Today, the cutting-edge work in operating systems
languages, desktop user interfaces, relational databases and many
other areas is being done either within the open-source community or
in cooperation with it by academics. These prodigious efforts of
imagination dwarf any “prediction” produced by closed-source software
development in the last two decades.

Steve’s “open source is reactive” claim strikes me as ironically
funny, because I can remember when the standard knock on my crowd was
that we’re great at innovation but can’t actually field product. How
quickly they forget…

He’s right enough about the difficulty of planning and high cost
of face-to-face meetings, though. These are real problems. It’s
a testimony to the power of our practices that we manage to ship large
volumes of high-quality software despite these obstacles.

What Steve called “player-killer” tactics have been tried — there
was a famous incident a few years back in which a TCP-wrappers
distribution was Trojaned. The crack was detected and the community
warned within hours. The black hats don’t seem to bother trying this
any more; our reaction time is too fast for that game to be very
rewarding. The technical design of Linux helps here in ways that
I won’t go into here — suffice it to say that it’s intrinsically
much harder to get a Trojan to do anything interesting than it
is under Windows or other single-user operating systems.

So far, the supply of open-source developers seems to be pretty
elastic — we’re not limited much by lacking bodies. Other factors
loom much larger; patents, the DMCA, intrinsically hard technical
problems. I don’t understand why this is as well as I’d like to, but
the facts are undeniable; the community is ten times the size my
wildest high-end scenarios predicted a decade ago and seems to be
growing faster as it gets larger.

Steve’s whole argument that open-source can’t win in embedded
systems is very curious, since it predicts exactly the opposite of
what is actually happening out there. Linux is taking over in
embedded systems — in fact, many observers would say it has already
won that space. If Steve had worked in the field within the last
three years he would probably know this.

Here are some data about the demand; the only non-general-purpose
open-source software magazine in existence is the Linux Embedded
Systems Journal. Open-source embedded developers like Monta Vista
Software are bucking the recession by growing like crazy. The first
cell-phone prototype running entirely open-source software just
entered beta testing.

I was in California to meet Steve partly because Real Networks
wanted me to be on stage when they announced the open-sourcing of
their RTSP engine. Their CEO, Rob Glaser, was quite frank about the
immediate business reasons: they needed to get ports to forty
different Nokia cellphones and just couldn’t figure out how to muster
the resources for that short of inviting every stakeholder on the
planet to hack the problem. Scaling bites. Hard.

In fact, some of the very characteristics that Steve thinks make
embedded systems like cellphones safe for closed development seems to
be the factors that are driving increased open-sourcing. The close
tie to hardware actually decreases the value of secrecy,
because it means the software is typically not easily re-usable by
hardware competitors. Thus open sourcing is often a great way to
recruit help from customer engineers without a real downside risk of

In fact, it’s an open secret in the industry that the most
important reason most closed-source embedded and driver software
remains closed is not nerves about plagiarism but fear of patent
audits on the source code. Graphics-card manufacturers, in
particular, routinely swipe patented techniques from their competitors
and bury them in binaries. (This is generally believed to be the
reason nVidia’s drivers aren’t open.)

Another trend that’s driving Linux and open-sourcing in embedded
stuff is the shift from specialty embedded 8-bit processors to 32-bit
chips with general-purpose architectures. Turns out the development
costs for getting stuff to run on the 8-bit chips are sickeningly high
and rising — partly because the few wizards who can do good work on
that hardware are expensive. The incremental cost for
smarter hardware has dropped a lot; it’s now cheaper to embed
general-purpose chips running Linux because it means you have a
larger, less expensive talent pool that can program them. Also,
when your developers aren’t fighting hardware limits as hard,
you get better time to market (which, as Steve observes, is

Steve is right about the comparative difficulty of applying
open-source methods to vertical applications. But the difficulty is
only comparative; it’s happening anyway. The metalab archive carries
a point-of-sale system for pizza parlors. I know of another case in
which a Canadian auto dealership built specialized accounting software
for their business and open-sourced it. The reasons? Same as usual;
they wanted to lay off as much as possible of the development and
maintainance cost on their competitors.

This is the same co-opetition logic that makes the Apache Software
Foundation work — it’s just as powerful for vertical apps, though
less obviously so. Each sponsoring company sees a higher payoff from
having the software at a small fraction of the manpower cost for a
complete in-house development. The method spreads risk in a way
beneficial to all parties, too, because the ability of separate
companies to sustain development tends to be uncorrelated — unless
they all sink, the project endures.

The way to solve the problem of not exposing your business logic to
competitors is to separate your app into an open-source engine and a
bunch of declarative business-rule schemas that you keep secret.
Databases work this way, and websites (the web pages and CGIs are the
schema). Many vertical apps can be partitioned this way too — in
fact, for things like tax-preparation software they almost have to be,
because the complexity overhead of hacking executable code every time
the rules change is too high.

Steve thinks the differences between Apache and Mozilla are bigger
than they are. In fact, the core groups of both projects are
full-time pros being funded by large users of the software.

So, let’s address Steve’s objections point by point:

For embedded software, OSS has the following problems:

  • It can’t be scheduled; timely delivery can’t be relied

    Timely delivery can’t be relied on for any software; see
    De Marco and Lister’s excellent book Peopleware: Productive
    Projects and Teams
    on the delusion of deadlines, especially
    the empirical evidence that the “wake me up when it’s done” strategy
    of not setting them actually gets your project done faster (also the
    implication of a recent Harvard Business School study of software
    project outcomes).

    Open source is at least not noticeably worse than closed-source on this
    axis. Arguably it’s better, because the rapid release cycles allow users
    to pick up on project results as soon as they’re good enough.

  • Debugging requires access to custom hardware which usually
    can’t easily be accessed across the net.

    There aren’t good solutions to this problem yet, but the increasing
    use of “overpowered” 32-bit processors using standard busses is
    tending to reduce it in scope. The development tools and interface
    hardware used in embedded stuff are rapidly getting more generic and closer
    to what’s used in general-purpose computers.

  • Active participation even for junior people requires substantial
    amounts of project-specific knowledge which isn’t easily acquired,
    especially remotely.

    This one puzzles me, because I think Steve ought to be right about
    it — but I’m not hearing the kinds of noises that I’d hear if it were
    slowing down the move to Linux and open source significantly.

    At least part of the answer is that embedded-systems work is
    getting de-skilled in a particular sense — more of it’s being done by
    application specialists who are training up to the required level of
    programming, rather than programmers who have acquired expensive
    application-specific knowledge.

  • A great deal of proprietary information is usually involved in
    the process, and if that’s released the company can be seriously

    It’s a question of tradeoffs. As RealNetworks found out when
    costing its Nokia contract, the choice is increasingly between giving
    up control of some of your proprietary IP and being too resource-bound
    to ship at all.

    There is no market for secrecy. There’s a market for product. If
    you can’t ship product, or your customers aren’t confident that you
    can maintain it after shipping, all that proprietary IP amounts to is
    a millstone around your neck.

    There will be more stories like RTSP in the future. Count on it.
    In fact, the day will come when most of your contract partners simply
    won’t accept the business risks of having someone else hold
    proprietary rights on the embedded software they use.

  • It’s nearly impossible to do embedded software without
    common impromptu face-to-face meetings with co-workers, either to ask
    questions or to brainstorm. Doing this electronically is sufficiently
    different as to not be practical.

    Yeah. They used to think that about operating systems, too. Obviously
    the Linux kernel is impossible, and therefore doesn’t exist.

    (At which point Oolon Colluphid disappeared in a puff of logic.)

For vertical apps, the objections are:

  • Security, security, security. You want me to trust my
    billing system to code written by anyone who happens to come along and
    volunteer to work on it, without any kind of check of credentials or
    checks on trustworthiness?

    One of the lessons the business world has been absorbing is that
    open-source projects are dramatically more secure than their
    closed-source competition — anybody who compares the Bugtraq records
    on Apache vs. ISS defacements, or Linux vs. Windows remote exploits,
    will notice that real fast.

    It’s not hard to understand why this is — I’ve found that even
    corporate executives grok the theory pretty quickly. I won’t do the whole
    argument here, but this article on Kerckhoff’s
    holds the crucial clue. When you rely on the obscurity of source
    code for security, it means that the bad guys find the bugs faster than
    you can plug them — there are more of them, and they have entropy on
    their side. Open source evens the odds for the good guys.

  • Recruitment: for most of the kind of people involved in
    OSS, vertical apps are boring. (Unless they want to figure out how to
    steal from it.)

    This remains a problem. On the other hand, open source makes it
    easier to train domain specialists to be good enough programmers to
    get the job done. It’s easier for physicists to learn to hack than
    it is for hackers to learn physics.

  • It takes a lot of knowledge of the specific aspects of the
    problem to make a significant contribution, which means things like
    observing the actual process of guests checking in at the front desk
    of the hotel.

    This just reinforces the tendency for vertical-app developers to be
    obsessives about something else who learn to program, rather than obsessives
    about programming who learn something else.

    Professional programmers tend to bridle at this thought. Well, better
    learn to live with it. As software becomes more pervasive, the amount
    of it done by application-specialist “amateurs” is going to increase.

  • The industry is full of horror stories of vertical apps
    which ran badly over budget and over schedule; the idea scares the
    hell out of business people. They’re unlikely to be very enthused by
    the use of a process which by its nature *cannot* be reliably
    scheduled. (Remember that Mozilla ran two years long.)

    Schedules — and the belief that deadlines make software happen
    faster — are a delusion in the mind of management, one not supported
    by the actual evidence about project outcomes. This delusion is
    so entrenched that managers fail to interpret the 70% rate of
    project failures correctly. It’s as if people were so determined
    to believe the Earth is flat that they ignore what their eyes tell
    them when ships sink over the horizon.

    No software larger than toy programs can be scheduled.
    Tactics aimed at doing so normally have the actual effect of
    increasing the time to market. `Aggressive’ schedules
    effectively guarantee failure. The sooner we learn these objective
    truths, and that the illusion of control that schedules give is not
    worth the real costs, the sooner rates of outright project failure
    will dip below 70%.

    Go read Peopleware. Now.

For short life apps:

  • Schedule is everything. If you’re six months late, you’re dead.

    See above. There are reasons open sourcing is less applicable to short-life
    applications, but this turns out not to be one of them.

  • Secrecy is everything else. If you’re on time but your
    competitor knows what you’re doing a year ahead, he’ll wipe you

    This argument has more force for short-life apps than for Steve’s other
    categories, but remember that increasingly the alternative to open source
    is not being able to ship at all. Your competitor is in the same boat
    you are.

  • How do you make money selling what anyone can get for free
    from any developer? If your product was developed out in the open, who
    exactly buys it afterwards?

    Steve has a stronger point here. It’s one that people used to
    think applied to almost all software, but which turns out to be mainly
    a problem for short-life apps. Actually the distinguishing
    characteristic isn’t expected lifetime per se, but something
    correlated with it — whether the product needs continued downstream
    work (maintainance and upgrades) or not.

    Long-life, high-maintainance apps create niches for service businesses.
    That’s the main way you make money in an open-source world. It’s
    harder to make that work with a short-life app. Sometimes it’s
    impossible. Life is hard.

For long life apps:

  • Will the participants be willing to work on what our
    marketing analysis says we need, or will they insist that they know
    what is required and try to add that instead? We don’t need feature
    creep, or people trying to change the direction we’re moving.

    In open-source projects, the function of “marketing analysis” tends to
    be taken be direct interaction with the user community. We find we
    do better work without a bunch of marketroids getting between us and
    our customers.

  • There is major learning curve involved in making a
    reasonable contribution to these kinds of programs; you don’t learn
    how a circuit board router works in a few days of study. In most cases
    you have to be conversant with the way that the package’s customers do
    what they do, and most programmers don’t know these things and can’t
    easily learn them.

    See my previous remarks about application specialists and the
    democratization of programming. And every time you’re tempted to
    say “But they couldn’t possibly get away with that in application
    area X” remember that they once said that about all the areas where
    open source now dominates.

    It’s just not smart to bet against the hackers. Not smart at all.
    We generally end up having the last laugh on the naysayers. As recently
    as 1990, “serious analysts” laughed at the idea of ubiquitous Internet.
    As late as 1996, they said Unix was dead. We showed them — and there
    are more of us now, with better tools, than ever.

Steve is right that one of the most effective ways to head off bugs
is to have a core group of professional engineers do a clean design.
Where he’s mistaken is in believing this truth has anything to tell
us about open vs. closed development. Us open-source guys, it turns
out, are really good at clean design.

This something to do with the fact that, as individuals, we tend to
be exceptionally capable and self-motivated — an elite selected by
dedication to the art of programming. It has more to do with not
having managers and marketroids pissing in the soup constantly,
telling us what tools to use, imposing insane deadlines, demanding
endless checklist features that don’t actually benefit anyone.

But mostly it has to do with the ruthless, invaluable pressure of
peer review — the knowledge that every design decision we make will
be examined by thousands of people who may well be smarter than we
are, and if we fail the test our effort will be pitilessly
discarded. In that kind of environment, you get good or you get

Blogspot comments


  1. I agree that “`Aggressive’ schedules effectively guarantee failure.”

    But in real life the programming is usually only part of a business project, and it’s often necessary for the software’s completion date to have a specific relationship to other activities. At the large end, the software may have to be completed before some large-scale construction work is completed. On a smaller scale, production of manuals, promotional and districution have their own lead times and managers need to know first when the software will be stable enough for the technnical writers to use and then when it will be ready for shipping – the best-known example of these issues in computer games.

    And of course sometimes the deadline is beyond the control of the organisation’s management – regulator-driven projects are usually hell.

Leave a comment

Your email address will not be published. Required fields are marked *