Hacking and Refactoring

In 2001, there was a history-making conference of software-engineering
thinkers in Snowbird, Colorado. The product of that meeting was a remarkable
document called the Agile Manifesto,
a call to overturn many of the assumptions of traditional software development.
I was invited to be at Snowbird, but couldn’t make it.

Ever since, though, I’ve been sensing a growing convergence between
agile programming and the open-source movement. I’ve seen agile
concepts and terminology being adopted rapidly and enthusiastically by
my colleagues in open-source-land—especially ideas like
refactoring, unit testing, and design from stories and personas. From
the other side, key agile-movement figures like Kent Beck and Martin
Fowler have expressed strong interest in open source both in published
works and to me personally. Fowler has gone so far as to include
open source on his list of agile-movement schools.

I agree that we belong on that list. But I also agree with
Fowler’s description of of open source as a style, rather than a
process. I think his reservations as to whether open source can be
described as just another agile school are well-founded. There is
something more complicated and interesting going on here. and I
realized when I read Fowler’s description of open source that at some
point I was going to have to do some hard thinking and writing in an
effort to sort it all out.

While doing research for my forthcoming book, The Art of Unix
Programming, I read one particular passage in Fowler’s
Refactoring that finally brought it all home. He
writes:

One argument is that refactoring can be an alternative to up-front
design. In this scenario, you don’t do any design at all. You just
code the first approach that comes into your head, get it working, and
then refactor it into shape. Actually, this approach can work. I’ve
seen people do this and come out with a very well-defined piece of
software. Those who support Extreme Programming often are portrayed
as advocating this approach.

I read this, and had one of those moments where everything comes
together in your head with a great ringing crash and the world assumes
a new shape—a moment not unlike the one I had in late 1996
when I got the central insight that turned into The Cathedral
and the Bazaar. In the remainder of this essay I’m going to
try to articulate what I now think I understand about open source,
agile programming, how they are related, and why the connection should
be interesting even to programmers with no stake in either movement.

Now I need to set a little background here, because I’m going
to need to have to talk about several different categories which are
contingently but not necessarily related.

First, there is Unix programmer. Unix is the operating
system with the longest living tradition of programming and design.
It has an unusually strong and mature technical culture around it, a
culture which originated or popularized many of the core ideas and
tools of modern software design. The Art of Unix
Programming is a concerted attempt to capture the craft wisdom
of this culture, one to which I have successfully enlisted quite a few
of its founding elders.

Second, there is hacker. This is a very complex term, but
more than anything else, it describes an attitude—an
intentional stance that relates hackers to programming and other
disciplines in a particular way. I have described the hacker stance
and its cultural correlates in detail in How To Become A
Hacker.

Third, there is open-source programmer. Open source is a
programming style with strong roots in the Unix tradition and the
hacker culture. I wrote the modern manifesto for it in 1997, The
Cathedral and the Bazaar, building on earlier thinking by
Richard Stallman and others.

These three categories are historically closely related. It is
significant that a single person (accidentally, me) wrote touchstone
documents for the second and third and is attempting a summum
bonum of the first. That personal coincidence reflects a larger
social reality that in 2003 these categories are becoming increasingly
merged — essentially, the hacker community has become the core
of the open-source community, which is rapidly re-assimilating the
parts of the Unix culture that got away from the hackers during
the ten bad years after the AT&T divestiture in 1984.

But the relationship is not logically entailed; we can imagine
a hacker culture speaking a common tongue other than Unix and C (in
the far past its common tongue was Lisp), and we can imagine an
explicit ideology of open source developing within a cultural and
technical context other than Unix (as indeed nearly happened several
different times).

With this scene-setting done, I can explain that my first take on
Fowler’s statement was to think “Dude, you’ve just described
hacking!”

I mean something specific and powerful by this. Throwing together
a prototype and refactoring it into shape is a rather precise
description of the normal working practice of hackers since that
culture began to self-define in the 1960s. Not a complete one, but it
captures the most salient feature of how hackers relate to code. The
open-source community has inherited and elaborated this practice,
building on similar tendencies within the Unix tradition.

The way Fowler writes about design-by-refactoring has two huge
implications for the relationship between open source and agile
programming:

First, Fowler writes as though he didn’t know he was describing
hacking. In the passage, he appears unaware that design by
repeated refactoring is not just a recent practice semi-accidentally
stumbled on by a handful of agile programmers, but one which hundreds
of thousands of hackers have accumulated experience with for over three
decades and have in their bones. There is a substantial folklore, an
entire craft practice, around this!

Second, in that passage Fowler described the practice of hacking
better than hackers themselves have done. Now, admittedly,
the hacker culture has simply not had that many theoreticians, and if
you list the ones that are strongly focused on development methodology
you lose Richard Stallman and are left with, basically, myself and
maybe Larry Wall (author of Perl and occasional funny and illuminating
ruminations on the art of hacking). But the fact that we don’t have a
lot of theoreticians is itself an important datum; we have always
tended to develop our most important wisdoms as unconscious and
unarticulated craft practice.

These two observations imply an enormous mutual potential, a gap
across which an arc of enlightenment may be beginning to blaze. It
implies two things:

First, people who are excited by agile-programming ideas can
look to open source and the Unix tradition and the hackers for the
lessons of experience. We’ve been doing a lot of the stuff the
agile movement is talking about for a long time. Doing it in a
clumsy, unconscious, learned-by-osmosis way, but doing it
nevertheless. I believe that we have learned things that you agile
guys need to know to give your methodologies groundedness. Things
like (as Fowler himself observes) how to manage communication and
hierarchy issues in distributed teams.

Second, open-source hackers can learn from agile programmers
how to wake up. The terminology and conceptual framework of
agile programming sharpens and articulates our instincts. Learning to
speak the language of open source, peer review, many eyeballs, and
rapid iterations gave us a tremendous unifying boost in the late
1990s; I think becoming similarly conscious about agile-movement ideas
like refactoring, unit testing, and story-centered design could be
just as important for us in the new century.

I’ve already given an example of what the agile movement has to
teach the hackers, in pointing out that repeated redesign by
refactoring is a precise description of hacking. Another thing we can
stand to learn from agile-movement folks is how to behave so that we
can actually develop requirements and deliver on them when the
customer isn’t, ultimately, ourselves.

For the flip side, consider Fowler’s anecdote on page 68-69, which
ends “Even if you know exactly what is going on in your system,
measure performance, don’t speculate. You’ll learn something, and
nine times out of ten it won’t be that you were right.” The Unix guy
in me wants to respond “Well, duh!“. In my tribe, profiling
before you speculate is DNA; we have a strong tradition of
this that goes back to the 1970s. From the point of view of any old
Unix hand, the fact that Fowler thought he had to write this down is a
sign of severe naivete in either Fowler or his readership or both.

In reading Refactoring, I several times had the
experience of thinking “What!?! That’s obvious!” closely followed
by “But Fowler explains it better than Unix traditions do…” This may
be because he relies less on the very rich shared explanatory context
that Unix provides.

How deep do the similarities run? Let’s take a look at what the
Agile Manifesto says:

Individuals and interactions over processes and tools. Yeah,
that sounds like us, all right. Open-source developers will toss out
a process that isn’t working in a nanosecond, and frequently do, and take
gleeful delight in doing so. In fact, the reaction against heavyweight
process has a key part of our self-identification as hackers for
at least the last quarter century, if not longer.

Working software over comprehensive documentation. That’s
us, too. In fact, the radical hacker position is that source code of
a working system is its documentation. We, more than any
other culture of software engineering, emphasize program source code as
human-to-human communication that is expected to bind together
communities of cooperation and understanding distributed through time
and space. In this, too, we build on and amplify Unix tradition.

Customer collaboration over contract negotiation. In the
open-source world, the line between “developer” and “customer” blurs
and often disappears. Non-technical end users are represented by
developers who are proxies for their interests—as when, for
example, companies that run large websites second developers to
work on Apache Software Foundation projects.

Responding to change over following a plan. Absolutely.
Our whole development style encourages this. It’s fairly unusual for
any of our projects to have any plan more elaborate than “fix
the current bugs and chase the next shiny thing we see”.

With these as main points, it’s hardly surprising that so many of
the Principles
behind the Agile Manifesto read like Unix-tradition and hacker
gospel. “Deliver working software frequently, from a couple of weeks
to a couple of months, with a preference to the shorter timescale.
Well, yeah—we pioneered this. Or “Simplicity—the art of
maximizing the amount of work not done—is essential.” That’s
Unix-tradition holy writ, there. Or “The best architectures,
requirements, and designs emerge from self-organizing teams.”

This is stone-obvious stuff to any hacker, and exactly the sort of
subversive thinking that most panics managers attached to big plans,
big budgets, big up-front design, and big rigid command-and-control
structures. Which may, in fact, be a key part of its appeal to
hackers and agile developers—because at least one thing that points
agile-movement and open-source people in the same direction is a drive
to take control of our art back from the suits and get out from under
big dumb management.

The most important difference I see between the hackers and the
agile-movement crowd is this: the hackers are the people who never
surrendered to big dumb management — they either bailed out of the
system or forted up in academia or industrial R&D labs or
technical-specialty areas where pointy-haired bosses weren’t permitted
to do as much damage. The agile crowd, on the other hand, seems to be
composed largely of people who were swallowed into the belly of the
beast (waterfall-model projects, Windows, the entire conventional
corporate-development hell so vividly described in Edward Yourdon’s
books) and have been smart enough not just to claw their way out but
to formulate an ideology to justify not getting sucked back in.

Both groups are in revolt against the same set of organizational
assumptions. And both are winning because those assumptions are
obsolete, yesterday’s adaptations to a world of expensive machines and
expensive communications. But software development doesn’t need big
concentrations of capital and resources anymore, and doesn’t need the
control structures and hierarchies and secrecy and elaborate rituals
that go with managing big capital concentrations either. In fact, in
a world of rapid change, these things are nothing but a drag. Thus
agile techniques. Thus, open source. Converging paths to the same
destination, which is not just software that doesn’t suck but a
software-development process that doesn’t suck.

When I think about how the tribal wisdom of the hackers and the
sharp cut-the-bullshit insights of the agile movement seem to be
coming together, my mind keeps circling back to Phil Greenspun’s brief
but trenchant essay Redefining
Professionalism for Software Engineers. Greenspun proposes,
provocatively but I think correctly, that the shift towards
open-source development is a key part of the transformation of
software engineering into a mature profession, with the dedication to
excellence and ethos of service that accompanies professionalism. I
have elsewhere suggested that we are seeing a close historical analog
of the transition from alchemy to chemistry. Secrets leak out, but
skill sustains; the necessity to stop relying on craft secrecy is one
of the crises that occupational groups normally face as they attain
professional standing.

I’m beginning to think that from the wreckage of the software
industry big dumb management made, I can see the outline of a mature,
humane discipline of software engineering emerging — and
that it will be in large part a fusion of the responsiveness and
customer focus of the agile movement with the wisdom and groundedness
of the Unix tradition, expressed in open source.

Blogspot
comments