Oct 22

How to write narrative documentation

The following is a very lightly edited version of email I wrote to my apprentice Ian Bruene after he wrote documentation for his new Kommandant project that was, alas, as awful as I generally expect from programmers. I’m not training Ian for mere coding competence; he’s too talented for that and anyway I have higher standards. This is my way of insisting that he do documentation well – and it was he who suggested it would make a good blog post.

Continue reading

Sep 23

On holy wars, and a plea for peace

I just posted the following to the Linux kernel mailing list.

Most of you know that I have spent more than a quarter century analyzing the folkways of the hacker culture as a historian, ethnographer, and game theorist. That analysis has had large consequences, including a degree of business and mainstream acceptance of the open source way that was difficult to even imagine when I first presented “The Cathedral and the Bazaar” back in 1997.

I’m writing now, from all of that experience and with all that perspective, about the recent flap over the new CoC and the attempt to organize a mass withdrawal of creator permissions from the kernel.

I’m going to try to keep my personal feelings about this dispute off the table, not because I don’t have any but because I think I serve us all better by speaking as neutrally as I can.

First, let me confirm that this threat has teeth. I researched the relevant law when I was founding the Open Source Initiative. In the U.S. there is case law confirming that reputational losses relating to conversion of the rights of a contributor to a GPLed project are judicable in law. I do not know the case law outside the U.S., but in countries observing the Berne Convention without the U.S.’s opt-out of the “moral rights” clause, that clause probably gives the objectors an even stronger case.

I urge that we all step back from the edge of this cliff, and I weant to suggest a basis of principle on which settlement can be negotiated.

Before I go further, let me say that I unequivocally support Linus’s decision to step aside and work on cleaning up his part of the process. If for no other reason than that the man has earned a rest.

But this leaves us with a governance crisis on top of a conflict of principles. That is a difficult combination. Fortunately, there is lots of precedent about how to solve such problems in human history. We can look back on both tragic failures and epic successes and take lessons from them that apply here.

To explain those lessons, I’m going to invite everybody to think like a game theorist for a bit.

Every group of humans trying to sustain cooperation develops an ethos, set of norms. It may be written down. More usually it is a web of agreements that one has to learn by observing the behavior of others. The norms may not even be conscious; there’s a famous result from experimental psychology that young children can play cooperative games without being able to articulate what their rules are…

Every group of cooperating humans has a telos, a mutually understood purpose towards which they are working (or playing). Again, this purpose may be unwritten and is not necessarily even conscious. But one thing is always true: the ethos derives from the telos, not the other way around. The goal precedes the instrument.

It is normal for the group ethos to evolve. It will get pulled in one direction or another as the goals of individuals and coalitions inside the group shift. In a well-functioning group the ethos tends to evolve to reward behaviors that achieve the telos more efficiently, and punish behaviors that retard progess towards it.

It is not normal for the group’s telos – which holds the whole cooperation together and underpins the ethos – to change in a significant way. Attempts to change the telos tend to be profoundly disruptive to the group, often terminally so.

Now I want you to imagine that the group can adopt any of a set of ethoi ranked by normativeness – how much behavior they require and prohibit. If the normativeness slider is set low, the group as a whole will tolerate behavior that some people in it will consider negative and offensive. If the normativeness level is set high, many effects are less visible; contributors who chafe under restriction will defect (usually quietly) and potential contributors will be deterred from joining.

If the normativeness slider starts low and is pushed high, the consequences are much more visible; you can get internal revolt against the change from people who consider the ethos to no longer serve their interests. This is especially likely if, bundled with a change in rules of procedure, there seems to be an attempt to change the telos of the group.

What can we say about where to set the slider? In general, the most successful – most inclusive – cooperations have a minimal ethos. That is, they are just as normative as they must be to achieve the telos, *and no more so*. It’s easy to see why this is. Pushing the slider too high risks internal factional strife over value conflicts. This is worse than having it set too low, where consensus is easier to maintain but you get too little control of conflict between *individuals*.

None of this is breaking news. We cooperate best when we live and let live, respecting that others may make different choices and invoking the group against bad behavior only when it disrupts cooperative success. Inclusiveness demands tolerance.

Strict ethoi are typically functional glue only for small groups at the margins of society; minority regious groups are the best-studied case. The larger and more varied your group is, the more penalty there is for trying to be too normative.

What we have now is a situation in which a subgroup within the Linux kernel’s subculture threatens destructive revolt because not only do they think the slider been pushed too high in a normative direction, but because they think the CoC is an attempt to change the group’s telos.

The first important thing to get is that this revolt is not really about any of the surface issues the CoC was written to address. It would be maximally unhelpful to accuse the anti-CoC people of being pro-sexism, or anti-minority, or whatever. Doing that can only inflame their sense that the group telos is being hijacked. They make it clear; they signed on to participate in a meritocracy with reputation rewards, and they think that is being taken way from them.

One way to process this complaint is to assert that the CoC’s new concerns are so important that the anti-CoC faction can be and should be fought to the point where they withdraw or surrender. The trouble with this way of responding is that it *is* in fact a hijacking of the group’s telos – an assertion that we ought to have new terminal values replacing old ones that the objectors think they’re defending.

So a really major question here is: what is the telos of this subculture? Does the new CoC express it? Have the objectors expressed it?

The question *not* to get hung up on is what any individual’s choice in this matter says about their attitude towards, say, historically underepresented minorities. It is perfectly consistent to be pro-tolerance and pro-inclusion while believing *this* subculture ought to be all about producing good code without regard to who is offended by the process. Not every kind of good work has to be done everywhere. Nobody demands that social-justice causes demonstrate their ability to write C.

That last paragraph may sound like I have strayed from neutrality into making a value claim, but not really. It’s just another way of saying that different groups have different teloi, and different ethoi proceeding from them. Generally speaking (that is, unless it commits actual crimes) you can only judge a group by how it fulfills its own telos, not those of others.

So we come back to two questions:

1. What is our telos?

2. Given our telos, do we have the most inclusive (least normative) ethos possible to achieve it?

When you have an answer to that question, you will know what we need to do about the CoC and the “killswitch” revolt.

Email archive thread at: https://lkml.org/lkml/2018/9/23/212

Sep 13

Hacker culture and the politics of process defense

In my last two blog posts, on the attempted hijack of the Lerna license and speech suppression in the Python documentation, I have both urged the hacker culture to stay out of political issues and urged what some people will interpret as “political” stance with regard to political correctness and “diversity”-driven speech demands.

The expected “gotcha!” comments that “ESR is saying hacker projects should stay clear of politics while arguing politics” have duly followed. While the way this sort of objection is usually posed barely rises above the level of a stupid rhetorical trick, there is an actual issue of principle here that deserves exploring.

Continue reading

Sep 12

Slaves to speech suppression are masters of nothing

Comes the news that the Python project has merged a request to partially eliminate the terms “master” and “slave” from its documentation on “diversity” grounds. Sensibly, Guido van Rossum at least refused to sever the project from uses of those terms in documentation of the underlying Unix APIs.

I wish Guido had gone further and correctly shitcanned the change request as political bullshit up with which he will not put. I will certainly do that if a similar issue is ever raised in one of my projects.

The problem here is not with the object-level issue of whether the terms “master” and “slave” might be offensive to some people. It’s with the meta-level of all such demands. Which the great comedian George Carlin once summed up neatly as follows: “Political correctness is fascism pretending to be manners.”

That is, the demand for suppression of “politically” offensive terms is never entirely or usually even mostly about reducing imputed harms. That is invariably a pretense covering a desire to make speech and thought malleable to political control. Which is why the first and every subsequent attempt at this kind of entryism needs to be kicked in the teeth, hard.

Continue reading

Aug 22

Unix != open source

Yesterday a well-meaning hacker sent me a newly-recovered koan of Master Foo in which an angry antagonist berated Master Foo for promoting an ethic of open-source software at the expense of programmers’ livelihoods.

Alas, I knew at once that he had been misled by a forgery, or perhaps some dreadful chain of copying errors, at whatever venerable monastic library had been the site of his research. Not because the economics was wrong – Master Foo persuades the antagonist that his assumption is in error – but because the koan conflates two things that were not the same. Actually, at least three things that are not the same.

Eighteen years into the third millennium, long after the formative events of Master Foo’s time, many people fail to understand how complex and contingent the relationship between the Unix tradition and the open-source ethos actually was in the old days. Too readily we project today’s conditions backwards in a way that impedes understanding of history.

Here’s how it was…

Continue reading

Mar 11

How to get started on the UPSide project

The current state of play is: We have a high-level system design and a map of the behavior states. We have a capacity target (300W for 15 mins) and a peak-continuous-load spec (400W) We know we’re going to build a double-conversion design and we’re considering a couple of alternative topologies. We pretty much know the external-interface specs (some details may change).

I’m expecting both my prototype copy of the forebrain Unix SBC (an Olimex LIME2) and the interface contract for the high-power subsystem to land on my desk tomorrow.

Interest in this project continues to be huge. Another company wants in as of this morning. The volume of feature requests is high enough that I’m buckling under the editing load.

The rest of this post is instructions to potential contributors about how to get on board.

1, Get an ID on GitLab. Tell me what it is so I can add you to the project group.

2. If you have a feature request, please Don’t post it on this blog. Add it to the “General feature request thread” on the tracker.

3. Read the wiki. Read the tracker issues. I try to keep both pruned so the volume is not overwhelming. Read the Rejected Ideas page on the wiki, too.

4. Read the design documents in the project wiki. The important one is the transaction design; the I2C message inventory will change, but the basic state diagram probably won’t.

5. Participate in the design discussion. This takes place in tracker threads.

6. When we’re ready to breadboard a prototype, throw some parts money in the tip jar we don’t have yet. If you must contribute before then the PayPal blogbutton works fine.

7. Prototype builds will probably go down at PA Makerspace in Phoenixville, PA. If you are within driving distance and a competent electrics tech, consider joining us for a build.

8. Once we have a full design with a PC board and enclosure: if you have a shop facilities for it, try to replicate the build. We’ll know we have the build recipe debugged when other people can do it.

9. If your favorite hardware feature request doesn’t appear in the version 1 prototype, relax, We may think it’s a good idea but be holding off till v2 out of a desire to keep v1 simple and launch fast.

10. If your favorite software feature request doesn’t appear in the version 1 prototype, pitch in and make it happen. A Unix SBC is not a difficult programming environment – the OS on this one is a Debian port.

After step 10 and a couple of design iterations the future becomes less clear. maybe try to get it into volume manufacturing through a partnership with an established vendor.

Feb 04

How “open source” was coined

Yesterday was the 20th anniversary of the promulgation of the term “open source”. Three days before that, Christine Peterson published How I coined the term ‘open source’ which apparently she hd written on 2006 but been sitting on since.

This is my addition to the history; I tried to leave an earlier version as a comment on her post but it disappeared into a moderation queue and hasn’t come out.

The most important point: Chris’s report accurately matches my recollection of events and I fully endorse it. There are, however, a few points of historical interest that can be added.

Continue reading

Nov 07

The long goodbye to C

I was thinking a couple of days ago about the new wave of systems languages now challenging C for its place at the top of the systems-programming heap – Go and Rust, in particular. I reached a startling realization – I have 35 years of experience in C. I write C code pretty much every week, but I can no longer remember when I last started a new project in C!

If this seems completely un-startling to you, you’re not a systems programmer. Yes, I know there are a lot of you out there beavering away at much higher-level languages. But I spend most of my time down in the guts of things like NTPsec and GPSD and giflib. Mastery of C has been one of the defining skills of my specialty for decades. And now, not only do I not use C for new code, I can’t clearly remember when I stopped doing so. And…looking back, I don’t think it was in this century.

That’s a helluva thing to have sneak up on me when “C expert” is one of the things you’d be most likely to hear if you asked me for my five most central software technical skills. It prompts some thought, it does. What future does C have? Could we already be living in a COBOL-like aftermath of C’s greatest days?

Continue reading

Jul 17

A teaching story

The craft of programming is not a thing easily taught. It’s not so much that the low level details like language syntaxes are difficult to convey, it’s more that (as I’ve written before) “the way of the hacker is a posture of mind”.

The posture of mind is more essential than the details. I only know one way to teach that, and it looks like this…

Continue reading

Jul 05

Gift vs. reputation in hacker culture

A G+ follower pointed me at Note on Homesteading the Noosphere by Martin Sústrik. He concludes saying this:

In short: Labeling open source communities as gift cultures is not helpful. It just muddles the understanding of what’s actually going on. However, given that they are not exchange economies either, they probably deserve a name of their own, say, “reputation culture”.

I’m going to start by saying that I wish I’d seen a lot more criticism this intelligent. It bothers me that in 20 years nobody seems to have refuted or seriously improved on my theories – I see this as a problem, both for the study of hacker culture and in the field of anthropology.

That said, I think Sústrik gets a couple things wrong here. And don’t want them to obscure the large thing he’s gotten right.

First (possible) mistake: I have not observed that, as a matter of language, the term “gift culture” is as hard-edged and specific as he thinks it is. There’s a way we could both be right, though – it might be that terminology has shifted since I wrote HtN. Possibly this came about as part as the revival of interest in the concept that I seem to have stimulated.

But: one piece of evidence that anthropologists are still using “gift culture” in the inclusive sense Sústrik criticizes me for enmploying is that Sústrik himself feels, at the end of the article, that he needs to propose a contrasting term rather than citing one that is already established.

This so far is all about map rather than territory. As a General Semanticist I know better than to get over-invested in it.

Here’s the territory issue: Sústrik is not quite right about expectations of direct reciprocal exchange not being a shaping force. True enough that they aren’t salient at the macrolevel the way they were among the Kwakaka’wakwe. But if I download a piece of open source, and it’s useful to me, and I find a bug in it, I do indeed feel a reciprocal obligation to the project owner (not just an attenuated feeling about the culture in general) to gin up a fix patch if it is at all within my capability to do so – an obligation that rises in proportion to the value of his/her gift.

I should also point out that the cultures Sústrik think are paradigmatic for his strict sense of “gift culture” are mixed in the other direction. There is certainly an element of generalized reputation-seeking in the way individual Kwakaka’wakwe discharged their debts. There, and in the New Guinea Highlands, the “big man” is seen to have high status by virtue of his generosity – he overpays, on the material level, to buy reputation.

In the terms Sústrik wants to use, open-source culture is reputation-driven at both macro-level and microlevel, and also sometimes driven by gift reciprocation in his strict sense at microlevel. The macro-level reputation-seeking and micro-level gift reciprocity feed and reinforce each other.

This brings me to the large thing that Sústrik gets right. I think his distinction between “gift” and “reputation” cultures is fruitful – both testable and predictively useful. While I’m still skeptical about it being in general use among anthropologists, I rather hope I’m mistaken about that – better if it were.

Yes, real-world cultures are probably never pure examples of one or the other. But differentiating the mechanisms – and observing that the Kwakaka’wakwe and hacker culture are near opposing ends of the spectrum in how they combine – that is certainly worthwhile.

As a minor point, Sústrik is also quite right about reciprocal licenses being a red herring in this discussion. But I think he has the reason for their irrelevance mostly wrong. The important fact is they’re not mainly intended to regulate in-group behavior; they’re mainly a lever on the behavior of outsiders coming into contact with the hacker culture.

(It was actually my wife Cathy – a pretty sharp-eyed observer herself, and not coincidentally a lawyer – who brought this to my attention.)

Bottom line, however, is that this was high-quality criticism that got its most central point right. In fact, if I were writing HtN today I would use – and argue for – Sústrik’s distinction myself.

Jun 30

Open Adventure 1.1, and some thoughts on software preservation

Open Adventure 1.1 has shipped. There are a lot more changes under the hood than are readily apparent. In fact there have been no changes in gameplay at all, and only minor changes to the UI (reversible with the -o oldstyle switch).

We (Jason Ninneman, Per Vorpaev, Aaron Traas, Peje Nilsson and I) could have taken the approach of changing the original rather ugly C code (mechanically translated from FORTRAN) as little as possible, simply packaging it for compilation and release in a modern environment.

I elected not to do that, one reason being that I think we honor hacker tradition better by bring the code forward as a dynamic, living artifact that invites being hacked on than museumizing it as a static one. There’s also the fact that the extreme obscurity of the code made it difficult to appreciate what a work of genius Adventure actually was. (The code we inherited had over 350 gotos in it – rather hard to see past those.)

So we’ve taken a different path. We’ve translated the code into (almost) fully idiomatic C (but not trying to introduce pointer idioms; that should make translation to future languages easier). We’ve replaced the rather cryptic custom text database file that used to define the dungeon with a YAML document that is orders of magnitude easier to read and modify. We haven’t hesitated to use technology that wasn’t even a gleam in anyone’s eye when Adventure originated – the YAML is compiled to C structures at build time by a Python script.

The effect (we hope) is Adventure as it would have been written if Crowther & Woods had had today’s tools to do it – the same vision and design logic, expressed in modern coding idioms. Worth doing, because there are still some things to be learned from this design.

Probably the single cleverest thing in it – which pretty much has to go back to Crowther, Woods couldn’t have bolted it on afterwards – is the way movement in the dungeon is handled. The dungeon’s topology is expressed by a kind of pseudocode broadly resembling the microcode found underneath a lot of processor architectures; movement consists of dispatching to the sequence of opcodes corresponding to the current room and figuring out which one to fire depending not only on the motion verb the user entered but also on conditionals in the pseudocode that can test for the presence or absence of objects and their state.

It was hard to fully understand and appreciate this before, because the code was a spaghetti tangle in what looks today like a shockingly primitive style. The abstraction of the dungeon topology into a declarative specification that – in effect – loads microcode into the game engine was a thing you could half-see, but the impact was blunted by the unreadability of both the code and the specification format. Lifting the specification to YAML was like polishing a rough diamond, revealing beauty and brilliance.

And that’s before we even get to Adventure considered as a work of communicative art. It’s had so many successful descendants – like, every dungeon-crawling game ever, and every text adventure ever – that it’s difficult to see with fresh eyes. But if you make the effort, it is astonishing how mature the wry, quirkily humorous, slightly surrealistic style of this very first game seems. The authors weren’t fumbling for an idiom that would be greatly improved by later artists more sure of themselves; instead, they achieved a consistent and (at the time, unique) style that would be closely emulated by pretty much everyone who followed them in text adventures, and not much improved on as style, even though the technology of the game engines improved by leaps and bounds.

I don’t know how they did it, and the authors would probably not be able to explain if we asked. But I think it is damned impressive how well this game has aged – the code may have needed a refresh, but the design still shines. I’m proud to have helped restore it, and hope I have brought it to a state where it can be forward-ported to future languages for as long as programming is a living art.

May 14

The advent of ADVENT

A marvellous thing has just occurred.

Colossal Cave Adventure, the original progenitor of the D&D-like dungeon-crawling game genre from 1977 and fondly remembered as ADVENT by those of us who played it on PDP-10s, is one of the major artifacts of hacker history.

The earliest version by Crowther and Woods (sometimes known as 350-point Adenture) was ported to C by Jim Gillogly in ’77 just after it first shipped. That has been part of the bsd-games collection forever.

What I have have just received Crowther & Wood’s encouragement to polish up and ship under a modern open-source license is not the Gillogly port; it’s Crowther & Woods’s last version from 1995. It has 18 years of work in it that the Gillogly version doesn’t.

I feel rather as though I’d been given a priceless Old Master painting to restore and display. Behooves me to be careful stripping off the oxidized varnish.

Apr 18

You shall judge by the code alone

I support the open letter by Drupal developers protesting the attempted expulsion of Larry Garfield from the Drupal commmunity.

As a Drupal contributor who has never in any respect attempted to tie the project to his beliefs or lifestyle, Garfield deserves the right to be judged by his code alone. That is the hacker way; competence is all that matters, and no irrelevance like skin color or shape of genitals or political beliefs or odd lifestyle preference should be allowed to matter.

That I even need to say this in 2017 is something of a disgrace. The hacker culture already had judge-by-the-code-alone figured out forty years ago when I was a n00b; the only reason it needs to be said now is that there’s been a recent fashion for “social justice” witch hunting which, inevitably, has degenerated into the sort of utter fiasco the Drupal devs are now protesting.

Thomas Paine said it best: “He that would make his own liberty secure, must guard even his enemy from oppression; for if he violates this duty, he establishes a precedent that will reach to himself.”

It doesn’t matter how much you dislike Larry Garfield’s personal kinks. If you don’t defend him now, you may have nobody to defend you when some self-declared commissar of political and sexual correctness – or just a censorious project lead like Dries Buytaert – decides that you should be declared an unperson.

You shall judge by the code alone. That is the only social equilibrium that doesn’t degenerate into an ugly bitchfest with expulsions controlled by whatever happens to be politically on top this week. It was the right community norm forty years ago, and remains so today.

Apr 12

PSA: “E-Shielder Security” and “CyberSec Buzz” are gangs of idiotic scum

This is a public service announcement: E-Shielder Security, describing itself as “leading importers and suppliers of high end electronic technology solution systems” is a gang of idiotic scum.

Yesterday they posted a Hacktivists on the rampage in 2017, which largely reproduced my Hacker Archetypes post.

They did so in obvious ignorance of who the hackers I was referring to actually are, going off on a tear about “hacktivists”. That term is, in general, a flare-lit clue that the person using it is either an idiot or a vandal trying to cloak destructive behavior in respectability – real hackers are proud of what they do, take responsibility for it, and don’t wear masks (with a limited exception for those under direct threat from totalitarian governments). In this case it was clearly idiocy.

Mere idiocy turned into something nastier. I left a comment on the post pointing out their error, something I had clear standing to do as the author of the article they were quoting.

The comment was suppressed. That was scummy behavior; thus “idiotic scum”.

Don’t do business with these clowns. Warn your friends. Propagate this widely, the clowns deserve some serious reputation damage.

Addendum: Title amended because the article may have originated at CyberSec Buzz, another ‘security’ blog run by drivelheads who obviously have no fscking idea what they’re talking about. It has been taken down where I originally found it.

Apr 03

Hacker Archetypes

There’s a book about martial arts called On the Warrior’s Path that tries to understand the differing psychologies of martial artists through the lens of half a dozen archetypes – Seeker, Ronin, Tribal Warrior, and others.

I have not yet read the book, but my friend and regular A&D commenter Susan Sons reports having found it very effective for motivating young and newbie martial artists. “It gave them their first glimpse of what they were trying to become,” she reports, “They both knuckled down not just in the obvious physical parts of training, but in the mental aspects, far more than they had before and far more than their age/experience peers.”

So, Susan had the idea that it might be a good idea to develop a parallel gallery of hacker archetypes to help motivate newbies. We brainstormed this on IRC for a while. One thing that had been blocking Susan is that, by her own report, she sucks at naming things. I, on the other hand, am pretty good at that; I was able to come up with names that helped the archetypes develop more definition.

We don’t think this is a complete set, and some of the names might change. But it’s enough of a start for some public brainstorming.

Also note: no hacker is only one of these, but in talking about a number of mutual friends we found it was always pretty easy to agree on both the friend’s dominant archetype and the secondary one that they display most after it. I think this is an indication that we are, even if imperfectly, zeroing in on real traits.

Here they are. Descriptions mostly Susan, names mostly me.

Continue reading

Mar 08

How to change the world in Zen easy lessons

This morning I stumbled over a comment from last September that I somehow missed replying to at the time. I suspect it’s something more than one of my readers has wondered about, so here goes…

Edward Cree wrote:

If I’m really smart enough to impress esr, I feel like I ought to be doing more with myself than toy projects, games, and an obscure driver. It’s not that I’m failing to change the world, it’s that I’m not even trying. (Not for want of causes, either; there are plenty of things I’d change about the world if I could, and I suspect esr would approve of most of them.)

Obviously without Eric’s extroversion I won’t be as influential as him, but… dangit, Eric, what’s your trick? You make having a disproportionate effect on the course of history look easy! Why can I never find anything important to hack on?

There are several reasons people get stuck this way. I’ve experienced some of them myself. I’ve seen others.

If this sounds like you, dear reader, the first question to ask yourself is whether you are so attached to having a lot of potential that you fear failing in actuality. I don’t know Edward’s age, but I’ve seen this pattern in a lot of bright young people; it manifests as a lot of project starts that are potentially brilliant but a failure to follow through to the point where you ship something that has to meet a reality test. Or in an opposite way: as self-constraining to toy projects where the risk of failure is low.

So my first piece of advice is this: if you want to have “a disproportionate effect on the course of history”, the first thing you need to do is give yourself permission to fail – as long as you learn something from every failure, and are ready to keep scaling up your bets after success.

The second thing you need to do is finish something and ship it. No, more than that. You need to make finishing and shipping things a habit, something you do routinely. There are things that can be made to look easy only by cultivating a lot of self-discipline and persistence. This is one of them.

(The good news is that once you get your self-discipline to the required level it won’t feel like you have to flog yourself any more. It’ll just be habit. It’ll be you.)

Another thing you need to do is actually pay attention to what’s going on around you, at every scale. 99% of the time, you find important things to hack on by noticing possibilities other people have missed. The hard part here is seeing past the blinding assumptions you don’t know you have, and the hard part of that is being conscious of your assumptions.

Here’s my favorite example of this from my own life. After I described the many-eyeballs-make-bugs-shallow effect, I worried for years at the problem of why nobody in the hacker culture had noticed it sooner. After all, I was describing what was already a decades-old folk practice in a culture not undersupplied with bright people – why didn’t I or anybody else clue in faster?

I remember vividly the moment I got it. I was pulling on my pants in a hotel in Trondheim, Norway, idly chewing over this question yet again. It was because we all thought we knew why we were simultaneously innovating and achieving low error rates – we had an unexamined, unconscious explanation that suited us and we never looked past it.

That assumption was this: hackers write better software because we are geniuses, or at least an exceptionally gifted and dedicated elite among programmers. Our culture successfully recruits and selects for this.

The insidious thing about this explanation is that it’s not actually false. We really are an exceptionally gifted elite. But as long as you don’t know that you’re carrying this assumption, or know it and fail to look past it because it makes you feel so good, it will be nearly impossible to notice that something else is going on – that the gearing of our social machine matters a lot, and is an evolved instrument to maximize those gifts.

There’s an old saw that it’s not the things you don’t know that hurt you, it’s the things you think you know that ain’t so. I’m amplifying that: it’s the things you don’t know you think that hurt you the most.

It’s not enough to be rigorous about questioning your assumptions once you’ve identified them. The subtler work is noticing you have them. So when you’re looking for something important to hack on, the question to learn to ask is: what important problems are everybody, including you, seeing right past? Pre-categorizing and dismissing?

There’s a kind of relaxed openness to what is, a seeing past preconceptions, that is essential to creativity. We all half-know this; it’s why hackers resonate so strongly with Zen humor. It’s in that state that you will notice the problems that are really worth your effort. Learn to go there.

As for making it look easy…it’s only easy in the same way that mastery always looks a skill easier than it is. When someone like John Petrucci or Andy Timmons plays a guitar lick with what looks like simple, effortless grace, you’re not seeing the years of practice and effort they put into getting to where that fluency and efficiency is natural to them.

Similarly, when you see me doing things with historical-scale consequences and making it look easy, you’re not seeing the years of practice and effort I put in on the component skills (chopping wood, drawing water). Learning to write well. Learning to speak well. Getting enough grasp on what makes people tick that you know how to lead them. Learning enough about your culture that you can be a prophet, speak its deepest yearnings and its highest aspirations to it, bringing to consciousness what was unconscious before. These are learnable skills – almost certainly anyone reading this is bright enough to acquire them – but they’re not easy at all.

Want to change the world? It’s doable. It’s not magic. Be aware. Be courageous. And will it – want it enough that you accept your failures, learn from them, and never stop pushing.