The infrastructure gnomes of tomorrow

Regular TomA continues a hot streak by asking, in response to my post on Holding Up The Sky, “Is the hacker support system robust?”

That is: having noticed that open-source volunteers now have a large and increasing role in maintaining critical shared infrastructure like the Internet, is there a sustainability issue here? Once the old guard who were involved in the early days (people like Jim Gettys and Dave Taht and myself) dies off, are we going to be able to replace them?

I shall set forth my reasons for optimism.

My principal reason for optimism is that the hacker culture has gotten extremely good at recruiting new talent. By “hacker culture” I mean anybody for which this is either a look in the mirror or an aspiration. Another good test is, as I’ve written before, is RFC1149. If you find it funny you may be part of the hacker culture, and if you find a report of implementing it with actual pigeons hilarious you almost certainly are.

Consider: in the late 1970s when I first began to identify with this culture, you could almost certainly have fit every hacker in the United States in a medium-sized auditorium. If you were willing to have people standing in the aisles, every hacker in the world.

Today, many Linux user groups can easily fill a hall that size, even in a Third World country. I know because they do it when I give talks! Those people aren’t swooning over my rugged masculine charm; they come to hear me because they want to be a part of what I represent to them (I know this directly from audience reactions). Even if only one in ten of the people in my audiences writes code, and only one in a hundred dedicates him- or herself to a piece of key software infrastructure, we’ll be able to sustain the numbers to meet our responsibilties.

Now go watch #commits on freenode for a while. CIA only monitors a small fraction of the active projects out there, but watching will give you a feel for the huge volume and breadth of commits to public repositories going on all the time. This is shit getting real, people not just yakking on chat channels or listening to the likes of me rant at a LUG but writing code. Hacking. Creating. Perfecting their craft.

There’s a helluva lot of momentum out there, and if anything the pace is increasing in tandem with Internet deployment, rather than slowing down. I think we’re also seeing a positive network effect; as the hacker culture gets larger and more diverse, the attractive value of the options it presents to potential hackers rises. Today’s flood of newbies will give us tomorrow’s coders and the next decade’s hard-core infrastructure gnomes.

Thus my optimism. The social machine that trains and motivates our sky-upholders is in rude, vibrant good health. if anything we can live with a lot more lossiness in the long pipeline from newb to infrastructure gnome today than we could have decades ago when our intake was orders of magnitude smaller.

I think it would take a disruption of historic proportions to stop that social machine from cranking out upholders of the sky. And it’s actually pretty difficult for me to think of a disruption in the right intensity range – that is, large enough to stall out the hacker-culture social machine without being a civilization-wrecker so total that not being able to find skilled help for stuff like the Bufferbloat project would be the least of our problems.

The usual run of everybody’s favorite looming threats don’t worry me in the long term. We’ve survived software patents and the Microsoft monopoly in style. We just handed Big Media its ass over SOPA/PIPA and now we know how to do that again if we have to. Overtly political repression by any single government or plausible coalition of governments would just push activity to jurisdictions where it’s less controllable. Pirate Bay isn’t us, but it’s a useful straw in the wind; even the weight of Big Media and several cooperating governments couldn’t take them out, and our network is much better dispersed.

Actually the only kind of disrupter I can imagine actually screwing up the supply of future infrastructure gnomes would be some kind of superstimulus that would be so much more attractive than the hacker culture that it would outcompete us for geek attention. I could sort of distantly imagine the hackerspace crowd and their 3D printers doing that, maybe, except that (a) atoms are harder to push around than bits, creating friction costs we don’t have, and (b) they’re actually us anyway!

Maybe garage nanotech? But I have a strong hunch that when that gets here, the people who make it happen will be us, too. What the hacker culture has actually become is an attractor that both pulls into itself and seeds the maker communities around any new software-intensive technologies that arise near it. Early minicomputers, the Internet, Unix, microcomputers, smartphone modding, 3D fabrication, open-source software and hardware – the hacker posture of mind, and the cultural signifiers that have evolved to express and transmit it, spans all of these not as a matter of accident but of essence.

This, too, helps explain the population explosion. And gives me confidence about the answer to TomA’s question. Crossover among the technologies we hack is constant, and everyone drawn into this attractor is implicitly in training to hold up a piece of the sky.

42 comments

  1. Eric, I am absolutely laughing my ass off over here from reading:

    “Another good test is, as I’ve written before, is RFC1147. If you find it funny you may be part of the hacker culture, and if you find a report of implementing it with actual pigeons hilarious you almost certainly are.”

    Um… That’s RFC1149, not RFC1147. I am even more astonished to be the first one to point it out.

    1. >When you link to RFC1149, I would suggest you should mention RFC4824 too.

      ROFL. I somehow did not know of this one.

  2. While you’ve corrected the typo, the link still goes to 1147 for me. I was quite confused for a few seconds, as I wasn’t finding it amusing at all.

  3. The Internet is here to stay because it is too big to be brought down by anyone:
    http://www.techflash.com/seattle/2012/03/huge-internet-economy-worlds-fifth.html

    All the attempts of $EvilCompany and $EvilOrganization to bring it down were thwarted by the simple fact that there are too many other people who really need the Internet. And way too many people in the world’s University with the time and expertise to reroute around any obstacle.
    (I say Universities, because I think in in practice, only students have both the time and know-how to be recruited fast)

  4. I don’t think that you should be so optimistic about Internet infrastructure hacking. Hacker culture will definately survive, but it will change. When I was young, I was an electronics hacker, which meant radio and similar analog electronics. It led me to my career, but *now* that’s all passe. There’s still radio and TV, but it’s common and unexciting. I go to hamfests and the vast majority of the attendees are around my age – very few youngsters. (This is a complete reversal from the first hamfests I attended in the 1960s.) It’s *old*.

    In a similar way, I see computers and the Internet as becoming old and unexciting, especially as the exploits of Anonymous and state-sponsored cracking forces ‘trusted computing’ paradigms on the world. The hackers of tomorrow will be hacking biology and space-related things. They will all need computers, but only as tools, not as things interesting in themselves.

    I see your job as, not so much training the next computer hacker generation, but as securing the ‘net against failure modes (hardware, software and human-induced) so that the next generation of hackers can simply take it for granted. They will have bigger fish to fry.

  5. We just handed Big Media its ass over SOPA/PIPA and now we know how to do that again if we have to.

    Sorry, but as successful as the campaign against SOPA/PIPA were, in the end it won’t count for much. In the U.S., anyway, the largest ISPs which control virtually all of internet infrastructure are either owned by (Time Warner, Comcast) or can be bought off by Big Media. Big Media is the internet, and starting on July 12, the internet infrastructure itself will spy on you and punish you for suspected piracy — no legislation needed.

    1. >the internet infrastructure itself will spy on you and punish you for suspected piracy

      Yeah, right. Time until this monitoring is defeated via systematic use of VPNs and other measures, on the order of two weeks.

  6. “Actually the only kind of disrupter I can imagine actually screwing up the supply of future infrastructure gnomes would be some kind of superstimulus that would be so much more attractive…” – Eric

    “Space… the final frontier, these are the voyages of the Starship [Open Source]…” – Kirk

    http://www.orbithangar.com/
    http://shatters.net/celestia/
    http://www.astronautix.com/
    http://www.openluna.org/
    http://tech.groups.yahoo.com/group/marsdrivemission/

    (Note: Orbiter and AGI STK not Open Source yet, so not linked.)

    On a more serious note, to find the sort of disaster that could disrupt the gnome supply sufficiently, Charles Sykes points out the education system:

    http://www.amazon.com/Dumbing-Down-Our-Kids-Themselves/dp/0312148232/ref=sr_1_1?ie=UTF8&qid=1332350970&sr=8-1

    Full title: “Dumbing Down Our Kids: Why American Children Feel Good About Themselves But Can’t Read, Write, or Add [Paperback]”

    Not only is it a potential “civilization-wrecker so total that not being able to find skilled help for stuff like the Bufferbloat project would be the least of our problems,” it is already a bigger problem than Bufferbloat, and has been since before the Internet was created (not long before the Internet was created though.) And of course, hacker culture developed in spite of (maybe even partly because of) this problem.

    The rest are gnarlier: nuclear war, killer asteroid, Yosemite blows, “The Big One”, the New Madrid “Big One”, Alex Jones (i.e. http://www.infowars.com) turns out to be perfectly correct, etc..

    Finally, if you think RFC1149 was funny, these young system administrators didn’t quite turn out as successfully:

    http://www.youtube.com/watch?v=2pScmvyzNQE

  7. The open source community is certainly robust. There is no shortage of new blood coming in.

    The possible issue I see is where that new blood concentrates its efforts. A sort of Darwinian “survival of the fittest” takes place. Some open source projects, like Linux and Apache, become hugely popular, with competition to get in (where “in” is “commit access to the VCS used.”) Others never get a following, and when the founders lose interest/go away, they become orphaned projects languishing in obscurity.

    Most folks don’t get paid to work on open source code. They do so to scratch a personal itch or for geek cred (or both) and do other things to pay the bills.

    I think the question is “What open source components are critical to the infrastructure, and what needs to happen to make sure *those components* have the needed warm bodies to maintain them?” A related issue in maintenance, resulting in what Jamie Zawinski called the “”Cascade of Attention-Deficit Teenagers” programming model. Fixing bugs isn’t *fun*. Writing *new* code is fun. So instead of getting bugs fixed, we get whole new versions of existing code, with a whole new set of bugs that won’t be fixed. (Gnome, anyone?)

  8. I would have thought that an additional factor for optimism would be the lower barriers of entry being represented by the App market. Building apps is easier than ever, and because of the low barriers to entry, it is derivatively easy to define, discover, and respond to a market with very focused functionality. That puts the hacker -> useable tool at a much lower threshold. Hackers are commensurately closer to consumers.

    Note that in this formulation, the currency for a given market may or may not be dollars, and can include hacking cred and competency. This is especially true when you realize how the App market in general is redefining functionality in particular industry verticals, and thus allows hacker cred to be focused into these verticals. That is a more refined feedback loop than, for example, Jim Gettys or yourself could expect 20 years ago.

  9. @IGnatius T Foobar, @esr I’m going to object to the idea that the problem is the “smart hyphens” (which it doesn’t actually contain) and characters like dashes, non-breaking spaces, curly etc (which it does) – which browsers today (even text mode browsers five years ago, IIRC) handle perfectly when served correctly, and say the real problem is the fact that it mysteriously is detected by browsers as Latin-1 even though it is in UTF-8 and has both an ?xml? and a meta content-type tag showing it as UTF-8. Further remedies to try might be get the catb.org server to respond a proper content-type header, to include a UTF-8 “byte order” mark, or to have it generate all non-ASCII characters as HTML entities.

  10. Game theory modeling of societal “organisms” and its component archetypes now includes a hacker variant in both symbiosis and competition with the other social elements. We’re beginning to see a few scenarios where hacker robustness is significantly challenged. There may be no real world corollary to this, but it’s illuminating nonetheless. In some models where relative freedom in society declines, hacker vitality surges briefly and then they disproportionately come under attack. I interpret this to mean that hackers may become easy scapegoats if the shit hits the fan and tyrants need someone to blame.

  11. I think the demand for expertise has outstripped supply, despite watching the #commits fly by.

    In part, this is why you see the googles of the world sucking all the sysadmins into the cloud – your typical sysadmin rather than a somewhat comfortable job working with a team, managing a 100 or 200 machines (and their respective (l)users) now has the more hairy job managing 10,000s, because nobody can afford the expertise to manage 100, as the invisible economic model of advertising for free sysadmin is outcompeting local intelligence and expertise. Similarly all the other infrastructure work duct taped and held together with chewing gum holding our society together, not just in the computer field, but in construction, legal, politics, name it…

    I emphasis with Tinny-Peete, greatly, very often.

    http://en.wikipedia.org/wiki/The_Marching_Morons

    Tho I wouldn’t mind a trip to mars, if the rockets actually worked.

  12. > Yeah, right. Time until this monitoring is defeated via systematic use of
    > VPNs and other measures, on the order of two weeks.

    One of my few self-allowed tinfoil-hat behaviors for the last five or so years has been the slow collection of modems. I like them in a retrocomputing sense, but also I have haha-only-serious explained it to friends (who give me weird looks anyway) by claiming that, when The Network goes bad, I’ll just lease a decent line from one of the telcos not involved in this BS and use said modems to start up a new local ISP that never logs anything and acts purely as a connection point. Modems are slow, but when you KNOW They are spying on you, or trying to, it may be a reasonable trade-off for some people. It might even pay for itself, I dunno.

    It’d be nice to live somewhere with more choices than Time Warner for high speed cable or AT&T for shitty DSL.

  13. My questions about robustness are different. I’m not especially worried about having enough hackers to “hold up the sky”.

    I’m more worried about the scalability and robustness of hacker methods and tools. Look at all the cruft associated with repositories, for instance.

    And managing the ever-larger number of coders and testers required by ever-larger software packages. Can purely social mechanisms do that well enough to avert major failures?

    Interactions between the massively expanding population of packages. This is an intrinsically intractable problem, which (ISTM) would be even harder to manage informally.

    There are reliability issues in computing which a lot of people don’t even realize exist. See
    http://perspectives.mvdirona.com/2012/02/26/ObservationsOnErrorsCorrectionsTrustOfDependentSystems.aspx
    for a discussion of how errors continually creep into RAM and disk storage, corrupt databases, and kill services and enterprises. “Best practices” (religious use of checksums and ECC) can defeat this – but not everyone does it.

    Scenario: the volunteer hosts of a repo holding a major project save money by using non-ECC motherboards. They host for three or four years. During this time, the repo database develops significant internal corruption, but no one sees it yet. Then the project moves to a different repo system on a different server.

    It runs there for a while – then the corruption manifests, wrecking development history and making it impossible even to determine what is the current release. This at a time when the project has problems with new features colliding with other projects’ changes. Some of the past key developers on the project have left, and can’t or won’t come back. The project has been low-activity for a while, and lacks prestige or glamor.

    But the project program has become real-world essential, and end-users can’t wait while the project team recruits new developers, rebuilds or replaces the damage, and brings the project back to health. The current project leaders aren’t really up to that job, and no one else is volunteering for it. (Those who might be up to it are fully engaged in other projects or real life.)

    I see this scenario coming to pass in the not-distant future, and I don’t see it ending well.

  14. @TomA I can believe that scenario, and there is real-world precedent for it. I believe once there was some sort of epidemic in a city in Russia. The Czar noticed the city had disproportionately many doctors, so he “fixed” the problem by having them all killed.

  15. Personally, I’m quite fond of RFC 3093, ‘Firewall Enhancement Protocol’, which provides a mechanism for IP over HTTP. In this case the joke is that it is sometimes useful…

  16. > Yes. It’s a problem with my Docbook-XML toolchain that I haven’t had time to solve yet.

    Actually, it looks like your toolchain is working just fine — the problem is the Unicode encoding. Docbook is producing XHTML encoded with UTF-8, which is the closest we have to a Right Thing as Unicode encodings go, but your web server mistakenly insists that the encoding is ISO-8859-1. See for yourself with this command:

    $ curl -vv http://www.catb.org/~esr/faqs/hacker-howto.html | head -n 10

    In particular, notice the Content-Type header:

    Content-Type: text/html; charset=iso-8859-1

    … and then look at the first line of the HTML:

    <?xml version=”1.0″ encoding=”UTF-8″?>

    I looked at the bytes in the file, and confirmed that it’s actually UTF-8. Fixing it should be a one-line change in an Apache config file somewhere, but I’m not familiar enough with Apache to know what line that would be.

    1. >Fixing it should be a one-line change in an Apache config file somewhere, but I’m not familiar enough with Apache to know what line that would be.

      Yes, I figured this out a couple hours ago and posted a fix request to my sysadmins on ibiblio.

  17. ” Similarly all the other infrastructure work duct taped and held together with chewing gum holding our society together, not just in the computer field, but in construction, legal, politics, name it…”

    Duct tape and chewing gum? We’re in better shape than I thought ;)

  18. Actually, as much as I love you ‘old guard’ guys (heh…even though we’re about the same age)….I don’t think you’re as important as you think you are. That’s not intended to be mean.

    Sure, you provide a wonderful collective ‘wiseman’ role that both informs and advises the ‘hacker’ culture, as well as perpetuating a certain cult-like mythos that enchants novices…..but when I look at what is really driving technologies that make the ‘net pulse, I’m seeing offshoot private/hacker funded collaborations and a powerful marketplace driven by a rather profound meritocracy – people bolstering a resume with OSS work.

    These market-driven incentives are, I think, a more powerful motivator than a bunch of greybeards.

    Of course, I wish the greybeards to continue providing valuable insight for as long as possible…….perhaps you really should invest in one of those clunky first-gen robot bodies…..but make sure a taxidermist preserves your mustache or we may not recognize you ;)

    1. >I don’t think you’re as important as you think you are. That’s not intended to be mean.

      No offense taken. Neither Dave nor I has claimed that the infrastructure-gnome thing makes us important.

      Sometime I actually am important, but that’s wearing a different hat. And one I generally avoid putting on unless whatever problem I’m working requires it.

      >These market-driven incentives are, I think, a more powerful motivator than a bunch of greybeards.

      All interesting behavior is overdetermined. :-)

  19. Eric, have you considered adding Javascript to the list of “recommended” languages for hackers? The question occurs to me mainly because Java is still on the list (though now deprecated as a recommended first language), and the main reason I like Javascript is that I would rather swim through molasses than program in Java, and Javascript plus the appropriate other tools allows me to avoid having to use Java to build Android apps. :-) But including Javascript could also be justified nowadays on the same grounds as you give for including HTML.

  20. @Peter Donis

    Please…for the sake of all that is sweet and precious on our beautiful globe….tell me you are joking…..

    ;)

  21. Sometime I actually am important, but that’s wearing a different hat.

    That’s for damned sure. I know that you know I was not challenging the truth of that. Entirely different kettle of fish etc.

    All interesting behavior is overdetermined. :-)

    Ah…OK…ya got me confused there….would you expand upon that comment please?

    1. >Ah…OK…ya got me confused there….would you expand upon that comment please?

      It’s something I learned from animal ethology. An “overdetermined” behavior is one for which there are multiple sufficient explanations. To unpack: “For every interesting behavior of animals and humans there is more than one valid and sufficient causal theory.” Evolution likes overdetermined behaviors; they serve multiple functions at once.

      In this case I’m claiming that the example of graybeards and market incentives are both sufficient hypotheses to explain an overdetermined behavior.

  22. Gotcha.

    I certainly think that ‘greybeards’ are valuable, but when I view the vast amount of code being generated through rentacoder.com et al, I have to doubt whether any but a small slice of them know what an “ESR” is ;)

    True, that isn’t OSS-specific, but it does exemplify a vibrant open ‘bazarre’ marketplace for hackers/code monkeys that gives me reason to wonder if the greybeard-culture is truly a motive force anymore.

    I love you guys, but I think the OSS globe would continue turning absent your contributions.

  23. @Dan:
    > Please…for the sake of all that is sweet and precious on our beautiful globe….tell me you are joking…..

    I wasn’t, actually, but I can see how you would wish I was. :-)

  24. >the simple fact that there are too many other people who really need the Internet.

    Which also makes it a particularly dangerous single point of failure. Another Carrington Event today would do much more damage to our civilization than necessary precisely because of the widespread internet dependence.

  25. >All interesting behavior is overdetermined. :-)

    My most popular recent LessWrong comment in a thread about how to catch yourself rationalizing was about that:

    You can find multiple, independent considerations to support almost any course of action. The warning sign is when you don’t find points against a course of action. There are almost always multiple points both for and against any course of action you may be considering.

  26. “You can find multiple, independent considerations to support almost any course of action. The warning sign is when you don’t find points against a course of action. There are almost always multiple points both for and against any course of action you may be considering.”

    Thus was summarized what is probably my #1 complaint about discussions of American politics.

  27. Mike Earl quoth:

    Personally, I’m quite fond of RFC 3093, ‘Firewall Enhancement Protocol’, which provides a mechanism for IP over HTTP. In this case the joke is that it is sometimes useful…

    RFC 3093 is absolutely NOT a joke. It (and some interesting derivatives using DNS UDP) are in active use by the folks who want to steal your ebanking details. I forget which trojan it is – could be ZeuS, Spyeye or LDSS (or possibly all of them) – that has an encrypted version of RFC 3093 built in as standard. Oh and they also use port 443 because that makes it harder for IDSes to detect badness.

    (stopping this is my day job)

  28. @FrancisT: That’s the thing about April 1st RFCs. Many of them actually work. That’s part of the humor. If you don’t get it, you don’t have the hacker mindset.

  29. Hi, I would like to introduce you to unwitting network infrastructure gnome Steve Hindi, who is working very hard to maintain the network infrastructure for RFC1149 and RFC2549 networks, specifically in the states of Pennsylvania and South Carolina. At various locations (Wing Pointe in Berks County, PA, Philadelphia Gun Club in PA, and Broxton Bridge Plantation in Colleton County, SC), potential packet carriers for these networks are shut down using multiple projectile firing devices known as “shotguns” after being launched from spring bottomed boxes which interfere with their inherent route selection and maintenance logic routines. Shut down procedures are usually incomplete, and proper signal 9 procedures are not exercised at these locations; packet carriers often require hours to days to finish shutting down. Steve Hindi’s organization has even reversed the shutdown routines of 21 potential packet carriers and returned them to full network service functionality. When potential packet carriers are retrieved by those shutting them down (or attempting to shut them down), they are not put to alternative uses (such as the pie recipe passed down to Ted by his Polish grandmother or less dignified consumption by cats.) They are usually burned, but on at least one occasion, left in an open pile, as shown in one Youtube video. I should caution potential RFC1149 and RFC2549 network administrators that this video contains graphic imagery of over one hundred shut down packet carriers and three still in their shutdown routines:

    http://www.youtube.com/watch?v=uLu4PxCC6qw

    I’m posting it here so that interested parties not yet aware of the careless destruction of our RFC1149 and RFC2549 network infrastructure and Steve Hindi’s efforts to protect it may become aware of them. Thank you.

  30. Hacker “Nachwuchs”?

    Hackable: Ubuntu boots on 8-bit microcontroller
    http://www.h-online.com/open/news/item/Hackable-Ubuntu-boots-on-8-bit-microcontroller-1495707.html

    All that’s left is that pesky 32-bit CPU & MMU requirement. Well the AVR has no MMU and is 8-bit. To conquer this obstacle, I wrote an ARM emulator. ARM is the architecture I am most familiar with, and it’s simple enough that I could comfortably write an emulator for it. Why write one instead of porting one? Well, porting someone else’s code is no fun, plus none of the emulators I saw out there were written in a way that would make them easy to port to an 8-bit device. One of the factors: AVR compiler insists on making ints 16-bit so something as simple as “(1 << 20)" will get you in trouble, producing zero. Instead you need to do "(1UL << 20)". Needless to say trawling someone else's unknown codebase looking for all places where ints are assumed and would fail would be a disaster. Plus I wanted a chance to write a nice modular ARM emulator. So I did.

    This is in line with this goldy oldy:
    Run Linux on… JavaScript?!
    http://www.readwriteweb.com/hack/2011/05/run-linux-on-javascript.php

    Fabrice Bellard built an x86 PC emulator using JavaScript. He’s even gone so far as to install Linux kernel 2.6.20 on top of it. “The code is written in pure Javascript using Typed Arrays which are available in recent browsers,” Bellard wrote. It works in Firefox and Chrome 11, but not not Chrome 12.

    Eric once wrote he planned a post on this. I am still curious for his comments.

Leave a Reply to Winter Cancel reply

Your email address will not be published. Required fields are marked *