<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Armed and Dangerous</title>
	<atom:link href="http://esr.ibiblio.org/?feed=rss2" rel="self" type="application/rss+xml" />
	<link>http://esr.ibiblio.org</link>
	<description> Sex, software, politics, and firearms. Life's simple pleasures...</description>
	<lastBuildDate>Sat, 21 Nov 2009 23:12:32 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.8.4</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>Hiding  the Decline: Prologue</title>
		<link>http://esr.ibiblio.org/?p=1431</link>
		<comments>http://esr.ibiblio.org/?p=1431#comments</comments>
		<pubDate>Sat, 21 Nov 2009 23:12:32 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=1431</guid>
		<description><![CDATA[According to the summaries I&#8217;ve seen, the 61 megabytes of email and documents net-jacked from the Climate Research Unit a few days ago do not &#8212; quite &#8212; reify conservatives&#8217; darkest fantasies about &#8220;the team&#8221; (as the network of professional anthropogenic-global-warming alarmists communicating through CRU likes to style itself). To do that, they&#8217;d have to [...]]]></description>
			<content:encoded><![CDATA[<p>According to the summaries I&#8217;ve seen, the 61 megabytes of email and documents net-jacked from the Climate Research Unit a few days ago do not &#8212; quite &#8212; reify conservatives&#8217; darkest fantasies about &#8220;the team&#8221; (as the network of professional anthropogenic-global-warming alarmists communicating through CRU likes to style itself). To do that, they&#8217;d have to contain marching orders from the Socialist International.</p>
<p>However, the excerpts I&#8217;ve seen are already quite damning enough; among other things, they are evidence of criminal conspiracy to violate the Freedom Of Information act. And I no longer have to speculate about the rest; I&#8217;ve downloaded the documents from Pirate Bay and will study them myself.</p>
<p>For those of you who have been stigmatizing AGW skeptics  as &#8220;deniers&#8221; and dismissing their charges that the whole enterprise is fraudulent? Hope you like the taste of crow, because I do believe there&#8217;s a buttload of it coming at you. Piping hot.</p>
<p>Am I going to blog about it?  Heh&#8230;try to stop me&#8230;</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&amp;p=1431</wfw:commentRss>
		<slash:comments>5</slash:comments>
		</item>
		<item>
		<title>Barbecue kings!</title>
		<link>http://esr.ibiblio.org/?p=1426</link>
		<comments>http://esr.ibiblio.org/?p=1426#comments</comments>
		<pubDate>Mon, 16 Nov 2009 20:53:58 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=1426</guid>
		<description><![CDATA[John Birmingham writes from Australia:

Even, and this is gonna hurt, the Americans have it all over us when it comes to cooking with fire, iron and tongs. In fact it&#8217;s arguable the American barbecue, or rather its plethora of regional variations on barbecue, set the gold standard worldwide for applying heat to meat while out [...]]]></description>
			<content:encoded><![CDATA[<p>John Birmingham <a href="http://www.brisbanetimes.com.au/opinion/blogs/blunt-instrument/flame-me-if-you-will-but-we-suck-at-grills/20091028-hkr7.html">writes from Australia</a>:</p>
<blockquote><p>
Even, and this is gonna hurt, the Americans have it all over us when it comes to cooking with fire, iron and tongs. In fact it&#8217;s arguable the American barbecue, or rather its plethora of regional variations on barbecue, set the gold standard worldwide for applying heat to meat while out of doors. While the popular image of American cooking, at least as practised by average Americans, involves squeezing a plastic sauce packet over something nasty in a chain restaurant, the truth is their barbecue specialists would put ours to shame. Undying, unutterable shame.
</p></blockquote>
<p>Alas, John, it is so.  I have eaten barbecue all over the U.S. and the world, and the kings of the genre are in this country.  Not in my part of it; I&#8217;m a Boston-born northerner and most barbecue where I live is as bland and bad as you describe.  As a general rule in the U.S. the further south you go the better the barbeque gets, with the acme reached in south Texas.  (Though the area around Memphis, further north, is a contender.)</p>
<p>Internationally, almost nobody even competes with the Southern U.S. for the barbecue crown.  Brazilian churrasco is the one exception I can think of &#8211; that stuff can give good ol&#8217; Texas &#8216;cue a run for its money.  But you ex-British-Empire types aren&#8217;t even properly in the running.  I&#8217;ve been to a backyard braai in South Africa and, while the spirit was there, the seasoning and cooking technique was sadly lacking, much like what you describe.</p>
<p>American cooking in general gets a bad rap internationally that it doesn&#8217;t deserve.  It&#8217;s as though foreigners think it&#8217;s still 1965 here or something. I can remember a time in my childhood when the slams were richly deserved &#8212; heck, I remember returning here from Europe in 1971 and having to wait more than a decade before I saw a decent piece of bread! But Americans got a clue about food in the 1980s and haven&#8217;t lost it since.  I learned this when I was traveling intensively around the turn of the century; most places I visited, even the &#8220;high-quality&#8221; food was inferior to what I ate at home.</p>
<p>UPDATE: I suppose it&#8217;s worth noting that Brazilian-style churrascarias have become the most recent high-end restaurant fad in the U.S., suggesting that other Americans generally agree with me that the style competes well with native &#8216;cue. Sadly, Korean barbecue failed to become naturalized here when that was tried in the 1990s. </p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&amp;p=1426</wfw:commentRss>
		<slash:comments>132</slash:comments>
		</item>
		<item>
		<title>The pragmatics of webscraping</title>
		<link>http://esr.ibiblio.org/?p=1411</link>
		<comments>http://esr.ibiblio.org/?p=1411#comments</comments>
		<pubDate>Sun, 15 Nov 2009 22:24:17 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=1411</guid>
		<description><![CDATA[Here&#8217;s an amplification of my previous post, Structure Is Not Meaning.  It&#8217;s an except from the ForgePlucker HOWTO on writing code to web-scrape project data out of forge systems.

Your handler class&#8217;s job is to extract project data.  If you are lucky, your target forge already has an export feature that will dump everything [...]]]></description>
			<content:encoded><![CDATA[<p>Here&#8217;s an amplification of my previous post, <a href="http://esr.ibiblio.org/?p=1387">Structure Is Not Meaning</a>.  It&#8217;s an except from the <a href="http://home.gna.org/forgeplucker/">ForgePlucker</a> HOWTO on writing code to web-scrape project data out of forge systems.</p>
<blockquote><p>
Your handler class&#8217;s job is to extract project data.  If you are lucky, your target forge already has an export feature that will dump everything to you in clean XML or JSON; in that case, you have a fairly trivial exercise using BeautifulStoneSoup or the Python-library JSON parser and can skip the rest of this section.</p>
<p>Usually, however, you&#8217;re going to need to extract the data from the same pages that humans use.  This is a problem, because these pages are cluttered with all kinds of presentation-level markup, headers, footers, sidebars, and site-navigation gorp &#8212; any of which is highly likely to mutate any time the UI gets tweaked.</p>
<p>Here are the tactics we use to try to stay out of trouble:</p>
<p>1. When you don&#8217;t see what you expect, use the framework&#8217;s self.error() call to abort with a message.  And put in <b>lots</b> of  expect checks; it&#8217;s better for a handler to break loudly and soon than to return bad data.  Fixing the handler to track a page mutation won&#8217;t usually be hard once you know you need to &#8211; and knowing you need to is why we have regression tests.</p>
<p>2. Use peephole analysis with regexps (as opposed to HTML parsing of the whole page) as much as possible.  Every time you get away with matching on strictly local patterns, like special URLs, you avoid a dependency on larger areas of page structure which can mutate.</p>
<p>3. Throw away as many irrelevant parts of the page as you can before attempting either regexp matching or HTML parsing. (The most mutation-prone parts of ppsages are headers, footers, and sidebars; that&#8217;s where the decorative elements and navigation stuff tend to cluster.) If you can identify fixed end strings for headers or fixed start strings for footers, use those to trim (and error out if they&#8217;re not there); that way you&#8217;ll be safe even if the headers and footers mutate.  This is what the narrow() method in the framework code is for.</p>
<p>4. Rely on forms. You can assume you&#8217;ll be logged in with authentication and permissions to modify project data, which means the forge will display forms for editing things like issue data and project-member permissions. Use the forms structure, as it is much less likely to be casually mutated than the page decorations.</p>
<p>5. When you must parse HTML, <a href="http://www.crummy.com/software/BeautifulSoup/documentation.html">BeautifulSoup</a> is available to handler classes.  Use it, rather than hand-rolling a parser, unless you have to cope with markup so badly malformed that it cannot cope.</p>
</blockquote>
<p>Actual field experience shows that throwing out portions of a page that are highly susceptible to mutation is a valuable tactic.  Also, think about where in the site a page lives.  Entry pages and other highly visible ones tend to get tweaked the most often, so the tradeoffs push you towards peephole methods and not relying on DOM structure.  Deeper in the site , especially on pages that are heavily tabular and mostly consist of one big form, relying on DOM structure is less risky.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&amp;p=1411</wfw:commentRss>
		<slash:comments>38</slash:comments>
		</item>
		<item>
		<title>Ego is for little people</title>
		<link>http://esr.ibiblio.org/?p=1404</link>
		<comments>http://esr.ibiblio.org/?p=1404#comments</comments>
		<pubDate>Tue, 10 Nov 2009 04:27:14 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=1404</guid>
		<description><![CDATA[When I got really famous and started to hang out with people at the top of the game in computer science and other fields, one of the first things I noticed is that the real A-list types almost never have a major territorial/ego thing going on in their behavior.  The B-list people, the bright [...]]]></description>
			<content:encoded><![CDATA[<p>When I got really famous and started to hang out with people at the top of the game in computer science and other fields, one of the first things I noticed is that the real A-list types almost never have a major territorial/ego thing going on in their behavior.  The B-list people, the bright second-raters, may be all sharp elbows and ego assertion, but there&#8217;s a calm space at the top that the absolutely most capable ones get to and tend to stay in.</p>
<p>I&#8217;m going to be specific about what I mean by &#8220;ego&#8221; now, because otherwise much of this essay may seem vague or wrongheaded.  I specifically mean psychologial egotism, not (for example) ethical egoism as a philosophical position.  The main indicators of egotism as I intend it here are are  loud self-display, insecurity, constant approval-seeking, overinflating one&#8217;s accomplishments, touchiness about slights, and territorial twitchiness about one&#8217;s expertise.  My claim is that egotism is a disease of the incapable, and vanishes or nearly vanishes among the super-capable.</p>
<p>It&#8217;s not only scientific fields where this is true.  For various reasons (none of which, fortunately, have been legal troubles of my own)  I&#8217;ve had to work with a lot of lawyers.  I&#8217;m legally literate, so a pattern I quickly noticed is this: the B-list lawyers are the ones who get all huffy about a non-attorney expressing opinions and judgments about the law.  The one time I worked with a stratospherically supercompetent A-list firm (I won&#8217;t name them, but I will note they have their own skyscraper in New York City)  they were so relaxed about recognizing capability in a non-lawyer that some language I wrote went straight into their court filings in a lawsuit with multibillion-dollar stakes.</p>
<p>This sort of thing has been noted before by other people and is almost a commonplace.  I&#8217;m bringing it up to note <em>why</em> that&#8217;s true, speaking from my own experience.  It&#8217;s not that people at the top of their fields are more virtuous. Well&#8230;actually I think people at the top of their fields <em>do</em> tend to be more virtuous, for the same reason they tend to be be more intelligent, less neurotic, longer-lived, better-looking, and physically healthier than the B-listers and below.  Human capability does not come in nearly divisible chunks;  almost every individual way that humans can excel is tangled up with other ways at a purely physiological level, with immune-system capability lurking behind a surprisingly large chunk of the surface measures. But I don&#8217;t think the mean difference in &#8220;virtue&#8221;, however you think that can actually be defined,  explains what I&#8217;m pointing at.</p>
<p>No.  It&#8217;s more that ego games have a diminishing return.  The farther you are up the ability and achievement bell curve, the less psychological gain you get from asserting or demonstrating your superiority over the merely average, and the more prone you are to welcome discovering new peers because there are so damn few of them that it gets <em>lonely</em>.  There comes a point past which winning more ego contests becomes so pointless that even the most ambitious, suspicious, external-validation-fixated strivers tend to notice that it&#8217;s no fun any more and stop.  </p>
<p>I&#8217;m not speaking abstractly here.  I&#8217;ve always been more interested in doing the right thing than doing what would make me popular, to the point where I generally figure that if I&#8217;m not routinely pissing off a sizable minority of people I should be pushing harder.  In the language of psychology, my need for external validation is low; the standards I try hardest to live up to are those I&#8217;ve set for myself.  But one of the differences I can see between myself at 25 and myself at 52 is that my limited need for external validation has decreased.  And it&#8217;s not age or maturity or virtue that shrunk it; it&#8217;s having nothing left to prove.</p>
<p><span id="more-1404"></span></p>
<p>I&#8217;m going to use myself as an example now, mainly because I don&#8217;t know anyone else&#8217;s story well enough to make the point I want to with it.  I&#8217;m the crippled kid who became a black-belt martial artist and teacher of martial artists. I&#8217;ve made the New York Times bestseller list as a writer. You can hardly use a browser, a cellphone, or a game console without relying on my code.  I&#8217;ve been a session musician on two records.  I&#8217;ve blown up the software industry once, reinvented the hacker culture twice, and am without doubt one of the dozen most famous geeks alive.  Investment bankers pay me $300 an hour to yak at them because I have a track record as a shrewd business analyst.  I don&#8217;t even have a BS, yet there&#8217;s been an entire academic cottage industry devoted to writing exegeses of my work.  I could do nothing but speaking tours for the rest of my life and still be overbooked.  Earnest people have struggled their whole lives to change the world less than I routinely do when I&#8217;m not even really trying.  Here&#8217;s the point: In what way would it make sense for me to be in ego or status competition with <em>anybody</em>?</p>
<p>And yet, there are people out there who are going to read the previous paragraph and think &#8220;Oh, that&#8217;s Eric&#8217;s ego again.  The blowhard.&#8221;  I&#8217;ve had a lot of time to get used to such reactions over the last decade, but it&#8217;s still hard for me not to collapse in helpless laughter at the implied degree of Not Getting It.  Now (limiting myself to a small random sample of the A-listers I&#8217;ve actually met and taken the measure of) Alan Kay or Terry Pratchett or David Friedman or Freeman Dyson&#8230;<em>they</em> would understand why I was laughing.  Because real A-listers are sui generis, and usually polymaths; they tend to have constellations of talent so extreme and idiosyncratic that they couldn&#8217;t even really be in ego competition with each other, let alone with those much less capable.  That&#8217;s supposing they wanted to be.</p>
<p>And generally they don&#8217;t want to be.  If you&#8217;re the kind of person who can make it to the top even in a <em>single</em> field (law or CS or whatever) you may not have started out with better things to do than compete for attention and glory, but by the time you make the A-list you&#8217;ve almost certainly discovered subtler games to play that are much more fun.  You&#8217;ll maintain a reputation because a reputation is a useful tool, but it&#8217;s not the <em>point</em> any more.  If it ever was.  In my experience this is even more true of polymaths, possibly because their self-images as competent people.have broader and more stable bases.</p>
<p>I think there are a couple of different reasons people tend to falsely attribute pathological, oversensitive egos to A-listers. Each reason is in its own way worth taking a look at.</p>
<p>The first and most obvious reason is projection.  &#8220;Wow, if I were as talented as Terry Pratchett, I know I&#8217;d have a huge ego about it, so I guess he must.&#8221; Heh.  Trust me on this; he doesn&#8217;t.  This kind of thinking reveals a a lot about somebody&#8217;s ego and insecurity, alright, but not Terry&#8217;s.</p>
<p>There&#8217;s a flip side to projection that I think of as the &#8220;Asimov game&#8221;. I met Isaac Asimov just a few months before he died.  Isaac had long been notorious for broadly egotistical behavior and a kind of cheerful bombast that got up a lot of peoples&#8217; noses.  But if you ever met him, and you were at all perceptive, you might see that it was all a sort of joke. Isaac was laughing inside at everyone who took bis &#8220;egotism&#8221; seriously &#8211; and, at the same time, watching hungrily for people who could see through the self-parody, because they might &#8211; <em>might</em> &#8211; actually be among the vanishingly tiny minority that constituted his actual peers.  The Asimov game is a constant temptation to extroverted A-listers; I&#8217;ve been known to fall into it myself.  It&#8217;s not really anybody&#8217;s fault that a lot of people are fooled by it.</p>
<p>Another confusing fact is that though A-listers may not be about ego or status competition, they will often play such games ruthlessly and effectively when that gets them something they actually want.  The something might be more money from a gig, or a night in the hay with an attractive wench, or whatever; the point is, if you catch an A-lister in that mode, you might well mistake for egotism some kinds of display behavior that actually serve much more immediate and instrumental purposes.  Your typical A-lister in that situation (and this includes me, now) is blithely unconcerned that a bystander might think he&#8217;s egotistical; the money or the wench or the whatever is the goal, not the approval or disapproval of bystanders.</p>
<p>Finally, a lot of people confuse arrogance with ego.  A-listers (and I am including myself, again, this time) are, as a rule, colossally arrogant. That is, they have utter confidence in their ability to meet challenges that would humble or break most people.  Do not be fooled by the self-deprecating manner that many A-listers cultivate; it is a mask adopted for social purposes, mostly to avoid freaking out the normal monkeys.  But this arrogance is not the same as egotism; in fact, in many ways it is the opposite.  It is possible to be arrogant about one&#8217;s abilities compared to the statistically average human being and the range of challenges one is likely to encounter, but deeply and genuinely humble when dealing with peers or contemplating the vastness of one&#8217;s own ignorance and incapability relative to what one could imagine being. In fact, this combination of attitudes is completely typical of the A-listers I have known.</p>
<p>The behaviors most people think of as &#8220;egotism&#8221; tend to be driven out by arrogance rather than motivated by it.  If you really believe bone-deep that you are superior, you don&#8217;t act insecure and twitchy and approval-seeking, because you just aren&#8217;t!  Arrogance doesn&#8217;t even have to be <em>justified</em> to drive out egotism &#8211; it just has to be there.  It&#8217;s all the more powerful an egotism-banisher when the arrogance is actually well-justified by the A-lister&#8217;s track record.  Thus, egotists are usually people who have not yet established their capability to themselves, or who had that confidence in the past but are beginning to doubt it.</p>
<p>Finally, I think a lot of people need to believe that A-listers invariably have flaws in proportion to their capabilities in order not to feel dwarfed by them.  Thus the widely cherished belief that geniuses are commonly mentally unstable; it&#8217;s not true (admissions to mental hospitals per thousand <em>drop</em> with increasing IQ and in professions that select for intelligence, with the lowest numbers among mathematicians and theoretical physicists) but if you don&#8217;t happen to be a genius yourself it&#8217;s very comforting.  Similarly, a dullard who believes A-listers are all flaky temperamental egotists can console himself that, though he may not be smarter than them, he is <em>better</em>.  And so it goes.</p>
<p>Ego is for little people.  I wish I could finish by saying something anodyne about how we&#8217;re all little when you come down to it, but I&#8217;d be fibbing.  Yeah, we&#8217;re all little compared to a supernova, but that&#8217;s beside the point.  And yeah, the most capable people in the world are routinely humbled by what they don&#8217;t know and can&#8217;t do, but that is beside the point too.  If you look at <em>how humans relate to other humans</em> &#8211; and in particular, how they manage self-image and &#8220;ego&#8221; and evaluate their status with respect to others&#8230;it really is different near the top end of the human capability range.  Better.  Calmer.  Sorry, but it&#8217; s true.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&amp;p=1404</wfw:commentRss>
		<slash:comments>134</slash:comments>
		</item>
		<item>
		<title>Maybe if moral cowardice cost money, it would be less common?</title>
		<link>http://esr.ibiblio.org/?p=1398</link>
		<comments>http://esr.ibiblio.org/?p=1398#comments</comments>
		<pubDate>Fri, 06 Nov 2009 01:10:24 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=1398</guid>
		<description><![CDATA[Heh.  State representative Fred Maslack of Vermont has proposed a bill under which non-gun-owners would have to register and pay a fee. Entertainingly enough, there is actual justification for this in a careful reading of the Vermont state constitution.
The Hon. Rep. Maslack is joking.  I think.  And I&#8217;m against requiring people who [...]]]></description>
			<content:encoded><![CDATA[<p>Heh.  State representative Fred Maslack of Vermont has proposed a bill under which <a href="http://www.resistnet.com/profiles/blogs/a-novel-idea-register-nongun">non-gun-owners would have to register and pay a fee</a>. Entertainingly enough, there is actual justification for this in a careful reading of the Vermont state constitution.</p>
<p>The Hon. Rep. Maslack is joking.  I think.  And I&#8217;m against requiring people who don&#8217;t want to bear arms to do so.  But gad, how tempting &#8211; because underlying his argument is a truth that the drafters of the Vermont and U.S. constitutions understood.  People who refuse to take arms in defense of themselves and their neighbors are  inflicting a cost on their communities far more certainly than healthy people who refuse to buy medical insurance (and yes, I do think that proposed mandate is an intended target of Maslack&#8217;s jab).  That externality is measured in higher crime rates, higher law-enforcement and prison budgets, and all the (dis)opportunity costs associated with increased crime.  And that&#8217;s before you get to the political consequences&#8230;</p>
<p>I&#8217;ve never made a secret of my evaluation that refusal to bear arms is a form of moral cowardice masquerading as virtue. Real adults know how precious human life is, when they are ethically required to risk it on behalf of others, and when killing is both necessary and justified. Real adults know that there is no magic about wearing a police or military uniform; those decisions are just as hard, and just as necessary, when we deny we&#8217;re making them by delegating them to others. Real adults do not shirk the responsibility that this knowledge implies.  And the wistful thought Rep Maslack&#8217;s proposal leaves me with is&#8230;maybe if moral cowardice cost money and humiliation, there would be less of it.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&amp;p=1398</wfw:commentRss>
		<slash:comments>163</slash:comments>
		</item>
		<item>
		<title>Structure Is Not Meaning</title>
		<link>http://esr.ibiblio.org/?p=1387</link>
		<comments>http://esr.ibiblio.org/?p=1387#comments</comments>
		<pubDate>Thu, 05 Nov 2009 15:42:21 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=1387</guid>
		<description><![CDATA[So, I announce ForgePlucker, and within a day I&#8217;ve got some guy from Y Combinator sneering at me for using regular expressions to parse HTML.  Says it&#8217;s &#8220;crappy code&#8221;.  The poor fool&#8230;he has fallen victim to a conceptual trap which I, fortunately, learned to avoid decades ago.  I could spout a freshet [...]]]></description>
			<content:encoded><![CDATA[<p>So, I announce ForgePlucker, and within a day I&#8217;ve got some guy <s>from Y Combinator</s> sneering at me for using regular expressions to parse HTML.  Says it&#8217;s &#8220;crappy code&#8221;.  The poor fool&#8230;he has fallen victim to a conceptual trap which I, fortunately, learned to avoid decades ago.  I could spout a freshet of theory about it, but instead I&#8217;m just going to utter a maxim: <b>Never confuse structure with meaning</b>.</p>
<p><span id="more-1387"></span></p>
<p>When you parse an XML or HTML document into a DOM, you&#8217;re not getting meaning.  You&#8217;re only getting structure.  The DOM structure.  And by doing that, you add a kind of dependency to your code that it didn&#8217;t have before &#8211; you become vulnarable to changes in the document&#8217;s parse structure that don&#8217;t change the meaning and wouldn&#8217;t have bitten you if you had stuck to doing local pattern-matching.  </p>
<p>That may be an acceptable or even necessary risk if the meaning you&#8217;re trying to extract is closely tied to the structure of the document. If what you&#8217;re looking at is pure high-church XML, that&#8217;s often the case.   But HTML?  It&#8217;s tag soup.  The structure the DOM tries to capture may not be well-formed at all.  Even if it is, some UI designer fiddling with presentation-level tags can easily mutate the structure in a way that that points your beautiful theoretically-neat XPath queries off into la-la land.</p>
<p>Yes, sometimes lxml or BeautifulSoup is the right thing.  I&#8217;ll probably use lxml when I get around to writing the parser for the XML state-dump format I&#8217;ve started to define.  That will be appropriate, because the generators for that format can guarantee well-formedness and aren&#8217;t going to be changed casually by UI designers who innocently believe they&#8217;re doing no harm.  </p>
<p>But that&#8217;s a rather different use case from HTML generated by someone else&#8217;s code.  It still could be that DOM-based parsing is the smart thing, if the HTML is stable and the generator was designed by somebody fastidious.  Best not to count on it, though.  Here&#8217;s an example of why&#8230;</p>
<p>At one point in my code, I have to parse HTML that presents as a two-column table on a bug index page.  First column is bug IDs, wrapped in anchor markup so they hotlink to bug detail pages.  Second column is the bug summary.  This table is embedded in huge amounts of gorp &#8211; page headers, page footers, a sidebar, various text annotations.  All I care about is the bug IDs, though.  I just want a list of them.</p>
<p>If I did what my unwise critic says is right, I&#8217;d take a parse tree of the entire page and then write a path query down to the table cell and the embedded link.  Which would be cute, and I could do that, and I&#8217;m sure code like that got him an A in college &#8211; but would leave me shit out of luck the second after, say, anything in the page&#8217;s  top-level structure changed.</p>
<p>So instead, I walk through the page looking for anything that looks like a hotlink wrapping literal text of the form #[0-9]*.  Actually, that oversimplifies; I look for a hotlink with a specific pattern of URL in a hotlink that I know points into the site bugtracker.</p>
<p>Now, ask yourself: what&#8217;s more likely to remain stable: the tag soup on the page, or the path structure of the forge site? Which is the more reliable cue to the data I actually want to extract?</p>
<p><b>Never confuse structure with meaning.</b></p>
<p>Sometimes, the brute-force hack-at-it-with-regexps approach really is best.  It looks stupid, but it gets the job done.</p>
<p>CORRECTION: The snotty kid (or snotty-kid soundalike) isn&#8217;t from Y Combinator.  That&#8217;s actually a relief &#8211; I know Paul Graham, I like Paul Graham, and the crew around him usually has more sense.  Anyway, this post was supposed to be about the critic&#8217;s mistake, not the critic.</p>
<p>UPDATE: I&#8217;ve had more to say on this topic in <a href="http://esr.ibiblio.org/?p=1411">The Pragmatics of Webscraping</a>.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&amp;p=1387</wfw:commentRss>
		<slash:comments>31</slash:comments>
		</item>
		<item>
		<title>Announcing ForgePlucker</title>
		<link>http://esr.ibiblio.org/?p=1369</link>
		<comments>http://esr.ibiblio.org/?p=1369#comments</comments>
		<pubDate>Thu, 05 Nov 2009 03:50:59 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=1369</guid>
		<description><![CDATA[I&#8217;ve been strongly hinting in recent blog entries that I planned to do something concrete about the data-jail problems of present open-source hosting sites.  Because I believe in underpromising and overperforming, I decided at the outset not to announce a project until I could not only show working code, but code with wide enough [...]]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve been strongly hinting in recent blog entries that I planned to do something concrete about the data-jail problems of present open-source hosting sites.  Because I believe in underpromising and overperforming, I decided at the outset not to announce a project until I could not only show working code, but code with wide enough coverage to make it crystal-clear that the project goals are achievable  with a relatively modest amount of effort.</p>
<p>That time has arrived.  I am very pleased to announce <a href="http://home.gna.org/forgeplucker/">ForgePlucker</a>, a project aimed at developing project-state extractor software for backup, offline analysis, and (eventually) re-importation.  The proof-of-concept code can extract complete issue-tracker state from Berlios, Gna!, or Savane &#8212; and issue trackers are probably the hardest part of the job.  I expect extraction of repository histories and developer permissions tables to be easier.  Extraction of mailing-list state  is probably a bit trickier than either of those, but doable.</p>
<p><span id="more-1369"></span></p>
<p>The code as it exists now is just 1100-odd lines of pure Python; it can dump tracker state in either JSON or an XML format.  Notice that this is already a production-ready tool for, among other things, examining bug lists offline. One of the goals in the project plan is: useful tools at every step of the way. This is <em>not</em> going to be a project where the developers toil in obscurity for years until releasing Epic Code That Changes Everything; the project plan lays out smaller deliverables that can be used to build cooperation with forge-system designers and other people interested in some of the data-transport problems associated with forges.</p>
<p>Accordingly, I&#8217;ve actually put as much effort into documentation than I have into code &#8211; there&#8217;s a project plan, a HOWTO on writing forge handler classes, and even a draft ontology of forge state we can use to translate among the data models of different forges.</p>
<p>Yes, I&#8217;m looking for co-developers.  What the project especially needs is people interested in taking responsibility for developing and maintaining the handler classes associated with different forge types.  Presently, I own the Savane and Berlios handlers; I&#8217;d like to give those away so I can concentrate on the framework code, and I hope to recruit owners for SourceForge, GForge/FusionForge, Launchpad, and Trac.</p>
<p>There&#8217;s lots of interesting work to be done here.  Much of it will be code, some of it will be standards and documentation. Relevant skills and technologies include Python, JSON, RDF, HTTP forms, web-scraping, test-driven development. As always, being highly motivated to address the problem is more important than knowing any specific toolset at the outset.  I expect this project to be genuine fun; it breaks naturally into small substeps on which you get rapid feedback at each stage, and testing for correctness will be relatively easy.</p>
<p>Once we field a forgeplucker that can pull complete project state off a suitably wide variety of forges, it will be time to think about tackling the re-import problem.  We may solve that by bolting importers onto existing forge systems one by one, or by launching a forge-development project that builds on our extractor tools.  Another direction we might go in is supporting scripted interaction with these systems.  At every stage, useful tools!</p>
<p>To participate, get an account on <a href="http://gna.org">Gna!</a> and apply to join ForgePlucker. You can also examine the <a href="https://gna.org/svn/?group=forgeplucker">source-code repository</a> and l<a href="https://gna.org/task/?group=forgeplucker">list of open tasks</a>.  You will probably want to join the mailing list.  I&#8217;m planning to set up an IRC channel as well.</p>
<p>(Yes, I&#8217;d rather have used a distributed VCS; I did my proof-of concept in Mercurial. But Subversion is the best we can do on Gna!, where I have the advantage of being a site administrator.)</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&amp;p=1369</wfw:commentRss>
		<slash:comments>27</slash:comments>
		</item>
		<item>
		<title>The future of software forges</title>
		<link>http://esr.ibiblio.org/?p=1359</link>
		<comments>http://esr.ibiblio.org/?p=1359#comments</comments>
		<pubDate>Fri, 30 Oct 2009 01:39:42 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=1359</guid>
		<description><![CDATA[I&#8217;m still not going to talk about my attack on the forge infrastructure problems quite yet; the software is coming along nicely, but I intend to announce only after it handles its fourth forge type (yes, that was a tease).  But I will say this: I now think I know what the future of [...]]]></description>
			<content:encoded><![CDATA[<p>I&#8217;m still not going to talk about my attack on the forge infrastructure problems quite yet; the software is coming along nicely, but I intend to announce only after it handles its fourth forge type (yes, that was a tease).  But I will say this: I now think I know what the future of forges looks like.  It&#8217;s called <a href="http://roundup.sourceforge.net/index.html">Roundup</a>, and it is <em>astonishingly</em> elegant and potentially more powerful than anything out there. <em>Anything</em>, not excluding the clever decentralized  systems like Fossil or Bugs Everywhere.</p>
<p>Here are the big wins:</p>
<p>1. Mailing lists, issue trackers, and online forums unify into *one* message queue that can be filtered in various ways.</p>
<p>2. Scriptable via XML-RPC or an email responder &#8216;bot.</p>
<p>3. Small base system with good extensibility &#8211; just three base classes (User. Msg, File) and the ability to define new classes.  &#8216;Issue&#8217; is a class built on top of these.</p>
<p>4. Arbitrary attributes per issue is basically free, with baked-in support for defining controlled vocabularies.</p>
<p>5. There&#8217;s a uniform way, called &#8220;designators&#8221;, for messages and other objects to refer to each other in text.</p>
<p>6. Small, clean implementation written in Python.</p>
<p>There are some things it needs, though&#8230; (Read the <a href="http://roundup.sourceforge.net/docs/design.html">Roundup design document</a> before continuing.)</p>
<p><span id="more-1359"></span></p>
<p>First, namespaces. A namespace is a set of User, Msg, and File objects that belong together. Each namespace gets its own hyperdatabase blob. Each namespace numbers objects separately. Each namespace has a unique name used as a prefix in designators.  Thus, if I say [gpsd/msg23], I&#8217;m asking for the 23rd msg object in namespace gpsd.  Everyone talking to a system system has, at all times, a current namespace.  If I just say [msg23], I am designating the 23rd message object in my current namespace.</p>
<p>Why the slash?  because, in the most general case, designators will be URIs &#8211; as, for example, [https://developer.berlios.de/gpsd/msg23].  Presto! There is now an internet-wide namespace of Roundup objects <em>all of which can refer to each other!</em></p>
<p>No prize for guessing that a namespace is a project. I&#8217;ve actually done two things above &#8211; I&#8217;ve defined all the required semantics for multiple projects to coexist in a forgelike system, and I&#8217;ve  created a way for projects to refer to each others&#8217; data even if they&#8217;re not on the same machine.  Both features will be sources of power.</p>
<p>The subtler thing about namespaces is that they are perfectly orthogonal to the system&#8217;s other primitive concepts.  This will pay off huge in some ways I can already foresee and many that I cannot yet.</p>
<p>I think the design also needs a primitive object class to encapsulate repositories.  But I may be wrong about that;   I&#8217;m still considering it.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&amp;p=1359</wfw:commentRss>
		<slash:comments>10</slash:comments>
		</item>
		<item>
		<title>Hacker superstitions about software licensing</title>
		<link>http://esr.ibiblio.org/?p=1350</link>
		<comments>http://esr.ibiblio.org/?p=1350#comments</comments>
		<pubDate>Mon, 26 Oct 2009 13:51:52 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=1350</guid>
		<description><![CDATA[Hackers have a lot of odd superstitions about software licensing.  I was reminded of this recently when a project maintainer asked me whether he needed to get a sign-off from each and every one of his contributors before switching from Apache v1 to Apache v2.  Here&#8217;s  what I told him:
My opinion is [...]]]></description>
			<content:encoded><![CDATA[<p>Hackers have a lot of odd superstitions about software licensing.  I was reminded of this recently when a project maintainer asked me whether he needed to get a sign-off from each and every one of his contributors before switching from Apache v1 to Apache v2.  Here&#8217;s  what I told him:</p>
<p>My opinion is this.  Under U.S. law &#8212; and I believe European codes are not different in this respect, because both are controlled by the Berne convention &#8212; a license change on a collection is grounds for protest or legal action only if the rights of the contributors are materially affected by the change.  That is, a court would have to be persuaded that the change caused a monetary loss or at least damage to a contributor&#8217;s public reputation.  If there is no such possibility, then there is no harm and no grounds for complaint.</p>
<p><span id="more-1350"></span></p>
<p>It is clear that there is such a claim when a license is changed from open source to proprietary, or from proprietary to open source, without the author&#8217;s consent (the legal categories that apply are &#8220;unjust enrichment&#8221; and perhaps &#8220;conversion&#8221;). But no such claim can plausibly be made about Apache v1 to v2.  A court would laugh at you if you tried.  The applicable rule in English and American common law is called &#8220;De minimis non curat lex&#8221; &#8211; &#8220;The law does not concern itself with trifles.&#8221;</p>
<p>I think the closest an open-source license change might come to meeting the &#8220;materially-affected&#8221; test would be a change from an infectious license like GPL to a non-infectious one.  Even that, I think, is doubtful.</p>
<p>Hackers have some weird superstitions in this area &#8211; they behave as though they think modifying a license even trivially is some sort of soul-stealing evil voodoo against the person who attached it, and they think the law treats license attachments as sacred and immutable.  It doesn&#8217;t &#8211; certainly not for collective works.</p>
<p>So I&#8217;m telling you that you may have gone beyond what the law requires by asking about the GPL-to-Apache-v1 license change, and you are <em>certainly</em> beyond it in worrying about Apache v1 to v2,</p>
<p>I think you did right in respecting hacker customs by going beyond the law in the first case, but to worry about the second would be excessive.</p>
<p>UPDATE: Yes, the maintainer had previously changed the project from GPL to Apache v1.  Then, another project that he wanted to amalgamate code with switched to Apache v2.</p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&amp;p=1350</wfw:commentRss>
		<slash:comments>25</slash:comments>
		</item>
		<item>
		<title>Fearing what might be true</title>
		<link>http://esr.ibiblio.org/?p=1337</link>
		<comments>http://esr.ibiblio.org/?p=1337#comments</comments>
		<pubDate>Fri, 23 Oct 2009 20:36:27 +0000</pubDate>
		<dc:creator>esr</dc:creator>
				<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://esr.ibiblio.org/?p=1337</guid>
		<description><![CDATA[I am not generally unhappy with my model of how the universe works.  Yes, it would be pleasant if there really were a beneficient creator-god and an afterlife; it would be nice if &#8220;good government&#8221; were actually a sustainable possibility rather than a fond but deluded hope in the minds of statists;  it [...]]]></description>
			<content:encoded><![CDATA[<p>I am not generally unhappy with my model of how the universe works.  Yes, it would be pleasant if there really were a beneficient creator-god and an afterlife; it would be nice if &#8220;good government&#8221; were actually a sustainable possibility rather than a fond but deluded hope in the minds of statists;  it would be just peachy if wishes were horses and everybody could be happy and rich.  But I&#8217;m reconciled to these things being not true because it seems to me they are <em>necessarily</em> not true &#8211; that is, for example, I cannot imagine a universe in which the actual existence of an intervening creator-god is actually compatible with the observed regularity of natural law, nor a universe in which scarcity  and agency problems do not imply that governments are subject to the <a href="http://esr.ibiblio.org/?p=984">iron laws of political economics</a>. </p>
<p>Other people can imagine theism or statism to be true, and so could I at one time &#8211; before I understood enough to grok the  contradictions they would entail.  But I&#8217;m not actually writing to argue for anarchism or atheism today; I brought those up as examples to make a more subtle point. It seems to me it&#8217;s actually more difficult to deal with unpleasant possible truths that are only contingent, because you can imagine how things could be different if you lived in a universe that was slightly different but recognizable.</p>
<p>I&#8217;ll give two examples of this that I&#8217;m still struggling with.  One: I fear feminism may be dangerously wrong.  Two: I fear that the crowd calling for the abolition of all forms of intellectual property may be dangerously right.</p>
<p><span id="more-1337"></span></p>
<p>I like living in a society where women are, generally speaking, as free to choose their own path in life as I am.  I like strong women, women who are confident and look me in the eye and see themselves as my equals. But I wonder, sometimes, if sexual equality isn&#8217;t doomed by biology.  The relevant facts are (a) men and women have different optimal reproductive strategies because of the asymmetry in energy investment &#8211; being pregnant and giving birth is a <em>lot</em> more costly and risky than ejaculating, and (b) a woman&#8217;s fertile period is a relatively short portion of her lifetime.  Following the logic out, it may be that the consequence of sexual equality is demographic collapse &#8212; nasty cultures which treat women like brood mares are the future simply because the nice cultures that don&#8217;t do that <em>stop breeding at replacement rates</em>.  </p>
<p>I&#8217;ve written before about how property rights are underpinned by Schelling points &#8212; places where the cost of rights enforcement rises discontinuously, creating boundaries that rival claimants can agree on even if they can&#8217;t signal each other reliably.  I&#8217;m disturbed by the fact that even though I&#8217;ve been thinking about the matter for years, I haven&#8217;t found any Schelling points in the theory of IP rights that look really stable.  This suggests that the IP abolitionists may win the argument in the end.  That could be very bad, because there are important kinds of creative work I don&#8217;t see how to fund without IP rights that allow creaters to capture positive externalities.  I&#8217;m not worried about software, because that can be funded from its value as an intermediate  good; I&#8217;m worried about music and novels and artistic goods with economics like those.</p>
<p>The difference between these two cases and my first two is this: I know nothing about the universe that makes it impossible that women should have longer fertile periods.  And there might be Schelling points in IP that I haven&#8217;t found yet because I&#8217;m looking in the wrong places. I hope so, anyway. </p>
<p>I haven&#8217;t gone into depth about any of these arguments because the specific beliefs I&#8217;m examining are not really the point of this essay, and you can&#8217;t address my issue by attacking them (that&#8217;s a hint to commenters, yes indeed it is!). What I&#8217;m actually poking at here  is the nexus among belief, emotion, and imagination.  The question  I <em>would</em> like to hear from commenters about is this: If there is something that you don&#8217;t want to be true, but  fear might be, and (like me) you&#8217;re pretty compulsive about following the evidence and the logic even if it leads you to unpleasant conclusions &#8212; how do <em>you</em> cope?  </p>
<p>Am I alone in feeling like unpleasant but necessary truths are easier to live with than unpleasant but contingent ones, or does this tell us something general and interesting about the psychology of belief maintainence?</p>
<p>UPDATE: Some people have gotten the wrong impression.  I didn&#8217;t ask &#8220;How do <em>you</em> cope?&#8221; because I&#8217;m having an existential crisis or anything, it&#8217;s that I&#8217;m curious about the range of cognitive and emotional strategies involved. I&#8217;m an engineer, I cope by finding the largest problem I can wrap my head around and attacking it with vigor; if we all do that, maybe we&#8217;ll win. It could be said that my strategy for emotional coping is  &#8220;don&#8217;t be passive&#8221;. </p>
]]></content:encoded>
			<wfw:commentRss>http://esr.ibiblio.org/?feed=rss2&amp;p=1337</wfw:commentRss>
		<slash:comments>210</slash:comments>
		</item>
	</channel>
</rss>
