The Great Beast, designed for converting large CVS repos, is now in full production. It hasn’t killed off any specimens in the wild yet (and I’ll explain why in a bit), but it’s doing spectacularly well on our test repositories.
As a representative large example, the entire Emacs CVS history, 1985-2009, 113309 CVS commits, lifts clean in 37 seconds at a sustained rate of 3K CVS commits a second. Yes, three thousand.
The biggest beast known to us, the NetBSD src repository, converts in 22 minutes. To give some idea of what a speedup this is, the first time I ran a lift on it – on one of Wendell’s Xeon machines – it took a bit under six hours. That’s about a factor of seventeen, there.
Judging by performance on the other project devs’ machines the Beast is good for a 2x to 3x speedup over a conventionally-balanced PC design (that is, one with worse RAM latency, narrower caches, more cores but somewhat lower single-thread speed). That’s a big enough advantage to validate the design and be practically significant on large repositories.
Cathy and I passed our Level 6 test in kuntao last night.
That’s the hybrid martial art we study, part traditional wing chun and part Philippine kali. The empty hand stuff is mostly wing chun, a South Chinese close-fighting style which … OK, if you don’t know much about martial arts just imagine the fights in The Matrix without the high kicks. The weapons stuff is mostly kali, knife and stick and (relatively short) sword.
The good folks from TekSyndicate showed up yesterday with a pile of parts and did final assembly of the Beast in my dining room. A&D regular John Bell remoted in last night to finish the setup. I’m actually blogging on it now as the last of my work environment transfers over from the old snark.
What a beautiful machine it is! The interior of the NZXT case is even more impressive live than it is in photos. It runs whisper-quiet.
During the next several hours we’ll be filming documentary and interview footage. I’ll announce here when and where the video is available.
UPDATE: I have now had a chance to profile performance on some of the benchmark repos. I’m seeing speedups of between a factor of three (on Emacs CVS) and twenty (on groff CVS). The entire NetBSD src repository, 288K commits and 37GB of content, converts in 22 minutes.
Recently, in New York City, a man named Eric Garner was strangled to death on the street by police. It was all caught on video. It was a nightmare sequence that made me think of George Orwell’s description of the future in 1984: a boot stamping on a human face, forever.
Eric Garner was black. The policeman who choked him to death was white.
Some people want to make this horror about race. I find myself wishing they were right – that just once, the racial grievance peddlers weren’t basically making up inflammatory crap that canonizes thug trash like Trayvon Martin and Michael Brown. Because as bad as violent racism is, I’m afraid that what actually killed Eric Garner was something far worse.
I’ve gotten questions from a couple of different quarters recently about my relationship to the the rationalist community around Less Wrong and related blogs. The one sentence answer is that I consider myself a fellow-traveler and ally of that culture, but not really part of it nor particularly wishing to be.
The rest of this post is a slightly longer development of that answer.
How To Learn Hacking: Version 1.2, with a new section on being original. Incorporates more feedback from here and G++
For those of you who wondered why this didn’t just become a major section in How To Become A Hacker, it’s because I think it might become long enough to make that document too bulky to read at one sitting.
A lot of U.S. economic policy is distorted by the belief that manufacturing jobs are a magic bullet against declining incomes. Manufacturing’s false promise of a decent payday punctures that illusion.
One of the dumb, predictable responses to articles like this is “We need a stronger union movement”. Sorry, but no. Declining manufacturing wages aren’t an effect of the weakening of unions and can’t be reversed by strengthening them. Explanation follows.
OK, this is interesting: From some tabloid, we have the following quote:
The unidentified witness wrote that the 18-year-old Brown “has his arms out with attitude,” while “The cop just stood there.” The witness added, “Dang if that kid didn’t start running right at the cop like a football player. Head down.”
This is exactly how I reconstructed the event in This picture tells a shooting story. I said: the reason I’m sure Brown was moving is the extreme torso angle suggested by the lack of exit wounds on the back. A human trying to do that standing still would overbalance and fall, which is why I think he was running or lunging when he took the bullets.
The witness said “arms out with attitude”. I said “with his right arm stretched forward […] probably while Brown was grabbing for Wilson or the pistol with his right hand.”
So much for “Hands up – don’t shoot.” It’s as I thought: Brown autodarwinated, bull-rushing an armed policeman he had already injured once.
UPDATE: I failed to make clear before that this account was part of the evidence dump from the grand jury proceedings, not just some random the tabloid turned up.
I had meant to blog-announce version 1.0 of How To Learn Hacking but got distracted at a crucial moment. So it went out on G+, where I got some useful feedback.
This version is revised and very slightly expanded. Enjoy and critique.
I just shipped SRC 0.9, and you no longer need to be adventurous to try it. It has a regression-test suite and real users.
Remarkably, SRC has had real users since 0.3, two days after it was born. Even more remarkably, the count of crash reports and botched operations from those users is zero. Zero. This is what you can gain from keeping code simple – I have has a couple of bug reports but they were both about filename quoting in the fast-export code, which is not a central feature.
Next, I’ll make a couple of what I think are important points about writing for zero defects. Then I’ll talk about a subtle issue or two in the design, and our one known behavioral glitch.
Finally. After ten months of work, it’s done. Emacs is fully converted to git. You can clone from git://git.sv.gnu.org/emacs.git and if you have commit rights you can push to it and the changes will stick. The bzr repo is still up but only as an archive.
A&D regular Mike Swanson did such a nice job on this that I want you all to see it.
My low-power, low-overhead version control system, SRC, is no longer just a stake in the ground. It is still a determinedly file-oriented wrapper around RCS (and will stay that way) but every major feature except branching is implemented and it has probably crossed the border into being useful for production.
The adventurous can and should try it. You’re safe if it blows up because the histories are plain RCS files. But, as previously noted, it’s RCS behind an interface that’s actually pleasant to use. (You Emacs VC-mode users pipe down; I’m going to explain why you care in a bit.)
The main developments today include a fairly complete regression-test suite (already paying large dividends in speeding up progress) and a “src status” command that will look very familiar to Subversion/git/hg users. There’s a hack behind that status command I’m rather proud of; I’ll talk about that, too.
I wrote a version-control system today. Yes, an entire VCS. Took me 14 hours.
Yeah, you’re looking at me like I’m crazy. “Why,” you ask, quite reasonably, “would you want to do a thing like that? We’re not short of powerful VCSes these days.
That is true. But I got to thinking, early this morning, about the fact that I haven’t been able to settle on just one VCS. I use git for most things, but there’s a use case git doesn’t cover. I have some document directories in which I have piles of things like HOWTOs which have separate histories from each other. Changes in them are not correlated, and I want to be able to move them around because I sometimes do that to reorganize them.
What have I been using for this? Why, RCS. The ancient Revision Control System, second oldest VCS in existence and clinging tenaciously to this particular niche. It does single-file change histories pretty well, but its UI is horrible. Worse than git’s, which is a pretty damning comparison.
Then I got to thinking. If I were going to design a VCS to do this particular single-file, single-user job, what would it look like? Hm. Sequential integer revision numbers, like Subversion and Mercurial used locally. Lockless operation. Modern CLI design. Built-in command help. Interchange with other VCSes via git import streams. This sounds like it could be nice…
Then, the idea that made it inevitable. “I bet.” I thought, “I could write this thing as a Python wrapper around RCS tools. Use them for delta storage but hide all the ugly parts.”
Thus, SRC. Simple Revision Control, v0.1.
I’ve just shipped a new version of cvs-fast-export, 1.26. It speeds the tool up more, more, more – cranking through 25 years and 113300 commits of Emacs CVS history, for example in 2:48. That’s 672 commits a second, for those of you in the cheap seats.
But the real news this time is a Python wrapper called ‘cvsconvert’ that takes a CVS repository, runs a conversion to Git using cvs-fast-export, and then – using CVS for checkouts – examines the CVS and git repositories side by side looking for translation glitches. It checks every branch tip and every tag.
Running this on several of my test repos I’ve discovered some interesting things. One such discovery is of a bug in CVS. (Yeah, I know, what a shock…)
Here’s an interesting article with a stupid and misleading title on the role of what the author calls “cognitive disinhibition” – a fancy term for “allowing oneself to notice what others miss” – in enabling creative genius.
While in many ways I could be a poster child for Simonton’s thesis (and I’ll get to those) I also think there are some important things missing from his discussion, which is why I’m blogging about it. The most crucial problem is that his category of “madness” is not sharp enough. I know how to fine it down in a way that I think sheds considerable light on what he is trying to analyze.
Lately I’ve been wrestling with various members of an ancient and venerable open-source development group which I am not going to name, though people who regularly follow my adventures will probably guess which one it is by the time I’m done venting.
Why it so freaking hard to drag some people into the 21st century? Sigh…
I’m almost 56, an age at which a lot of younger people expect me to issue semi-regular salvos of get-off-my-lawn ranting at them. But no – I find, that, especially in technical contexts, I am far more likely to become impatient with my age peers.
A lot of them really have become grouchy, hidebound old farts. And, alas, it not infrequently falls to me to be the person who barges in and points out that practices well-adapted for 1995 (or, in the particular case I’m thinking of, 1985) are … not good things to hold on to decades later.
Why me? Because the kids have little or no cred with a lot of my age peers. If anyone’s going to get them to change, it has to be someone who is their peer in their own perception. Even so, I spend a lot more time than seems just or right fighting inertia.
Young people can be forgiven for lacking a clue. They’re young. Young means little experience, which often leads to unsound judgment. It’s more difficult for me to forgive people who have been around the track often enough that they should have a clue, but are so attached to The Way It’s Always Been Done that they can’t see what is in front of their freaking noses.
Some people on the NetBSD tech-repository list have wondered why I’ve been working on a full NetBSD repository conversion without a formal request from NetBSD’s maintainers that I do so.
It’s a fair question. An answer to it involves both historical contingency and some general issues about moving and mirroring large repositories. Because of the accident that a lot of people have recently dropped money on me in part to support an attack on this problem, I’m going to explain both in public.
Wendell Wilson over at TekSyndicate had a good idea – run the NetBSD repo conversion on a machine roughly comparable to the Great Beast design. The objective was (a) to find out if it freakin’ worked, and (b) to get a handle on expected conversion time and maximum working set for a really large conversion.
The news is pretty happy on all fronts.
I’ve attempted to summarize the discussion of build options for the repository-surgery machine. You should see a link at the top of the page: if not, it’s here
I invite all the commenters who have shown an interest to critique these build proposals. Naturally, I’d like to make sure we have a solid parts list with no spec conflicts before we start spending money and time to build this thing.