NTPsec is preparing for a release, which brought a question to the forefront of my mind. Are tarballs obsolete?
High on my list of Things That Annoy Me When I Hack is sourcefiles that contain huge blobs of license text at the top. That is valuable territory which should be occupied by a header comment explaining the code, not a boatload of boilerplate that I’ve seen hundreds of times before.
Hackers have a lot of superstitious ideas about IP law and one is that these blobs are necessary for the license to be binding. They are not: incorporation by reference is a familiar concept to lawyers and courts, it suffices to unambiguously name the license you want to apply rather than quoting it in full.
This is what I do in my code. But to make the practice really comfortable for lawyers we need a registry of standardized license identifiers and an unambiguous way of specifying that we intend to include by reference.
Comes now the Software Package Data Exchange to solve this problem once and for all. It’s a great idea, I endorse it, and I will be using it in all my software projects from now on.
Sometimes you find performance improvements in the simplest places. Last night I improved the time-stepping precision of NTP by a factor of up to a thousand. With a change of less than 20 lines.
The reason I was able to do this is because the NTP code had not caught up to a change in the precision of modern computer clocks. When it was written, you set time with settimeofday(2), which takes a structure containing seconds and microseconds. But modern POSIX-conformant Unixes have a clock_settime(2) which takes a structure containing seconds and nanoseconds.
The following is a comment I just filed on FCC Docket 15-170, “Amendment of Parts 0, 1, 2, 15, and 18 of the Commission’s Rules et al.”
Thirty years ago I had a small hand in the design of the Internet. Since then I’ve become a senior member of the informal collegium that maintains key pieces of it. You rely on my code every time you use a browser or a smartphone or an ATM. If you ever ride in a driverless car, the nav system will critically depend on code I wrote, and Google Maps already does. Today I’m deeply involved in fixing Internet time service.
I write to endorse the filings by Dave Taht and Bruce Perens (I gave Dave Taht a bit of editorial help). I’m submitting an independent comment because while I agree with the general thrust of their recommendations I think they may not go far enough.
One of my followers on G+ asked me to comment on a Vox article,
What no politician wants to admit about gun control.
I’ve studied the evidence, and I don’t believe the effect of the Australian confiscation on homicides was significant. You can play games with statistics to make it look that way, but they are games.
As for the major contention of the article, it’s simply wrong. 80% of U.S. crime, including gun violence, is associated with the drug trade and happens in urban areas where civil order has partially or totally collapsed.
Outside those areas, the U.S. looks like Switzerland or Norway – lots of guns, very little crime. Those huge, peaceful swathes of high-gun-ownership areas show that our problem is not too many guns, it’s too many criminals.
I’ve been pretty quiet on what’s going on with NTPsec since posting Yes, NTPsec is real and I am involved just about a month ago. But it’s what I’m spending most of my time on, and I have some truly astonishing success to report.
The fast version: in three and a half months of intensive hacking, since the NTP Classic repo was fully converted to Git on 6 June, the codebase is down to 47% of its original size. Live testing on multiple platforms seems to indicate that the codebase is now solid beta quality, mostly needing cosmetic fixes and more testing before we can certify it production-ready.
Here’s the really astonishing number…
shimulate, vt.: To insert a shim into code so it simulates a desired standardized ANSI/POSIX facility under a deficient operating system. First used of implementing clock_gettime(2) under Mac OS X, in the commit log of ntpsec.
I checked first use by Googling.
Who are the programmers who are famous for doing programmer things?
I’m wondering about this because my wife Cathy asked me a simple question last night, and I realized I didn’t have an answer to it. “Are you” she asked “the most famous programmer in the world?”
(This copies a comment I left on Derek Lowe’s blog at Science Magazine.)
I was the foundational theorist of open-source software development back in the 1990s, and have received a request to respond to your post on open-source pharma.
Is there misplaced idealism and a certain amount of wishful thinking in the open-source pharma movement? Probably. Something I often find myself pointing out to my more eager followers is that atoms are not bits; atoms are heavy, which means there are significant limiting factors of production other than human attention, and a corresponding problem of capital costs that is difficult to make go away. And I do find people who get all enthusiastic and ignore economics rather embarrassing.
On the other hand, even when that idealism is irrational it is often a useful corrective against equally irrational territoriality. I have observed that corporations have a strong, systemic hunker-down tendency to overprotect their IP, overestimating the amount of secrecy rent they can collect and underestimating the cost savings and additional options generated by going open.
I doubt pharma companies are any exception to this; when you say “the people who are actually spending their own cash to do it have somehow failed to realize any of these savings, because Proprietary” as if it’s credulous nonsense, my answer is “Yes. Yes, in fact, this actually happens everywhere”.
Thus, when I have influence I try to moderate the zeal but not suppress it, hoping that the naive idealists and the reflexive hunker-downers will more or less neutralize each other. It would be better if everybody just did sound praxeology, but human beings are not in general very good at that. Semi-tribalized meme wars fueled by emotional idealism seem to be how we roll as a species. People who want to change the world have to learn to work with human beings as they are, not as we’d like them to be.
If you’re not inclined to sign up with either side, I suggest pragmatically keeping your eye on the things the open-source culture does well and asking if those technologies and habits of thought can be useful in drug discovery. Myself, I think the long-term impact of open data interchange formats and public, cooperatively-maintained registries of pre-competitive data could be huge and is certainly worth serious investment and exploration even in the most selfish ROI terms of every party involved.
The idealists may sound a little woolly at times, but at least they understand this possibility and have the cultural capital to realize it – that part really is software.
Then…we see what we can learn. Once that part of the process has been de-territorialized, options to do analogous things at other places in the pipeline may become more obvious,
P.S: I’ve been a huge fan of your “Things I Won’t Work With” posts. More, please?
I created a Patreon page just before leaving for vacation on 2 Aug. The background to this is that while I’m now getting some regular bucks for working on NTPsec, it’s not a lot. Royalties from my books have been dwindling and my wife Cathy isn’t making all that much from legal contract gigs that are all she can get since Obamacare costs killed her full-time law job. Add the fact that our eight-year-old car has developed problems that would cost more to fix than its book value, and the house needs a new roof, and it’s looking pretty broke out.
When I built the Great Beast of Malvern, it was intended for surgery on large repositories. The specific aim in view was to support converting the NetBSD CVS to git, but that project is stalled because the political process around NetBSD’s decision about when to move seems to have seized up. I’ve got the hardware and software ready when they’re ready to move.
Now I have another repo-conversion job in the offing – and it does something I thought I’d never see. The working set exceeds 32GB! For comparison, the working set of the entire NetBSD conversion tops out at about 18GB.
What, you might well ask, can possibly have a history that huge? And the answer is…GCC. Yes, they’re looking to move from Subversion to git. And this is clearly a job for the Great Beast, once the additional 32GB I just ordered from Newegg arrives.
Sometimes, the best encouragement you can get in a martial-arts class is silence.
Once a month my school, which normally teaches a combination of wing chun kung fu and Philippine blade/stick fighting, gets a visit from Sifu Jerry Devone, who teaches pure traditional Wing Chun at a level a bit higher than our Sifu Dale Yeager.
Sifu Jerry is a nice guy, but it’s not difficult to find videos of him almost casually destroying other kung fu players in ring fights. He shows the same soft/hard combination as an instructor – never yells at anyone, but demands precision and perfection and often gets it, even from unpromising students.
Tonight he told the four senior students (including Cathy and myself) to line up facing him and do the Siu Nim Tao form with him watching for defects. From some instructors this would be a terrifying prospect, with anticipation of a humiliating ass-chewing to follow. While Sifu Dale wouldn’t exactly humiliate someone who screwed up, he might make snarky theatrical jokes about bad performance in a half-laughing-with, half-laughing-at manner. Neither of these is Sifu Jerry’s style – he’d just quietly correct in a way that would make you grimly determined to get it right next time.
Still, I felt rather stressed. I know the motions of Siu Nim Tao – it’s not a complex form, and it doesn’t require anything I’m bad at like high kicking – but it’s subtle. There are fine details in it, and the devil is in those details, and in getting the overall flow and timing right.
A couple of stories by Charles Babcock and (my coincidentally old friend) Steven J. Vaughan-Nichols have mentioned the existence of an ‘NTPsec’ project being funded by the Core Infrastructure Initiative as an alternative and perhaps eventual replacement for the reference implementation of Network Time Protocol maintained by Harlan Stenn and the Network Time Foundation.
I confirm that NTPsec does exist, and that I am deeply involved in it.
The project has not yet officially released code, though you can view a preliminary web page at ntpsec.org. For various complicated political reasons a full public discussion of the project’s genesis and goals should wait until we go fully public. You probably won’t have to wait long for this.
I can, however, disclose several facts that I think will be of interest to readers of this blog…
So here’s how my day went….
I started off trying to convert a legacy manual page to asciidoc. Found that pandoc (which could be the target of a whole separate rant, because it totally sucks at translating anything with tables in it) won’t do that.
But it will convert DocBook to asciidoc. OK, so I can use my doclifter tool to convert the manual page to DocBook, then DocBook to asciidoc via pandoc I try this, and doclifter promptly loses its cookies.
Huh? Oh, I see an [nt]roff request in there I’ve never seen before. Must fix doclifter to handle that. Hack hack hack – done. Push the fix, think “I ought to ship a doclifter release”
I look at the doclifter repo. I see that the commit graph has an ugly merge bubble in it from where I forgot a –rebase switch when I was pulling last. It’s the pointless kind of bubble where someone else’s patch commutes with mine so the history might as well be linear and easier to read.
You know, before I ship it…I was planning to move that repo from the downstairs machine to GitLab anyway, I might as well fix that bubble in the process….
Now how do I do that? Hm, yeah, this patch-replay sequence will do it. I ought to can that procedure into a script because I’m doubtless going to have to do it again. (I hates pointless merge bubbles, I hates them forever…) Script done. Hm. I should publish this.
OK, throw together a man page and README and Makefile. Oh, right, I need a project logo for the web page. What should it be? Into my mind, unbidden, enters the image of the scrubbing bubble animated characters from an advertising campaign of my childhood. Win!
I google for images, find one, and GIMP out a suitable piece within individual scrubbing bubble in it. Snrk, yeah, that’s funny. Scale it to 64×64 and go.
Funny logo achieved.
OK, version 1.0 of git-debubble gets published.
git-debubble gets applied to the doclifter repo,
…which I then publish to GitLab.
Now I can convert the manual page to DocBook XML…
…which can then be converted to asciidoc.
I have a lot of days like this.
I think there ought to be a term for a sequence of dependency fulfillments that is a lot like yak shaving, except that something useful or entertaining independently of the original task gets emitted at each stage.
If you hate pointless merge bubbles as much as I do, you’ll kick yourself because you didn’t think of this.
But don’t feel bad. I should have, a lot sooner, myself.
I think it’s weird that I have to write this post in 2015, but earlier today I had to explain to someone with the technical skills to submit a good patch that he was doing the process wrong in some basic and extremely annoying ways.
Googling revealed that most explanations of patch etiquette are rather project-specific in their advice. So I’m going to explain the basics of patch submission that apply to just about any open-source project, with a focus on how to do it right when you aren’t a regular committer (that is, it’s what’s often called a drive-by patch). Here we go…
Welcome to my first new-project release of the year, git-weave. It’s a polished and documented version of the script I used to reconstruct the early history of INTERCAL five years ago – see Risk, Verification, and the INTERCAL Reconstruction Massacree for the details on that one.
git-weave can be used to explode a git repository into a sequence of per-commit directory trees accompanied by a metadata file describing parent-child linkage, holding committer/author/timestamps/comment metadata, and carrying tags.
Going in the other direction, it can take the same sequence of trees plus metadata file and reconstruct the live repository. Round-tripping is lossless.
What it’s really useful for is reconstructing a partial but useful ancient history of a project from before it was put under version control. Find its release archives, synthesize a metadata file, apply this tool, and you get a repository that can easily be glued to the modern, more continuous history.
Yes, you only get a commit for each release tree or patch you can dig up, but this is better than nothing and often quite interesting.
Nifty detail: the project logo is the ancient Egyptian hieroglyph for a weaver’s shuttle.
In my last post, I inveighed against using git-svn to do whole-repository conversions from Subversion to git (as opposed to its intended use, which is working a Subversion repository live through a git remote).
Now comes the word that hundreds of projects a week seem to be fleeing SourceForge because of their evil we’ll-hijack-your-repo-and-crapwarify-your installer policy. And they’re moving to GitHub via its automatic importer. Which, sigh, uses git-svn.
I wouldn’t trust that automatic importer (or any other conversion path that uses git-svn) with anything I write, so I don’t know how badly it messes things up.
But as a public service, I follow with a description of how a really well-done repository conversion – the kind I would deliver using reposurgeon – differs from a crappy one.
This is a public-service warning.
It has come to my attention that some help pages on the web are still recommending git-svn as a conversion tool for migrating Subversion repositories to git. DO NOT DO THIS. You may damage your history badly if you do.
Reminder: I am speaking as an expert, having done numerous large and messy repository conversions. I’ve probably done more Subversion-to-git lifts than anybody else, I’ve torture-tested all the major tools for this job, and I know their failure modes intimately. Rather more intimately than I want to…