Yes, it’s been a month since I posted here. To be more precise, 30 Days in the Hole – I’ve been heads-down on a project with a deadline which I just barely met. and then preoccupied with cleanup from that effort.
The project was reposurgeon’s biggest conversion yet, the 280K-commit history of the Gnu Compiler Collection. As of Jan 11 it is officially lifted from Subversion to Git. The effort required to get that done was immense, and involved one hair-raising close call.
I was still debugging the Go translation of the code four months ago when the word came from the GCC team that they has a firm deadline of December 16 to choose between reposurgeon and a set of custom scripts written by a GCC hacker named Maxim Kyurkov. Which I took a look at – and promptly recoiled from in horror.
The problem wasn’t the work of Kyurkov himself; his scripts looked pretty sane to me, But they relied on git-svn, and that was very bad. It works adequately for live gatewaying to a Subversion repository, but if you use it for batch conversions it has any number of murky bugs including a tendency to badly screw up the location of branch joins.
The problem I was facing was that Kyurkov and the GCC guys, never having had their noses rubbed in these problems as I had, might be misled by git-svn’s surface plausibility into using it, and winding up with a subtly damaged conversion and increased friction costs for the rest of time. To head that off, I absolutely had to win on 16 Dec.
Which wasn’t going to be easy. My Subversion dump analyzer had problems of it own. I had persistent failures on some particularly weird cases in my test suite, and the analyzer itself was a hairball that tended to eat RAM at prodigious rates. Early on, it became apparent that the 128GB Great Beast II was actually too small for the job!
But a series of fortunate occurrences followed. One was that friend at Amazon was able to lend me access to a really superpowered cloud machine with 512GB. The second and much more important was in mid-October when a couple of occasional reposurgeon contributors, Julien “__FrnchFrgg__” Rivaud and Daniel Brooks showed up to help – Daniel having wangled his boss’s permission to go full-time on this until it was done. (His boss whose company is critically dependent on GCC flourishing…)
Many, many hours of hard work followed – profiling, smashing out hidden O(n**2) loops that exploded on a repo this size, reducing working set, fixing analyzer bugs. I doubled my lifetime consumption of modafinil. And every time I scoped what was left to do I came up with the same answer: we would just barely make the deadline. Probably.
Until…until I had a moment of perspective after three week of futile attempts to patch the latest round of Subversion-dump analyzer bugs and realized that trying to patch-and-kludge my way around the last 5% of weird cases was probably not going to work. The code had become a rubble pile; I couldn’t change anything without breaking anything.
It looked like time to scrap everything downstream of the first-stage stream parser (the simplest part, and the only one I was completely sure was correct) and rebuild the analyzer from first principles using what I had learned from all the recent failures.
Of course the risk I was taking was that come deadline time the analyzer wouldn’t be 95% right but rather catastrophically broken – that there simply wouldn’t be time to get the cleaner code working and qualified. But after thinking about the odds a great deal, I swallowed hard and pulled the trigger on a rewrite.
I made the fateful decision on 29 Nov 2019 and as the Duke of Wellington famously said, “It was a damned near-run thing.” If I had waited even a week longer to pull that trigger, we would probably have failed.
Fortunately, what actually happened was this: I was able to factor the new analyzer into a series of passes, very much like code-analysis phases in a compiler. The number fluctuated, there ended up being 14 of them, but – and this is the key point – each pass was far simpler than the old code, and the relationships between then well-defined. Several intermediate state structures that had become more complication than help were scrapped.
Eventually Julien took over two of the trickier intermediate passes so I could concentrate on the worst of the bunch. Meanwhile, Daniel was unobtrusively finding ways to speed the code and slim its memory usage down. And – a few days before the deadline – the GCC project lead and a sidekick showed up on our project channel to work on improving the conversion recipe.
After that formally getting the nod to do the conversion was not a huge surprise. But there was a lot of cleanup, verification, and tuning to be done before the official repository cutover on Jan 11. What with one thing and another in was Jan 13 before I could declare victory and ship 4.0.
After which I promptly…collapsed. Having overworked myself, I picked up a cold. Normally for me this is no big deal; I sniffle and sneeze for a few days and it barely slows me down. Not this time – hacking cough, headaches, flu-like symptoms except with no fever at all, and even the occasional dizzy spell because the trouble spread to my left ear canal.
I’m getting better now. But I had planned to go to the big pro-Second Amendment demonstration in Richmond on Jan 20th and had to bail at the last minute because I was too sick to travel.
Anyway, the mission got done. GCC has a really high-quality Git repository now. And there will be a sequel to this – my first GCC compiler mod.
And posting at something like my usual frequency will resume. I have a couple of topics queued up.
Glad to hear you are feeling better. Too much of that going around this season.
Glad you’re on the mend, and selfishly looking forwards to increased post frequency.
I’d love to know more about that thinking process, if it’s something you’d care to share.
>I’d love to know more about that thinking process, if it’s something you’d care to share.
I wish I could articulate it. I’ll give that some thought.
>And posting at something like my usual frequency will resume. I have a couple of topics queued up.
Oh, I’ll just bet.
Take care of your health, man. It’s the only one you’ve got.
Wow. Great read! Congratulations!
Sounds like you got bitten by what we’re calling the “12 Days of Christmas Cold” — between friends and family (and myself) it’s hit at least a half-dozen in my small circle in Maryland.
Great job on the conversion – a big job indeed! Welcome back!
I think the surname should be Kuvyrkov, not Kyurkov.
Maybe Kryukov? That sounds right and is pretty widespread.
I have taken the surname from the GCC mailing list here: https://gcc.gnu.org/ml/gcc/2019-12/msg00533.html (see the From: field).
Kuyrkov is perfectly correct Russian surname. Kuyrok means “firearm’s hammer”, so his ancestors were in firearm’s manufacture. Probably.
Thought your link was going to take me to a version of: https://www.youtube.com/watch?v=f8hT3oDDf6c :)
Why actually care about this? If the GCC folks had gone ahead and done yet-another-stupid thing, so what? Were they paying you?
There’s no reason they couldn’t simply put the old repo into read-only mode for historical interest and simply check in a fixed version into git and move forward. Anything beyond that is a bonus for them.
You underrate the importance of history. Too many people are already facing too much frustration because the early change history of many projects has been lost. Nobody was actually archiving a complete set of tarball hosts to the old archives like simtel et al, and so much has been lost.
Why the frustration, though? Sure – archivists might care in the abstract. But I’m still not getting the day-to-day need.
>Why actually care about this?
I told you there was going to be a sequel. All will eventually become clear.
Sorry for your having to miss Richmond. From reports I’ve read, it would’ve been a reaffirming experience, aside from the legislature then ignoring it…
I especially loved the media telling everyone it was a gathering of white nationalists.
. . .Including the Black Panthers.
I (a Virginia resident) had the offer of a
ride to the demonstration (from a fellow
member of a ham radio club — hams tend to
skew more conservative than programmers or
SF fans do). I turned it down. I don’t see
why I should support anyone’s RKBA if they
won’t support mine. I permanently lost mine
because I was falsely convicted of a
non-violent crime 43 years (!) ago. My
record is otherwise perfectly clean before
and since. The NRA supports Virginia’s
“Project Exile,” i.e. a mandatory five-year
prison sentence without parole if I’m
ever alone in a home, office suite, or
vehicle in which there’s an unsecured
firearm, even if I didn’t know it was there.
That’s like the AAA demanding the same
sentence for a passenger alone in a car
with the keys in the ignition while its
driver steps out to use a toilet, if the
passenger had been ticketed for riding a
bike too fast half a century ago. After
all, cars kill more Americans than guns do,
even though there are more guns than cars in
the US. So I am not an NRA supporter.
>The NRA supports Virginia’s “Project Exile,”
The NRA had nothing to do with the Lobby Day demonstration and declined to support it, because the NRA gets the screaming fantods at the thought of armed citizens making a public assertion of their rights. You pointlessly snubbed people who are much harder-core about opposing bullshit laws and would support your grievance.
I was “snubbing” the person who invited me and offered me a ride, as
he was most certainly an NRA member. As for VCDL, the sponsoring
organization of the Richmond demonstration, nowhere on their website
do they state any position on the rights of “felons.” If their
position is different from that of the best-known pro-RKBA
organization, the burden is on them to say so, since otherwise people
will assume it’s the same. Anyhow, the demonstration had no effect,
except to start a rather silly conversation about conservative
counties seceding from Virginia and joining West Virginia, which is
certainly never going to happen since it would require the approval
of both states and of both houses of the US Congress.
>If their position is different from that of the best-known pro-RKBA
organization, the burden is on them to say so, since otherwise people
will assume it’s the same.
Uh. No. Only people utterly ignorant of everything going on politically in the gun culture will assume that. The NRA is the squishy, compromising end of the activism spectrum. You can pretty much take it as a given that any other 2A organization will be harder-line on pretty much any related issue.
>I was “snubbing” the person who invited me and offered me a ride, as
he was most certainly an NRA member.
Did you ask him whether he supported “Project Exile?” Odds are he doesn’t – the NRA’s members are not in general as squishy as the NRA brass.
I don’t think I asked him specifically, but the usual answer from NRA
members is some variant of the Shirley Exception or the Just World
Fallacy, i.e. they support Project Exile, but are sure that somehow it
doesn’t apply to me, only to bad people. Or that all I have to do is
tell the state that they made a mistake and it will promptly apologize
and reverse my conviction. In my experience NRA members tend to be
*more* “squishy” than the NRA. That’s why the NRA split off the
NRA-ILA — so that people who oppose the NRA’s “extreme” politics can
pay for member services without also paying for the lobbying. So,
what’s *your* opinion? When should convicted felons get their gun
>So,what’s *your* opinion? When should convicted felons get their gun
I never thought they should lose those rights in the first place.
I believe the only actual point of that provision was as battlespace prep for making all firearms rights creatures of legislative fiat, preparatory to abolishing them.
In fact, one would naively expect the American _left_ to be fighting in the trenches on this issue, given the disproportionate number of black Americans with felony convictions. (E.g. the difference in treatment of cocaine vs. crack convictions).
But they’re not. Which is telling.
Well they’re solution to that problem is to simply not enforce the laws against blacks.
It’s impressive and inspiring that you still have so much fire in your belly. You deserve a million dollar bonus. In lieu of that, here’s one — attaboy!
>It’s impressive and inspiring that you still have so much fire in your belly. You deserve a million dollar bonus. In lieu of that, here’s one — attaboy!
If you want to be helpful, join Loadsharers!
When you come up for air, might you be able to comment on the UPSide project? I was excited about that but the site looks like everyone has abandoned it.
There was a pro-second amendment rally in Richmond? When was that? All I heard about was some massive gathering of white supremacists and Trump supporters trying to intimidate the poor, noble, brilliant blackface governor!
20,000+ klansmen were there waving ‘feddy flags and burning negroes!
A good time was had by all.
Except the niggers, of course.
Great job! Refactoring the code is often difficult, even more so under time crunch deadline.
Anyway, if / when you have time could you work on improving tutorial-style reposurgeon documentation, so it is used more widely? See for example this article (though it was more about trying to use reposurgeon outside of its intended area of use): https://git.github.io/rev_news/2019/08/21/edition-54/#reposurgeon
>Anyway, if / when you have time could you work on improving tutorial-style reposurgeon documentation
It’s not really practical. See my next blog post.
Big warm hugs for you :)
Take care & get well.
You were almost bankrupted by the technical debt in existing tools.
Ubergeek: a bankruptcy manager for someone with too much technical debt.
The cultural decay is profound. Great code has an aesthetic – is “beautiful” in addition to good and true. What top coders understand is Truth, goodness, and beauty coexist or not at all. The spaghetti code that handles the test cases will break, and will cost more to fix than replace.
I suspect Boeing is NOT learning this.
The worst part is when you see some horrid blob and have to decide to debug, refactor, or discard.
Truth, goodness, and beauty can’t always coexist. Sometimes the problem domain doesn’t allow for anything approaching “beauty”. Those of us lucky enough to work in “Enterprise computing” are painfully aware of bespoke applications (or heavy customizations to COTS apps) to accommodate Byzantine business logic, and more recently regulatory impacts of SOX, HIPAA, ACA, STIG, and GDPR,
Perhaps the canonical example of Irreducible Suckage is laid out in Tom Scott’s epic time zone rant (to which any of us who have had to deal with that problem space, including our gracious host, can only nod in agreement while wincing in remembered pain from his own battle injuries). The closest to “beauty” anyone can hope for in such situations is “not butt-ugly” and “just call this library and let its maintainers worry about the problem”.
A clean API to a well-implemented (if ugly in the details) library is indeed a thing of beauty.
Off topic: I believe Prof. Jussim was working on this while you were still hip-deep in GCC bits, but it’s right up your alley (and it even makes use of one of your best coinages):
Nice that it ends with
Pity that it didn’t link back to the A&D article that introduced the term.
Did it get edited? I see a link there on “A rhetorical move”.
Also: holy crap, there are a lot of pingbacks to “Kafkatrapping”.
There is absolutely a link to that specific article in there.
The underline was so faint I didn’t see it at first.
Also, that’s not the end of the list.
Glad to hear your feeling better!
OT: Oracle v. Google has made it to SCOTUS. Thoughts?
I dunno how apropos it is to post here, and if asked I will stop discussing the topic.
I couldn’t help but notice how all the amici — even Microsoft! — were in support of Google, except for a handful who were expressing their wish for the Supreme Court to preserve fair use or some other aspect of copyright law and submitted briefs “in favor of neither party”.
It’s far from a guarantee of success for Google; in the lower courts, there has been a tendency to interpret copyright law very broadly, and the decision may ultimately come down to how many justices have hit the links with Larry Ellison. But the fact that literally everyone with a dog in the hunt does not share Oracle’s interpretation of copyright law is a significant factor in how the case will turn.
>But the fact that literally everyone with a dog in the hunt does not share Oracle’s interpretation of copyright law is a significant factor in how the case will turn.
I think the most likely outcome is that the Federal Circuit ruling in Oracle’s favor will be vacated and the case remanded with a rebuke that they failed to apply the Altai precedent on merger and abstraction doctrine correctly.
Sorry to necropost, but…
> I think the most likely outcome is that the Federal Circuit ruling in Oracle’s favor will be vacated and the case remanded with a rebuke that they failed to apply the Altai precedent on merger and abstraction doctrine correctly.
Uh, yeah, about that… It seems a group called the “Internet Accountability Project” has filed an amicus brief on behalf of Oracle. And now the Trump admin has weighed in on Oracle’s side.
Honestly, I think the IAP and the Trump admin are in the wrong here. They’re so consumed with Google hatred that they’ll take any chance they can get to take them down a peg–even if it would have massive repercussions throughout our industry.
Yeah, maybe in the Clinton years. With Trump’s conservative, pro-business Supreme Court? It’s not so cut and dry. Think of it from a conservative’s perspective: Oracle, American success story, defending its IP (the lifeblood of American business) from the commie- and SJW-infested Google.
Barring that, think of it from a conservative justice’s perspective: Whether or not APIs should be specially excluded from eligibility for copyright protection is a matter for Congress, not the courts, to decide. Lower court ruling affirmed. Pay up, Google.