30 Days in the Hole

Yes, it’s been a month since I posted here. To be more precise, 30 Days in the Hole – I’ve been heads-down on a project with a deadline which I just barely met. and then preoccupied with cleanup from that effort.

The project was reposurgeon’s biggest conversion yet, the 280K-commit history of the Gnu Compiler Collection. As of Jan 11 it is officially lifted from Subversion to Git. The effort required to get that done was immense, and involved one hair-raising close call.

I was still debugging the Go translation of the code four months ago when the word came from the GCC team that they has a firm deadline of December 16 to choose between reposurgeon and a set of custom scripts written by a GCC hacker named Maxim Kyurkov. Which I took a look at – and promptly recoiled from in horror.

The problem wasn’t the work of Kyurkov himself; his scripts looked pretty sane to me, But they relied on git-svn, and that was very bad. It works adequately for live gatewaying to a Subversion repository, but if you use it for batch conversions it has any number of murky bugs including a tendency to badly screw up the location of branch joins.

The problem I was facing was that Kyurkov and the GCC guys, never having had their noses rubbed in these problems as I had, might be misled by git-svn’s surface plausibility into using it, and winding up with a subtly damaged conversion and increased friction costs for the rest of time. To head that off, I absolutely had to win on 16 Dec.

Which wasn’t going to be easy. My Subversion dump analyzer had problems of it own. I had persistent failures on some particularly weird cases in my test suite, and the analyzer itself was a hairball that tended to eat RAM at prodigious rates. Early on, it became apparent that the 128GB Great Beast II was actually too small for the job!

But a series of fortunate occurrences followed. One was that friend at Amazon was able to lend me access to a really superpowered cloud machine with 512GB. The second and much more important was in mid-October when a couple of occasional reposurgeon contributors, Julien “__FrnchFrgg__” Rivaud and Daniel Brooks showed up to help – Daniel having wangled his boss’s permission to go full-time on this until it was done. (His boss whose company is critically dependent on GCC flourishing…)

Many, many hours of hard work followed – profiling, smashing out hidden O(n**2) loops that exploded on a repo this size, reducing working set, fixing analyzer bugs. I doubled my lifetime consumption of modafinil. And every time I scoped what was left to do I came up with the same answer: we would just barely make the deadline. Probably.

Until…until I had a moment of perspective after three week of futile attempts to patch the latest round of Subversion-dump analyzer bugs and realized that trying to patch-and-kludge my way around the last 5% of weird cases was probably not going to work. The code had become a rubble pile; I couldn’t change anything without breaking anything.

It looked like time to scrap everything downstream of the first-stage stream parser (the simplest part, and the only one I was completely sure was correct) and rebuild the analyzer from first principles using what I had learned from all the recent failures.

Of course the risk I was taking was that come deadline time the analyzer wouldn’t be 95% right but rather catastrophically broken – that there simply wouldn’t be time to get the cleaner code working and qualified. But after thinking about the odds a great deal, I swallowed hard and pulled the trigger on a rewrite.

I made the fateful decision on 29 Nov 2019 and as the Duke of Wellington famously said, “It was a damned near-run thing.” If I had waited even a week longer to pull that trigger, we would probably have failed.

Fortunately, what actually happened was this: I was able to factor the new analyzer into a series of passes, very much like code-analysis phases in a compiler. The number fluctuated, there ended up being 14 of them, but – and this is the key point – each pass was far simpler than the old code, and the relationships between then well-defined. Several intermediate state structures that had become more complication than help were scrapped.

Eventually Julien took over two of the trickier intermediate passes so I could concentrate on the worst of the bunch. Meanwhile, Daniel was unobtrusively finding ways to speed the code and slim its memory usage down. And – a few days before the deadline – the GCC project lead and a sidekick showed up on our project channel to work on improving the conversion recipe.

After that formally getting the nod to do the conversion was not a huge surprise. But there was a lot of cleanup, verification, and tuning to be done before the official repository cutover on Jan 11. What with one thing and another in was Jan 13 before I could declare victory and ship 4.0.

After which I promptly…collapsed. Having overworked myself, I picked up a cold. Normally for me this is no big deal; I sniffle and sneeze for a few days and it barely slows me down. Not this time – hacking cough, headaches, flu-like symptoms except with no fever at all, and even the occasional dizzy spell because the trouble spread to my left ear canal.

I’m getting better now. But I had planned to go to the big pro-Second Amendment demonstration in Richmond on Jan 20th and had to bail at the last minute because I was too sick to travel.

Anyway, the mission got done. GCC has a really high-quality Git repository now. And there will be a sequel to this – my first GCC compiler mod.

And posting at something like my usual frequency will resume. I have a couple of topics queued up.

51 thoughts on “30 Days in the Hole

  1. Glad you’re on the mend, and selfishly looking forwards to increased post frequency.

    But after thinking about the odds a great deal

    I’d love to know more about that thinking process, if it’s something you’d care to share.

    • >I’d love to know more about that thinking process, if it’s something you’d care to share.

      I wish I could articulate it. I’ll give that some thought.

  2. >And posting at something like my usual frequency will resume. I have a couple of topics queued up.

    Oh, I’ll just bet.

  3. Sounds like you got bitten by what we’re calling the “12 Days of Christmas Cold” — between friends and family (and myself) it’s hit at least a half-dozen in my small circle in Maryland.

    Great job on the conversion – a big job indeed! Welcome back!

  4. Why actually care about this? If the GCC folks had gone ahead and done yet-another-stupid thing, so what? Were they paying you?

    There’s no reason they couldn’t simply put the old repo into read-only mode for historical interest and simply check in a fixed version into git and move forward. Anything beyond that is a bonus for them.

    • You underrate the importance of history. Too many people are already facing too much frustration because the early change history of many projects has been lost. Nobody was actually archiving a complete set of tarball hosts to the old archives like simtel et al, and so much has been lost.

      • Why the frustration, though? Sure – archivists might care in the abstract. But I’m still not getting the day-to-day need.

  5. Sorry for your having to miss Richmond. From reports I’ve read, it would’ve been a reaffirming experience, aside from the legislature then ignoring it…

    • I especially loved the media telling everyone it was a gathering of white nationalists.

      . . .Including the Black Panthers.

    • I (a Virginia resident) had the offer of a
      ride to the demonstration (from a fellow
      member of a ham radio club — hams tend to
      skew more conservative than programmers or
      SF fans do). I turned it down. I don’t see
      why I should support anyone’s RKBA if they
      won’t support mine. I permanently lost mine
      because I was falsely convicted of a
      non-violent crime 43 years (!) ago. My
      record is otherwise perfectly clean before
      and since. The NRA supports Virginia’s
      “Project Exile,” i.e. a mandatory five-year
      prison sentence without parole if I’m
      ever alone in a home, office suite, or
      vehicle in which there’s an unsecured
      firearm, even if I didn’t know it was there.
      That’s like the AAA demanding the same
      sentence for a passenger alone in a car
      with the keys in the ignition while its
      driver steps out to use a toilet, if the
      passenger had been ticketed for riding a
      bike too fast half a century ago. After
      all, cars kill more Americans than guns do,
      even though there are more guns than cars in
      the US. So I am not an NRA supporter.

      • >The NRA supports Virginia’s “Project Exile,”

        The NRA had nothing to do with the Lobby Day demonstration and declined to support it, because the NRA gets the screaming fantods at the thought of armed citizens making a public assertion of their rights. You pointlessly snubbed people who are much harder-core about opposing bullshit laws and would support your grievance.

        • I was “snubbing” the person who invited me and offered me a ride, as
          he was most certainly an NRA member. As for VCDL, the sponsoring
          organization of the Richmond demonstration, nowhere on their website
          do they state any position on the rights of “felons.” If their
          position is different from that of the best-known pro-RKBA
          organization, the burden is on them to say so, since otherwise people
          will assume it’s the same. Anyhow, the demonstration had no effect,
          except to start a rather silly conversation about conservative
          counties seceding from Virginia and joining West Virginia, which is
          certainly never going to happen since it would require the approval
          of both states and of both houses of the US Congress.

          • >If their position is different from that of the best-known pro-RKBA
            organization, the burden is on them to say so, since otherwise people
            will assume it’s the same.

            Uh. No. Only people utterly ignorant of everything going on politically in the gun culture will assume that. The NRA is the squishy, compromising end of the activism spectrum. You can pretty much take it as a given that any other 2A organization will be harder-line on pretty much any related issue.

          • >I was “snubbing” the person who invited me and offered me a ride, as
            he was most certainly an NRA member.

            Did you ask him whether he supported “Project Exile?” Odds are he doesn’t – the NRA’s members are not in general as squishy as the NRA brass.

            • I don’t think I asked him specifically, but the usual answer from NRA
              members is some variant of the Shirley Exception or the Just World
              Fallacy, i.e. they support Project Exile, but are sure that somehow it
              doesn’t apply to me, only to bad people. Or that all I have to do is
              tell the state that they made a mistake and it will promptly apologize
              and reverse my conviction. In my experience NRA members tend to be
              *more* “squishy” than the NRA. That’s why the NRA split off the
              NRA-ILA — so that people who oppose the NRA’s “extreme” politics can
              pay for member services without also paying for the lobbying. So,
              what’s *your* opinion? When should convicted felons get their gun
              rights back?

              • >So,what’s *your* opinion? When should convicted felons get their gun
                rights back?

                I never thought they should lose those rights in the first place.

                I believe the only actual point of that provision was as battlespace prep for making all firearms rights creatures of legislative fiat, preparatory to abolishing them.

                • In fact, one would naively expect the American _left_ to be fighting in the trenches on this issue, given the disproportionate number of black Americans with felony convictions. (E.g. the difference in treatment of cocaine vs. crack convictions).

                  But they’re not. Which is telling.

  6. It’s impressive and inspiring that you still have so much fire in your belly. You deserve a million dollar bonus. In lieu of that, here’s one — attaboy!

    • >It’s impressive and inspiring that you still have so much fire in your belly. You deserve a million dollar bonus. In lieu of that, here’s one — attaboy!

      If you want to be helpful, join Loadsharers!

  7. There was a pro-second amendment rally in Richmond? When was that? All I heard about was some massive gathering of white supremacists and Trump supporters trying to intimidate the poor, noble, brilliant blackface governor!

    • 20,000+ klansmen were there waving ‘feddy flags and burning negroes!
      A good time was had by all.
      Except the niggers, of course.

    • >Anyway, if / when you have time could you work on improving tutorial-style reposurgeon documentation

      It’s not really practical. See my next blog post.

  8. You were almost bankrupted by the technical debt in existing tools.

    Ubergeek: a bankruptcy manager for someone with too much technical debt.

    The cultural decay is profound. Great code has an aesthetic – is “beautiful” in addition to good and true. What top coders understand is Truth, goodness, and beauty coexist or not at all. The spaghetti code that handles the test cases will break, and will cost more to fix than replace.

    I suspect Boeing is NOT learning this.

    The worst part is when you see some horrid blob and have to decide to debug, refactor, or discard.

    • Truth, goodness, and beauty can’t always coexist. Sometimes the problem domain doesn’t allow for anything approaching “beauty”. Those of us lucky enough to work in “Enterprise computing” are painfully aware of bespoke applications (or heavy customizations to COTS apps) to accommodate Byzantine business logic, and more recently regulatory impacts of SOX, HIPAA, ACA, STIG, and GDPR,

      Perhaps the canonical example of Irreducible Suckage is laid out in Tom Scott’s epic time zone rant (to which any of us who have had to deal with that problem space, including our gracious host, can only nod in agreement while wincing in remembered pain from his own battle injuries). The closest to “beauty” anyone can hope for in such situations is “not butt-ugly” and “just call this library and let its maintainers worry about the problem”.

      • “just call this library and let its maintainers worry about the problem”

        A clean API to a well-implemented (if ugly in the details) library is indeed a thing of beauty.

    • I dunno how apropos it is to post here, and if asked I will stop discussing the topic.

      I couldn’t help but notice how all the amici — even Microsoft! — were in support of Google, except for a handful who were expressing their wish for the Supreme Court to preserve fair use or some other aspect of copyright law and submitted briefs “in favor of neither party”.

      It’s far from a guarantee of success for Google; in the lower courts, there has been a tendency to interpret copyright law very broadly, and the decision may ultimately come down to how many justices have hit the links with Larry Ellison. But the fact that literally everyone with a dog in the hunt does not share Oracle’s interpretation of copyright law is a significant factor in how the case will turn.

      • >But the fact that literally everyone with a dog in the hunt does not share Oracle’s interpretation of copyright law is a significant factor in how the case will turn.

        I think the most likely outcome is that the Federal Circuit ruling in Oracle’s favor will be vacated and the case remanded with a rebuke that they failed to apply the Altai precedent on merger and abstraction doctrine correctly.

        • Sorry to necropost, but…

          > I think the most likely outcome is that the Federal Circuit ruling in Oracle’s favor will be vacated and the case remanded with a rebuke that they failed to apply the Altai precedent on merger and abstraction doctrine correctly.

          Uh, yeah, about that… It seems a group called the “Internet Accountability Project” has filed an amicus brief on behalf of Oracle. And now the Trump admin has weighed in on Oracle’s side.

          Honestly, I think the IAP and the Trump admin are in the wrong here. They’re so consumed with Google hatred that they’ll take any chance they can get to take them down a peg–even if it would have massive repercussions throughout our industry.

        • Yeah, maybe in the Clinton years. With Trump’s conservative, pro-business Supreme Court? It’s not so cut and dry. Think of it from a conservative’s perspective: Oracle, American success story, defending its IP (the lifeblood of American business) from the commie- and SJW-infested Google.

          Barring that, think of it from a conservative justice’s perspective: Whether or not APIs should be specially excluded from eligibility for copyright protection is a matter for Congress, not the courts, to decide. Lower court ruling affirmed. Pay up, Google.

Leave a Reply to Krzesimir Nowak Cancel reply

Your email address will not be published. Required fields are marked *