Trials of the Beast

This last week has not been kind to the Great Beast of Malvern. Serenity is restored now, but there was drama and (at the last) some rather explosive humor.

For some time the Beast had been having occasional random flakeouts apparently related to the graphics card. My monitors would go black – machine still running but no video. Some consultation with my Beastly brains trust (Wendell Wilson, Phil Salkie, and John D. Bell) turned up a suitable replacement, a Radeon R360 R7 that was interesting because it can drive three displays (I presently drive two and aim to upgrade).

Last Friday I tried to upgrade to the new card. To say it went badly would be to wallow in understatement. While I was first physically plugging it in, I lost one of the big knurled screws that the Beast’s case uses for securing both cards and case, down behind the power supply. Couldn’t get it to come out of there.

Then I realized that the card needed a PCI-Express power tap and oh shit the card vendor hadn’t provided one.

Much frantic running around to local computer stores ensued, because I did not yet know that Wendell had thoughtfully tucked several spares of the right kind of cable behind the disk drive bays when he built the Beast. Which turns out to matter because though the PCI-E end is standardized, the power supply end is not and they have vendor-idiosyncratic plugs.

Eventually I gave up and tried to put the old card back in. And that’s when the real fun began. I broke the retaining toggle on the graphics card’s slot while trying to haggle the new card out. When I tried to boot the machine with the old card plugged back in, my external UPS squealed – and then nothing. No on-board lights, no post beep, no sign of life at all. I knew what that meant; evidently either the internal PSU or the mobo was roached.

Exhausted, pissed off, and flagellating myself for my apparent utter incompetence, I went to bed. Next morning I called Phil and said “I have a hardware emergency.” Phil, bless him, was over here within a couple of hours with a toolkit and a spare Corsair PSU.

I explained the whole wretched sequence of events including the lost case screw and the snapped retaining clip and the external UPS squealing and the machine going dead, and Phil said “First thing we’ll do is get that case screw out of there.” He then picked up the Beast’s case and began shaking it around at various angles.

And because Phil’s hardware-fu is vastly greater than mine, we heard rattling and saw a screw drop into view in short order. But it was the wrong screw! Not the big knurled job I’d dropped earlier but a smaller one.

“Aha!” says Phil. “That’s a board-mount screw.” Sure enough we quickly found that the southeast corner of the mobo had a bare hole where its securing screw ought to be. I figured out what must have happened almost as soon as Phil did; we gazed at each other with a wild surmise. That screw had either worked itself loose or already been loose due to an assembly error, and it had fallen down behind the motherboard.

Where it had bothered nobody, until sometime during my attempt to change cards I inadvertently jostled it into a new position and that little piece of conductive metal shorted out my fscking motherboard. The big knurled screw (which he shook out a few seconds later) couldn’t have done it – that thing was too large to fit where it could do damage.

Phil being Phil, he had my NZXT PSU out of the case and apart almost as fast as he could mutter “I’m going to void your warranty.” Sure enough, the fuse was blown.

This was good on one level, because it meant the mobo probably wasn’t. And indeed when we dropped in Phil’s Corsair the Beast (and the new card, and its monitors!) powered up just fine. And that was a relief, because the ASUS X99 motherboard is hellaciously more expensive than the PSU.

Almost as much of a relief was the realization that I hadn’t been irredeemably cackhanded and fatally damaged the Beast through sheer fucking incompetence. Hadn’t been for that little screw, all might have gone smoothly.

I also got an informative lecture on why the innards of the PSU looked kinda dodgy. Hotglue in excessive use, components really crowded in, flywiring (that’s when you have wires unsupported in space except by their ends, looping or arcing around components, rather than being properly snugged down).

But Phil was also puzzled. “Why didn’t this thing crowbar?” he wondered. The fuse, you see, is a second-order backup. When a short draws too much power, the PSU is supposed to shut itself down before there’s been time for the fuse to blow.

Phil took it home to replace the fuse with a circuit breaker, leaving the Corsair in the Beast. Which is functioning normally and allowing me to write this blog post, none the worse for wear except for the broken retaining clip.

He texted me this morning. Here’s how it went, effectively verbatim. I feel this truly deserves to be preserved for posterity:

Phil: So, I did a very nice repair job, installing the circuit breaker in the power supply. Then, as I was about to connect it up to my machine, I thought “you know, it really _should_ have crowbarred – not blown a fuse.” So, I powered it up sitting on the floor instead, saying to Ariel “either it’ll work, or it’ll catch fire.”

Phil: So, you may as well hang on to the Corsair power supply – I’ve already picked up a replacement to go in my machine.

Me: Are you telling me it caught fire?

Phil: No, no, no, no, no! Well, a little bit, yes. Just some sparks, and a small amount of smoke – no actual _flames_ as such, not on the outside of the box, at least. Hardly a moment’s trouble, really, once it cooled down enough to toss it in the bin…

Me: The magic smoke got out.

Phil: You might well say that, yes. In fact, all the smoke got out – the magic, the mystic, the mundane, all gone up in, well, smoke, as it were.

Me: “No actual _flames_ as such, not on the outside” Cathy was highly amused. I suspect she was visualizing you behind a blast shield, a la Jamie Hyneman.

Phil: I was trying more for Monty Python’s dead parrot sketch. In retrospect, a blast shield may have been warranted, had I only thought to use one…

Me: What about all those extra cables in the Beast? Are they obsolete now?

Phil: Yes, yes they are. At some convenient point I’ll tidy up all the cabling and make sure you have proper spares for the next time we dare disturb the Beast…

No, I did not make any of this up. Yes, Phil is really like that. And thus endeth the tale of the Trials of the Beast.

39 thoughts on “Trials of the Beast

  1. Once upon a time, I was doing some work on an Alienware PC at work when the 750W PSU decided to blow on start up … while I had my head about a foot away from it. Suffice to say that I was somewhat started when 4″ blue sparks shot out the back of the PSU. No spares on hand, but Microcenter was only a 20 minute drive away, so all I lost was a little time.

  2. The pessimist curses the fact he no longer has a working computer;
    the optimist celebrates the fact he now has a free single use smoke machine.

  3. My employer makes industrial automation systems. We also make big honkin’ electrical panels full of heavy-duty switches and contractors and circuit breakers and suchlike.

    Standard procedure when installing a big panel like that is to stand to one side the first time you close the main power switch, lest a wiring error on, say, the 480V/60A three-phase main power circuit make it fail explosively. It keeps the blast out of your face.

    I’ve got a similar annoyance…my inexpensive Linux box has a problem where too much of a graphics load will make the thing reboot. It’ll run fine for a while, then trbootb itself, then run fine for a while longer… I’m guessing the fault is in the GTX570 card itself. Might be an excuse to get a new GTX 1060 or so. If that doesn’t fix it, I’ve got another machine I can swap in. Need to upgrade to Mint 18 anyway.

  4. That ‘problem with a motherboard standoff, intermittent motherboard shorts’ used to happen much more often. Or at least you used to hear about it much more often. Personally I haven’t seen someone describe having it happen to *them* in over a decade.

    I suppose it was more common in the era of universal enthusiast homebuilts and fly by night PC ‘manufacturers’ that were really white box outfits with marketing budgets and aspirations. Now things are consolidated and mature.

  5. Stories where somebody gorilla-fists components until bits snap – and leaves loose screws lying around inside – have a tendency to shake out this way.

    Can’t say I haven’t been there myself, shamefully…I’ve caused my fair share of magical smoke emissions ;)

  6. I did get Python, though not that particular sketch.

    Or Douglas Adams; I wasn’t quite sure which, which Adams will probably take as a compliment.

    I also got Jerry Pournelle; I did love Chaos Manor.

    I’ve posted this column around; you might see some newbies…

  7. I generally wouldn’t put a Radeon in a Linux box anyways. They’re utter crap with OpenGl workloads even on Windows. My own “orthanc”* was originally built with a Radeon, and had some glitchiness with compiz, which I tolerated until I discovered that that the drivers, at a reproducible –and indeed unavoidable– point in one of my favorite games would wedge in kernel mode and hang the entire graphics stack, leaving X unkillable and the system with a blank screen without any ability to even switch out to a virtual terminal. Fortunately, the machine being a multicore system, only one cite was hung, and ssh access and a clean shutdown were still possible. I filed a bug with AMD, and they closed it because the *current LTS release for Ubuntu* was not on their list of supported platforms for their Linux driver. At that point I ordered an Nvidia card.

    *orthanc is so named because of its Corsair case: It’s a big, black tower.

  8. esr:

    …a Radeon R360 R7 that was interesting because it can drive three displays (I presently drive two and aim to upgrade).

    Let me guess: the third screen is going to be all Mutt-and-WeeChat, all the time. Is that correct? :o)

    And now, a few corrections (all emphasis is mine):

    1. “This last week was not been kind to the Great Beast of Malvern.”
    2. “…except for the the broken retaining clip.”
    3. “…behind a blast shield, a la Jamie Hynemann.” If you were referring to this gentleman, you wrote one N too many.
    4. You closed the post with a comma instead of a period: “And thus endeth the tale of the Trials of the Beast,”
    5. “This entry was posted in Technology and tagged Bease

  9. The moral I get from this story is the need for hardware fu.

    The question with a fancy high-end machine built by other people for you is whether you have the skill to fix it yourself if you have a problem. The answer is, you may not, so you are without a working machine till someone with the skill can get to you.

    At some point, it might be worth your while to build a cheap much lower end system from components yourself, just to acquire more hardware fu.
    ______
    Dennis

  10. I think Radeons on Linux are fine-ish if you use the open-source drivers and never, ever plan on doing anything intensive with OpenGL, which I suspect is our host’s use case. Otherwise, yeah, AMD’s OpenGL support is so notoriously awful that there are instances of AAA games being ported to Linux with the devs saying outright “we’re only supporting Nvidia cards”.

  11. Makes me wonder why plastic screws aren’t standard for this kind of thing. Or why some other non-vibration-sensitive fastener/mounting method isn’t used for motherboards.

    Then I remember that most “why that design?” questions about consumer gear end up being answered with “because it’s cheaper”.

  12. Makes me wonder why plastic screws aren’t standard for this kind of thing. Or why some other non-vibration-sensitive fastener/mounting method isn’t used for motherboards.

    The screws usually connect the ground plane of the board to the case, and for that purpose they need to be made of conductive materials (both the screw and the spacer mounted on the side of the case that the screw screws into).

  13. What does “crowbarred” in this context mean?

    It’s probably a term derived from the crowbar circuit, although it may have drifted slightly from its original meaning as used in this context. Crowbar circuits intentionally short out the main power leads in order to protect against an overvoltage condition. A screw in the wrong place will create the same effect, but the crowbar circuit will not be active in that case.

    PC power supplies are required to provide overcurrent protection circuits, both by the usual electrical safety standards, and because some parts of PC design rely on overcurrent protection being implemented to work properly. This is supposed to be in addition to fuses. Shorting out a healthy PC power supply for a brief interval shouldn’t do permanent damage; hence the confusion on Phil’s part when it apparently did.

    Many of the exotic connectors that connect a motherboard to its power sources are wired so that if you connect them wrong (e.g. backwards or offset by one column), they short out the main power leads. This is supposed to trigger the PSU’s overcurrent protection circuits and shut the power supply down before the misconnected device can power up, hopefully preventing any lasting damage to the equipment.

  14. Jay Maynard:

    Need to upgrade to Mint 18 anyway.

    They always tell you to ask yourself whether you really need to upgrade, and I always ignore that just to see the shiny new toys. Learned the hard way on an upgrade from 17.3 to 18, because the shipped version of GVFS no longer supports the ftpobex backend, never having been ported from version 4 (used in Mint 17.3) to 5 (in Mint 18) of BlueZ. So if you upgrade to 18, you lose the ability to mount your bluetooth device as a filesystem, which was enough of a pain in the butt that I eventually just restored 17.3. It’s supported until 2019, and I can always upgrade if they decide to stop stripping out features I use.

  15. I’m running 17.1, never upgraded. The problem I’m having right now is that the proprietary Nvidia driver package won’t upgrade. My primary use for that system is as a Second Life client, and so I do push the graphics hard.

    And no, no ATI graphics ever again on Linux. I’ve got three ATI cards, two 5770s and a 6970, and they’re sitting on the shelf in their little plastic bags, where they will remain. I do have an R9-270X in the Mac Pro, though; Second Life on OS X El Capitan has a couple of serious breakages with Nvidia graphics. (Due, sadly, to Apple’s backing away from OpenGL and refusing to do OpenGL 2 compatibility in OpenGL 4 mode, unlike Windows. Porting the Second Life renderer to OpenGL 4, never mind Apple’s preferred replacement for it, Metal, is an impossibility for someone like me. There are maybe five people on the planet who understand the Second Life renderer. Two of them work for Linden Lab. The other three are all certifiable.)

    Anyway, at this point, I suspect it’s a GTX 570 breakage. The one thing that has me questioning that is that occasionally, when it crashes, instead of rebooting, it just gives me the five beeps that say the CPU’s got a microcode error – though after a power cycle, it always reboots correctly.

    I have often lamented Eric’s lack of hardware-fu. If I lived closer than a thousand miles away, I’d attempt to do something about it.

  16. “I also got an informative lecture on why the innards of the PSU looked kinda dodgy.”

    Well? You can’t just leave us hanging! Why are they so dodgy?

  17. >Well? You can’t just leave us hanging! Why are they so dodg

    Er, you didn’t read the rest of the paragraph?

  18. > Er, you didn’t read the rest of the paragraph?

    I read that as defining what “looking dodgy” meant in this context. I was expecting an explanation of *why* they make them that way.

  19. Another tale of orthanc (which seems to be similar to the Beast in both specifications and adventures), related to your present woes is the late night / early morning (I forget which it was for me) that I was sitting in the living room talking to my mother (for whom it was early morning) when I heard a *BANG* from my computer desk behind me. I got up and started looking behind the desk, suspecting a fallen pile of CD cases or old floppies. When I found none I said “what was that?”

    Mom: I don’t know, but I saw a flash from behind your desk.

    Me: (noticing that orthanc’s power light is out) Oh, *CHKDSK*!

    Fortunately, the power lights on my monitor and speakers were out as well, and when I unplugged orthanc from my UPS and hooked it up directly to the wall, I was able to power it on. For a moment, though, I was certain something had shorted in orthanc itself.

    Another comment on orthanc’s name: a year or two after I built it, I saw a machine with the exact same case in a photo in an Ars Technica article. The caption gave the machine’s name as “monolith”, so apparently someone used a very similar thought process in naming their machine.

  20. @Jon Brase —

    I generally wouldn’t put a Radeon in a Linux box anyways…. At that point I ordered an Nvidia card.

    I suspect it would be a colder-than-absolute-zero day in Hell before you could get Eric to even consider a Nvidia card. Notwithstanding their stupidity in not open-sourcing their drivers for the Chinese market about four years back, he’s had his own run of nasty troubles with them. As he put it – “I will never again install an nVidia card unless forced at gunpoint, and if that happens I will find a way to make my assailant eat the fucking gun afterwards.”

  21. esr:

    Excuse me. I hate to be compulsive, but how come this is the only thread tagged “Great Beast”? It’s not like this is the first thread about your current desktop computer; take a look at these search results. For some reason, you didn’t tag the previous Beast threads. You’ll want to either tag them or “untag” this one (unless you consider single-element sets useful ;-P).

  22. @John D. Bell:

    I see. And I tend to be more of an open source purist than Eric, but given the experience I’d had with them before, the only reason I bought a Radeon for orthanc was that I liked ATIs open source policy better and thought I’d give them another chance. But it wasn’t the bug itself that made me throw up the Torvalds finger to them, it was the fact that they wouldn’t touch the bug because they weren’t currently supporting the current LTS release of Ubuntu.

  23. Yikes! I hope I wasn’t the jinx by asking you the week before on how the Great Beast was doing. :-(

  24. d5xtgr:

    A move to Mint 18 did reveal that annoyance (I can transfer files TO my phone, but not FROM which makes no sense). Alas, Mint 17 (Xfce) has another annoying bluetooth breakage in that in order to get a bt headset to work one must issue “sudo pactl load-module module-bluetooth-discover” after every boot (it has been annoyingly resistant to _my_ attempts to automate it.) So my initial checks, against my Known Bug was that 18 fixed things. Not quite.

    I need to see if the installed (not run from USB stick on slow port) PCLinuxOS (Xfce) has similar breakages. There are many things Mint gets right and it’s nice how (most) stuff “just works” yet there are times I’ve been tempted to go Full Slackware – and I really do not want to go Full Slackware.

  25. @Jakub: I’ll add to the crowbar circuit stuff in the cited wikipedia article, the origin of the term:

    Workers on electric railways when working on a section of track where the power was supposed to be off would lay a crowbar from the third rail to the closest other rail. Then, if someone inadvertently turned the power back on, the short circuit would trip the overload protection and keep the power off.

    @esr: Looks like Phil is worth his weight in gold, while you are simply worth less :-) .

  26. Back in the days before personal locks for isolating circuits became common (along with it being a firing offence for removing someone else’s lock), electricians would do a similar thing on when working on LV (<1000V) – twist the incomer live and neutral together so if some bastard turned the power on it would trip straight out.

    ESR, I'm very much enjoying your forays into hardware, if only because it makes this engineer happy that you're not completely infallible. Please try and avoid anything higher voltage than you'll find inside a computer case though :)

  27. Also, this is why you have a fuse *anyway*, despite the power supply supposedly being designed to prevent the supply of current into a short circuit. Think of it as the ultimate exception trap.

  28. Oh god I’ve had that screw problem happen more than once.

    I just had that while helping a friend. It wasn’t even a lost screw. Everything else was accounted for. Hell, I checked all holes on the board and case (and cards) and still have no clue where it came from. It’s not even any standard size I’ve ever seen before. It’s now in a little baggie with a note saying “mystery screw we found while installing watercooling”.

    Needless to say, it took the both of us to shake that whole full-tower case until I was satisfied there was no rattling.

    I used to have a story about a blood sacrifice when opening a case, because they were so sharp inside. Now I’m going to start one about my shake test and video card re-seating.

  29. On the plus side, at least the next time someone accuses you of having a screw loose, you can demonstrate that the problem was found and remedied.

  30. latest stable PCLinuxOS (Xfce) will pair with the phone, but the connection won’t hold. Trying to set things up the system only offers the option ‘access point’ and nothing more. But the phone can transfer files to the computer over bluetooth. It’s NOT reliable and often takes multiple attempts. A connection is established and if things go well ( “Gentlemen, place your bets.”) the transfer is done and the connection dropped again. This is slightly better than Mint 18’s not doing it at all, but still not what it really ought to be. I’m keeping PCLOS on that machine, but another partition is apt to get another distribution.

  31. Orvan Taurus

    KDE’s bluedevil seems to be more reliable than Cinnamon’s blueberry. Can’t speak to XFCE. Might be worth a shot.

  32. Orvan Taurus

    I’ve had some success with KDE’s bluedevil over Cinnamon’s blueberry. I can’t speak to XFCE, but you may give the KDE version a shot- it hit release in Mint about a week ago.

  33. > along with it being a firing offence for removing someone else’s lock

    A former co-worker had this happen.

    IMO it should not be a firing office, it should be attempted homicide.

  34. > IMO it should not be a firing office, it should be attempted homicide.

    Attempted homicide is generally a firing offense. :-P

  35. The stupidity of some manufacturers is amazing. I needed to replace the power supply on an old Dell server today and apparently Dell uses a standard ATX connector on the motherboard but changes all of the voltages around. Of the 24 wires on the connector only one matches the standard ATX pin-out. It looks as if Dell tried to be as contrary as possible: 12 volt wires going to 3.3 volt pins, 5 volt wires going to ground pins, etc. Imagine what could happen to the motherboard if you plugged a standard ATX supply into it and the supply was able to power up.

    It’s bad enough when companies try to lock in customers with proprietary components, but violating standards to do so is insane. Dell is now on my never buy list.

  36. I appreciate the beast, but what about this?
    From Schneier on Security

    Someone Is Learning How to Take Down the Internet

    Over the past year or two, someone has been probing the defenses of the companies that run critical pieces of the Internet. These probes take the form of precisely calibrated attacks designed to determine exactly how well these companies can defend themselves, and what would be required to take them down. We don’t know who is doing this, but it feels like a large nation state. China or Russia would be my first guesses.

    https://www.schneier.com/blog/archives/2016/09/someone_is_lear.html

  37. Eric, if you have a ritual that you perform when entering the dojo, or going to the range, put something inside your case that will remind you to perform it before you get at the innards.

    The PCIe retainer breakage was caused by rushing when you should have been carefuling. I’ve lost data “fixing” MD raid sets before, always from careless mistakes caused by rushing, despite me knowing enough to navigate the situation correctly, had I been careful instead of hasty. Everyone has stories like that, and situations that make them uncomfortable because they are acting beyond their perceived skill level, and they end up making silly mistakes that are actually far below their skill. You have the advantage of years of practice in dangerous situations that command your focus – you don’t need to learn the skill of calmness from scratch, you just need to remind yourself when to apply it in a novel way.

    The rest of your story is just good clean fun. Armed with the flip and shake method, lost computer screws will never again trouble you – and now you know how the hardware guys spend their days. Just make sure the screw actually falls out. If you can’t hear it any more, but still can’t see it, it has hunkered down, waiting for the right time to launch a counterattack.

Leave a Reply

Your email address will not be published. Required fields are marked *