May 09

Embrace the SICK

There’s a very interesting article just out, C Is Not a Low-level Language;. in which David Chisnall punctures the comforting illusion that C is really a “close-to-the-metal” language and relates this illusion to the high costs of Spectre and other processor-level bugs.

Those of us who think seriously about language design have long been aware that C’s flat-address-space model is increasingly at odds with the real world of memory-caching hierarchies. Chisnall’s main contribution is to notice that speculative execution, the feature at the bottom of the Spectre and Meltdown bugs, is essentially a hack implemented to allow C programmers to maintain the illusion that they’re running on a really fast serial machine.  But he has other interesting points as well.

I recommend reading Chisnall’s article before you go further with this post.

It’s no news to my regulars that I’ve been putting increasing investment into the Go language and now believe it a plausible candidate to replace C and C++ over most of C/C++’s range – that is, outside  of kernels and hard realtime.  So the question that immediately occurred to me upon reading the article was: Is Go necessarily productive of the same kind of kludge that Chisnall is calling out?

Because if it is – but something else isn’t – that could be a reason not to overcommit to Go.  The twin pressures of demand for lower security defects and the increasing complexity costs of speculative execution are bound to toll heavily against Go if it does demand massive speculative execution and there’s any realistic alternative that does not. Do we need something much more divergent from C (Erlang? Ocaml? Even perhaps Haskell?) for systems programming to follow where the hardware is going?

So let’s walk through Chisnall’s discussion points, bounce Go off each one, and see what we can see.  What we’ll find implies, I think, some more general conclusions about what will and won’t work in matching language design to real-world workloads and processor architectures.

Continue reading

Mar 16

UPSide needs a battery technologist

The design of UPSide is coming together very nicely. We don’t have a full parts list yet, but we do have a functional diagram of the high-power subsystem most of which can be expanded into a schematic in a pretty straightforward way.

If you want to see what we have, clone the repo, cd to design-docs, make transactions.html, and view that in a browser. Note that the bus message inventory is out of date; don’t pay a lot of attention to it, one of the design premises has changed but I haven’t had time to rewrite that section yet.

We’ve got Eric Baskin, a very experienced power and signals EE, to do the high-power electronics. We’ve got me to do software and systems integration. We’ve got a lot of smart kibitzers to critique and improve the system design, spotting problems the two Eric’s might have missed. It’s all going well and smoothly – except in one key area.

UPSide needs a battery technologist – somebody who really understands all the tradeoffs among battery chemistries, how to spec battery types for different applications, and especially the ins and outs of battery management systems.

Eric Baskin and I are presently a bit out of our depth in this. Given time we could educate ourselves up to the required level, but the fact that that portion of the design is lagging the rest tells me that we ought to recruit somebody who already knows the territory.

Any takers? No money in it, but you get to maybe disrupt the whole UPS market and and certainly work with a bunch of interesting people.

Mar 11

How to get started on the UPSide project

The current state of play is: We have a high-level system design and a map of the behavior states. We have a capacity target (300W for 15 mins) and a peak-continuous-load spec (400W) We know we’re going to build a double-conversion design and we’re considering a couple of alternative topologies. We pretty much know the external-interface specs (some details may change).

I’m expecting both my prototype copy of the forebrain Unix SBC (an Olimex LIME2) and the interface contract for the high-power subsystem to land on my desk tomorrow.

Interest in this project continues to be huge. Another company wants in as of this morning. The volume of feature requests is high enough that I’m buckling under the editing load.

The rest of this post is instructions to potential contributors about how to get on board.

1, Get an ID on GitLab. Tell me what it is so I can add you to the project group.

2. If you have a feature request, please Don’t post it on this blog. Add it to the “General feature request thread” on the tracker.

3. Read the wiki. Read the tracker issues. I try to keep both pruned so the volume is not overwhelming. Read the Rejected Ideas page on the wiki, too.

4. Read the design documents in the project wiki. The important one is the transaction design; the I2C message inventory will change, but the basic state diagram probably won’t.

5. Participate in the design discussion. This takes place in tracker threads.

6. When we’re ready to breadboard a prototype, throw some parts money in the tip jar we don’t have yet. If you must contribute before then the PayPal blogbutton works fine.

7. Prototype builds will probably go down at PA Makerspace in Phoenixville, PA. If you are within driving distance and a competent electrics tech, consider joining us for a build.

8. Once we have a full design with a PC board and enclosure: if you have a shop facilities for it, try to replicate the build. We’ll know we have the build recipe debugged when other people can do it.

9. If your favorite hardware feature request doesn’t appear in the version 1 prototype, relax, We may think it’s a good idea but be holding off till v2 out of a desire to keep v1 simple and launch fast.

10. If your favorite software feature request doesn’t appear in the version 1 prototype, pitch in and make it happen. A Unix SBC is not a difficult programming environment – the OS on this one is a Debian port.

After step 10 and a couple of design iterations the future becomes less clear. maybe try to get it into volume manufacturing through a partnership with an established vendor.

Feb 18

In the face of uncertainty, buy options.

Yesterday I posted about how the streetlight effect pulls us towards bad choices in systems engineering. Today I’m going to discuss a different angle on the same class of challenges, one which focuses less on cognitive bias and more on game theory and risk management.

In the face of uncertainty, buy options. This is a good rule whether you’re doing whole-system design, playing boardgames, or deciding whether and when to carry a gun.

Continue reading

Feb 16

Announcing: The UPSide project

A week ago I argued that UPSes suck and need to be disrupted. The response to that post was astonishing. Apparently I tapped into a deep vein of private discontents – people who had been frustrated and pissed off with UPS gear for years or decades but never quite realized it wasn’t only their problem.

Many people expressed an active desire to contribute to a kickstarter aimed at this problem. I got one offer from someone actually willing to hire an engineer to work on it. Intelligent feature suggestions – often framed as gripes about the deficiencies of what you can buy out there – came flooding in.

Perhaps most remarkably, the outlines of a coherent design began to emerge. We identified a battery technology we could buy COTS that would improve on the performance and lifetime of lead-acid but without the explosion risk of lithium-ion. The way that safety and regulatory requirements would require a partition between low- and- high-power electronics became clearer. A feature list solidified. We took in good ideas and rejected some not-so-good ones.

Therefore, even though we don’t yet have a lead hardware engineer, I have initiated Project UPSide. There’s no code or schematics yet; we’re still developing requirements and architecture. By “architecture” I mean, for example, what specific kinds of information the hardware subsystems need to exchange.

All interested parties are welcome to browse the wiki and apply for write access. Roles we are especially looking for:

* Lead hardware engineer – needs to be able to do overall design and systems integration.

* Someone who knows how to program USB endpoints. (It will land on me to learn this if we can’t find someone with experience.)

* Someone who understands battery-state modeling. (Again, I’ll learn this if nobody steps up.)

My own job is, basically, product manager – keeper of the requirements list and recruiter of talent.

UPDATE: If you want to request features or changes to the design wiki, the best way to do that is by opening an issue in the tracker. That way the discussion stays on record for later viewers.

Feb 08

UPSes suck and need to be disrupted

Warning: this is a rant.

I use a UPS (Uninterruptible Power Supply) to protect the Great Beast of Malvern from power outages and lightning strikes. Every once in a while I have to buy a replacement UPS and am reminded of how horribly this entire product category sucks. Consumer-grade UPSes suck, SOHO UPSs suck, and I am reliably informed by my friends who run datacenters that no, you cannot ascend into a blissful upland of winnitude by shelling out for expensive “enterprise-grade” UPSes – they all suck too.

The lossage is extra annoying because designing a UPS that doesn’t suck would be neither difficult nor expensive. These are not complicated devices – they’re way simpler than, say, printers or scanners. This whole category begs to be disrupted by an open-hardware design that could be assembled cheaply in a makerspace from off-the-shelf components, an Arduino-class microcontroller, and a PROM.

How badly do UPSes suck? Let me count the ways…

Continue reading

Mar 28

Odlyzko-Tilly-Raymond scaling

I’ve been ill with influenza and bronchitis for the last week. Maybe this needs to happen more often, because I had a small but fundamental insight into network scaling theory a few minutes ago.

I’m posting it here because I think my blog regulars cast a wide enough net to tell me if I’ve merely rediscovered a thing in the existing literature or, in fact, nobody quite got here before.

Continue reading

Sep 13

Trials of the Beast

This last week has not been kind to the Great Beast of Malvern. Serenity is restored now, but there was drama and (at the last) some rather explosive humor.

For some time the Beast had been having occasional random flakeouts apparently related to the graphics card. My monitors would go black – machine still running but no video. Some consultation with my Beastly brains trust (Wendell Wilson, Phil Salkie, and John D. Bell) turned up a suitable replacement, a Radeon R360 R7 that was interesting because it can drive three displays (I presently drive two and aim to upgrade).

Last Friday I tried to upgrade to the new card. To say it went badly would be to wallow in understatement. While I was first physically plugging it in, I lost one of the big knurled screws that the Beast’s case uses for securing both cards and case, down behind the power supply. Couldn’t get it to come out of there.

Then I realized that the card needed a PCI-Express power tap and oh shit the card vendor hadn’t provided one.

Much frantic running around to local computer stores ensued, because I did not yet know that Wendell had thoughtfully tucked several spares of the right kind of cable behind the disk drive bays when he built the Beast. Which turns out to matter because though the PCI-E end is standardized, the power supply end is not and they have vendor-idiosyncratic plugs.

Eventually I gave up and tried to put the old card back in. And that’s when the real fun began. I broke the retaining toggle on the graphics card’s slot while trying to haggle the new card out. When I tried to boot the machine with the old card plugged back in, my external UPS squealed – and then nothing. No on-board lights, no post beep, no sign of life at all. I knew what that meant; evidently either the internal PSU or the mobo was roached.

Continue reading

Apr 15

The midrange computer dies

About five years ago I reacted to a lot of hype about the impending death of the personal computer with an observation and a prediction. The observation was that some components of a computer have to be the size they are because they’re scaled to human dimensions – notably screens, keyboards, and pointing devices. Wander outside certain size extrema and you get things like smartphone keyboards that are only good for limited use.

However, what we normally think of as the heart of a computer – the processing and storage – isn’t like this. It can get arbitrarily small without impacting usability at all. Consequently, I predicted a future in which people would carry around powerful computing nodes descended from smartphones and walk them to docking stations bundling a screen, a pointing device, and a real keyboard when they need to get real work done.

We’ve now reached an interesting midway point on that road. The (stationary) computers I use are in the process of bifurcating into two classes: one quite large, one very small. I qualify that with “stationary” because laptops are an exception for reasons which, if not yet obvious, will be in a few paragraphs.

Continue reading

Apr 07

Too clever by half

The British have a phrase “Too clever by half”, It needs to go global, especially among hackers. It can have any of several closely related meanings: the one I mean to focus on here has to do with overconfidence in one’s intelligence or skill, and the particular bad consequences that can have. It’s related to Nassim Taleb’s concept of a “fragilista”.

Continue reading

Apr 03

Sometimes I should give in to my impulses

For at least five years now I’ve been telling myself that, as nifty as it would be to play with the hardware, I really shouldn’t spend money on a small-form-factor PC.

This was not an easy temptation to resist, because I found little systems like the Intel NUC fascinating. I’d look over the specs for things like that in on-line stores and drool. Replacing a big noisy PC seemed so attractive…but I always drew back my hand, because that hardware came with a premium pricetag and I already have working kit.

Then, tonight, I’m over at my friend Phil Salkie’s place. Phil is a hardware and embedded-programming guy par excellence; I know he builds small-form-factor systems for industrial applications. And tonight he’s got a new toy to show off, a Taiwanese mini-ITX box called a Jetway.

He says “$79 on Amazon”, and I say “I’ve thought about replacing my mailserver with something like that, but could never cost-justify it.” Phil looks at me and says “You should. These things lower your electric bills – it’ll pay itself off inside of a year.”

Oh. My. Goddess. Why didn’t I think of that?

Continue reading

Oct 07

The FCC must not lock down device firmware!

The following is a comment I just filed on FCC Docket 15-170, “Amendment of Parts 0, 1, 2, 15, and 18 of the Commission’s Rules et al.”

Thirty years ago I had a small hand in the design of the Internet. Since then I’ve become a senior member of the informal collegium that maintains key pieces of it. You rely on my code every time you use a browser or a smartphone or an ATM. If you ever ride in a driverless car, the nav system will critically depend on code I wrote, and Google Maps already does. Today I’m deeply involved in fixing Internet time service.

I write to endorse the filings by Dave Taht and Bruce Perens (I gave Dave Taht a bit of editorial help). I’m submitting an independent comment because while I agree with the general thrust of their recommendations I think they may not go far enough.

Continue reading

Mar 11

The Great Beast is armored!

All my readers should be aware of the Rowhammer attack by now.

It gives me great pleasure to report that thanks to our foresight in specifying ECC memory for the design, the Great Beast of Malvern has armor of proof against this attack. The proof being over a thousand runs of the Rowhammer test.

Thank you, everyone who threw money into the Beast’s build budget. If y’all hadn’t been so generous, the build team might have had to make compromises. One of the most likely items to be cut would have been ECC…because registered ECC DRAM at the Beast’s speeds is so freaking expensive that the memory was about a third of the entire build budget. And now we’d have a vulnerable machine.

As it is, the Beast roars in triumph over the Rowhammer.

Oh, and what I’m currently doing with the Beast? Why, I’m repairing the very fabric of time..itself! Explanation to follow, probably early next week.

Feb 25

Beware cut-price Korean monitors!

There have been a flood of big, cheap monitors (2560×1440 and up) becoming available on TigerDirect and other similar sites recently. But I’m here to tell you that these should come with warning labels, and explain why.

I’ve had some dolorous experiences with the no-name pair of big flatscreens I bought back in 2013 – the things called Aurias that I described in The Agony, the Ectasy, the Dual Monitors. Very recently I finally got both to finally work with enough stability that I’m sure I’ll be keeping them for a while.

But the troubleshooting process was arduous. Along the way I learned some important things, mainly because two friends who are unusually capable hardware troubleshooters actually took the suckers apart in my presence and explained things about the internals and the surrounding market.

Here are some of the things I learned…

Continue reading

Oct 20

Building the perfect beast

I’ve attempted to summarize the discussion of build options for the repository-surgery machine. You should see a link at the top of the page: if not, it’s here

I invite all the commenters who have shown an interest to critique these build proposals. Naturally, I’d like to make sure we have a solid parts list with no spec conflicts before we start spending money and time to build this thing.

Continue reading