The build-a-better-UPS project is progressing nicely. About a week ago we had first hardware lightup; I successfully threw messages over an I2C bus to the 20×4 LCD we plan to use as a status display. Hey, it’s not the power plane (yet) but it’s something.
Much has changed in UPSide over the last week. Ground has been broken on the software; one key piece of the control daemon, the policy state machine, now exists.
The design of UPSide is coming together very nicely. We don’t have a full parts list yet, but we do have a functional diagram of the high-power subsystem most of which can be expanded into a schematic in a pretty straightforward way.
If you want to see what we have, clone the repo, cd to design-docs, make transactions.html, and view that in a browser. Note that the bus message inventory is out of date; don’t pay a lot of attention to it, one of the design premises has changed but I haven’t had time to rewrite that section yet.
We’ve got Eric Baskin, a very experienced power and signals EE, to do the high-power electronics. We’ve got me to do software and systems integration. We’ve got a lot of smart kibitzers to critique and improve the system design, spotting problems the two Eric’s might have missed. It’s all going well and smoothly – except in one key area.
UPSide needs a battery technologist – somebody who really understands all the tradeoffs among battery chemistries, how to spec battery types for different applications, and especially the ins and outs of battery management systems.
Eric Baskin and I are presently a bit out of our depth in this. Given time we could educate ourselves up to the required level, but the fact that that portion of the design is lagging the rest tells me that we ought to recruit somebody who already knows the territory.
Any takers? No money in it, but you get to maybe disrupt the whole UPS market and and certainly work with a bunch of interesting people.
Somebody using the handle JeffyPoooh (yes, ‘ooo’) left a comment on The Register that is an only slightly exaggerated summary of the reasons for UPSide:
The current state of play is: We have a high-level system design and a map of the behavior states. We have a capacity target (300W for 15 mins) and a peak-continuous-load spec (400W) We know we’re going to build a double-conversion design and we’re considering a couple of alternative topologies. We pretty much know the external-interface specs (some details may change).
I’m expecting both my prototype copy of the forebrain Unix SBC (an Olimex LIME2) and the interface contract for the high-power subsystem to land on my desk tomorrow.
Interest in this project continues to be huge. Another company wants in as of this morning. The volume of feature requests is high enough that I’m buckling under the editing load.
The rest of this post is instructions to potential contributors about how to get on board.
1, Get an ID on GitLab. Tell me what it is so I can add you to the project group.
2. If you have a feature request, please Don’t post it on this blog. Add it to the “General feature request thread” on the tracker.
3. Read the wiki. Read the tracker issues. I try to keep both pruned so the volume is not overwhelming. Read the Rejected Ideas page on the wiki, too.
4. Read the design documents in the project wiki. The important one is the transaction design; the I2C message inventory will change, but the basic state diagram probably won’t.
5. Participate in the design discussion. This takes place in tracker threads.
6. When we’re ready to breadboard a prototype, throw some parts money in the tip jar we don’t have yet. If you must contribute before then the PayPal blogbutton works fine.
7. Prototype builds will probably go down at PA Makerspace in Phoenixville, PA. If you are within driving distance and a competent electrics tech, consider joining us for a build.
8. Once we have a full design with a PC board and enclosure: if you have a shop facilities for it, try to replicate the build. We’ll know we have the build recipe debugged when other people can do it.
9. If your favorite hardware feature request doesn’t appear in the version 1 prototype, relax, We may think it’s a good idea but be holding off till v2 out of a desire to keep v1 simple and launch fast.
10. If your favorite software feature request doesn’t appear in the version 1 prototype, pitch in and make it happen. A Unix SBC is not a difficult programming environment – the OS on this one is a Debian port.
After step 10 and a couple of design iterations the future becomes less clear. maybe try to get it into volume manufacturing through a partnership with an established vendor.
Yesterday I posted about how the streetlight effect pulls us towards bad choices in systems engineering. Today I’m going to discuss a different angle on the same class of challenges, one which focuses less on cognitive bias and more on game theory and risk management.
In the face of uncertainty, buy options. This is a good rule whether you’re doing whole-system design, playing boardgames, or deciding whether and when to carry a gun.
A week ago I argued that UPSes suck and need to be disrupted. The response to that post was astonishing. Apparently I tapped into a deep vein of private discontents – people who had been frustrated and pissed off with UPS gear for years or decades but never quite realized it wasn’t only their problem.
Many people expressed an active desire to contribute to a kickstarter aimed at this problem. I got one offer from someone actually willing to hire an engineer to work on it. Intelligent feature suggestions – often framed as gripes about the deficiencies of what you can buy out there – came flooding in.
Perhaps most remarkably, the outlines of a coherent design began to emerge. We identified a battery technology we could buy COTS that would improve on the performance and lifetime of lead-acid but without the explosion risk of lithium-ion. The way that safety and regulatory requirements would require a partition between low- and- high-power electronics became clearer. A feature list solidified. We took in good ideas and rejected some not-so-good ones.
Therefore, even though we don’t yet have a lead hardware engineer, I have initiated Project UPSide. There’s no code or schematics yet; we’re still developing requirements and architecture. By “architecture” I mean, for example, what specific kinds of information the hardware subsystems need to exchange.
All interested parties are welcome to browse the wiki
and apply for write access. Roles we are especially looking for:
* Lead hardware engineer – needs to be able to do overall design and systems integration.
* Someone who knows how to program USB endpoints. (It will land on me to learn this if we can’t find someone with experience.)
* Someone who understands battery-state modeling. (Again, I’ll learn this if nobody steps up.)
My own job is, basically, product manager – keeper of the requirements list and recruiter of talent.
UPDATE: If you want to request features or changes to the design wiki, the best way to do that is by opening an issue in the tracker. That way the discussion stays on record for later viewers.
Warning: this is a rant.
I use a UPS (Uninterruptible Power Supply) to protect the Great Beast of Malvern from power outages and lightning strikes. Every once in a while I have to buy a replacement UPS and am reminded of how horribly this entire product category sucks. Consumer-grade UPSes suck, SOHO UPSs suck, and I am reliably informed by my friends who run datacenters that no, you cannot ascend into a blissful upland of winnitude by shelling out for expensive “enterprise-grade” UPSes – they all suck too.
The lossage is extra annoying because designing a UPS that doesn’t suck would be neither difficult nor expensive. These are not complicated devices – they’re way simpler than, say, printers or scanners. This whole category begs to be disrupted by an open-hardware design that could be assembled cheaply in a makerspace from off-the-shelf components, an Arduino-class microcontroller, and a PROM.
How badly do UPSes suck? Let me count the ways…
I’ve been ill with influenza and bronchitis for the last week. Maybe this needs to happen more often, because I had a small but fundamental insight into network scaling theory a few minutes ago.
I’m posting it here because I think my blog regulars cast a wide enough net to tell me if I’ve merely rediscovered a thing in the existing literature or, in fact, nobody quite got here before.
This last week has not been kind to the Great Beast of Malvern. Serenity is restored now, but there was drama and (at the last) some rather explosive humor.
For some time the Beast had been having occasional random flakeouts apparently related to the graphics card. My monitors would go black – machine still running but no video. Some consultation with my Beastly brains trust (Wendell Wilson, Phil Salkie, and John D. Bell) turned up a suitable replacement, a Radeon R360 R7 that was interesting because it can drive three displays (I presently drive two and aim to upgrade).
Last Friday I tried to upgrade to the new card. To say it went badly would be to wallow in understatement. While I was first physically plugging it in, I lost one of the big knurled screws that the Beast’s case uses for securing both cards and case, down behind the power supply. Couldn’t get it to come out of there.
Then I realized that the card needed a PCI-Express power tap and oh shit the card vendor hadn’t provided one.
Much frantic running around to local computer stores ensued, because I did not yet know that Wendell had thoughtfully tucked several spares of the right kind of cable behind the disk drive bays when he built the Beast. Which turns out to matter because though the PCI-E end is standardized, the power supply end is not and they have vendor-idiosyncratic plugs.
Eventually I gave up and tried to put the old card back in. And that’s when the real fun began. I broke the retaining toggle on the graphics card’s slot while trying to haggle the new card out. When I tried to boot the machine with the old card plugged back in, my external UPS squealed – and then nothing. No on-board lights, no post beep, no sign of life at all. I knew what that meant; evidently either the internal PSU or the mobo was roached.
About five years ago I reacted to a lot of hype about the impending death of the personal computer with an observation and a prediction. The observation was that some components of a computer have to be the size they are because they’re scaled to human dimensions – notably screens, keyboards, and pointing devices. Wander outside certain size extrema and you get things like smartphone keyboards that are only good for limited use.
However, what we normally think of as the heart of a computer – the processing and storage – isn’t like this. It can get arbitrarily small without impacting usability at all. Consequently, I predicted a future in which people would carry around powerful computing nodes descended from smartphones and walk them to docking stations bundling a screen, a pointing device, and a real keyboard when they need to get real work done.
We’ve now reached an interesting midway point on that road. The (stationary) computers I use are in the process of bifurcating into two classes: one quite large, one very small. I qualify that with “stationary” because laptops are an exception for reasons which, if not yet obvious, will be in a few paragraphs.
The British have a phrase “Too clever by half”, It needs to go global, especially among hackers. It can have any of several closely related meanings: the one I mean to focus on here has to do with overconfidence in one’s intelligence or skill, and the particular bad consequences that can have. It’s related to Nassim Taleb’s concept of a “fragilista”.
For at least five years now I’ve been telling myself that, as nifty as it would be to play with the hardware, I really shouldn’t spend money on a small-form-factor PC.
This was not an easy temptation to resist, because I found little systems like the Intel NUC fascinating. I’d look over the specs for things like that in on-line stores and drool. Replacing a big noisy PC seemed so attractive…but I always drew back my hand, because that hardware came with a premium pricetag and I already have working kit.
Then, tonight, I’m over at my friend Phil Salkie’s place. Phil is a hardware and embedded-programming guy par excellence; I know he builds small-form-factor systems for industrial applications. And tonight he’s got a new toy to show off, a Taiwanese mini-ITX box called a Jetway.
He says “$79 on Amazon”, and I say “I’ve thought about replacing my mailserver with something like that, but could never cost-justify it.” Phil looks at me and says “You should. These things lower your electric bills – it’ll pay itself off inside of a year.”
Oh. My. Goddess. Why didn’t I think of that?
The following is a comment I just filed on FCC Docket 15-170, “Amendment of Parts 0, 1, 2, 15, and 18 of the Commission’s Rules et al.”
Thirty years ago I had a small hand in the design of the Internet. Since then I’ve become a senior member of the informal collegium that maintains key pieces of it. You rely on my code every time you use a browser or a smartphone or an ATM. If you ever ride in a driverless car, the nav system will critically depend on code I wrote, and Google Maps already does. Today I’m deeply involved in fixing Internet time service.
I write to endorse the filings by Dave Taht and Bruce Perens (I gave Dave Taht a bit of editorial help). I’m submitting an independent comment because while I agree with the general thrust of their recommendations I think they may not go far enough.
All my readers should be aware of the Rowhammer attack by now.
It gives me great pleasure to report that thanks to our foresight in specifying ECC memory for the design, the Great Beast of Malvern has armor of proof against this attack. The proof being over a thousand runs of the Rowhammer test.
Thank you, everyone who threw money into the Beast’s build budget. If y’all hadn’t been so generous, the build team might have had to make compromises. One of the most likely items to be cut would have been ECC…because registered ECC DRAM at the Beast’s speeds is so freaking expensive that the memory was about a third of the entire build budget. And now we’d have a vulnerable machine.
As it is, the Beast roars in triumph over the Rowhammer.
Oh, and what I’m currently doing with the Beast? Why, I’m repairing the very fabric of time..itself! Explanation to follow, probably early next week.
There have been a flood of big, cheap monitors (2560×1440 and up) becoming available on TigerDirect and other similar sites recently. But I’m here to tell you that these should come with warning labels, and explain why.
I’ve had some dolorous experiences with the no-name pair of big flatscreens I bought back in 2013 – the things called Aurias that I described in The Agony, the Ectasy, the Dual Monitors. Very recently I finally got both to finally work with enough stability that I’m sure I’ll be keeping them for a while.
But the troubleshooting process was arduous. Along the way I learned some important things, mainly because two friends who are unusually capable hardware troubleshooters actually took the suckers apart in my presence and explained things about the internals and the surrounding market.
Here are some of the things I learned…
Today, for the first time ever in my life, I used a soldering iron.
Remarkably, I neither conflagrated my house nor inflicted horrible burns on sensitive portions of my anatomy. Cat and wife are looking visibly relieved at this. As well they might.
Wendell Wilson over at TekSyndicate had a good idea – run the NetBSD repo conversion on a machine roughly comparable to the Great Beast design. The objective was (a) to find out if it freakin’ worked, and (b) to get a handle on expected conversion time and maximum working set for a really large conversion.
The news is pretty happy on all fronts.
I’ve attempted to summarize the discussion of build options for the repository-surgery machine. You should see a link at the top of the page: if not, it’s here
I invite all the commenters who have shown an interest to critique these build proposals. Naturally, I’d like to make sure we have a solid parts list with no spec conflicts before we start spending money and time to build this thing.
Something of significance to the design discussion for the Great Beast occurred today.
I have finally – finally! – achieved significant insight into the core merge code, the “black magic” section of cvs-fast-export. If you look in merge.c in the repo head version you’ll see a bunch of detailed comments that weren’t there before. I feel rather as Speke and Burton must have when after weeks of hacking their way through the torrid jungles of darkest Africa they finally glimpsed the source of the Nile…