UPSide progress report

The build-a-better-UPS project is progressing nicely. About a week ago we had first hardware lightup; I successfully threw messages over an I2C bus to the 20×4 LCD we plan to use as a status display. Hey, it’s not the power plane (yet) but it’s something.

Eric Baskin is making progress on the power plane. He has started ordering parts for a…not exactly breadboard rig, but something analogous that you do with higher-power electronics.

The control software is in very good shape. Having identified the kinds of sensors and switches we’ll need, I designed a file format that can describe sequences of events coming off the power plane. The daemon’s access to hardware is sealed off behind a Go class interface that has two implementations; the one that’s fully written interprets parsing those event files.

This means I can feed an event file to the daemon on standard input and see a log of its actions on standard output. Accordingly I don’t have to guess that the business logic is correct – I know it is. (And of course every such file is a stringent regression test, verifying everything but the low-level hardware interface.)

Another thing I did towards correctness was avoiding hand-hacking. And thereby hangs a tale.

Before I wrote the first line of Go by hand I sat down and composed a state-transition diagram. The states are (for example) “mains power up” and “on battery dwell time, below threshold”. The events are sensor notifications like “good line voltage”. The transitions can include throwing control switches or repainting the LCD.

I first composed the diagram in DOT, the markup language of the graphviz toolkit. The raw DOT was repetitive and confusing, so I refactored it as a sequence of calls of two text macros – state() and transition() – that expand into DOT markup.

Then I had a thought: I have in this sequence of macros a complete description of of UPSide’s business logic. Wouldn’t it be smart to generate the code for the corresponding state machine from it? And that’s exactly what I did. The macro calls turned into method invocations in a little Python program that can implement them either by emitting DOT or by emitting state-machine code in Go.

Notice what this means. If the business logic needs to be modified, I can do almost the whole job by looking at, thinking about, and editing the state-transition diagram. Sure, I sometimes have to write hook functions to be fired on particular transitions – like “when you pass from ‘waiting for charge threshold’ to ‘mains power up’, enable all AC outlets”, but those hooks are almost trivial and easy to audit.

The defect attractor is not the hooks but the state machine that calls them – 8 states, 7 event types, 14 transitions. Not so much because it would be intrinsically complicated to write by hand, but because bugs and divergences from the state-transition diagram would be difficult to spot amidst all that boilerplate code. Any time you can compile this kind of thing from a higher-level declarative description, you win.

The choice of Go as an implementation language is looking like an excellent call. We haven’t yet really started to collect what I think the big long-term win will be – long-term reliability due to the language’s immunity to a large class of memory-management errors – but development on the daemon has definitely gone faster and smoother than it would have in C. The Go type system and compiler error messages are actually useful; who’d a thunk it?

The project does have one serious problem, though. We can’t find a COTS battery pack that meets our specs.

Neither Eric B. nor I is an expert on battery state modeling, nor do we want to be. To avoid wandering into this swamp, we’ve made the same choice laptop designers do. We’re assuming that the battery will be sitting on the other side of a a fuel-gauge chip that does all that modeling itself and controls the battery’s trickle charger, so all we have to do is poll it frequently and let it tell us state of charge, state of battery, and expected dwell time.

So far, all the COTS batteries we’ve found that have an integrated fuel gauge chip are lithium-ion, designed for applications like flying drones where power-to-weight ratio is a big deal. Which is really annoying because explosion hazard.

What we want to do is use a variant of that technology called LiFePO – Lithium Iron Phosphate. This should greatly reduce the odds of kaboom (and lower costs) at the cost of less stored power per kilogram – but that downside would be OK because a UPS is a stationary application in which weight is much less important.

Alas, we have not yet found a product that is (a) commercial off the shelf (so we don’t have to fret about extra UL compliance issues beyond the power plane itself). (b) LiFePO, and (c) has the fuel gauge built in.

(Why not lead-acid? Well, they’re cheap, but we want better lifetime than you can get from those – besides, dumping lead and sulphuric acid into the waste stream is no favor to anyone. And we can’t find them with a built-in fuel gauge, either.)

We can evade this issue for a while. Eric B. can test the mains-conversion part of the design without a battery subsystem. Technology is moving fast; by the time the battery subsystem becomes critical-path, maybe there’ll be a COTS solution. If not, there are outfits that will do semi-custom battery packs to spec. That would be best avoided, though, as it would be certain to drive the unit cost way up.

29 comments

  1. Just FYI, via RSS today this post was truncated.

    And about lead-acid: I’m wondering, why the short lifespan on current batteries? I previously mentioned the 1980 vintage UPS I had that used a standard wheelchair battery — lifespan about 10-12 years each.

    And is LiFePO a net environmental gain?

    Is it possible to have a design that could swing both ways, using whichever type of battery was available? I came across this:
    http://www.batteryspace.com/Li-Ion-Batteries-To-Replace-Lead-Acid-Batteries.aspx

    1. >I’m wondering, why the short lifespan on current batteries?

      At least part of it is that a lot of UPSes constantly trickle-charge the battery even when it’s at full capacity, leading to premature boil-off of the elecrolyte. This could be avoided with a proper battery management system (BMS) consisting of a fuel gauge and charger, but nobody seems to actually build SLAs with a BMS inboard.

      Some “smart charging” UPSes probably mate an equivalent of a BMS built into the UPS with dumb SLAs, but replicating that would be a rathole because we’d still have to (a) have state-of-charge sensors running into the battery, (b) do all the BMS tuning and calibration ourselves (ugh, this is the are where our skills are weakest), and (c) take on safety and regulatory risks we”d really rather be someone else’s problem.

      >And is LiFePO a net environmental gain?

      Longer life = less waste. Almost anything that could plausibly go in a battery is less nasty than waste lead. Except cadmium.

      >Is it possible to have a design that could swing both ways, using whichever type of battery was available?

      Probably not, unless we can find multiple packs with inboard BMS but differing battery chemistries. And that takes us right back to the headline problem; we can’t find anything with inboard BMS that isn’t lithium-ion.

      1. There is, however, very little waste lead, as the lead in batteries is trivially recoverable, and the recycling rate for lead-acid batteries is extremely high. (Some estimates are 98% or better, though I suspect the numbers are far worse for UPS and emergency lighting batteries than for car and motorized equipment batteries, due to core charge programs for the latter.)

        Certainly the achieved lifespan for lead-acid batteries in COTS UPSes is not good, something on the order of 3-5 years or less in many cases. With the added disadvantage that their “self test” can’t tell that the battery capacity has degraded, so unless you’re on a replacement schedule you only find out when the power fails and your UPS does, too.

        1. Indeed! People out here in the country regularly have to buy new batteries for gates, because there’s a booming market in recycling stolen batteries. They’re worth about $20 each, so scumbags go around hopping fences and stealing them out of electric gates. I’ve lost two so far.

        2. For the last ~10 years, I haven’t had a UPS last longer than two years.

          Before that, I had the same three for over a decade, and the only reason one of ’em died was a serious zapping by a lightning strike.

      2. It is not hard to keep a lead acid battery correctly charged, even though almost no one ever does it. You just keep the charging voltage a hairsbreadth below the point that you are generating hydrogen and oxygen.

        Which if you knew exactly what that voltage was, would be easy.
        Unfortunately the critical voltage varies with temperature and acid concentration. However, battery resistance and the variation in battery voltage gives you an indication. As you approach the point where hydrogen and oxygen is about to be generated, a small amount of additional charge causes a large change in battery voltage, as you pass that point, a large amount of additional charge causes a small change in battery voltage. So you want to find the voltage at which there is a kink in the graph representing state of charge versus voltage, and charge to just short of that voltage.

        What a lead acid charger does, if you are lucky, is shoot some charge into the battery, monitor the ensuing voltage, and rest a bit if the voltage is too too high. Which is great if its guess of what the voltage should be is correct.

        1. This. Telcos have been using banks of flooded lead-acid batteries for backup on float charge for decades, and they last for decades when treated appropriately. Overcharging is what kills them, and I suspect existing UPS vendors have a vested interest in not caring about short battery lifetimes if they’re the ones selling replacements.
          Even modern “maintenance-free” vehicle starter batteries can be moderately abused by being vibrated, discharged brutally several times a day and charged to a nominal 14.4 or 14.6V regardless of cell temperature and still last for years.
          My bikeshedding inclination, being totally inexpert in these matters, would be at least to look at modern non-spill AGM lead-acid batteries, rather than the SLA types which are often supplied for UPS and are very intolerant of being either under- or over-charged to any meaningful degree. I’m prepared to accept that the form factor may be inappropriate for this application though.

      3. Thanks for the info.

        > At least part of it is that a lot of UPSes constantly trickle-charge the battery even when it’s at full capacity, leading to premature boil-off of the elecrolyte.

        Wait, is that the funny smell I sometimes get around my computers? kinda like an overheated CRT ??

        And maybe that explains why my original Tripp-Lite that lasted forever (died of a lightning strike) ran ice-cold, but the several short-lived replacements were hot ALL the time.

  2. Eric, based on your description in this post of the sort of work you’re doing, it sounds like you might profit from learning how to use Alloy. Think of it as a high-level interface to a SAT solver. Now that you’ve formally specified your state machine, you can proceed to write out system safety constraints: logical propositions that, so long as they hold, will prevent your UPS from doing anything nasty like frying your computer or needlessly cutting power to it. Obviously, do this part in consultation with your hardware team to ensure that your constraints are sufficient. Encode all this into Alloy’s language, and it’ll search for inputs that lead to those constraints being violated.

    N.b., the tool is open source, but you’ll need to buy a copy of the associated book in order to get anywhere with it.

    1. >N.b., the tool is open source, but you’ll need to buy a copy of the associated book in order to get anywhere with it.

      I’ll take the easy way in and read the case studies first.

      At the moment, my informal demonstration of correctness for the software looks like this:

      1. The Python pseudocode calling state() and transition() is definitionally equivalent to the state/transition diagram I can see,

      2. The relationship between the Python pseudocode and the state-machine code is simple and easy to audit. It is within the cognitive capacity of a human to be effectively certain that the generated Go correctly translates the pseudocode.

      3. Therefore we can have high confidence that the Go correctly captures the intent of the state-transition diagram.

      If I put in enough work, maybe Alloy can deliver a more formal demonstration. I suspect that the effort cost will be quite high for a relatively small increment of confidence, but I’ll probably do it anyway to learn the tool.

        1. Yep…TLA+ is what Amazon use for their High Availability systems.

          I used Z for years, and TLA+ is a close relative..it was pretty trivial for me to make the switch – and it has a much nicer toolset.

  3. @eric

    “What we want to do is use a variant of that technology called LiFePO – Lithium Iron Phosphate”.

    To be correct, and I admit this is a scientific nit-pick, Lithium Iron Phosphate is LiFePO4 – that last 4 is a subscript, but I don’t t know how to insert that in a comment.
    I also don’t know what LiFePO is, but it most definitely is not Lithium Iron Phosphate. This has been mentioned before. Incidentally, in the literature where this chemistry is mentioned, the accepted abbreviation appears to be LFE – equally bad, but not an erroneous chemical formula.

    Regards,

    Geoff

    1. Hey, it beats hell out of “LiFePo”, which is how he originally wrote it, and could lead one to think we’re using batteries with Polonium in them. I don’t think <sub> works in comments; I know I couldn’t get it to work on several of the Wiki pages on GitLab either, so I decided that “LiFePO” was good enough, as it at least names the correct elements, even if it doesn’t give their relative proportions.

      A much better three-letter abbreviation is “LFP” (Lithium FerroPhosphate).

  4. I think you might find yourself needing to forgo having smart built in controller chips that come with the batteries. Yah, it sucks and makes things much more complicated, but if you do it on your end, you can be vastly more flexible about batteries and the batteries will be cheaper without your users being any worse off. You may then also be able to support more battery technologies — LiIon 18650s, automotive lead acid batteries, etc. — which would be both a boon to the hackers using this and would allow more experimentation.

    1. > Yah, it sucks and makes things much more complicated

      Probably infeasably so. At minimum, to homebrew a BMS we’d have to run voltage sensors into each cell.

      1. As I understand your original reason for doing this project, proper battery management is the core problem you were trying to solve. Marketing it with a nice front-end, etc. was the icing.

        The reason that Li-Ion batteries come with management chips is that they contain the thermometer that protects the cells. By necessity, this needs to be inside the battery, near the cells that it’s protecting, and it needs to have guarantees about how fast the protection can react. Other battery technologies are less difficult to work with and, as such, building all the safety into the battery itself is not as critical.

        I think the core value of your product comes from having a good battery management solution. You can get away with a custom build battery (if it exists) for R&D but, as you say, it’s really going to push the price up. For production I think you’ll want to design your own in order to meet your specification and price points. Of course, you can apply Production Engineering and leave that latter change for a revision when you’re working on margin but the tradeoff is that the first revision is probably going to be a prototype in so much as it’ll be expensive and single-source.

  5. BTW, one other thing, Go is only “mostly” safe. You can still get arbitrarily nasty undefined behavior because of concurrency screws. (Rust has the interesting feature that it’s concurrency safety is enforced at compile time by a static checker, but Go lacks the sort of linear types needed to enforce ownership that way.)

    1. >BTW, one other thing, Go is only “mostly” safe. You can still get arbitrarily nasty undefined behavior because of concurrency screws.

      I believe that can only happen if you do multi-thread access to something that’s not a channel.

      Not an issue for UPSide, it turned out to be simple enough to write the daemon with only one thread of control.

  6. Motorcycle Batteries. You should allow for any Lead-acid or Lithium version, but may not have access to the chip in the latter case.

    Do you have a link to the state diagram or description of the 8 states?

    To go back to Pb v.s Li, the former are cheaper and ubiquitous and are less picky about charging. Beyond that there is an error in you are looking to “cheap” batteries that are only cheap because they are capable of cranking an engine.

    Lithium cells are just as problematic and toxic as Lead. And 2.5x or more expensive (I can find recycled Lead batteries easily and have them as backups for my solar panel system).

    1. >Do you have a link to the state diagram or description of the 8 states?

      git clone git@gitlab.com:esr/upside.git
      cd upside/docs
      make view

      Have graphviz and dpic installed before you do this.

    2. But possess 4 times the energy density and half the weight of the Lead Acids.
      Moreover, Lead-Acids possess a problem. Peukert’s Law.

      https://en.wikipedia.org/wiki/Peukert%27s_law

      As you pull current from the Lead-Acid battery, the internal resistance of your plates will bleed you of your charge capacity. At 2C, you don’t have Amp-Hours, for example, you have HALF of them.

      A Lithium Ion battery DOES NOT HAVE THIS PROBLEM.

      And, better yet, how many deep-cycles of your Lead-Acids do you have? 100? 200? 500 at best. A Lithium Ion goes for 2k or more. You’re also replacing them at 4-6 times the rate of the Lithium batteries.

      It’s better to understand the real issues instead of removing any doubts…please don’t remove any more of those…

  7. To expand, if the only batteries this can work with is some kind of lithium cell, I would consider it an EPIC FAIL. ( aslo consider where Lithium originates, the processing…)

    If you can find a spare well functioning Lead Acid battery somewhere, you won’t be able to use it because you have snooty envrionmental SJWs on the team insisting you ought not?

    Wouldn’t it be better to recycle Lead batteries into UPS units?

    1. >To expand, if the only batteries this can work with is some kind of lithium cell, I would consider it an EPIC FAIL.

      Brace yourself for disappointment, then. Unless you or someone else finds me an SLA with a fuel-gauge/BMS built in, lead-acid is off the table.

      >If you can find a spare well functioning Lead Acid battery somewhere, you won’t be able to use it because you have snooty envrionmental SJWs on the team insisting you ought not?

      Where in the freaking fuck did you get that idea? No, the major reasons for avoiding those is (a) we can’t find one with a smart BMS, and (b) short lifetime. I like the idea of burdening the waste stream less but it’s not dispositive.

      1. Each battery chemistry has its own family of discharge curves.

        Here’s a discharge curve for Li-Ion:
        http://siliconlightworks.com/image/data/Info_Pages/Li-ion%20Discharge%20Voltage%20Curve%20Typical.jpg

        Here’s some curves for LiFePO4:
        https://www.powerstream.com/z/life-discharge-series-a.png

        Here are some for Lead Acid:
        https://i.stack.imgur.com/BMCnJ.jpg

        As you can see, the Voltage of lithium based technologies doesn’t vary much until the cell is almost empty. This means that it’s not straightforward to work out how much capacity is remaining.

        On the other hand, lead acid, whilst not completely linear, has a much more proportional relationship between voltage and remaining capacity. This means that the circuit to gauge the remaining capacity is comparatively simple. The tradeoff is that you might need a more sophisticated regulator for supplying the power to the load. However, if you’re using a switch mode power supply with a boost topology then it ends up being pretty similar for both the lithium technologies and the lead acid ones as the voltage output to the load is always more than the input.

        The BMS should also provide you with over-current protection. For lithium technologies this is, again, rather complicated. For lead acid a simple circuit is possible.

        You are rightly cautious about the lithium technologies: their thermal runaway behaviour makes them rather unpredictable. Whilst still not risk free, lead acid is hardy and easy to work with.

  8. Please let me know once you have enough hardware/software design to have a viable product. My current desktop UPSs are basically worthless at this point. I either need to replace the batteries or replace the UPS.

    If there’s enough data/power capacity in the design, I’m happy to order/build one to help get some real-world consumer data for this. I’m reasonably handy with a soldering iron/multi-meter to be able to assemble things, if needed, though I don’t have the time to make cases/PCBs from scratch.

  9. What’s the current status (pun intended) of this project? I haven’t seen any more recent updates.

    1. >What’s the current status (pun intended) of this project? I haven’t seen any more recent updates.

      It got stalled some months back because our power engineeer (Eric Baskin) had to build a farm. Literally.

      However, I spoke with him about 10 days ago. He’s back to work on a prototype of the high-power electronics.

      A lot of the control software has been written and tested. Not hooked to live hardware yet, but to test loads that describe sequences of power events as they would be registered by on-board sensors. I have successfully lit up an I2C status connected to a BeagleBone display with that code.

      Here’s the Upside repository

Leave a comment

Your email address will not be published. Required fields are marked *