upside wants a firmware dev

The UPSide project, announced here two weeks ago, has come together with amazing speed.

We now have:

* A hardware lead – A&D regular Eric Baskin – with thirty years of experience as a power and signals engineer. He is so superbly qualified for this gig that my grin when I think about it makes my face hurt.

* A high-level system design (about which more below) that promises to be extremely capable, scalable, flexible, and debuggable.

* A really sharp dev group. Half a dozen experts have shown up to help spec this thing. critique te design docs, and explain EE things to ignorant me.

* Industry participation! We have a friendly observer who’s the lead software architect for one of the major UPS vendors.

* A makerspace near me where the owner recruited himself onto the project and is looking forward to donating bench time and skilled hands to the hardware build.

All this helpfulness almost – but not quite – fills in my deficits as a designer/implementer. I don’t really know from hardware design, so I’m attacking the problem with the modularity and information-hiding principles I know from software.

Here is how the design looks:

An I2C bus that ties together a “forebrain” which is a Unix SBC, almost certainly at this point an Olimrx LIME2, with a “midbrain” that is an Arduino-class microcontroller.

The midbrain is mechanism – a simple state machine whose job is to control the high-power subsystem (inverters, battery, AC input and output). Policy decisions and stuff like battery state modeling will live in the forebrain. It will also run the USB and Ethernet interfaces, and host the development environment for the firmware

The forebrain will talk to a 20×4 LCD panel over I2C, and various other controls like alarm mute and self-test buttons via GPIO pins.

I’ve actually written the spec for the I2C bus messages already. And here’s your cute hack for the day…

I realized early on that one of the first things I needed to do was draw a state/action diagram for the system so I could pin down its behavior in response to any given transition in its environment (mains power up, mains power down, battery dwell limit approaching, those sorts of things). So I reached for one of my favorite tools, a graph-drawing DSL called dot.

Only when I write the first version of the graph, I found the dot markup cluttered and repetitive. So I wrote a couple of cpp macros named “state” and “action” that expand to dot markup, and expressed the graph as a sequence of macro calls.

Then I blinked, looked again, and realized…hey, I could compile these calls to C source code for a state machine! And now it is done – I can already generate the tricky part of the application logic for the midbrain directly from the state/action diagram. (The action functions are stubs but the control flow is all there.)

(If the fact that I just solved a design problem by writing a DSL to generate code in another DSL and provably correct equivalent C application logic seems weird to you, you must be be new here. This is how I think all the time. It is obedience to the Unix wisdom: never hand-hack code you can generate from a higher-level description.)

This, however, does not solve the entire firmware problem by any means. The midbrain’s going to need system logic to do things like receive and send I2C messages, poll A2D converters from sensors watching the mains and battery voltage, and so forth.

Accordingly, we need a firmware developer. I’ll learn how to do this if nobody steps up (which is why I said “wants” in the post title) but the whole process will doubtless go faster and more smoothly if we have someone with experience. So:

WANTED: One firmware hacker. Must be familiar with AVR-class microcontrollers and the Linux toolchains for them. Experience with I2C and low-level programming of USB endpoints would be a plus. Perks of the job include getting one of the first UPSides made, your name in lights, and working with a dev crew that is impressive even by my elevated standards.

EDIT: Well, that didn’t take long. A&D regular Jay Maynard has signed on.

55 thoughts on “upside wants a firmware dev

  1. I wish I could help. I’m a bit short on the 3 requirements there. I mostly worked on Rabbit Semiconductor and Microchip stuff when I was still working on the microcontroller stuff, and mostly with Modbus RTU and RS485 rather than I2C.

  2. Why are you doing USB at the midbrain level instead of the forebrain?

    What you describe for the midbrain is what I do for a living, although on Atmel SAM devices (ARM-architecture) rather than AVR. Having the state machine done already makes the midbrain design and code a lot simpler. I speak mainly Modbus, though I2C shouldn’t be that hard to pick up, and it’s native on current parts.

    • >Why are you doing USB at the midbrain level instead of the forebrain?

      We’re not. The USB host port is attached to the forebrain. I just added that because I don’t know how to program USB ebdpoints and it would be nice to recruit someone who does.

    • Jay, I just coincidentally have been looking at some SAM devices for a side project (comparing to NXP/Freescale i.MX and others). Would you mind having Eric put us in touch via email so I can solicit your advice? I didn’t see any way to post privately through the blog reply options, so I’ll just hang it out here (I assume that as administrator Eric can see my address; are address scrapers still a thing? I can never remember if it is still a bad idea to put your primary email on a public blog comment or not)

  3. My experience specifically with AVR is somewhat limited sadly, most of my control software work is very bespoke. I’ve worked on embedded control software for Rolls Royce engines, but that’s remarkably different to AVR with I2C.

    Happy to offer my skills if they’re needed, however, and I’m happy to learn a new toolchain. The basic principles should transfer.

    • >Happy to offer my skills if they’re needed, however, and I’m happy to learn a new toolchain. The basic principles should transfer.

      This could be a good situation, then. Jay Maynard, who does know these toolchains, just volunteered. If you like, you can second him and learn from what he’s doing – it sure wouldn’t hurt to have a code reviewer who knows how things are done in microcontroller-land better than I do and has a different background than Jay.

  4. Will the device include an audio alarm? If so, could you make sure that you eschew the traditional beep and create a sound that is easier to locate?

    Many years ago, I read reports on binaural directionality tests. The reports found that a sound source was easiest to locate by a standard auditor when it has some white noise in it. A pure sine wave was found the hardest to directionally acquire. Square or triangular wave forms weren’t much better. Adding a percentage of white noise in the sound made the localization much easier, even if it was brief (100-ms) pulses.

    When I read that, I was expecting these findings to take the industry by storm. Instead, we are flooding with indistinguishable beeps that are almost impossible to locate. Ever had something beep in a rack that has about 30 devices?

    Please kindly add this to the requirement list.

    • >Will the device include an audio alarm? If so, could you make sure that you eschew the traditional beep and create a sound that is easier to locate?

      Your report on directionality tests is interesting. Thank you for posting it; I learned a thing!

      I am not, however, sure that better directionality is a significant win for the deployments we’re expecting. UPSide-1 is intended to be a personal and SOHO device, not something that will sit in a datacenter rack.

      Thus, I think whether we will do this depends mainly on the incremental parts cost. We were indeed planning to use a buzzer for alarms. Plugging speakers into the SBC’s audio output would certainly be possible but seems like overkill. What’s the smallest, cheapest device we could put inboard that would be able to generate white noise?

      • > Plugging speakers into the SBC’s audio output would certainly be possible but seems like overkill. What’s the smallest, cheapest device we could put inboard that would be able to generate white noise?

        Frankly, a little (1.5″) speaker is not much more expensive than a buzzer (maybe even cheaper), and if you drive it from a pin on the “midbrain” computer with an appropriately pulse-width-modulated signal, you could make it talk. The hardest part there is getting the audio recorded and encoded in the first place.

        Possibly get your wife to record short snippets for us? ;-} (Research shows that female voices are more closely paid attention to than male ones – the USAF Strategic Air Command discovered this one in the ’50s.)

        • >and if you drive it from a pin on the “midbrain” computer with an appropriately pulse-width-modulated signal,

          There’s a system-partitioning issue here. If I’m really going to ship PCM over a GPIO pin I’d rather do it direct from the forebrain – shoving that volume of data over I2C doesn’t make any sense to me and I refuse to embed PCM data in the midbrain where it can’t be easily changed. But that means no alarm will be available on a failure to boot. I guess we need a buzzer after all.

          Cathy says she’ll do it. :-)

          For reasons that should be obvious to you (though they may be lost on my more SF-ignorant readers) may brain is now vividly imagining Cathy’s voice saying “I am a thirty-second bomb. I am a 29-second bomb…”

          • Change this to “Pssst – I am a thirty-second bomb…” and the requirement shall be deemed satisfied!

            Did you notice that humans use a short burst of white noise when they want to quietly attract someone’s attention? Is it a social convention or an adaptation to the innate directionality advantage of such sound?

          • There’s a system-partitioning issue here. If I’m really going to ship PCM over a GPIO pin I’d rather do it direct from the forebrain – shoving that volume of data over I2C doesn’t make any sense to me and I refuse to embed PCM data in the midbrain where it can’t be easily changed. But that means no alarm will be available on a failure to boot. I guess we need a buzzer after all.

            Yup. OK, if you do want verbal notifications, use a simple resistive summer at the input to the speaker (possibly buffered with an op-amp, or even a single transistor). One input is the PCM signal from the forebrain, with all the sophisticated messages we want. The other input is a simple pulse train from the forebrain, signaling “BOOT FAILURE, you’ve been a lovely contestant, thank you for playing.” Maybe the two-tone siren that the European police use?

            For reasons that should be obvious to you (though they may be lost on my more SF-ignorant readers) [my] brain is now vividly imagining Cathy’s voice saying “I am a thirty-second bomb. I am a 29-second bomb…”

            Johnny Rico (and RAH) would be proud ;-}

          • Cathy’s voice saying “I am a thirty-second bomb. I am a 29-second bomb…”

            I pray that feature remains hypothetical. If someone who’s not in on the joke hears that and calls the Feds, that’s a great way to get them to invoke the Spell of National Security and project a constitutional-rights-nullifying field around your home while they search it for incriminating evidence. The real fun begins when they find your guns, swords, and other stuff.

          • Rather than an alarm on failure to boot, how about a pleasant status message indicating the boot is complete and the Sanity Check has passed, indicating that all daemons are up and appear to be operating normally.

            The Sanity Check is a pre-requisite for writing to the SD card that a new kernel/etc. is good rather than “on probation” (which would mean that the boot loader should go back to the previous known-good boot configuration the next time it tried to boot)

        • >> (Research shows that female voices are more closely paid attention to than male ones – the USAF Strategic Air Command discovered this one in the ’50s.)

          Do you have a reference to the study in question? My first intuition on reading that sentence is that the USAF might have only had straight male listeners in the subject pool. (After all, at the time it was only going to be male pilots, male control room operators, etc..) It would make more sense to me if the brain prioritized Other Gender voices or Gender of Attraction voices, then for both genders to favor female voices. Although I am prepared to be proven wrong.

      • Try to have a audible alarm that is not in the 3,000 Hz (or higher) level, many of us that are older lose or have lost that range.

  5. I really like this project.

    If I wasn’t up to my ear-balls with other things, I’d be happy to contribute real effort to this project but I don’t expect to have that kind of time until perhaps the fall.

    In the mean time, I’d like to see (didn’t find it) a short statement of “desires” (“requirements” can be scary) about external comms: I’d like the capability for the thing to tell me it’s having problems via one or more (configurable, addable later, whatever) “standard” interfaces. I don’t want to have to look in the back of the closet at the blinking “warning” LEDs or audible alarm. Perhaps something like:

    The hardware [will | may | should] be provisioned with a common standard async serial output (e.g. RS-232/422) and/or Ethernet for (but not limited to) the ability to provide status information to external devices/networks.

    Some “user sentences”:
    – I would like to be able to remotely connect to the UPS over the network and see a history of power failure times and dates.
    – I would like to be able to remotely connect to the UPS and learn about charge level, battery status, current (instantaneous) and average load on the output and the presence (or not) of supply power.
    – In a second generation device, I would like to be able to connect remotely and launch a process that would turn off, wait some number of seconds, and turn back on) a single output receptacle (e.g. power cycle some external device.)

    • >a short statement of “desires”

      Pretty much everything you want is on the roadmap, with the single exception that we will probably not support RS-232/242 for anything but console debugging. We do plan to support the USB HID profile for UPS, however.

      When and in what order these things will happen depends on what kind of help recruits itself in. All external comms is going to be handled by the UNIX SBC which is of course a very friendly development environment for this sort of thing and comes with host USB and Ethernet ports.

      Want to make your features happen? Join the gang, build a prototype once we have the hardware recipe ready, then dive in and do Linux programming like you probably already know how to do.

  6. I’m retired after 50+ years as a hardware designer… and I’m enjoying it.

    But I will throw out one tip that I discovered the hard way: Build a hardware reset method for your I2C parts. Many parts do not have a hardware reset pin, so I wound up retrofitting a switch on the power bus when a expansion electromechanical device threw enough noise onto the bus to lock it up. The level of redesign necessary to protect the bus was not feasible within the time and cost constraints, but tacking in the power circuit was. If your design is going to allow expansion with as yet undefined and unknown devices a reset capability for the I2C bus is needed.

    • >If your design is going to allow expansion with as yet undefined and unknown devices a reset capability for the I2C bus is needed.

      Fortunately, that’s not the case. We presently envision only a small number of well-studied devices on the bus – the Unix SBC, an ARM microcontroller, a 20x4LCD, and a temperature sensor. I will bear the issue in mind, though. With the inventory I have I’m guessing the most likely bad actor would be a dodgy cheap LCD panel?

      • In my case, it was noisy stepper motor. The I2C bus does not use differential signals, so it’s very susceptible to induced spikes. The expansion device had both a motor and an I2C temperature sensor on the end of an 18 inch cable. Shielding and separating the signals didn’t stop the lockup, which occured when a slave would lock the data bus low. Neither of the two slave devices that did this had a reset pin but fortunately were on the same power pcb trace.

        • Many (most?) modern devices will get out of that condition if you have the master toggle the clock line enough times to step through the slave device state machine enough times that it will get back to the idle state no matter which state it started in. For most devices that ends up being some integer multiple of 8 bits if you are reading bytes out of registers, so toggle the clock line 8 or 16 or 32 etc. times.

  7. Pingback: UPSide Project Update | Irreal

  8. The choice of AC input power plug provides for 20 amps input, but isn’t this a problem for a lot of potential users? Even folks that own their own houses may not want to run a 20 amp circuit to the room with the computer…. or is this mostly for a server in basement where it is relatively easy to run another AC circuit?

    • That specification is for the power connector not melting until more than 20A is passing through it (probably over time, in hot rooms, and bad conditions etc etc.) It doesn’t have an effect on the specification of how much power the device will want to draw – just how much it is able to safely.

  9. Hey, can your state machine cope with the case of a person who wants to preferentially use aux power (solar) when it is available and the batteries are full-ish? Or is that outside the scope of the project?

    My basement server room uses something like a kilowatt constantly, and I’m always looking for ways to reduce that bill. For various reasons, a normal solar system is impractical right now, so I’m looking for workarounds that technically meet code requirements.

    One hilarious option I’ve considered is setting some solar panels outside on the lawn with microinverters and running an extension cord through the wall and into a non-hardwired transfer switch. But it would be a lot slicker if I could plug a car battery (or 4) into my main UPS’s external battery port and tell the UPS to switch to battery when the battery goes over 90% and back to utility power when the external battery discharges to 80% or less.

    • >Hey, can your state machine cope with the case of a person who wants to preferentially use aux power (solar) when it is available and the batteries are full-ish? Or is that outside the scope of the project?

      It is. That would make UPSide a “transfer device” which has much stiffer regulatory barriers.

      • I think the best way for UPSide to deal with this is to say that such “transfer devices” will be external components (just as the hindbrain is assumed to be in the battery pack) controlling the AC Line In power, and that a future version of UPSide may talk to such things as part of WINTERMUTE support.

        Westar Energy defines a “momentary outage” as one lasting less than five minutes. What they clearly mean by that is “Don’t call us to tell us your power is out until it’s been five minutes, because automatic circuit-breaker resets fix problems that last less than that.” (And they’re almost certainly padding it because impatient people think three minutes is five minutes.) So, it’s a reasonable assumption that when UPSide has been running on battery for five minutes, there’s a good chance the outage will go beyond the 15 minutes for which SUMMERTIDE is designed. Therefore, if the transfer device itself can’t be pre-programmed to kick in after a few minutes, we’d want UPSide to be able to tell a generator to fire up, or a switchover to such extended battery bank as the source for its upstream power.

        If this is successful, without a way to know status of the transfer device, UPSide will think grid power has been restored, the upstream battery will be used to recharge the UPSide battery bank, and when the former runs out of power, we’ll be back to SUMMERTIDE again.

        • >So, it’s a reasonable assumption that when UPSide has been running on battery for five minutes, there’s a good chance the outage will go beyond the 15 minutes for which SUMMERTIDE is designed. Therefore, if the transfer device itself can’t be pre-programmed to kick in after a few minutes, we’d want UPSide to be able to tell a generator to fire up, or a switchover to such extended battery bank as the source for its upstream power.

          Policy to be done in the controller daemon. Ping me about this again when you’ve examined a working prototype.

          • > Policy to be done in the controller daemon

            Yes, but is there anything we need to provide hardware-wise to be able to do this? Do “transfer devices” normally provide a USB interface or what?

            • >Yes, but is there anything we need to provide hardware-wise to be able to do this? Do “transfer devices” normally provide a USB interface or what?

              I don’t know enough about that category to say.

              • Many of them have serial or network interfaces. Typically very basic stuff – switch source on command, set which source (if any) is the preferred source, read basic stats or conditions, etc. When they are integrated with a port-switchable PDU, the control interface will often have options to control ports, set load-shedding, ramp-up and ramp-down order, etc – the stuff you’d normally find in an addressable PDU.

                I don’t have experience with expensive units, but I will say that low-end units have incredibad interfaces, pretty much universally. They often have terrible text menus which makes automation nearly impossible. The network ones expose everything through SNMP which is great if you are paying $15,000 a year for a managed SNMP suite, but shit for everyone else. If there is a web interface, it usually defies all standards and common sense, using javascript to “encrypt” passwords to POST into the login form or some other moronic scheme.

                The management hardware is usually unstable as hell. I’ve scrapped units before because I needed to physically visit it to unplug and replug the management card every time I wanted to use the network interface to cycle a port.

                APC PDUs are pretty decent – those are the $500 power strips. Every other brand I’ve tried is crap. But APC isn’t universally good, I have a couple of APC transfer switches which are apparently permanently brain damaged. The display shows random segments lit, and random status LEDs lit or unlit according to no rhyme or reason that I’ve been able to divine. Still works fine as an automatic transfer switch – just don’t look at it or touch it or log in to it.

      • Only in the senses in which it is already a transfer device. To be clear, I was not necessarily talking about an AC transfer. I’d be perfectly happy to run the inverter off of a secondary DC power connector. Most people would use that aux DC connector for more battery capacity.

        I’m thinking of a penny-pincher mode. In this mode, the internal or primary battery bank is charged from utility as usual and will be consumed during an outage (when the aux battery is low and utility power is absent/messed up). The secondary battery bank would NOT be charged by the UPS, but would need an external charge controller, and the UPS would be willing abandon a perfectly good utility source to run off of the external battery bank when it is close to full (when the sun is shining).

        Only one AC input and one set of AC outputs. Shouldn’t bring any new regulatory problems. Ground and neutral continuity maintained across the unit, and no chance of backfeeding beyond the main break-before-make relay that you have to use when you switch from input to inverter anyway.

        This would also make the unit more attractive for off-grid and expensive-grid uses.

  10. Then I blinked, looked again, and realized…hey, I could compile these calls to C source code for a state machine!

    And it’s for gems like this that keep me coming back.

    Ontopic: do keep the idea of a version for server racks in mind.

    • >Ontopic: do keep the idea of a version for server racks in mind.

      I shall. Not for UPSide-1 though. Scope creep is deadly dangerous when you’re trying to get to your first prototype; I’m being quite the bastard about feature requests.

      • But it’s good to put things on the roadmap for future development, which can inform decisions about current design.

        For instance, if we can see “SOHO with one computer and maybe some monitors” as the “profile” we’re targeting UPSide-1 for, we might write a bunch of parameters to soho.profile that would otherwise be hard-coded Magic Numbers, and when we need to support other use-cases we don’t have to refactor the whole code base. (Similar logic to writing strings to an EN_US/ localization file that for UPSide-1 would be the only such file, allowing others to be created with minimal friction when that time comes.)

        • >For instance, if we can see “SOHO with one computer and maybe some monitors” as the “profile” we’re targeting UPSide-1 for, we might write a bunch of parameters to soho.profile that would otherwise be hard-coded Magic Numbers, and when we need to support other use-cases we don’t have to refactor the whole code base.

          Monster, I do this kind of thing as a matter of course. You might as well be instructing me to eat or breathe.

          One of the things already in my plan, for example, is that the management daemon in forebrain won’t know any I2C addresses at boot time. Those will come from a config file describing the device’s actual hardware.

          • Good to know. In the prior discussion about string localization, you seemed to resist abstracting them out to a file, but that might have been a flat-out rejection (so no potential refactor) rather than a deferral to v2.

            > Those will come from a config file

            Given what you’ve said about such things in the past, I thought you’d be scanning the I2C bus at boot, enumerating the devices, and figuring out The Right Thing To Do Most Of The Time™ and only needing a config file to support overriding the defaults when that turned out to be wrong.

            • >Given what you’ve said about such things in the past, I thought you’d be scanning the I2C bus at boot, enumerating the devices, and figuring out The Right Thing To Do Most Of The Time™ and only needing a config file to support overriding the defaults when that turned out to be wrong.

              That’s my style when it’s possible. The material I’ve found on I2C bus scanning warns that this is risky, though – bad consequences ranging from bus lockups to corrupted EEPROMs(!) can ensue.

              Besides, that kind of autodetect is only weakly indicated when devices never hotswap. The I2C address inventory of any particular UPSide instance is going to be pretty static.

            • >In the prior discussion about string localization, you seemed to resist abstracting them out to a file

              I was resisting it. Still am. My experience with that kind of localization has been unpleasant, with complexity and headaches seeming way out of proportion to the added value. But we’ll see – depending on how the displays are designed, there could be little enough locale-dependent text to keep the pain down to a tolerable level.

  11. Jay and Eric:
    If you would be willing to take on another apprentice, I am available.
    I retired a little while back and would be able to put in some serious time on this project.
    I have done some avr programming, mostly arduino with excursions into low level stuff when I hit things where the standard libraries didn’t reach.

    My background is 30 years in a vinyl window manufacturing plant, first writing, then maintaining the entire software and data (and for quite a while, the network hardware/software) for the entire process from order entry to shipping and exporting the completed orders to an accounting system.

    The last task was converting the software from CA clipper 5.2 to python. This conversion being forced when windows quit supporting 16 bit dos programs with windows 7.
    I wanted to do it sooner, but couldn’t get the resources from management till then.

    Jim Hurlburt
    Bend, OR USA (formerly Yakima, WA)

    • >If you would be willing to take on another apprentice, I am available.

      The current state of UPSide is that we’re going to use hard logic rather than a microconrtroller; the regulatory implications of field-modifiable firmware turn out to be a crash landing.

      Still, the forebrain software is not going to be a trivial job. Are you willing to learn Go?

      • Well —
        In your writings you advise learning new languages. I’ll take your word for it that Go is a keeper and worthwhile. So, sure. Your advice on Python was certainly good.

        I have done a lot more work on things of the general nature of the forebrain — making differing systems talk to one another, scheduling and priorities.
        I’m sure that a UPS is going to be quite a lot different than manufacturing, but more in detail than in large.

        So —
        Any suggestions about where to start with Go? Beyond installing it on my Linux system and playing around with it some. I’ll also look back in your blog postings and see what I can find in recommended readings.

        Also, this will be my first open source developer experience. What is the process of getting started and involved? In this specific situation. It doesn’t look like submitting a patch is quite the way to start here.

        Jim

        • >I’ll take your word for it that Go is a keeper and worthwhile.

          I’m planning to do the forebrain software in Go. It’s good for writing service daemons.

          >Any suggestions about where to start with Go?

          “A Random Xoogler” had good recommendations.

          >What is the process of getting started and involved?

          Get a GitLab ID. Tell me what it is so I can add you to the project. Read the wiki, read the bugtracker issues, join in the design discussion.

          It’s not time to write software yet. But being in on the discussion will help equip you for when we do.

          The one major role we don’t have filled yet: we need somebody who knows how to write USB endpoint code. Yoiu might want to study USB programming.

  12. Even though that might seem to be overkill… what about interfacing with diesel generators?
    I know that a typical SOHO setup does not have a diesel, but maybe we should keep this option in mind anyways?
    Due to coincidence I got a reasonably sized diesel generator last week for 0$ as it had a fault that prevented it from working correctly, though that was trivial to fix.
    So now I am sitting on this device, maybe others are as well?

    I think two things would be required.
    1) Incoming power feed into UPSide
    2) Letting the generator know it shall start – I think a relay contact (both NO and NC) is the right weapon here
    2a) Maybe something more fancy is needed/wanted to understand whether the generator actually started successfully. That might be verified by measuring the power feed from the generator, but I guess there are generators that have a dedicated interface for that.

    What do you guys think?
    Not sure if that would significantly increase the BOM cost or can be made optionally.

    • >Even though that might seem to be overkill… what about interfacing with diesel generators?

      This has come up before. Out of scope for v1; a complicated problem without well-defined victory conditions.

Leave a Reply

Your email address will not be published. Required fields are marked *