I think this diagram is now stable enough to put on the record.
Both this diagram and the Go code for the policy logic are generated from this pseudocode:
render.state("DaemonUp", "Daemon running")
render.action("DaemonUp", "ChargeWait", CHARGING)
render.state("ChargeWait", "Charge wait")
render.action("ChargeWait", "MainsUp", CHARGED)
render.action("ChargeWait", "OnBattery", MAINSDROP)
render.state("MainsUp", "On mains power")
render.action("DaemonUp", "OnBattery", MAINSOFF)
render.state("OnBattery", "On battery power")
render.action("MainsUp", "OnBattery", MAINSDROP)
render.action("OnBattery", "Overtime", DWELLWARNING)
render.state("Overtime", "User warned of shutdown")
render.action("Overtime", "PreShutdown", DWELLTIMEOUT)
render.state("PreShutdown", "Awaiting power drop")
render.action("PreShutdown", "ChargeWait", RESTORED)
render.state("UPSCrash", "UPS goes dark")
render.state("HostDown", "Host has shut down")
render.action("PreShutdown", "HostDown", HOSTDOWN)
render.action("PreShutdown", "UPSCrash", BATTERYDRAIN, unreachable=True)
render.action("OnBattery", "ChargeWait", RESTORED)
render.action("Overtime", "ChargeWait", RESTORED)
render.action("HostDown", "MainsUp", RESTORED_LATE)
render.action("HostDown", "UPSCrash", BATTERYDRAIN, unreachable=True)
To see the full context of this, clone git@gitlab.com:esr/upside.git and explore the docs/ directory.
Hi,
Why does
—–
Trigger: Poll of mains reports good AC voltage
Means: Power came back after host shut down
Action: Cycle AC power to load host.
Alarm: UP
—–
…go to “On mains power” rather than “Charge wait”?
Aaand andyjpb wins the prize for being the first to spot my paste error.
This is why I post such things.
Only needed a one-token fix to the pseudocode. Changes the look of the graph pretty drastically, though – we ran into some kind of Thom catastrophe in the spring-layout algorithm’s behavior space there.
What is the difference between “Power came back after host shut down” and the “Daemon Running” state at the top, other than that the battery was able to keep the SBC running through the outage? Is that difference enough to justify a separate state, or should “Host has shut down” just send the system back to the “Daemon Running” state?
Related: “On mains power” does not distinguish between whether the host has powered up or not. Is that a relevant distinction? It seems that the only reason to care whether it’s powered down is to determine whether to try to abort the shutdown command (if that’s even possible).
>What is the difference between “Power came back after host shut down” and the “Daemon Running”
Please put these questions on the project issue tracker so the whole design group can discuss them. You might want to consider joining the group yourself.
https://gitlab.com/esr/upside/issues/64
You have one transition (“Pull of mains returns bad AC voltage”) twice, once with alarm and once without.
Is it UML for state diagrams?
>You have one transition (“Pull of mains returns bad AC voltage”) twice, once with alarm and once without.
Yes, that second one should fire the “DOWN” alarm.
>Is it UML for state diagrams?
I don’t understand the question.
>> Is it UML for state diagrams?
I see that it is not
> I don’t understand the question.
UML (Unified Modeling Language), beside well known graphic language to describe class hierarchy, includes graphical language to describe state transitions (to describe state machine). In UML statechart labelled nodes are states, and labelled edges are transitions (you have other shape of node as transitions, like for Petri nets). Edges can also be annotated with conditions.
Some advantages of UML statechart over traditional Finite State machine diagrams include hierarchically nested states (which allows to avoid repeated transition, if transition can apply to any state in given category/subset of states), orthogonal regions, enter and exit actions.
Though UPSide state diagram should not be very complicated.
I’m confused. Why does the state that emits UP go to the state that is labelled “On Mains Power” vs “Charge Wait”?
By the time that happens, the battery has run past the limit of shutdown time, so certainly can’t support shutdown plus margin.
So, if I understand correctly, if AC goes up after the host has received shutdown command but before shutting down, nothing would be done to wake it up.
>So, if I understand correctly, if AC goes up after the host has received shutdown command but before shutting down, nothing would be done to wake it up.
Well, it’s not down yet. We can tell when that happens by watching the load on the master socket.
There’s a problem here in that we don’t know if the host can receive a shutdown command – the USB cable might be unplugged. If it can’t, this is accidentally the right thing to do.
If it can, the worst case is that the host shuts down cleanly when mains power is back up up and shutdown could have been aborted. I can live with that outcome, especially since little dropouts tend to come in closely spaced bunches. It’s probably prudent of the host’s operator to leave it down for a few minutes.
We couldn’t actually do anything more clever unless HIDUPS has an “abort shutdown” command it can pass. I don’t know that it does.
> It’s probably prudent of the host’s operator to leave it down for a few minutes.
Anything that goes beyond a few minutes is a Big Deal, so waiting a while is definitely The Right Thing To Do™, as is not bringing the load sockets online until after the battery has reached AT LEAST enough for a SOFTLANDER dwell time. This, of course, ought to be user-configurable, and the status display should, in addition to “CHARGING”, indicate expected dwell time of the current charge level, and some button should allow an override to bring up the load sockets at less than the pre-set threshold.
Another thought on this is that it should be user-configurable whether a Host Down status should cause the AC outlets to be disabled (and then re-enabled once sufficient charge is established). Most BIOSes can be set to bring the machine up when the power is restored (a logical setting for a server), leave it in whatever state it was in before power failed, or leave it off regardless. This would allow UPSide to bring the host up even if the shutdown couldn’t be aborted.
Usually after receiving the “shutdown now” message from the UPS, the host sends the UPS a message that means “I acknowledge your shutdown request, I have prepared for power down, and now require the UPS to turn the host off for a minimum of 20 (or some configured value) seconds.”
After commanding the UPS to shut down, the host will typically put itself into some state that only a power cycle can get it out of (e.g. it goes into S5 state), and further communication between host and UPS will not be available. So that 20 seconds of off time is a minimum to be observed in all circumstances, even when AC power is available (sometimes this protocol is used as a “host reboot” feature, since it also achieves a reboot when AC power is on).
If AC power is still unavailable at the end of the 20 second delay, the host should not be turned on again until AC power (and optionally UPS charge) is restored. The host->UPS shutdown message is the host’s way of saying “I’m out of here, let’s try again when power is back to normal.”
I think you are expecting the host will change its load current for this signal. I’ve seen that idea fail enough times by now to stop bothering to try. Current sensors will either turn off an energy-efficient host too early, or fail to turn an earthburning host off at all. The explicit USB signalling is much more reliable.
I’ve been trying to read the UPS HID specs, and it sounds like what you’ve described here is writing “20” to the Usage ID 55 (“DelayBeforeReboot”) of Power Device Page (x84). Since ESR has already said we should fully support UPS HID profile, we’ll obviously respect these commands if given. The question is what should UPSide do when those commands are not given, because the USB cable is disconnected, or the
usbhid-upsdriver forupsd(or equivalent Windows device driver) isn’t installed/configured correctly. If UPSide hasn’t had anything talk to it via UPS HID within a reasonable amount of time, then it has to assume it’s on its own, and Do The Right Thing™ anyway.I was going to ask the same thing andyjpb did, but with that answered I’m wondering how this is an improvement over what any other ups does.
>I was going to ask the same thing andyjpb did, but with that answered I’m wondering how this is an improvement over what any other ups does.
“Any other UPS” is all too likely to crash your machine ungently because the absence of a BMS cripples its ability to project end of dwell time.
UPS, not GPS.
Don’t cross the projects. https://www.youtube.com/watch?v=jyaLZHiJJnE
*snrk* Corrected.
That collision is a hazard of my project history.
So if the machine goes from battery power to mains restored without first hitting the dwell-limit warning, it looks like it can only reach ‘Mains power’ again via ‘Charge wait’.
Does this mean the outlets necessarily go down with any power interruption? Or is the “outlets disabled” state only accessible from cold start?
>Does this mean the outlets necessarily go down with any power interruption? Or is the “outlets disabled” state only accessible from cold start?
The latter. The only time the logic shuts off the master socket power is during an explicit power cycle due to late restore.
So, your state machine will shutdown a host when the AC power dies and the battery has drained to a certain level. Then, if the AC power comes back, it will enter a charging state and only power up the host once the battery has sufficient charge to perform a shutdown in the event of a subsequent AC power failure.
Am I reading that right?
>Am I reading that right?
Yes.
Where is the part that detects a degraded battery during steady state AC operation? So you can be warned that your battery sucks before the power outage?
>Where is the part that detects a degraded battery during steady state AC operation? So you can be warned that your battery sucks before the power outage?
That’s why you need a smart BMS that can do what’s called “state of battery” (SOB) monitoring. This is distinct from “state of charge” (SOC) monitoring. SOB tells you what percent of the battery’s nominal capacity is still available at peak charge. SOC tells you how much dwell time is left.
The control software already accepts a simulation of SOB reports from an SBD-1.1-compliant BMS and throws them on the status display. Interpreting actual SOB results will happen when I have hardware to test on and can write the I2C driver to query the fuel gauge.
So this state machine doesn’t reflect the handling of SOB messages?
I’d have thought there would be another edge leaving “On mains power”, raising the alarm (of appropriate severity), and then coming back.
>So this state machine doesn’t reflect the handling of SOB messages?
It doesn’t have to. Pretty much by definition, if the SOB is down to a low enough percentage that you need to worry, you’re going to trigger a SOC (dwell-time) alarm first.
What if, while you’re on mains power, battery degradation causes the SOB to fall to such that your maximum dwell time becomes less than ShutdownTime?
See, this is why you need Alloy. It’ll catch stuff like this.
From the diagram, that only seems to happen whilst the system is in “On battery power”. The steady state of the system, and where it spends most of its time is “On mains power”. The state of the battery will change whilst the system languishes in “On mains power” and will only notice the SOB/SOC situation when a power outage causes it to transition into “On battery power”.
If the battery’s not too damaged you might have enough time to work through the states and allow an immediate but graceful shutdown but that’s not guaranteed.
How does your current simulation respond to the state of the battery changing behind its back in the different states?
>So this state machine doesn’t reflect the handling of SOB messages? I’d have thought there would be another edge leaving “On mains power”, raising the alarm (of appropriate severity), and then coming back.
I didn’t do it that way because I didn’t believe SOB can crash without raising a SOC warning first.
The way this works is that the SBC polls the BMS many times a second – yes, while it’s on mains power too. One of the responses it gets is the BMS’s time-to-empty estimate (if the battery is fully charged that will just peg at its current maximum dwell time). I didn’t imagine that the firmware would be dumb enough not update that based on both the SOC *and* SOB estimates from its state modeling. If it doesn’t do that, we’re probably screwed bo matter how we try to evade it.
You can estimate SOC without knowing the battery’s history if you know the shape of the discharge curve, but you can’t get SOB that way – you have to track the rate at which the discharge time required to reach a threshold output drops over discharge cycles. To be on the safe side, occasionally do a calibration drain.
See @andyjb’s comment; this diagram seems to only process a dwell time alarm if you’re on battery power.
> this diagram seems to only process a dwell time alarm if you’re on battery power.
I’ve added an arc to ensure that the SOC trigger is checked at each poll interval while on mains power.
Payoff from getting the software engineering right: the change was one line of pseudocode that I could verify instantly without even reading Go.
Trigger: Poll of host load sensor shows no current draw
Means: Host has shut down
Preferable?
Trigger: Poll of host load sensor shows current draw less than user defined host quiescent current draw + % threshold (say 5 or 10%).
Means: Host has shut down
Because many hosts will continue to draw power to keep ALOM/WOL and other functions alive. “no current draw” would be an edge case for most hosts these days.
>Because many hosts will continue to draw power to keep ALOM/WOL and other functions alive. “no current draw” would be an edge case for most hosts these days.
Good point. I shall so amend.
Sooo, is “fail to bypass” still in the works for a faulty battery/charge circuit? I ask because I just had yet another APC drop a circuit because the charge circuit went south. That’s the third one in about 6 years, and I don’t have _that_ many of these things.
>Sooo, is “fail to bypass” still in the works for a faulty battery/charge circuit?
Alas, no. After extended discussion among the brainstorming group we concluded that we could either support fail-to-bypass of soft-land your host system in the maximum range of cases, but those two goals can’t be reconciled.
The way to get closest to fail-to-bypass that’s compatible with safe landing is, I think, this: if the BMS stops returning data we treat this as battery death and ship shutdown to the host. We could do the same thing if the charge level is not currently both not at SOC maximum and is failing to increment over some configurable period.
Hmmm…. So y’all aren’t doing dual conversion, then? The big stuff I’m used to in Datacenters are something like this:
AC Input—–+–2–>Rectifier——+———————————+——>Inverter—-2—–>Load
| +-3-Charge/Discharge Controller–+ |
| +–4– Batteries —–+ |
+—————————————————————————+
1: DP/DT Fast Breaker w/ Trip circuit
2,3,4: Regular Breakers
Well crap that didn’t come out very well. :)
I’m sure y’all figured that out and looked at that already. Obviously, the quick-through tripped breaker is a no-go, since it would cost more than the rest of the UPS combined, but I did hope there was a way to do that with more modern technologies than manual switches and springs. :)
Either way, thanks for trying. I knew it was a big ask when I put it on the wish list ;).
>Hmmm…. So y’all aren’t doing dual conversion, then?
Actually we are. Go look at the design document, then we can continue this.
OK, I think I see. You’re electing to be able to sense power draw on the ports and shutdown only after the last component is powered off.
You could inject between the inverter (“DC-to-AC Converter 9”). You’d want to make sure that your inverter remained synced to mains, but you should be doing that anyway for safety. Your bypass relay/breaker goes between the inverter and the loads.
On the big gear, this isn’t a problem, since you come out of those UPSes and go into a step-down transformer and service panel (all the ones I worked with were 480V, stepped down to 408/3phase).
What’s the reasoning for that configuration not working?
>What’s the reasoning for that configuration not working?
I no longer remember, but you should be able to find that discussion by reading the open issue threads on the product tracker.
OK, read through the thread and responded. Actually, it looks more like y’all just gave up because you didn’t see the benefit. I’ve addressed that. Unfortunately, I have very little time these days to keep up, but I’ll try to follow along on that one and reply as I’m able.
Why does “Cycle AC power to load host” only depend on “Poll of mains reports good AC voltage”? Shouldn’t it require that _plus_ “TimeToEmpty > (ShutdownTime + WarningTime)”? Because otherwise, the UPS might not actually be ready to power the host. What am I missing here?
This might be a good application of Lamport’s TLA+. It seems like there are multiple asynchronous processes going on at once: battery state, mains state, (load?) and this controller. I’m just getting started with TLA+ but given what amounts to executable pseudocode spec of your system, it appears to exhaustively examine all combinations of states for safety (assure what I do and don’t want to happen) and liveness (what I want will eventually occur). Here’s some links to get stated.
https://learntla.com/introduction
https://lamport.azurewebsites.net/tla/tla.html
Indeed, I’ve identified 4 different battery capacity buckets (which I call Red, Orange, Yellow, and Green in the state discussion on Gitlab) 2 mains states, three load states (Disabled, Enabled [but only “watchkeeping” level to handle processing the “power” button, BoE, etc.], and Active, the latter two of which unfortunately can be difficult to distinguish from one another due to power-conservation measures taken by many machines) and potentially a couple of flags to track whether an SOS or Shutdown had been initiated from UPSide’s end. The ROY battery levels should be treated as one when the load is Disabled or Enabled, simplifying matters a bit.
Surprisingly, the mains state turns out to be largely orthogonal to the other state components; it only matters for what we display. Eventually, a change in mains state would be reflected in the battery level state, and that’s the transition that can drive an action (such as enabling the load socket when Green battery level is reached, issuing an SOS, or a Shutdown, disabling the load socket in ROY battery levels when the load drops below the defined threshhold).
But what we really care about is the transitions between states. Those are the things that should trigger actions. So a basic loop that collects the statuses and updates the display on a transition of Mains, and fires off the appropriate action when moving between the other states, should suffice.
>But what we really care about is the transitions between states.
Take a look at the way the diagram in the design document is generated in docs/flow.py. Your insights would be best expressed by a patch to that pseudocode, together with a rationale for the patch. Then the system description and the policy code would be guaranteed to change in sync.
I’m not even close to being ready to write a patch yet, I’m still struggling with whether we’ve defined our states coherently.
The basic idea that we poll the sensors to collect the component statuses that together define a “state” I think is sound, though.
@esr,
This is off topic, but I just saw something that reminded me of a comment of yours from several weeks ago in which you compared “warrior mind” to “programmer mind.” The comment was this one: http://esr.ibiblio.org/?p=7925#comment-1939825
I watched a video interview with Jonathan Blow, a man with a few decades of experience as a coder. He mentions that on a handful of occasions he’s blanked out for 20 minutes or so while working and came to with a program in front of him that he didn’t remember writing. It reminded me of your comment because you said that one can’t program by emptying his mind the way you do when fighting. It seems he’s internalized his craft to the point where he sometimes can (or at least is doing something similar). He doesn’t mention martial arts but he does contrast it with improv dancing.
At the end of your comment you said the subject might be worth a full blog post so I figured this would be interesting to you. The video is 22 minutes and change and you can find it here: https://www.youtube.com/watch?v=4Ej_3NKA3pk
>He mentions that on a handful of occasions he’s blanked out for 20 minutes or so while working and came to with a program in front of him that he didn’t remember writing.
Is this no-mind or a transient fugue state? I don’t think they’re equivalent. Combat no-mind isn’t a fugue or absence of awareness, it’s hyperacute if anything.
I couldn’t really say. I’ve never experienced anything like that, at least not with something as complex as coding. You two are both way beyond my league, though. Has that ever happened to you?
>Has that ever happened to you?
Combat no-mind? Yes.
Fugue while programming? No. And this is first report I’ve ever seen of it.
Having never experienced such fugue myself, I am reluctant to draw conclusions about it. I norte, however, that if I ever went no-mind while programming I would notice it because I do know what that is like..
I’d love to lock the two of you in a room together and have you compare notes; you both share an interest in how one’s own mind works.
But failing that, I do look forward to that full blog post on the subject if you ever decide to write it.
Just happened to think: Say we have a rack full of these built out in a DataCenter (or several per rack). Is there a state whereby we can implement a kill/battery disconnect in an emergency setting? Something that can connect to a master kill switch in the DC in case of fire?
> Is there a state whereby we can implement a kill/battery disconnect in an emergency setting?
Not in the plans. You can submit an issue about it if you like.
@esr Is there an update to the project, it’s been a while? Thanks
Yes, I posted about this. Project is on hiatus while our hardware lead finishes building a farm.
Are you sure? Where? I’ve been watching for status updates…
>Are you sure? Where? I’ve been watching for status updates…
Here. You’re not the first commenter to ask about it.
Help me out here, I can’t find any reference to building a farm in this thread, or any other thread. (Maybe I don’t know how to search).
In the comment where the h/w engineer says they are building a farm, was there an ETA on being complete? I have friends that farm in Iowa, they say the only time a farm is done is when the farmer is doing their last chore inside a pine box.
Thanks!
>Help me out here, I can’t find any reference to building a farm in this thread
No, you can’t. That’s what an unhappy Eric Baskin told me by phone when offering his resignation. I asked him if he really wanted to do the project, he said “Well yes, I do.” in a way that left me in no doubt, and I said “OK, we’re not on a deadline, we’ll resume when you can.” For which he was then obviously grateful.
If I had a more skilled power engineer handy I might have chosen differently. There aren’t many more skilled than he is.
I ping him every month or so.