In the face of uncertainty, buy options.

Yesterday I posted about how the streetlight effect pulls us towards bad choices in systems engineering. Today I’m going to discuss a different angle on the same class of challenges, one which focuses less on cognitive bias and more on game theory and risk management.

In the face of uncertainty, buy options. This is a good rule whether you’re doing whole-system design, playing boardgames, or deciding whether and when to carry a gun.

A useful way to sort the decision challenges we face is into situations of high uncertainty versus low uncertainty. These call for vary different adaptations. In a situation of low uncertainty there is a single optimal choice; your effort should go into determining what it is and then executing it as hard and fast as possible. Unless uncertainty rises during execution (for example because you discover you made a serious mistake in your problem analysis), deviation from plan is most likely to be a mistake. Buying options is wasteful.

In a situation of high uncertainty you don’t know what your best choice is up front; there’s a broad range of possible ones that might be optimal, and there may be choices you can’t yet see. In this situation, what you need to do is enable yourself to collect on as many of the options as you can identify and afford to buy. Your hope is to be able to narrow the range of conditions you need to cope with as you learn more.

This dichotomy is so fundamental that it has moral consequences. If, in any role other than military in a war zone, you are ever carrying a gun because you have a have a high-certainty expectation that you will use it against humans, your life choices have probably gone very badly wrong. On the other hand, carrying a gun as a hedge against uncertainty – for example, if you need to visit someone in a dicey enough part of town that you might have to defend yourself – makes both practical and moral sense.

In yesterday’s example, I described using a Unix SBC rather than an Arduino-class microcontroller as a way to counter-bias against our tendency to underestimate and underweight software-development costs we can’t estimate crisply. It is that; it is also a way to buy options as a hedge against uncertainty about what how the whole system should behave. Often we don’t actually know this until field testing. When we get it wrong, the pain of correcting is in direct proportion to the cost of changing the whole system’s behavior; it can be lowered if the controller is chosen for flexibility and low development costs.

This is how “overkill” can save your butt. Suppose you’re right that an Arduino-class chip is sufficient for every scenario you imagine in your planning phase; it may still be the case that you’ll be mugged by a field reality you didn’t anticipate. If you don’t have good judgement about how to hedge against this possibility, your designs will have more than your share of expensive failures.

The hardest part of this lesson to learn is that an early choice to buy hedges against uncertainty does not retrospectively turn into a mistake if you get lucky and everything goes as originally planned. You have to sum over all possible worlds; whether the choice to carry a gun or deploy a more flexible controller was wise depends only on the accuracy of your ex ante risk evaluation, not on whether you actually got attacked or the firmware Arduino-class controller is actually good enough on the first spin.

(Yes, I know. Firearms ethics and systems engineering in the same blog post. If you don’t find this amusing, go read something stupid.)

42 comments

  1. Of course, anything can be taken to YAGNI extremes. As you say: “collect on as many of the options as you can identify and afford to buy”.
    My father is fond of saying “Flexibility is the key to indecision.”

  2. > an early choice to buy hedges against uncertainty does not retrospectively turn into a mistake if you get lucky and everything goes as originally planned.

    Y2K springs to mind with all its silly, after the fact critics.

    1. At the time I was at a place where they were cleaning up a lot of Y2K mistakes in the central database. No one heard about it as these had all been corrected in time.

    1. Just remember: computing with a Taurus model 44 can get really expensive, even if you stick to .44 Special.

  3. This is chapter….oh I don’t know, two of the Autodidact’s Toolkit; it might warrant a lengthy discussion in the introduction.

    I might chat with Hanson about that project on Monday BTW, if that’s not a problem.

  4. Case study: Linksys WRT54GL – Early versions were more flexible, with lots of extra RAM and flash, which coincidentally made it a useful target for projects like OpenWRT. Later versions – once the problem and solution were well established – trimmed back on those expensive bits – coincidentally(?) making it impossible (or unuseful) as an OpenWRT target.

    1. >Later versions – once the problem and solution were well established – trimmed back on those expensive bits – coincidentally(?) making it impossible (or unuseful) as an OpenWRT target.

      I was affected by that crapstorm. I had a couple of those and maintained the FAQ on them. It’s still visible here:

      http://en.tldp.org/HOWTO/Linksys-Blue-Box-Router-HOWTO/

      The GL kept the extra flash and hackability; the G didn’t. They stopped making the GL.

    2. It’s the WRT54G that was trimmed back with the v.3-to-v.4 (32/8 to 16/4) and v.4-to-v.5 (16/4 to 16/2) version changes.

      The WRT54GL was a new SKU for the re-release of the v.4 hardware, and was never trimmed, precisely because its whole purpose was to support people who wanted to keep using it for custom firmware. Both GL versions, 1.0 and 1.1 had the same 16/4 config as the G v.4

      The interesting bit is that the WRT54GL is still a live product. The GL is on the Linksys website, available new at Amazon.com/Target.com/Walmart.com, et cetera.

      1. >The interesting bit is that the WRT54GL is still a live product

        How interesting! They must have revived it after I lost interest in the product line.

        UPDATE: Now that I think back, I may remember hearing this had occurred. Not sure; it was over 10 years ago.

  5. Most (interesting) things are not a clear cut choice between “explore” or “exploit”, but rather a mix.

    Tweaking the exact ratio, perhaps doing so dynamically as part of a self-correcting system, is one part engineering and one part art.

  6. Most linux capable single board computers are designed to support a keyboard and a video display. That is overkill

    You want a linux capable single board computer that is designed to support VNC over a usb port.

    1. Penny wise, pound foolish – and depending on the economy of scale of production implied by “Most”, might not even be penny wise.

    2. >You want a linux capable single board computer that is designed to support VNC over a usb port.

      Really, I’d settle for character terminal emulation.

      1. That’s probably quite a bit simpler to do. I can’t see how you’d get VNC – a fundamentally network thing – over USB. A serial terminal connection’s a lot simpler to do, and for your purposes should be more than adequate.

        Then again, if you do WiFi and DHCP, then the only thing you really need a serial connection for is to debug network failures. I don’t know if you’re planning to go that route or not. Might be a bit more than you need.

        1. Network over USB thing is actually not difficult and quite standard for modern OSes – just implement a CDC USB class with a virtual network segment with your VNC host being the only other endpoint.

          IP auto-configuration in non-conflicting way with other network adapters might be more tricky, but then for IPv6 link-local address might suffice. (I am not sure about that)

          Of course one can then channel SSH and any other network protocol over such a connection – I guess modern users are more accustomed to an SSH client such PuTTY than to a terminal emulator over a serial USB port.

              1. >I would just enable SSHD with strong defaults.

                Yup. Never used VNC myself, but know ssh/sshd like the back of my hand.

                1. Yup. Never used VNC myself, but know ssh/sshd like the back of my hand.

                  Print the UPS key fingerprint on the display when sshd receives an incoming connection.

      2. What UI is best for cases when people are seriously stressed out? Hint: learn from the military. For the really important stuff you want to do in the middle of the night during a blackout I would recommend glowing physical buttons and big letters on them, something you would see in a helicopter.

        For doing a calm setup at a newly bought device character terminal emulation, not like a shell but like a typical BIOS setup with text menus used with the arrow keys and Enter or the numbe would sound OK. If you want to or in some cases need to comply with suitable for blind or otherwise disabled people rules, that works pretty well. Pressing 3 to access menu item 3 is probably older than I am, but can’t think of a better thing for blind people for example.

  7. If, in any role other than military in a war zone, you are ever carrying a gun because you have a have a high-certainty expectation that you will use it against humans, your life choices have probably gone very badly wrong.

    That, or someone else’s bad choices are about to be visited upon them.

  8. I think part of the instinctual objection to a over-powered controller arises from it’s crackability. A RaspPi is smart enough to mine bitcoins, or spread viruses. That means a RaspPi hidden in a device that no one would expect to be smart is potential target. A lot of IoT devices are hideously vulnerable, and not upgradable. That can make them a stepping stone to other devices on the network. Who expects a TV to be an attack vector?

    In this way, the options you buy for yourself can become a hazard to everyone around you, who did not agree to the risk, and don’t know to expect it.

    This is also applicable carrying handguns.

    1. Yeah but a Pi is upgradeable and can be set to automatically get and install updates. If that process goes wrong then in the worst case you can recover by replacing the SD card with a new improved one. Your average industrial cut down linux OS isn’t anywhere near as easy to upgrade and you can brick it fairly easily if the upgrade process goes wrong. Since the flash is almost certainly soldered onto the board unbricking it probably requires unnatural acts with JTAG and a bookloader,

      And I know this because I just had to recover both a Linksys Router that I wanted to put openwrt on and a RaspPi with a duff flash card. The latter was a whole lot easier than the former

      1. > automatically get and install updates.

        For now.

        I was visiting my Dad for Christmas. It was the first time I’d been back in 5+ years. I have photos taken of the updates from Windows XP. In addition to taking hours to get the existing updates, no current updates are available.

        Automatic updates are great until end-of-support. At which point you are in the same position, really. Sure, a customer might be able to upgrade it themselves, but then it’s not a consumer product any more.

    2. Security of the device is relatively simple and boils down to not letting (at the iptables level) it talk to random places on the internet. In fact except under certain controlled conditions not letting it talk to anything outside its home subnet.

      1. This is unfortunately is not nearly enough.

        Modern JavaScript running in a browser with no additional privileges can do plenty of port scanning on the local network. There are some protections to block access to certain well-known ports, but this is security by exclusion.

        In general one, unfortunately, has to assume that your subnet can be accessed if you visit the wrong site (or a compromised one)

        1. I would prefer a browser without Javascript for secure browsing (e.g., dillo). Sadly, that has become much harder nowadays with random Javascript on every website.

  9. The first full-scale reactor, then known as an atomic pile, was built at the Hanford B site to produce plutonium for the Manhattan Project. When it first went into operation, it ran for three hours, began losing power, and stopped entirely after about a day. After a while, it started up again, by itself, and repeated the performance.

    What had happened was that unbeknownst to the scientists, fission produces as one of its by-products a certain amount of Xenon-135, which is an extremely effective neutron absorber. Enough 135Xe had built up to take out most of the neutrons propagating the chain reaction.

    But Xenon-135 is itself vigorously radioactive, with a half-life of about 9 hours. So if you leave it sitting around for a day or so, most of it is gone, and your reaction can proceed again. Luckily for the Manhattan scientists, the pile had been built twice as big as their calculations called for, so they could essentially use brute force to overcome the Xenon poisoning.

  10. This is how “overkill” can save your butt. Suppose you’re right that an Arduino-class chip is sufficient for every scenario you imagine in your planning phase; it may still be the case that you’ll be mugged by a field reality you didn’t anticipate. If you don’t have good judgement about how to hedge against this possibility, your designs will have more than your share of expensive failures.

    Although this is true, you’re thinking like an engineer rather than as an engineer *and* a product manager. That “field reality you didn’t anticipate” might be that you raised the BoM cost of your product over a threshold that prevented an entire class of potential customers from buying it. Engineering discipline is important, but the products that succeed the most in the marketplace are the ones where engineers and product managers work together instead of as adversaries.

    (To be certain: for *this* product, I agree that a Linux SBC makes more sense than a microcontroller. We’re speaking in generalizations here.)

    1. >Although this is true, you’re thinking like an engineer rather than as an engineer *and* a product manager. That “field reality you didn’t anticipate” might be that you raised the BoM cost of your product over a threshold that prevented an entire class of potential customers from buying it.

      Um, what made you think I wasn’t sensitive to that tradeoff? It’s a pretty obvious one.

      In the particular case of UPSide it’s true I’m not worried about it much. We’re not designing a mass-market project yet; this is more like a technology demonstrator that can be built in small runs at a price level that is not bargain-basement but pretty good value for dollar.

    2. >That “field reality you didn’t anticipate” might be that you raised the BoM cost of your product over a threshold that prevented an entire class of potential customers from buying it.

      But I think that is a good problem to have. You just create a budget version of your product, a dumbed-down Lite. From a marketing viewpoint that is an excellent situation to be in, your brand is considered upmarket and good quality, prestigious, the Lite version inherits it, and people who think it is crap will not even blame you, after all you told them it is less good, they kick themselves for not buying the real thing.

      If you have a cheap crap brand, it is hell of a difficult to sell a premium version. It inherits the bad rep of the cheap version. Huawei can make good phones and tablets but just because the name is Chinese people think it is not as good.

      In the longer run, the cheap version will likely be far more profitable. But still it is far better to enter a market positioned as a premium brand and then make your Lite version.

      People think Oculus Go (i.e. Lite) gonna be good just because it is Oculus. In reality it will have the same platform thus the same content as Samsung Gear, i.e. almost none, while people who own the real Oculus can play awesome stuff like Project Cars 2. But Lite-ing a premium brand sells any cheap crap because the expectations are high.

      Oh and a premium product also tickles our social status module. We want to think ourselves discerning, advanced customers who are in the know. And even buying the Lite version can make us think we are that but we are also frugal, so, like, doubly smart.

      1. Or, if you’re Apple or want to be the Apple of X, you can make the Lite version a revolutionary product in its own right. See: iMac, MacBook Air.

  11. The science of estimating is an entire field onto itself, and it serves an important and supportive role in both systems engineering and then ultimately the project management function. The environment also matters, in that mass produced products (like UPSide) are characteristically different from singular mega-projects (like building the Brooklyn Bridge). As you correctly point out, managing uncertainty can be a difficult task and the application of experience and intuition is one of the keys to doing it well. The nature of uncertainty is such all decisions involve some level of risk, and a real danger is what’s known as analysis paralysis (e.g. excessive resistance to making any decision in the face of uncertain uncertainty). If it helps any, one of the lessons of evolutionary history is that robustness works.

  12. Different but related is Rory Miller’s concept of fighting when you are weaker than your opponent. The stronger guy just wants to maintain the dynamic and either wear the other guy down or wait for him to give an opening. But the weaker guy wants to stir things up and shake up the dynamic, in the hope that he can get an opening, either to strike or to escape.

  13. There is always the matter of cost of options. And the cost is not just always monetary.

    Arduino-class as I understand is not really easier to program than Unix, so non-monetary cost does not apply. But I’ll take a different example, I want to go out to town and I can use the bus or drive. The driving gives me many more options, I am not ties to the schedule of the bus. But i I drive, I have to care about parking and, most significantly, I can not drink while out – which detracts from the value of the evening.

    A gun is at least as dangerous as a car, so carrying while drunk is unethical, whatever the law. Moreover, you need to be on heightened alert while carrying, otherwise you’re a sitting duck for “brick from behind to get the gun”. Is this always an acceptable price for the option?

    I have seen a teacher who owns guns and has a concealed carry permit argue against the “arm teachers” option with exactly this reasoning – that when in school, he wants to concentrate on education and not be in carry-alert mode.

Leave a comment

Your email address will not be published. Required fields are marked *