The British have a phrase “Too clever by half”, It needs to go global, especially among hackers. It can have any of several closely related meanings: the one I mean to focus on here has to do with overconfidence in one’s intelligence or skill, and the particular bad consequences that can have. It’s related to Nassim Taleb’s concept of a “fragilista”.
This came up recently when I posted about building a new mailserver out of a packaged fanless mini-ITX system. My stated goal was to reduce my mailserver’s power dissipation in order to (eventually) collect a net savings on my utility bill.
Certain of my commenters immediately crapped all over this idea, describing it as overkill and insisting that I ought to be using something with even lower power draw; the popular suggestion was a Raspberry Pi. It was when I objected to the absence of a battery-backed-up RTC (real-time clock) on the Pi that the real fun started.
The pro-Pi people airily dismissed this objection. One observed that you can get an RTC hat for the Pi. Some others waxed sarcastic about the maintainer of GPSD and the NTPsec tech lead not trusting his own software; a GPS to supply time, or NTP to take time corrections over the net, should (they claim) be a perfectly adequate substitute for an RTC.
And so they would be…under optimal conditions, with everything working perfectly, and a software bridge that hasn’t been written yet. Best case would be that your GPS hat has a solid satellite lock when you boot and sets the system clock within the first second. Only, oops, GPSD as it is doesn’t actually have the capability to set the system clock directly. It has to go through ntpd or chrony.
So now you have to have a time service daemon installed, properly configured, and running for the timestamps on your system logs to look sane. Well, unless your GPS doesn’t have sat lock. Or you’re booting without a network connection for diagnostic or fault isolation reasons. Now your cleverness has gotten you nowhere; your machine could believe it’s near 0 in the Unix epoch (Midnight, January 1st 1970) for an arbitrary amount of time.
Why is this a problem? One very mundane reason is that logfile analyzers don’t necessarily deal well with large jumps in the system clock, like the one that will happen when the system time finally gets set; if you have to troubleshoot boot-time behavior later. Another is cron jobs firing inappropriately. Yet another is that the implementations for various network protocols can get confused by large time skew, even if they’re formally supposed to be able to handle it.
And I left out the fact that outright setting the system clock isn’t normal behavior for an NTP daemon, either. What it’s actually designed to do is collect small amounts of drift by speeding up or slowing down the clock until system time matches NTP time. And why is it designed to do this? If you guessed “because too many applications get upset by jumping time” you get a prize.
You can certainly tell an NTP daemon to set time rather than skewing the clock rate. But you do have to tell it to do that. This is a configuration knob that can be gotten wrong.
Are we perhaps beginning to see the problem here?
Engineering is tradeoffs. When you optimize for one figure of merit (like low cost) you are likely to end up pessimizing another, like proliferating possible failure modes. This is especially likely if an economy measure like leaving out an RTC requires interlocking compensations like having a GPS hat and configuring your time-service daemon exactly right.
The “too clever by half” mindset often wants to optimize demonstrating its own cleverness. This, of course, is something hackers are particularly prone to. It can be a virtue of sorts when you’re doing exploratory design, but not when you’re engineering a production system. I’m not the first person to point out that if you write code that’s as clever as you can manage, it’s probably too tricky for you to debug.
A particularly dangerous form of too clever by half is when you assume that you are smart enough for your design to head off all failure modes. This is the mindset Nassim Taleb calls “fragilista” – the overconfident planner who proliferates complexity and failure modes and gets blindsided when his fragile construct collides with messy reality.
Now I need to introduce the concept of an incident pit. This is a term invented by scuba divers. It describes a cascade that depends with a small thing going wrong. You try to fix the small thing, but the fix has an unexpected effect that lands you in more trouble. You try to fix that thing, don’t get it quite right, and are in bigger trouble. Now you’re under stress and maybe not thinking clearly. The next mistake is larger… A few iterations of this can kill a diver.
The term “incident pit” has been adopted by paramedics and others who have to make life-critical decisions. A classic XKCD cartoon, “Success”, captures how this applies to hardware and software engineering:
Too clever by half lands you in incident pits.
How do you avoid these? By designing to avoid failure modes. This why “KISS” – Keep It Simple, Stupid” is an engineering maxim. Buy the RTC to foreclose the failure modes of not having one. Choose a small-form-factor system your buddy Phil the expert hardware troubleshooter is already using rather than novel hardware neither of you knows the ins and outs of.
Don’t get cute. Well, not unless your actual objective is to get cute – if I didn’t know that playfulness and deliberately pushing the envelope has its place I’d be a piss-poor hacker. But if you’re trying to bring up a production mailserver, or a production anything, cute is not the goal and you shouldn’t let your ego suck you into trying for the cleverest possible maneuver. That way lie XKCD’s sharks.
Great term. Many of the issues facing the country today are incident pits. A century ago the government started regulating medicine. Numerous iterations later, we have the headed-for-bankruptcy monstrosities of Medicare, Medicaid, and Obamacare, and we still haven’t fixed the problems they were supposed to solve. Or higher education: Hey, let’s help kids go to college! Then college tuition goes up, loans go up, the government takes over the loan program, and now vast numbers of people are up to their eyeballs in debt they can’t repay. Or home mortgages. Or the drug war. Etc….
Yeah, a real-time clock is very important, as long as it FREEZES when you pull the battery to replace it (put on a switch to freeze the RTC state in EEPROM and read it back). This is a lesson that Game Freak should have learned with Pokémon R/S/E.
Unrelatedly, I keep reading RTC as RTG, which keeps confusing me.
Civilization as a whole is in an incident pit. Got it.
(See “Genghis John”)
The Odroid series generally have RTC including the (no longer produced but available) Raspi clone the Odroid W – separate batteries for power (charge controller) and RTC backup. I use it with a FLiR and raspi camera for my drone because it is very compact.
Adafruit and Sparkfun have I2C 3232 https://www.maximintegrated.com/en/products/digital/real-time-clocks/DS3232.html with tcxo so it is 2ppm (2 seconds per year IIRC) and can be trimmed to tweak the accuracy and have notes on how to add it to the Raspberry Pi
Another possibility might be a cellular modem – even in E911 only (unactivated) it should provide the current time.
IMO the RasPi has other problems. I considered using mine for a mailserver as well, but at least the first generation had USB stability problems which would make it completely unacceptable unless you relied entirely on SD storage.
@tz’s last comment actually leads me to another think: Got a spare rootable cellphone around? If it’s less than around 3 years old, it may make a fine server: wifi, USB (via OTG), onboard flash, maybe microSD, low power consumption, battery backup, realtime clock… throw on a cyanogenmod kernel and whatever userspace you want and there you go.
An all-too-true example:
About 16 months ago, when I did the initial software load on the Great Beast of Malvern, I had written a multi-hundred line long shell script of gazillions of “apt-get install …” lines. Since I thought it wouldn’t throw any errors, I put -xe on the hash-bang invocation line.
Something else broke.
And instead of just removing those flags, I kept trying to hack the body of the script to ‘fix’ the problems. Lost 5 hours that way, and Eric almost didn’t get his system installed.
“Too Clever By Half”, indeed!
My fancy new mail server doesn’t need a clock, because it has a GPS hat and an NTP channel!
**strike that, reverse it:
My fancy new mail server doesn’t need a GPS hat or an NTP channel, because it has a built-in real-time clock!
Yes, that’s better.
To be fair, if you just freeze the clock when you’re off, that fixes many of the issues with not having an RTC battery. (Put in the superblock a note stating the time, then every second or so change it, and when booting, read that first thing.)
While I agree strongly with what you wrote above, it would still be my hope you’d have some fun playing with a raspberry pi. My “killer app” for it has been using it as a kodi server and home lighting controller (using some hacked up versions of these: https://mjg59.dreamwidth.org/40397.html ). Another app for it (for me) is merely having a test tool around for alternate architectures. And: 64 bit arm code, in particular, is in need of polish and porting to make full advantage of the new features in the architecture. I keep meaning to have the time to try to hook up some blinkenlights, weather sensors, or other forms of relays to mine, and simply haven’t got around to it.
Secondly, a lower power mail (or any home) server with an integral RTC, and ups, would be a nice piece of gear to have. Repurposing an android tablet for this does not quite cut it, and the current generation of hackerboards and cases are not quite a match, either. Something that could withstand a few minutes of power outage, work up to 70C, have an RTC, and that integral UPS, would be an ideal box to have in parts of the world that have poor electric infrastructure.
Lack of an RTC alone wouldn’t even doom the Pi as a solution, since you can buy an RTC hat for it. But the faster I/O bandwidth, faster speed in general, and the fact that hardware and software integration on a mini PC platform are a known quantity supported by every major distro still make it a more attractive option.
“Too clever by half” is a succinct summary of why I have major distrust for things like systemd.
>Secondly, a lower power mail (or any home) server with an integral RTC, and ups, would be a nice piece of gear to have. Repurposing an android tablet for this does not quite cut it, and the current generation of hackerboards and cases are not quite a match, either. Something that could withstand a few minutes of power outage, work up to 70C, have an RTC, and that integral UPS, would be an ideal box to have in parts of the world that have poor electric infrastructure.
Like Cambridge, MA. Sorry bad joke. I once lived about a block away from Draper Labs, a short walk from the east side of the MIT campus, and our power was 3rd world quality. Apparently very old infrastructure.
Repurpose an obsolete laptop, if a cheap smartphone isn’t good enough. What especially would you be inclined to use it for?
>Something that could withstand a few minutes of power outage
I doubt it takes more than a little skill with a soldering iron and some random 5v battery to run a pi for a few hours. (Current draw of pi <2A, 7Ah lead-acid batteries can be found for $30)
>My fancy new mail server doesn’t need a clock, because it has a GPS hat and an NTP channel!
**strike that, reverse it:
My fancy new mail server doesn’t need a GPS hat or an NTP channel, because it has a built-in real-time clock!
Yes, that’s better.
Which has more and more complex dependencies? And which has more and more frequent failure modes?
System reliability isn’t a social construct. ;)
P.S. Yes, that solution is probably too clever by half.
Speaking as an engineer, the pragmatic form of the maxim is more precisely stated as “the simplest thing that works reliably is always best.” And the first corollary, “only add complexity under extreme duress.”
Unfortunately, these principles are not taught in engineering schools and must be learned in the school of hard knocks. As such, in an ideal world, every engineer would apprentice in a blue collar occupation of related practice before being allowed anywhere near a design project. Sadly, that is not often the case, and the end result can be both expensive and deadly.
Yup, seen this sort of problem before.
(In my case, the use case that requires time is cron for an alarm clock, and the time setting problem was a bug in the init scripts that made each reboot cause a drift equal to the timezone offset…)
First connections usually take a long time and are not reliable, so any sort of “get something from somewhere else”-based scheme for startup is a bad idea. (Yes, this applies to cell phone modems.)
Additionally, having used USB stuff for about 10 years, I’ll note that it tends to be…less than completely reliable, even comparing flash drives to spinning rust that’s accessed much more frequently. Network ports connected via a single USB controller really doesn’t sound like a good idea.
And SD cards are slow.
EMF’s proposal theretically sounds reasonable for an “always-on” computer like a mailserver would be, unless you have memory with a limited number of read-write cycles or power-based limits on how often you can write.
But SD cards have a limited number of read-write cycles.
>I doubt it takes more than a little skill with a soldering iron and some random 5v battery to run a pi for a few hours. (Current draw of pi <2A, 7Ah lead-acid batteries can be found for $30)
Battery banks with 5v input and output through USB plugs of some kind are a commodity now. Here's a small cheap one that you might want to add to your travel kit:
https://www.fasttech.com/product/1137904-miller-ml-102-universal-usb-smart-charger-version
A single decent 18650 lithium ion rechargeable will run you a few bucks, and is generally good for somewhere in the region of 10W*h.
Lol indeed. You don’t seriously think that you’re the first one dealing with rtc-less systems in production? The problem you’ve described is already solved by systemd: upon boot it sets the clock to the time when the particular version was released. The leaves you with couple of years off time worst case instead of 45+. The rest you’ve described – systemd-timesyncd will compensate for that with proper skew and systemd’s timer units will fire properly instead of cron jobs.
Welcome to 21 century, old-timer :)
>The problem you’ve described is already solved by systemd
You lost me when you said “systemd”. Someone else in the thread has aptly described it as “too clever by half”.
In embedding the XKCD strip, the mouseover text was lost. It reads: “40% of OpenBSD installs lead to shark attacks. It’s their only standing security issue.”
> every engineer would apprentice in a blue collar occupation of related practice before being allowed anywhere near a design project.
I wholeheartedly agree, to the extent of learning how to MIG weld.
On that note, do US (or anywhere outside the UK, for that matter) engineering degrees often allow students to spend a year (usually between 2nd and 3rd) doing an internship in industry then resume the course where they left?
> Someone else in the thread has aptly described [systemd] as “too clever by half”.
Indeed. Rather, you could avoid much of the damage simply by (1) not starting cron or similar daemons until _after_ the clock is set; and (2) fixing logging services so that they can minimally cope with an unset clock. But there could still be pervasive issues with e.g. file creation or modification times which would be pretty hairy to solve. The real issue AIUI is that the OS was simply not designed with an awareness of this possibility. It could be worthwhile to try and change this, because the RTC (with its onboard battery) _is_ a potential point of failure in a system, and many stripped-down systems could sensibly omit it.
>The real issue AIUI is that the OS was simply not designed with an awareness of this possibility. It could be worthwhile to try and change this
Now that I think about it, this might not be all that difficult to change. One place to intervene would be in the bootloader. Add logic such that if it sees a system time before its build date it prompts for correct time and sets the clock…
Yes but skills like welding and machining don’t appear to be prerequisites. My father lamented the state of modern engineering students when he mentored a few from UConn. He said they’re damn smart but they don’t know how to do the basics of machining and that exacts a cost on the job because then the designers don’t know what the people on the shop floor are doing.
Of course with the shop floor increasingly dominated by NC mills, laser cutters, 3D printers, and the like, maybe that won’t mean so much going forward.
@Lambert –
At my school, it is required that engineering students do a co-op in “real” industry. Don’t know how effective it is in training practical sense into them.
@esr “Add logic such that if it sees a system time before its build date it prompts for correct time and sets the clock…”
This is a no-go for many use cases. A server shouldn’t require manual intervention just to boot up.
@ Lambert – “On that note, do US engineering degrees often allow students to spend a year doing an internship in industry then resume the course where they left?”
Summer internships are relatively common (especially between junior and senior year). Many industries use this technique to evaluate and recruit top talent. I spent mine working for a Kodak x-ray film manufacturing plant in Colorado.
An others have commented, a hands-on apprenticeship that culminates in the acquisition of a basic occupational skill is ideal. This builds real-world common sense and a respect for the guys that actually run these plants from the bottom up. Ivory-tower engineers that miss out on this type of experience tend to produce designs that are clever in theory and unworkable in practice. I’ve seen $100+ million projects fail abysmally because of this. And it seems to be getting worse.
> Why is this a problem? One very mundane reason is that logfile analyzers don’t necessarily deal well with large jumps in the system clock, like the one that will happen when the system time finally gets set;
Maybe timestamps aren’t the best format for a log file.
No, hear me out.
If you stored something like a unique value representing the boot time followed by a linear count of seconds since that time, you could retroactively establish the real time for those entries after you’ve figured out what time it is.
As one of the Pi fan boys referenced, I should like to point out that I was not being entirely serious – sadly tone of voice doesn’t come across well in internet posts.
[Though I do think you are being slightly too dismissive of the Pi. Even if you have a slight point when it comes to running a mail server on one]
The root cause of the financial crisis in the education sector is ballooning bureaucracy: backroom parasites who quarantine all income in their bottomless pockets while berating the teachers and researchers for not returning higher “productivity dividends” to them on zero investment.
@ FrancisT
> sadly tone of voice doesn’t come across well in internet posts.
Exactly. I often fear that my remarks and questions on this blog may be misinterpreted as arrogant or snide, and strive to prevent that by including “With all due respect…”, “No offense, but…”, etc. If we were communicating orally, I’d convey respect through voice tone instead.
OK. This is funny. On my doorstep today there arrived a CanaKit Raspberry Pi 3 32GB Ultimate starter kit. RaspPi, power supply, GPIO breakout board, case, breadboard, patch cables, and a buncha resistors and LEDs and diodes. With, even, adorable little stick-on heatsinks. Basically a Pi project lab in a box.
I don’t know who sent it. Clearly somebody wants me to learn Pi hacking.
EDIT: Turns out it was Dave Taht.
Current date is Tue 1-01-1980
Enter new date (mm-dd-yy): _
In all seriousness, putting this logic in the bootloader won’t help much unless said bootloader has a way to pass the date/time info to the OS. The system clock is generally implemented as a programmable interval timer and an ISR in the OS which, inter alia, updates the OS’s notion of what time it is. Without an RTC, that’s all you get, so any notion of wall-clock time is held and maintained locally to the OS.
Yes — by all means please do. The Pi may not be a fit for your mail server project, but I’ll be damned if it isn’t a nifty machine to hack on. To work on the machine itself, you will have to check your fancy desktop environments and IDEs at the door — shit gets mad slow even on the considerably beefier Pi 2 — and hack the old-school way, with editors and command line tools. But I don’t think that’ll be a problem for you.
>But I don’t think that’ll be a problem for you.
No. No, it won’t be.
I built a medium-sized clock (6 feet by 8 feet) using LEDs that project onto my kitchen wall, controlled from a Raspberry Pi. I priced an RTC clock at $17 (plus some integration effort) and a WiFi USB adapter at $11, so I picked the NTP solution.
You describe a number of problems that arise from trying to maintain assorted bits of Unix legacy practice. It’s a $25 headless network server, so there’s no need for a network-less diagnostic mode, in fact it doesn’t need to work at all without a network connection. If you get any trouble, just swap out the entire device (or more often just the SD card) with a freshly configured replacement, and do fault analysis on the old one offline.
Alas, the reason why Raspberry Pis make terrible mail servers is the SD cards. They tend to have short lives, and all the data on them goes away at once when they fail. That problem pretty much kills the whole idea of a mail server in practice.
Raspbian comes with a package ‘fake-hwclock’ preinstalled, which basically records the time in a file, and sets the clock to a few seconds after the time recorded in the file on next boot. That means the time will be off by roughly the length of the downtime. Still wrong, of course, but all the speculation here does seem to be wildly misinformed about how specifically wrong it’ll be if you just take one of the devices out of the box and plug it in.
> A single decent 18650 lithium ion rechargeable will run you a few bucks, and is generally good for somewhere in the region of 10W*h.
An ordinary 15Ah USB charger pack that you can buy at a drugstore can run a Raspberry Pi for 22 hours…longer, if you are not also using the same battery pack to power 3W of wearable LEDs.
The pi3 runs emacs *just fine*.
@ESR
I remember vaguely you addressed this in The Jargon File as well and Yudkowsky discussed this too but I don’t have the links at hand. Anyway, the important thing is to focus on the psychological mechanism for it.
It is a subset of the mechanism I call “status wireheading”. Everybody likes to be respected and have a good reputation. Smart people tend to do it through doing smart things and getting a reputation of being a smart problem-solver. The “wireheading” aspect is that is this gets quickly short-cut in the brain: you just do smart things and feel good about yourself, even if nobody knows about it and thus nobody gives you points for this. (This is why the worst cases of this are called intellectual masturbation: masturbation is a similar shortcut between action and brain-reward.) (There is a related and far more dangerous version of it: doing it with moral goodness, not cleverness, but “moral masturbation” does not belong to a tech discussion, so I am just mentioning it for completeness.)
This can easily get in the way of solving problems actually right. Therefore, it is often better to resist it. How to resist it is a difficult question, I guess being aware of and catching yourself when you do it it is step 1. Then it is good to mock it when others do it – depending on the situation it is the light, friendly mocking like you do now or in other cases it can be sharper. On the social level, incentives should be structured so that people get actual rewards like money and jobs not out of signalling smartness but actually solving problems. This is not just a public vs. private thing, it happens in some private jobs, too, a friend of mine works in a corporate research center and apparently it is not really possible there to get cred for actually solving problems and making products, don’t know why exactly but that smells like a bankruptcy coming, anyway, so they tend to sign their emails with their academic ranks and whatnot… and that’s not good. The point is when external incentives change, “wireheading”, the shortcut tends to follow it. Smart people aren’t stupid, they tend to masturbate with things that are actually pretty similar to the kind of things that actually get them cred and money and other goodies, so if you pay and praise people to jump high they are going to shortcut it at some point and just enjoy jumping and get a self-esteem boost out of doing it well. Finally I would recommend something like the Bene Gesserit Litany Against Fear. “Too-cleverness is the good-solution-killer…” just formulate it better than I can.
Too clever by half is not a good figure of speech. Sure, if someone really is too clever for their own good, that should be communicated to them succinctly. But when you actually hear that figure of speech, it’s usually from someone who thinks themselves more clever or experienced and is annoyed that you brought up a better solution which would require leaving their comfort zone. It’s just resistance to change. “This works and is good enough” is an excellent way of getting inferior solutions. It’s how banks got stuck with mainframes running Cobol code that nobody can touch without breaking something.
I have ARM systems that have been running without problems for years, with much more demanding loads than a mere mail server. They’re headless systems by design, which means I couldn’t attach a local console to set up or troubleshoot if I wanted to. Maybe that’s not your kind of thing, but there are ARM systems with HDMI and RTC which are still far cheaper and use less power than the anemic x86 system that you got. There are also far more powerful x86 systems with passive cooling and lower power consumption which are still less costly than the one you got, even if you insist on two wired network interfaces. Just because your buddy got the old crappy one and knows how to troubleshoot it should not make you shy away from a different system. Those are still just PCs.
>Maybe that’s not your kind of thing, but there are ARM systems with HDMI and RTC which are still far cheaper and use less power than the anemic x86 system that you got. There are also far more powerful x86 systems with passive cooling and lower power consumption which are still less costly than the one you got, even if you insist on two wired network interfaces.
Examples please.
@TheDividualist [emphasis mine]:
I’m not saying you’re wrong here–but the idea that we can tell in advance when we are “wireheading” and when we are functional feels, to me, as “being too clever”. The idea as I’ve heard expressed elsewhere was “Were P=NP to be true, then understanding a solution would be the same as finding it–anyone who could appreciate a symphony would be a Mozart”. Here, I feel as if you’ve taken hold of that same dichotomy of understanding v. solving, but are ignoring the complexity gulf between them. For a great many problems, the only means we may ever have to differentiate between actual workable solutions or clever-but-inoperable ones is by experience (ie: implementing some of the bad ones).
>I’m not saying you’re wrong here–but the idea that we can tell in advance when we are “wireheading” and when we are functional feels, to me, as “being too clever”.
It doesn’t sound so much like a prescriptive solution, as another anecdotal case of a little bit of self-awareness going a long way.
Damn self-awareness privilege.
Hey, COBOL is pert-near an ideal language in its solution space. Hackers may look down their noses at it and make jokes about the object-oriented variant being called ADD 1 TO COBOL GIVING COBOL, but hackers can be ignorant, arrogant dicks sometimes too.
And mainframes? Mainframes have I/O bandwidth out the wazoo — which it turns out is the real bottleneck in mainframe style applications. Back in the day they weren’t just good enough — nothing else could touch them when you needed data throughput to service hundreds or thousands of users. And IBM has provided an upgrade path since System/360 that maintains perfect backward compatibility, allowing those old applications to run bit-identical on modern hardware alongside Linux web server instances.
There’s a reason why mainframes are said to “run Western civilization”.
ESR mentioned SCUBA diving as a community intensely aware of incident pits. Rodale’s magazine has a column about SCUBA fatalities that makes for fascinating reading on the topic. Short concise articles about how small mistakes lead to tragedy.
http://www.scubadiving.com/tags/lessons-for-life
As far as Pi I/O goes: SD cards with standard Linux filesystems are problematic for any kind of heavy (or really, moderate) I/O for multiple reasons. A boot loader or minimal root filesystem, and the rest on a USB external drive is probably the thing for use cases where you care.
And, yeah, it’s a “toy”… but it’s a quad-core gigahertz CPU with 1024 MB of RAM and 100Mb/s ethernet. Not that many years ago that was serious business!
@Jack: The problem you’ve described is already solved by systemd: upon boot it sets the clock to the time when the particular version was released.
This is an extremely…interesting…usage of the word “solved”.
@Greg:
That’s exactly why I phrased my response as “not saying you’re wrong”. Self-awareness and precaution are critical with any untested idea–to that extent I agree 100% with TheDividualist. However, to the extent his phrasing suggests there is a “general case” system to sort out in advance all ideas which belong in the circular filing drawer–well, that concept “feels like” proving that P=NP to me.
Sometimes people sneer at perfectly serviceable ideas (see Jeff Read’s comment re: COBOL) and sometimes we fall in love with terrible ones (insert ⟨hated political ideology⟩ here). Sure, some ideas are obviously doomed to failure–you’ll never fly to the moon by jumping–but the only ways I know of to sort out which ones actually work is either direct experience (ie: “try it and see”) or learning from the people who already have.
On of my main reasons for getting involved with the ietf (again) a few years ago was that far too many ideas were “too clever by half” and the former model of “running code and rough consensus” seemed to have evolved to “proto code and the consensus of the insane”. I kind of hope to have pushed the balance back a bit, but more folk stepping up to the task of keeping the internet saner for the billions using it, would be nice.
Great post (and I’ve already reposted it). The older I get, and the more troubled/failed IT projects I review (both as a consultant and as an expert witness), the more I keep coming back to my favorite systems design maxim:
— Robert J. Spinrad, lecture at USC, 1988, cited in The Art of Systems Architecting (3rd ed.), Maier & Rechtin (CRC, 2009), p. 399.
In this context, iatrogenesis is a salient concept as well.
I was one of the clever boys in the first go around in the mail server thread. When power goes off the Pi’s and the rest of the house goes dark and the internet goes down. When things power back up everything boots and at some point my Pi’s connect to an NTP service, figure out what time it is and things move on from there.
Since AstroEnterprises is smaller than the vast conglomerate that is A&D that’s been good enough. I use Pi’s like little blocks. Need to run that as a service? Pull a Pi out of the box, slap in in a case, power, config, put it on the shelf and forget about it.
I’m a hardware guy so using weird hardware isn’t hard for me. And this is where Eric and I diverge. He isn’t a hardware guy. So his requirements list wasn’t a low powered PC to run a mail server on. It’s 24 * 7 hardware support and after that a low powered PC to run a mail server.
I suggested the PI, he countered with needing a RTC. There are few $12 RTC boards that plug into the Pi. They do require being plugged in and they need to be configured. But I agree it falls into the realm of hardware.
On a whim I ordered the PiFace RTC from MCM (hardware guy, just added it to the next order of stuff from them) and it showed up today. Its took me less time to config it than it did to write this post. It works and when it boots there is still little log weirdness since Linux needs to query the time. That happens in /etc/rc.local, so it’s well into the boot process. But since my server startups happen after that, they will all start with the right time.
But that weirdness my not sit well. It doesn’t bother me since all time sensitive stuff runs after I do the time set. I’m guessing that it does bother Eric since he took the time to write another post about it.
To me the Pi is just a pull it off the shelf and use it chunk of hardware. I’d stumble for a day trying to build an x86 board from parts.
Happy that you have hardware support and your new system will be up soon.
Super happy that Dave gifted you with a Pi kit that you can play with.
If you want an RTC for it, drop me a note, MCM delivers worldwide, I put orders in ever week.
A client of mine wanted a smaller physical box for an x86 Linux based firewall – which shall remain nameless since it’s probably considered irredeemably gauche in these circles, but it has been serving him (and me) very well for about 10 years now, so I stuck with what I knew – aside to Jeff Read: I went well out of my way to go non-Intel. Does that make any difference? Anyway, at first I thought of just getting the smallest PC I could find, but that was (with the necessary NICs) still pretty big. I found a (what turned out to be) very slick little headless box (apu1d4) small enough to make the client quite happy. The latest version (which I hadn’t ever used) of said firewall claimed to be installable on headless, but the procedure for doing so was, let’s say, less than sparkling clear. So, being a firm subscriber to the admonitions in ESR’s OP, I tried it on a burner PC I had lying around, going so far as to construct a jumper-configurable null-modem adapter first. To make a long story short, I finally got it to work, but the process was much easier without the pressure of “I’m going to look like a real chump if I have to tell the client all this hardware he just bought can’t be used.”
@esr “engineering a production system”
What makes something “production”? In the past decade I’ve spent my time working for companies where the failure of a single component, including the motherboard, should not stop current workloads from being handled. Your use seems to imply anything where 9 5s of uptime is expected or better, maybe. That strikes me as a dorm-room definition production system.
>That strikes me as a dorm-room definition production system.
All engineering is tradeoffs. You have much more stringent requirements for MTBF than I do, but that doesn’t make mine a “dorm-room” definition, just one fitted to a different economic context than yours.
In my experience, production means “I want it to work, I want it to keep working, and I don’t want it to wake me up at 3 AM when it stops working.” For some reason it’s always 3 AM when things stop working, regardless of your time zone.
How you make this happen is a vast and exciting topic all its own, but one thing most of the approaches have in common is that you want to avoid doing things that are obviously going to cause weird, hard-to-reproduce problems. Personally, I would recommend never dealing with time in any way whatsoever, except perhaps recording causal ordering of events, Lamport-style. If you deal with time, you should always only record the number of seconds since God created the heavens and the earth on January 1, 1970. Time zones are evil, and leap seconds are right out.
“Production” for Eric means reducing his total cost of ownership. Eric’s most limiting resource is his time, followed by attention-grabbing concerns.
The word “secure” comes from the Latin: literally “no care”. The best idiomatic translation is the Jamaican “no worries, mon”. As a state of mind, it’s unmeasurable and very personal.
What can we can measure and reason about? Resilience. And not just in the availability domain; integrity and confidentiality too. Each individual will weight those elements differently, particularly in the context of technical risks, budget and intent/goals for the implementation. Eventually there has to be a decision to proceed, balancing the evidence of resiliency elements viewed through the prism of experience.
For example: personally, I would weight the integrity provided by Amazon Web Services over anything I could provide at my personal home. I’m also convinced they would do a better job of maintaining confidentiality on my behalf because they have hundreds of millions of revenue dollars per month at stake if their customers perceive a confidentiality failure. And I know their data centers have far better more reliable power supplies than what I have at home.
Does this make me right and Eric wrong? Not at all. We can look at the same evidences of resiliency and weight them differently. For example, Eric explained he weights the possibility of forgetting to pay Amazon (an availability threat) enough to overwhelm the other threats to availability that using Amazon would mitigate. He knows himself and his situation better than any of us, and I believe he knows enough about Amazon to make a good decision.
It’s nice to see this term get picked up by a broader community. I have been using it for the last 6 years in building integration architecture to wire together b2b systems in legacy verticals. Inevitably you run into 2 sorts sitting across the table from you: data modelers and nativists.
Data modelers can’t comprehend why you want to pass simplistic json strings with 2 objects and a defined array instead of their perfect xml behemoth that represents, in essence, their entire persistence model. Why, they exclaim, that it is _you_ introducing complexity by over simplifying because (after all) they have built and tested their system over X years to support Y client and certainly that robustness counts for a lot more than passing strings around.
Then there are the nativists: usually either sysAdmin types who have been finagled into writing code or corporate 4GL types who have read the white paper from Borg Central describing how easy it is to use the Z Library over SOAP to integrate systems, and how this supports decoupling and unity of purpose. What is lost is that the decoupling is still more coupled than is wise (integration is an art of coupling the right things and decoupling system-specific things) and the unity of purpose is the Borg Central’s biz plan, as opposed to, say, building an API that supports the actual integration case.
Both of these types lead you to the inevitable exploration of the architectural foundations of their code base wherein they announce, after 3 months, that “our system was never designed to actually integrate with your system”. Ahem.
:Yeah, a real-time clock is very important, as long as it FREEZES when you pull the battery to replace it (put on a switch to freeze the RTC state in EEPROM and read it back).”
Meh. Simpler and smarter is to gang it with a capacitor that’ll hold the fort for a few seconds. I learned this from a smoke detector that chirped its freaking annoying low battery alarm for ten minutes after the battery was pulled. (Related pro tip: use only Duracell in smoke detectors, and if they are not available, use a no-name brand rather than Energizer; the Energizer 9-volt is hardly better than a turd.) But that might not be necessary either, since the typical Li-MnO2 clock battery will often power the clock long after other parts of the system have died, such as the cooling fan. But since we’ve eliminated the cooling fan…
“Unrelatedly, I keep reading RTC as RTG, which keeps confusing me.”
Well, if you had an RTG for an RTC, you’d never need to change it. On the other hand, you’d probably need a cooling fan, lol! You’d also need to fill out a ream of paperwork for the IAEA and a team of lawyers and physicists to help you with that, which’ll probably put ESR’s goal of “reduc[ing] my mailserver’s power dissipation in order to (eventually) collect a net savings on my utility bill.” That’s a situation that fits the “too clever by half” description, CMIIR!
@Alex K.
>I’m not saying you’re wrong here–but the idea that we can tell in advance when we are “wireheading” and when we are functional feels, to me, as “being too clever”.
There is one method that often works – KISS principle, start with the simplest solution, then increasingly more complex ones if they fail, thus keeping complexity to the minimum because complexity implies fragility.
The problem with the method is that we can rarely solve modern tech problems with the application of a stone hatched (although sometimes one is really tempted…) so the defaults of what the minimal viable complexity depends on actual experience, so you are right there, but as a psychological mechanism knowing that we tend to like smart and fragile complexity and trying to hold those horses back is usually a good approach.
>>There is one method that often works – KISS principle, start with the simplest solution, then increasingly more complex ones if they fail, thus keeping complexity to the minimum because complexity implies fragility.
I have been using the YAGNI heuristic a lot lately to define the upper boundaries. … if it doesn’t directly relate to the problem _as defined_ it probably falls outside of the boundaries of what is “extra” to solving the problem. This works really well with software, especially where we can define the problem well.
I think in hardware, future proofing the solution (buying extra disk space or a better processor or a “more comprehensive” power solution) is usually a wiser and ultimately cheaper expedient. Moore’s law, and 60 years of cheap energy, historically makes this a viable approach. But it’s interesting because a lot of forces are causing the pendulum to shift and YAGNI to begin to intrude on the hardware purchasing decision.
ESR’s issue here is interesting because it defines a boundary line between hardware and software build / buy parsimony.
If an educational ARM board is not to your taste, what about a 96boards.org compatible server? One of those should have something suitable, one would think. See: http://www.96boards.org/
I’ve spent most of my career writing software for miliitary systems where Lots of People May Die if things don’t work properly. It makes ones engineering tend to the conserative side. I live by Knuth’s observation – “Premature optimisation is the root of all evil”.
Five Nines is 5 minutes of downtime a year. That may or may not stop a workload from being handled, depending on how much latency is allowed in your process.
Sort of a followup – the least replaceable resource is time. If you’re burning time messing around and it’s not for fun or profit, then you’re wasting that most precious resource.
I’m currently in the process of repurposing a R-Pi I originally bought as a VPN server (but never used because I have a working solution, and getting the Pi working was both not-fun and taking time I wanted to use for other things). The new purpose? Being a game streamer to my TV. In one way, this is exactly as much a waste of time as trying to get it working as a VPN head – the same system that I currently use as my VPN head uses my TV as a monitor and has Steam installed on it. But, right now, the messing about is the “fun” part. If I can’t get it to work in a relatively rapid time, I’ll drop the project and move on.
> For some reason it’s always 3 AM when things stop working, regardless of your time zone.
I have gotten to the point that I just automatically wake up around 3 am and glance at the phone.
“I’ve spent most of my career writing software for miliitary systems where Lots of People May Die if things don’t work properly. It makes ones engineering tend to the conserative side. I live by Knuth’s observation – “Premature optimisation is the root of all evil”.”
I’ve spent the last 10 years writing/reviewing/etc software for military systems and have discovered in that process that rebooting the system every 24 hours is an acceptable work around for major weapon systems.
I spent the previous few years writing software for a major telecommunications company (now mostly dead) with employees drawn from Nortel, Lucent, etc and am now pleasantly surprised to get a dial tone when I pick up the phone. There were more WTF moments working in the “5 nines” industry than I thought possible.
Telecom, at least the company I work for, obtains “five nines or better” by brute force and massive overengineering and redundancy.
@nht, one of my friends worked at the Ericsson research center that invented BlueTooth. Design spec: don’t have more than 79 pairings close to each other. Seriously. A larger LAN party would kill it if everybody used BT keyboards and mice.
Yikes. Within reach I have 2x phones that are paired to a headset, my smartwatch, a mouse, the pen to my Surface, my Surface, a BT keyboard, and 2 laptops. Extend to within BT range and I add an iPad, another laptop and a BT speakerphone module. Not all of those are powered up, but still…
(Good thing it’s getting hard to find BT mice and keyboards, eh? Most wireless mice and KBs use proprietary 2.5gHx dongles instead of BT)
> But if you’re trying to
bringstand up a production mailserverNoveau vernacular.