In the wake of the Ars Technica article on NTP vulnerabilities, and Slashdot coverage, there has been sharply increased public interest in the work NTPsec is doing.
A lot of people have gotten the idea that I’m engaged in a full rewrite of the code, however, and that’s not accurate. What’s actually going on is more like a really massive cleanup and hardening effort. To give you some idea how massive, I report that the codebase is now down to about 43% of the size we inherited – in absolute numbers, down from 227KLOC to 97KLOC.
Details, possibly interesting, follow. But this is more than a summary of work; I’m going to use it to talk about good software-engineering practice by example.
The codebase we inherited, what we call “NTP Classic”, was not horrible. When I was first asked to describe it, the first thought that leapt to my mind was that it looked like really good state-of-the-art Unix systems code – from 1995. That is, before API standardization and a lot of modern practices got rolling. And well before ubiquitous Internet made security hardening the concern it is today.
Dave Mills, the original designer and implementor of NTP, was an eccentric genius, an Internet pioneer, and a systems architect with vision and exceptional technical skills. The basic design he laid down is sound. But it was old code, with old-code problems that his successor (Harlan Stenn) never solved. Problems like being full of port shims for big-iron Unixes from the Late Cretaceous, and the biggest, nastiest autoconf hairball of a build system I’ve ever seen.
Any time you try to modify a codebase like that, you tend to find yourself up to your ass in alligators before you can even get a start on draining the swamp. Not the least of the problems is that a mess like that is almost forbiddingly hard to read. You may be able to tell there’s a monument to good design underneath all the accumulated cruft, but that’s not a lot of help if the cruft is overwhelming.
One thing the cruft was overwhelming was efforts to secure and harden NTP. This was a serious problem; by late last year (2014) NTP was routinely cracked and in use as a DDoS amplifier, with consequences Ars Technica covers pretty well.
I got hired (the details are complicated) because the people who brought me on believed me to be a good enough systems architect to solve the general problem of why this codebase had become such a leaky mess, even if they couldn’t project exactly how I’d do it. (Well, if they could, they wouldn’t need me, would they?)
The approach I chose was to start by simplifying. Chiseling away all the historical platform-specific cruft in favor of modern POSIX APIs, stripping the code to its running gears, tossing out as many superannuated features as I could, and systematically hardening the remainder.
To illustrate what I mean by ‘hardening’, I’ll quote the following paragraph from our hacking guide:
* strcpy, strncpy, strcat: Use strlcpy and strlcat instead. * sprintf, vsprintf: use snprintf and vsnprintf instead. * In scanf and friends, the %s format without length limit is banned. * strtok: use strtok_r() or unroll this into the obvious loop. * gets: Use fgets instead. * gmtime(), localtime(), asctime(), ctime(): use the reentrant *_r variants. * tmpnam() - use mkstemp() or tmpfile() instead. * dirname() - the Linux version is re-entrant but this property is not portable.
This formalized an approach I’d used successfully on GPSD – instead of fixing defects and security holes after the fact, constrain your code so that it cannot have defects. The specific class of defects I was going after here was buffer overruns.
OK, you experienced C programmers out there are are thinking “What about wild-pointer and wild-index problems?” And it’s true that the achtung verboten above will not solve those kinds of overruns. But another prong of my strategy was systematic use of static code analyzers like Coverity, which actually is pretty good at picking up the defects that cause that sort of thing. Not 100% perfect, C will always allow you to shoot yourself in the foot, but I knew from prior success with GPSD that the combination of careful coding with automatic defect scanning can reduce the hell out of your bug load.
Another form of hardening is making better use of the type system to express invariants. In one early change, I ran through the entire codebase looking for places where integer flag variables could be turned into C99 booleans. The compiler itself doesn’t do much with this information, but it gives static analyzers more traction.
Back to chiseling away code. When you do that, and simultaneously code-harden, and use static analyzers, you can find yourself in a virtuous cycle. Simplification enables better static checking. The code becomes more readable. You can remove more dead weight and make more changes with higher confidence. You’re not just flailing.
I’m really good at this game (see: 57% of the code removed). I’m stating that to make a methodological point; being good at it is not magic. I’m not sitting on a mountaintop having satori, I’m applying best practices. The method is replicable. It’s about knowing what best practices are, about being systematic and careful and ruthless. I do have an an advantage because I’m very bright and can hold more complex state in my head than most people, but the best practices don’t depend on that personal advantage – its main effect is to make me faster at doing what I ought to be doing anyway.
A best practice I haven’t covered yet is to code strictly to standards. I’ve written before that one of our major early technical decisions was to assume up front that the entire POSIX.1-2001/C99 API would be available on all our target platforms and treat exceptions to that (like Mac OS X not having clock_gettime(2)) as specific defects that need to be isolated and worked around by emulating the standard calls.
This differs dramatically from the traditional Unix policy of leaving all porting shims back to the year zero in place because you never know when somebody might want to build your code on some remnant dinosaur workstation or minicomputer from the 1980s. That tradition is not harmless; the thicket of #ifdefs and odd code snippets that nobody has tested in Goddess knows how many years is a major drag on readability and maintainability. It rigidifies the code – you can wind up too frightened of breaking everything to change anything.
Little things also matter, like fixing all compiler warnings. I thought it was shockingly sloppy that the NTP Classic maintainers hadn’t done this. The pattern detectors behind those warnings are there because they often point at real defects. Also, voluminous warnings make it too easy to miss actual errors that break your build. And you never want to break your build, because later on that will make bisection testing more difficult.
Yet another important thing to do on an expedition like this is to get permission – or give yourself permission, or fscking take permission – to remove obsolete features in order to reduce code volume, complexity, and attack surface.
NTP Classic had two control programs for the main daemon, one called ntpq and one called ntpdc. ntpq used a textual (“mode 6”) packet protocol to talk to nptd; ntpdc used a binary one (“mode 7”). Over the years it became clear that ntpd’s handler coder code for mode 7 messages was a major source of bugs and security vulnerabilities, and ntpq mode 6 was enhanced to match its capabilities. Then ntpdc was deprecated, but not removed – the NTP Classic team had developed a culture of never breaking backward compatibility with anything.
And me? I shot ntpdc through the head specifically to reduce our attack surface. We took the mode 7 handler code out of ntpd. About four days later Cisco sent us a notice of critical DoS vulnerability that wasn’t there for us precisely because we had removed that stuff.
This is why ripping out 130KLOC is actually an even bigger win than the raw numbers suggest. The cruft we removed – the portability shims, the obsolete features, the binary-protocol handling – is disproportionately likely to have maintainability problems, defects and security holes lurking in it and implied by it. It was ever thus.
I cannot pass by the gains from taking a poleaxe to the autotools-based build system. It’s no secret that I walked into this project despising autotools. But the 31KLOC monstrosity I found would have justified a far more intense loathing than I had felt before. Its tentacles were everywhere. A few days ago, when I audited the post-fork commit history of NTP Classic in order to forward-port their recent bug fixes, I could not avoid noticing that a disproportionately large percentage of their commits were just fighting the build system, to the point where the actual C changes looked rather crowded out.
We replaced autotools with waf. It could have been scons – I like scons – but one of our guys is a waf expert and I don’t have time to do everything. It turns out waf is a huge win, possibly a bigger one than scons would have been. I think it produces faster builds than scons – it automatically parallelizes build tasks – which is important.
It’s important because when you’re doing exploratory programming, or mechanical bug-isolation procedures like bisection runs, faster builds reduce your costs. They also have much less tendency to drop you out of a good flow state.
Equally importantly, the waf build recipe is far easier to understand and modify than what it replaced. I won’t deny that waf dependency declarations are a bit cryptic if you’re used to plain Makefiles or scons productions (scons has a pretty clear readability edge over waf, with better locality of information) but the brute fact is this: when your build recipe drops from 31KLOC to 1.1KLOC you are winning big no matter what the new build engine’s language looks like,
The discerning reader will have noticed that though I’ve talked about being a systems architect, none of this sounds much like what you might think systems architects are supposed to do. Big plans! Bold refactorings! Major new features!
I do actually have plans like that. I’ll blog about them in the future. But here is truth: when you inherit a mess like NTP Classic (and you often will), the first thing you need to do is get it to a sound, maintainable, and properly hardened state. The months I’ve spent on that are now drawing to a close. Consequently, we have an internal schedule for first release; I’m not going to announce a date, but think weeks rather than months.
The NTP Classic devs fell into investing increasing effort merely fighting the friction of their own limiting assumptions because they lacked something that Dave Mills had and I have and any systems architect necessarily must have – professional courage. It’s the same quality that a surgeon needs to cut into a patient – the confidence, bordering on arrogance, that you do have what it takes to go in and solve the problem even if there’s bound to be blood on the floor before you’re done.
What justifies that confidence? The kind of best practices I’ve been describing. You have to know what you’re doing, and know that you know what you’re doing. OK, and I fibbed a little earlier. Sometimes there is a kind of Zen to it, especially on your best days. But to get to that you have to draw water and chop wood – you have to do your practice right.
As with GPSD, one of my goals for NTPsec is that it should not only be good software, but a practice model for how to do right. This essay. in addition to being a progress report, was intended as a down payment on that promise.
To support my work on NTPsec, please pledge at my Patreon or GoFundMe pages.
> Back to chiseling away code. When you do that, and simultaneously code-harden, and use static analyzers, you can find yourself in a virtuous cycle.
Aren’t you missing unit testing? I’m all for the rest, but leaving out some kind of automated testing gets my heckles up ;)
>Aren’t you missing unit testing?
Sadly, yes. There are some broken unit tests in the codebase. I’ve tried to get two different people on the team to fix them. Neither has delivered yet. It’s almost our only remaining release blocker
The first new “feature” I have planned is replicable end-to-end testing.
So if I understand it the core functionality and interface of NTP Classic is still there in this release of STPsec. The only real changes are killing the depreciated binary interface and the build assumption of POSIX compliance. The rest of the changes are basically invisible to the user code cleanup using modern libraries.
In other words NTP could use this upcoming release as a new start point on their version. :)
> The rest of the changes are basically invisible to the user code cleanup using modern libraries.
Not quite all. Some names have changed. Some dodgy features have been removed. The documentation is a lot better.
“This differs dramatically from the traditional Unix policy of leaving all porting shims back to the year zero in place because you never know when somebody might want to build your code on some remnant dinosaur workstation or minicomputer from the 1980s.”
Bravo. The Unix tradition dates from the time when there were a lot more odd hardware platforms in use, and hardware was a lot more expensive. One of the first assumptions I’d make on an effort like NTPSec is the one you made: that the code will be built and run on relatively current hardware and OSes. Don’t have a supported platform? Tough. Get one. Hardware is cheap.
______
Dennis
>Don’t have a supported platform? Tough. Get one. Hardware is cheap.
A standards-conformance filter is actually less stringent than that, as some legacy hardware has had modern POSIX/C99-conformant toolchains ported back to it.
So what’s the Windows port (assuming there is one?) going to look like? As an example of a “modern” OS that doesn’t support POSIX or C99.
> So what’s the Windows port (assuming there is one?) going to look like? As an example of a “modern” OS that doesn’t support POSIX or C99.
If there would be any such port, it would probably use MSYS / MinGW, just like modern Git for Windows (Git also makes heavy use of POSIX features).
>it would probably use MSYS / MinGW
MSYS looks like the most likely implementation path.
-I’m not sitting on a mountaintop having satori, I’m applying best practices.
+I’m not JUST sitting on a mountaintop having satori, I’m applying best practices.
-I do have an an advantage because I’m very bright
+I do have an an advantage because I’m enlightened
Please apply soonest.
As an experienced C programmer and unix hacker, what do you think about current pretenders to C’s position as the system language, such as Go, D or Rust (or modern C++)? If you were writing ntpd from scratch, would you consider languages other than C?
>If you were writing ntpd from scratch, would you consider languages other than C?
Yes. I admit I’d be thinking about Rust seriously. I dislike C++, D is proprietary, and the occasional GC delays in Go make it seem like a risky choice for this problem.
Assuming there is progress in the area of high-quality and standard interfaces, meaning we can abandon support for older and less popular operating systems. it raises the question: what is the next feature that isn’t common enough to assume support for it across various popular operating systems, that would simplify portability a lot? The definition of “feature” here could be very broad: it could be a function, or could be undocumented or variant behaviour. Time didn’t stand still in 1989, nor in 1999, even if C99 is very good. In the long run maybe some language other than C will be assumable, although that may be a long time in the future: C++ failed to declare C as obsolete, and today there are many up-and-coming languages, dividing the attention of the worldwide community. An immediate improvement for portability likely wouldn’t be switching languages, so I wonder what that is?
Portability could get worse, too. Progress requires the doing of new things, often different people doing the same thing in different ways. Standardization entails finding a common way of doing the new thing. Here is a metaquestion: how do we develop software and agree standards in a way that avoids a repetition of the Unix situation?
C++ as a systems programming language??!
OINK!!!!
Really, now…how heavyweight can you get?
@esr –
> you never know when somebody might want to build
> your code on some remnant dinosaur workstation
> or minicomputer from the 1980s
Not disputing your recommendation to ‘simplify to the universal standards’, but —
does the existence of tools like the Debian Porter Box farm help with testing across a wide range of unusual architectures? Is it enough? Does it justify leaving in (some? any?) of the “porting shims”?
>does the existence of tools like the Debian Porter Box farm help with testing across a wide range of unusual architectures?
It does. I used them to test GPSD extensively. They actually make it easier to code purely to standard by, for example, smoking out bugs due to size and endianness of integers.
Well, that was quite the layman’s armchair exposition!
From one of the mass of hungry peasants who stand outside the keep (and “digitally divided” mote) of this kingdom, I, for one, say, “Thank You!”
@John D Bell – aren’t all those running Debian though? AIUI most of the kind of thing that compatibility shims are designed for tends to apply more to rare operating systems than merely a rare CPU architecture.
It seems to me that any time i have done the configure/make/make install thing, a rediculous number of warnings have come out. Like 1 warning per file compiled is not noteworthy and not even the worst i’ve seen. I’d just assumed that either something in my toolchain setup is non-standard (always plausible) or that general portability expectations break the gcc warning mechanism or something.
Looking over the D site, it sure looks like it’s fully open source to me….including gcc and LLVM-based compilers.
>Looking over the D site, it sure looks like it’s fully open source to me….including gcc and LLVM-based compilers.
OK, that’s news to me. I’ll take another look at it.
It’s a bit oddball.
LDC (LLVM version) and GDC (gcc version) are both fully open source as would be expected. DMD (the original Walter Bright reference implementation) has an open front end but closed back end. The only information i could find on differences in either LDC/GDC were that the language itself was compiling fine but the libraries(e.g. phobos) for everything other than x86/x86_64 were full of issues.
I think Rust and D both try to hit the same market but I think i’m more interested in Rust than D right now.
As for your objection to Go’s garbage collection and resulting delays…it would seem to me that a garbage-collecting language implementation for systems programming would have an intrinsic that says “don’t garbage collect until I tell you you can”. After all, there are large chunks of code where a GC delay would be unremarkable and have no effect on the program’s performance.
>As for your objection to Go’s garbage collection and resulting delays…it would seem to me that a garbage-collecting language implementation for systems programming would have an intrinsic that says “don’t garbage collect until I tell you you can”.
Sure. But having that primitive is not the same as knowing when you can use it safely. There’s no way to guaranteee that a UDP packet won’t arrive while GC is arriving, imposing a large unpredictable latency. Which is about the last thing you want in a time-service daemon.
Go does allow temporarily disabling the GC at runtime: for instance using https://golang.org/pkg/runtime/debug/#SetGCPercent , although there may be a better way.
> Sure. But having garbage collecting control primitive is not the same as knowing when you can use it safely.
Or you can use garbage collection algorithm that do not require “stop the world” interrupts:
http://www.h-online.com/developer/features/Java-hiccups-and-how-to-beat-them-1627673.html
Though I think that in system programming carefully tracking ownership (like Rust does, like C++ can do with unique_ptr and shared_ptr) is a more performant solution anyway.
BTW, Eric, do NTPsec use pointers, and if it does, does it use C99 “restricted” keyword?
>BTW, Eric, do NTPsec use pointers, and if it does, does it use C99 “restricted” keyword?
It does use pointers, and it does not use restricted.
One of the lessons that Lispheads never quite learned was “GC = fail”. All of the use cases for garbage collection can be covered with a combination of scope-bounded object lifetimes (C++’s RAII pattern) and automatic reference counting (C++ smart pointers and the ARC mechanism in Objective-C); and those give you deterministic time and memory constraints. Btw, this is another reason why the Android runtime objectively sucks compared to iOS (and hence why mobile developers will target iOS first).
To be honest, I’m surprised I don’t see more hackers hacking Ada. The language itself has many more features to ensure type and memory safety than does C++, the SPARK static verifier is a thing, the compiler produces helpful error messages of the form “you used X; perhaps you meant to use Y here” or “you used an aliased pointer; if you really want this you need to mark it ‘aliased'”, and because the generic features are implemented a bit more smartly than naïve template reinstantiation, compile times even for generic code are fairly short. Much of the promise of Rust are things that Ada can give you today.
Automatic reference counting leaks cyclic data structures. There are workarounds, but it’s not pretty.
More importantly, lock/wait-free data structures without garbage collection are somewhere between “insanely difficult” and “actually impossible”. Last time I checked, a lot of this was an open research problem. Garbage collection might be a dog, but I know trying to do concurrency with locks is a bitch.
There’s no such thing as “lock-free”, just like there’s really no such thing as “no side effects”. Locking is how shit gets done in a concurrent environment. STM and similar use mutexes and locking inside to implement their transaction abstraction. They tend to fall over hard when running embedded or at scale (and scale comes around much sooner than you think).
Boost (and the C++11 standard library) provides standard, easy to use, cross-platform threading and locking primitives. It’s normally a simple matter of wrapping your critical section in curly braces and using boost::mutex::scoped_lock.
Bravo on eliminating so much of the legacy codebase. I am looking forward to the first public release and hopefully it will be adopted quickly by many open source operating system distributions.
Perhaps the better question is “when you are done with cleaning the codebase, will NTPsec use pointers and if so, will it use the C99 “restricted” keyword?”
>Perhaps the better question is “when you are done with cleaning the codebase, will NTPsec use pointers and if so, will it use the C99 “restricted” keyword?”
It will definitely still use pointers, trying to eliminate them would be way too hard.
Whether it uses restrict will depend on whether I learn a good method for adding such annotations. Are there any good white papers on this?
Merciful $DEITY, Jeff. Ada??! If you want PL/I or Pascal, you know where to find them.
Hackers tend not to use languages that have so much of the BDSM nature that they require safewords.
Do you mean this statement in a non-literal “Yes it’s a thing but it’s not worth the cost in complexity” way or do you actually think lock-free concurrency doesn’t exist?
If the former, then I mostly agree… nothing to see here.
If the latter, then i suggest you look at LMAX disruptor. Unless your definition of “lock” includes “using raw compare-and-swap operations” then that is a lock-free message queue in use in production.
Can RCU parallel primitive be considered lock-free? You can use high-level constructs that while using locks / semaphores in their implementation, are nevertheless much safer to use.
There is also parallel functional programming paradign (lock-free I think), with e.g. GPH (Glasgow/Glorious Parallel Haskell).
Nb. is there any futures / promises (for async) implementation for C?
Congratulations for shooting yourself into the foot by choosing yet another stupid buildsystem. Thanks for being a pain in the ass to all the people who actually need to build your software and bring it into a shape for a distribution.
https://lists.debian.org/debian-devel/2010/02/msg00714.html
>Perhaps the better question is “when you are done with cleaning the codebase, will NTPsec use pointers and if so, will it use the C99 “restricted” keyword?”
Why the emphasis on using the “restrict” keyword? This isn’t some magical thing that makes using pointers “safer”. It tells the compiler to /assume/ the pointers are not in overlapping regions of memory (which allows optimisations to be more aggressive because the alias analysis doesn’t have to be as conservative)
It does *not* enforce anything about the pointer’s relationships. If the pointers are overlapping and you’re not aware of it, dragons will still come out of your nostrils, they just might come out much faster :)
If you want to make life easy on distro maintainers, the build system to use is — wait for it — autotools. It works on almost every POSIX system, gets GNU software running on a combinatorical explosion’s worth of weird configurations, and if you include the generated scripts in your obsolete tarball, the distro maintainer doesn’t have to install or set up anything to build your software.
(That’s another thing: pervasive package management exists because diligent distro maintainers stand ready in the night to roll us debs and rpms. They don’t want to sync your damn Git repo; they want to wget a tarball, run a script, turn the crank, and get a package.)
So just use autotools or be prepared to be the target of downstream’s ire.
> […] and if you include the generated scripts in your obsolete tarball, the distro maintainer doesn’t have to install or set up anything to build your software.
Doesn’t WAF include itself in tarball, so that you don’t need to install WAF to build the project?
>Doesn’t WAF include itself in tarball, so that you don’t need to install WAF to build the project?
That is correct. This also answers Bernd Zeimetz’s objection; you know the version of waf is right because it’s the same one the developers built with. That’s a guarantee autoconf doesn’t provide.
Uhm, Jeff, distribution maintainers want to be able to suck down a package and build it with a minimum of work. Autotools does do that, true, but so do lots of other build systems. The key is that the package must be self-contained to the maximum extent possible, and dependencies must be clearly documented and themselves as easy to obtain as possible, preferably as standard parts of the distribution.
In fact, making a distribution maintainer run the autotools himself, instead of providing a ready-to-run configure script, is making it harder on him than, say, providing him a complete, ready-to-run scons recipe that requires no external tools at all.
FWIW, I was a distro packager once, long ago when autotools was used universally. I was on the Gentoo KDE package team for KDE 2.x and 3.x. And I cheered when KDE switched from autotools to cmake for 4.x (I was no longer a packager by then) – it reduced the workload greatly, killed a bunch of awful dsls, and incudentally made builds much faster and incremental builds more reliable.
Autotools is good mostly at making you spend your time debug build scripts instead of actual code. Of course YMMV.
And, yeah, dealing with incompatible versions of autotools which different packages need to have installed was a PITA – thanks for reminding me of those days…
> One of the lessons that Lispheads never quite learned was “GC = fail”.
Haha, such a categoric, contextless, and absolute refusal cannot be taken seriously. I’ll name it MMM: memory management manicheism.
> All of the use cases for garbage collection can be covered with [other means]; and those give you deterministic time and memory constraints.
That’s definitely true. And guess what, sometimes you don’t need this determinism at all. And you actually gain very significant things in exchange for it. Things that have been useful to millions of developers around the world… Ha, it had been a long time since I had seen such MMM.
> Much of the promise of Rust are things that Ada can give you today.
Ada can give much of them today, Rust gives them today.
It’s not in beta anymore, it’s in 1.3.0.
Why use a 35 year-old language when you can have modern features without legacy?
> you know the version of waf is right because it’s the same one the developers built with. That’s a guarantee autoconf doesn’t provide.
> In fact, making a distribution maintainer run the autotools himself, instead of providing a ready-to-run configure script
Er, am I completely off the mark in believing that most autotools-based packages do ship their ready-to-run configuration scripts, guaranteed to be built from the same version of autoconf that the developers used, and don’t even require you to have autoconf installed let alone run it?
>Er, am I completely off the mark in believing that most autotools-based packages do ship their ready-to-run configuration scripts
That’s normal in tarballs. But developers routinely build from repository checkouts or clones, and it is variable whether the generated files are included in those. Increasingly, normal practice has been not to include the generated files, and instead to have a bootstrap script (usually, but not always, called “autogen.sh”) that relies on a local copy of autotools.
This is exactly what Hercules did while I was maintaining it.
> It does *not* enforce anything about the pointer’s relationships. If the pointers are overlapping and you’re not aware of it, dragons will still come out of your nostrils, they just might come out much faster :)
But it allows other static tools to enforce this. And your C-level code might well make that same assumption.
@ Eric
Good work on the clean-up and basic methodology. Entertaining read just like your Cathedral and Bazaar paper I finally took the time to read rather than others’ summaries. A lot of stuff makes more sense now, esp parallel debugging concept.
Even better, though, was the “draining the swamp” reference. I was recently reviewing a paper with that title about VM’s or something. I told the crowd on Schneier’s blog (my main spot) it was a nice title but best for a paper on cleaning up a legacy UNIX codebases. Then, I see it here referencing exactly that. :)
@ Jeff
They probably push it because they know there’s been a ton of GC’s that eliminate its issues. And LISP machines had GC’s. We’re seeing people learn the idea again in the form of research into pauseless GC’s for embedded and server use. Azul System’s Vega machines do Java natively on 54 cores w/ pauseless GC’s. So, why not try to keep using such tech unless one truly can’t do otherwise?
https://www.azul.com/products/vega/
Best route, though, is providing a mix of options for memory management like Modula-3, Ada, and JX OS do. Might use none for performance-critical, inner loops or lowest level stuff. Might use some real-time or pauseless option for stuff that can’t take a hit. Might use a more bulk option for something that can pause such as servers in a load-balancing configuration. A well-designed language or platform should be able to mix these even in the same system.
@ Jay Maynard
Yet, it was the Ada developers writing working code on a regular basis in all the 90’s era studies that pitted them against C and C++ developers. The C++ people got close one time in defect counts. C users had way more. Meanwhile, Ada has gotten even safer and whatever you can express in SPARK can be proven free of common defects. That’s because every feature is systematically designed to eliminate real-world bugs. Harder to write but most software exists in maintenance mode. (see NTP Classic)
http://www.adacore.com/knowledge/technical-papers/safe-secure/
Now, if you want BDSM, you should try safe or secure programming in C. Heck, C was shown to be inferior garbage back when Hansen’ and Wirth’s languages did systems programming while being efficient, safe, easy-to-read, fast-to-compile, and portable. Both used them for clean, safer OS designs. Wirth added GC to OS and language later with it being usable (see A2 Bluebottle). People should’ve ditched C way back then but didn’t want to re-write code and tooling. Band-aids, cruft, and vulnerabilities accumulated instead.
Of course, if you knew how C was created, you’d know that (a) it wasn’t *designed* at all, (b) it wasn’t a system language really, (c) wasn’t a cross-platform assembler, and (d) there were better things in the 70’s-80’s that their hardware just couldn’t use. (a)-(c) being common myths I hear. Every feature of BCPL was designed *only* to compile & run fast on 1960’s hardware. C was a modified clone of it with same reasoning for PDP-11 with later extensions. Terrible foundation for systems programming vs later Modula-3 (used in CVSup) or even Hansen’s stuff (eg Component Pascal or Edison System) on same PDP-11. So, let’s not pretend it’s anything other than crap derived from weak hardware in the 1960’s and look for any reason to ditch it.
Brief, bullet-point history of C showing what it really is and why with references
http://pastebin.com/UAQaWuWG
Quoth The Raymond:
>> need to be isolated and worked around by emulating the standard calls.
I’m guessing the similarity to the OpenBSD boys coding to a “sane target” (their words on eg LibreSSL) and adding portability shims for the gubbins missing on a particular platform is essence, not accident, as you’re tackling (afaict) the same problem?
>I’m guessing the similarity to the OpenBSD boys coding to a “sane target” (their words on eg LibreSSL) and adding portability shims for the gubbins missing on a particular platform is essence, not accident, as you’re tackling (afaict) the same problem?
Yes, I would say the parallel is quite real. They could use the hell out of me on that project if I weren’t so busy and would prefer root canal to having to deal with Theo de Raadt for any length of time.
Sounds like a good challenge Eric, keep up the good work!
Going through grungy hairballs of C source code is one aspect of a certain long term project I’ve been avoiding. There’s a few projects that have just grown so “organically” they are like kudzu on a fence line. I’m tempted just to reverse engineer the useful bits and write them in a saner, more maintainable language. I’d use Python, but it’s not responsive enough.
If C isn’t the system programming language of the long term future, what is?
wrt build systems, any opinions on things like fabricate.py which automate the build and the dependencies so you can just script your build and it figures out deps for you (by watching the build with strace)? ?
>wrt build systems, any opinions on things like fabricate.py which automate the build and the dependencies
waf is like that. It works.
Rust is a likely candidate. Nick P is right: C should have been deprecated and abandoned decades ago, but its close association with Unix makes it a huge millstone around our necks. C++ is somewhat better, but cumbersome to use; but then again Stroustrup noted that there are languages people complain about and languages nobody uses.
Ideally some Pascal, Ada, or Modula variant would have become the bedrock for future systems in the 80s, and it almost happened: the entire Macintosh API assumed Pascal as an applications language, though most of the OS itself was written in 68k assembly. But, regrettably, the Unix heads won again and most dev kits late in the classic Mac OS years were for C and C++. Although, the most secure web server of the late 90s was not a Unix box but a Macintosh running WebSTAR, because Pascal strings > C strings.
I guess I’ll check out Rust since most the Pascal/Modula/Oberon family is on life-support and Ada doesn’t have enough traction in my realms. (Although Oberon is intriguing due to it’s nature.)
Interestingly enough, many of the finance programmers I know are going functional to omit errors and such. Ocaml and F# are growing among those communities, but they don’t really work as lower level languages.
@ Jeff Read
I have hopes for Rust to take that possition as it’s clever work. My old tools, from automatic programming days, did a mock-up of 3GL’s in LISP with macro’s for 4GL-like functionality. I could code up and test an application quickly using LISP’s incremental compilation. Once at a certain point, I could generate a full app for BASIC, C++, or C. Flip a switch to get safety/security or portability although annotations were required for latter. Lost that stuff with other work years ago in triple HD crash but figure it’s worth rebuilding. The thing to takeaway is that hacking and exploration should be easy for the developer with the final result easy to analyze or transform for robustness.
On that note, I suggest you take a look at Julia language. They’ve done amazing work combining productivity, performance, and robustness. They seem to be setting a new baseline for 3GL’s. Anyway, I mentioned my scheme on a Hacker News article about Julia and was told that (surprise) Julia is a LISP internally. The author(s) built femtolisp to be compact, fast, and powerful. Then, the Julia AST’s are just femtolisp S-expressions. So, interpreting and transforming it is as easy for the compiler as is writing regular LISP applications. For developers, they see an easy-to-use, fast 3GL with native C support. Smart combo.
In any case, we’ve already seen pieces of what system language needs in prior work. It must be easy to parse, easy to analyze, produce efficient code, have *optional* safety on every critical construct, allow control of represenation, support design-by-contract, and have safe linking. Modula-3, Ada, Eiffel, and Cyclone all have pieces of this with each leading to robust programs in practice. Three did that in industry for a long time. Macro’s are a must for zero-overhead abstractions for productivity, portability, or even security. Can even include custom type-systems where desirable (eg concurrency checks). Inline asm, esp with good wrapper support, would also help. I guess the end result I’m visualizing is a language with Modula-3-like features/readibility, as safe as Ada 2012, and as easy to write & extend as Julia. I don’t think that’s too hard as much could just be built into compiler front-end using a strategy like Julia’s.
Note: Can also keep in mind the compile-to-C strategy to leverage those compilers’ performance and portability. Fortuntely, LLVM work is making this less necessary as we can just compile to LLVM. I once proposed supporting inline LLVM bytecode in a language for a portable assembler for optimization reasons. What you think on that one?
“Although, the most secure web server of the late 90s was not a Unix box but a Macintosh running WebSTAR, because Pascal strings > C strings.”
It was definitely a smart move to write it in Pascal to reduce bug count via both strings and interface issues. However, an app’s security depends on TCB: each component that can break security policy. That includes the OS and it wasn’t secure by any means. The closest thing to a secure web server in 90’s was Schell’s that relied on GEMSOS security kernel: a high assurance system written in Pascal that survived years of NSA pentesting. You’d think modern security engineers concerned about NSA & hackers might imitate tech that stopped them in past or present, eh? ;)
http://www.iwia.org/2005/Schell2005.PDF
@ All
Far as safe programs, remember that there’s also substantial work in preventing attacks on programs via automated transformations. We should continue to invest in these given it’s push-button safety against the stuff most likely to mess us up. The best right now is probably Softbound + CETS:
http://www.cs.rutgers.edu/~santosh.nagarakatte/softbound/
Any approach like that has a heavy hit. However, it would be fine for many applications on today’s processors, server farms, and so on. However, if you need more performance, there’s more lightweight strategies such as Code Pointer Integrity with 1-10% hit on supported platforms.
http://dslab.epfl.ch/proj/cpi/
I endorse using and further developing tools like these as the default way to deploy C in security-critical settings. I mean, there’s simply so many ways to shoot oneself in C that even pro’s regularly screw up. A nice compromise is defaulting on the strongest protections with a way to turn them off for specific modules that are trustworthy. So, you might rework the fast path until it passes Astree Analyzer and everything else you throw at it. Its assumptions are encoded into contracts for development phase. Then, in production, you can turn off unnecessary checks for it for a performance boost because you know it doesn’t need them. Can use this approach to incrementally improve the app until it uses just the right amount of protections.
Thoughts?
> But it allows other static tools to enforce this. And your C-level code might well make that same assumption
The keyword there is static. The majority of these sorts of problems happen at runtime, dynamically allocated arrays / pointer arithmetic with statically unknown values, etc.
Interesting bit about the Julia runtime and femtolisp. It reminds me of the fact that the Scheme48 VM — much like the Squeak VM — is written in a subset of the language environment it’s supposed to support. This subset, called Pre-Scheme in Scheme48’s case, compiles neatly to readable C. I used it to write the screen-drawing hot path for a game engine I wrote, and got respectable performance even out of aging hardware.
Am I the only one who remembers that CVSup had serious problems getting any traction at all because it was written in an oddball language?
Like it or not, C and C++ are the standards, and niche languages like Rust and D and Julia have a long way to go to gain any sort of mindshare.
@ Jeff Read
PreScheme was also used in the VLISP project that did rigorous verification of Scheme48. A good tool for correct-by-construction approach to other tools (esp compilers) down to x86, PPC, and ARM which the work supported. Cool that you used it in a game.
@ Jay Maynard
Oh, I remember every time I look at the source of anything I use. Gabriel’s Worse is Better taught us well about the network effects of half-assed tools. Stayed true all this time. Hell, your point even supports COBOL and PHP use to large degrees. Still staying away from them. So, back to C/C++, my interim solution was better languages/tools that compiled to C or C++ for distribution. Long term approaches without that are a long way to traction as you pointed out.
However, there’s been a recurring counterpoint that has some merit. That is that the obscure languages could get lots of traction fast if one or more killer apps were written in them. The better ones are usually written in the popular languages instead. Enough attempts using better stuff might lead to a larger niche userbase of one of the better, but obscure, languages. So, there’s something people might keep trying. Still a long shot, though.
In the new world, LLVM is the standard and C and C++ are just relatively thin layers on top of that, which could easily be swapped out for any other layer (D, Rust, etc.) even while maintaining ABI backwards compatibility. We already see Apple headed in this direction — phasing out Objective-C in favor of Swift.
> In the new world, LLVM is the standard and C and C++ are just relatively thin layers on top of that
I don’t follow. LLVM had been designed and is currently developed primary for C and C++. It’s not as wonderfully general as people often claim. It has certainly made creating certain types of new languages /easier/. You say easily, but it’s not that straight-forward. C++ is not even close to a thin layer. Using it for languages with very different semantic models is even less straight-forward. Haskell has an LLVM backend for example, but last I checked it was a long chalk behind the custom C– backend for that type of programming language.
Swift is an interesting example. That was started in 2010 by Chris Lattner, the creator of LLVM, and later worked on for several years by members of the the Apple LLVM group, who as we know are one of the power-houses in LLVM development, with a great deal of compiler development experience. That’s 5 “expert” years in a huge company with lots of money to develop a language that while certainly possessing many novel merits, attacks a lot of the same design surface as C++. And it’s far from “complete”. This ain’t an area where Joe or Jill Random can take LLVM off the shelf and be competitive with their new ideas without a massive amount of work.
> even while maintaining ABI backwards compatibility
What do you mean? LLVM has no stake is ABI standards. It implements several, but it isn’t one itself. D can talk to C, but Lord help you if you want to talk to C++ in the general case.
Like everything in life, there is no free lunch here. You would need to put a massive amount of work into LLVM to gain mindshare with some “out of the box” language.
>Swift is an interesting example.
I forgot to mention as another point about this, in Swift’s case, they designed an entirely new IR (SIR) which sits on top of LLVM’s IR to model their new language more appropriately. That is no small add-on, and we know it’s completely necessary because the people who designed the LLVM IR designed SIR!
LLVM is a fantastic set of libraries, but it’s no silver-bullet for this problem we’re discussing.
>I once proposed supporting inline LLVM bytecode in a language for a portable assembler for optimization reasons. What you think on that one?
By byte code I suppose you mean IR? Either way, both the IR and the byte code are inherently platform specific, i.e., not portable at all. This post is a classic explanation for this common misconception:
http://lists.llvm.org/pipermail/llvm-dev/2011-October/043724.html
@ Jay Maynard
C and C++ may be the standards, but there has been enough bright people pointing out their shortcomings and actively trying to build tools that are more secure and better designed.
After looking at the alternatives, Rust may be that choice for me. With FFI interfaces to C, C++ and Python, nice documentation and a decent community, I’ ready to try it for awhile. Maybe build the responsive, time sensitive code in Rust and use Python as a scripting language for user interfaces. Maybe I can slowly replace the scrambled spaghetti C code with some maintainable logic before I drown my brain cells in locally produced rum.
>After looking at the alternatives, Rust may be that choice for me.
I am also finding Rust more attractive than its rivals. Haven’t made the jump yet, though.
@ Charlie
Thanks for the excellent link! Provides every reason in the world that I should never mention or even consider that possibility again lol.
Apple should assign a couple of interns to rewrite ntpd in Swift.
>Apple should assign a couple of interns to rewrite ntpd in Swift.
I assume you were mostly joking, but..no. Not unless those interns happen to be expert time metrologists who are already intimately familiar with at least one implementation.
Even leaving out the time-sync algorithms the network plumbing is pretty formidable. I won’t say you’d absolutely need someone as capable as me to lead a rewrite, but you’d best not try it with someone much less capable.
Charlie,
>What do you mean? LLVM has no stake is ABI standards. It implements several, but it isn’t one itself. D can talk to C, but Lord help you if you want to talk to C++ in the general case.
>Like everything in life, there is no free lunch here. You would need to put a massive amount of work into LLVM to gain mindshare with some “out of the box” language.
Not sure if you saw this, but a different approach towards interfacing D with C++ is under way. Clang being open source and the existence of the llvm-based D compiler ldc enable it, and it doesn’t appear to have required “a massive amount of work” yet:
http://syniurge.blogspot.com/2013/08/calypso-to-mars-first-contact.html
Of course, there’s always that last 10%. ;) If it’s not obvious, my bet’s on D.
Way off-topic, but posting here so ESR can read it, file it, and delete it.
“I just connected my Ubuntu phone to my projector, coupled a keyboard and a mouse via Bluetooth and it turned into desktop. I haven’t been this excited since I got to build my first supercomputer. The endless possibilities! The fun! The perspective of no longer having to carry a heavy laptop around!”
https://plus.google.com/u/0/102486542947898431342/posts/5ueteCgo3r2
Everything Is Proceeding As I Have Foreseen
@Random Observer –
> Apple should assign a couple of interns to rewrite ntpd in Swift.
Here’s another explanation of why not:
http://www.joelonsoftware.com/articles/fog0000000069.html
Eric may have been simplifying all the cruft and hair out of the existing codebase, but he didn’t throw away “all those collected bug fixes” that meant anything useful to the code. A new implementation (any language, any authors) would be full of new bugs.
I think ESR has said something similar about not rewriting cvs-fast-export(? – someone correct me if I’ve got it wrong) because of the hard-won real world experience embodied in it.
This, of course, assumes you’ve inherited something that works (such as ESR has) – not something that is bug-laden, badly-designed, possibly-undebuggable (removing 1 bug introduces 1 + epsilon bugs, epsilon strictly > 0), and needs to be strapped to a nuclear device which is subsequently initiated (I took over such a shipment of fail – and gleefully nuked it when its limits became apparent).
>I think ESR has said something similar about not rewriting cvs-fast-export(? – someone correct me if I’ve got it wrong) because of the hard-won real world experience embodied in it.
That is correct. In fact, now I think of it, the parallel is quite close.
For similar reasons, trying to rewrite GPSD from scratch would be a very bad idea.
@Marco
I think ESR wrote about this being the future a couple of years ago.
>I think ESR wrote about this being the future a couple of years ago.
Yes, I did. I could rightly say that all is proceeding as I have foreseen.
@Marco: I’ve been expecting this for a while. Hardware gets steadily smaller, faster and cheaper. Many folks already use their smartphone as their main device.
MS recently issued a new phone through the part of Nokia they bought that is designed to be plugged into a docking station with KB, mouse, monitor, network and NAS access, and become your principal machine. It doesn’t cover all bases, but it’s likely to handle about 80% of what most folks do with a computer.
My cell phone is deliberately the smallest, cheapest, dumbest model Samsung makes. All it does is calls and SMS, and that’s all I want it to do. The rest is something else’s job. The something else is increasingly an Android tablet, because most of what I might do really wants a larger screen than a practical phone can have. Add a portable KB, and the laptop is increasingly superfluous. It doesn’t do everything I do on the desktop, but it does what I need to do when traveling.
I don’t see myself getting the MS model or a Ubuntu phone for the same reason I have a dumb phone – I need a different form factor. But the fact that you now *can* do stuff like that is something I’ve anticipated for years.
______
Dennis
> Add a portable KB, and the laptop is increasingly superfluous
When you have a big enough tablet screen, and a big enough keyboard, then what you have is a (very top-heavy, since the battery is behind the screen rather than under the keyboard) laptop. Microsoft has a product line of these, I believe.
Ultimately, laptops that have been available for years are already pushing into the region of “smallest keyboard and screen I am willing to put up with”.
What might be nice, though, would be a way to put all the computing power and the battery (maybe even 2-3x the weight of a typical smartphone) in a separate box, and have a dumb screen/keyboard combo (same form factor as a laptop) as a peripheral that can be connected to it with a cord.
> Ultimately, laptops that have been available for years are already pushing into the region of “smallest keyboard and screen I am willing to put up with”.
But there is market for even 17″ laptops.
“esr on 2015-10-26 at 00:07:51 said:
>I’m guessing the similarity to the OpenBSD boys coding to a “sane target” (their words on eg LibreSSL) and adding portability shims for the gubbins missing on a particular platform is essence, not accident, as you’re tackling (afaict) the same problem?
Yes, I would say the parallel is quite real. They could use the hell out of me on that project if I weren’t so busy and would prefer root canal to having to deal with Theo de Raadt for any length of time.”
You realize, that OpenBSD, already wrote their own NTP daemon ages ago, right?
No, maybe you were too busy assuming Theo is an asshole. Have you met the fellow in person? I have! He is actually quite personable in person, yes, I agree that he can be an asshole online and does not suffer fools gladly, particularly on mailing lists for projects he actively works on, which he does so, without forcing himself into someone’s office or presence.
If I were to assume things based solely upon your code and writing, it would be *far* less gracious than what you have said about Theo, so I will not go too deeply into that for the time being.
“I report that the codebase is now down to about 43% of the size we inherited – in absolute numbers, down from 227KLOC to 97KLOC.”
“> # cd /usr/src/usr.sbin/ntpd/
> # for i in $(find . -name “*.[ch]”); do cat $i >> /root/allcode; done
> # egrep -v ‘[:blank:]*/?\*’ /root/allcode | grep -v “^ *$” | wc -l
> 2898
….
> > $ for i in $(find . -name “*.[ch]”); do cat $i >> allcode; done
> > $ egrep -v ‘[:blank:]*/?\*’ allcode | grep -v “^ *$” | wc -l
> > 192870
> >
> > This is ntp-4.2.8 A rough estimate but close enough if we are comparing to
> > a know solution that is <5000.
That is a factor of 66x. Shocking, it is larger than I remembered."
(the 2898 LoC ntp implementation being referenced was none other than http://openntpd.org/; clearly, the NTP daemon referenced at the time was not quite so big as the 227KLOC point you started at [why did it swell another 27KLoC in a mere 12 months from this post: http://marc.info/?l=openbsd-tech&m=141905854411370 would be a better question] but it is still *ORDERS OF MAGNITUDE* smaller than your currently claimed 97KLoC rewrite)
I know others have disparaged you in the past, and I don't really mean to make ad hominem attacks, but can you please; PLEASE, look at the numbers? Programming is at least, partially, derived from mathematics, which is what makes is useful and problems such as these tractable, so don't turn a complete blind eye to them.
Moreover, where is your code? Is there a repository somewhere? You haven't mentioned it here, or near as I can tell elsewhere in your posts on the subject. Don't get me wrong, I am OK with folks doing development in private, I do so myself, but I also don't actively blog about self-congratulatory things while doing so, when simultaneously not showing the code.
Let's compare what you're writing about, to another NTP alternative, TLSDate:
%git clone https://github.com/ioerror/tlsdate
Cloning into 'tlsdate'…
remote: Counting objects: 3998, done.
remote: Total 3998 (delta 0), reused 0 (delta 0), pack-reused 3998
Receiving objects: 100% (3998/3998), 1.53 MiB | 1.42 MiB/s, done.
Resolving deltas: 100% (2182/2182), done.
Checking connectivity… done.
%cd tlsdate/
%git ls-files | xargs cat | wc -l
19881
Not quite as small as OpenNTPD, but still certainly *much smaller* than your ongoing rewrite.
I do not mean to disparage your work, but I am certainly not going to compliment it (indeed, at the moment I cannot see it, so I can truly do nothing other than judge your work based upon what you have written *about* it thus far). You should take your attitude, comments about Theo and security and bite your tongue. People who actually give a shit about stuff like this, already saw there were problems, YEARS ago, and *did* something about it, the code is out there, today, for others to use.
Not with funding, not with patreon sponsorship links, not with endless blog posts and self congratulatory hype.
You are coasting on some laurels that I don't think were ever merited, and you are being called out, just a little bit.
Good luck with NTPSec, the world that cares, was already using something different, some time ago.
Believe me, I, like you, need income. Unlike you, I don't seem to be blogging about projects which even when they're done, will probably not be terribly useful or widely used.
But, who knows, maybe this NTP rewrite will fold space and time in ways that skape never foresaw with other worthwhile research into how time is parsed on computing: http://hick.org/~mmiller/presentations/toorcon05/toorcon05.pdf
If so, it may still be interesting; exploitation vectors have all sorts of wonderful unintentional properties.
>You realize, that OpenBSD, already wrote their own NTP daemon ages ago, right?
Yes. I was speaking of LibreSSL as the project on which I would be very useful, not OpenNTPd.
>No, maybe you were too busy assuming Theo is an asshole
I’m afraid it’s not an assumption, it’s experience.
>but can you please; PLEASE, look at the numbers?
I already have. I used David Wheeler’s sloccount tool to measure the line count at the fork point. I’m using it to measure NTPsec as it shrinks. It’s not going to shrink much more, I don’t think.
>Moreover, where is your code? Is there a repository somewhere?
There is. We will expose it soon. A principal reason we have not already is complications surrounding embargo requirements on vulnerabilities. I have no control over those, but we at NTPsec have been pushing hard for early disclosure.
>Good luck with NTPSec, the world that cares, was already using something different, some time ago.
CII doesn’t think that’s true. They’d hardly be funding the work if they did. We also have strong interest from major cloud providers such as Cisco and others I won’t disclose.
What I think you fail to understand is that the NTP userbase is very conservative, especially at Stratum 1. They prefer the devil they know and would much rather have a cleaned-up version of it than any of the alternatives. Perhaps this wouldn’t be the case in an ideal world where everything was judged on pure technical merit, but alas we do not live in that world.
> But there is market for even 17? laptops.
I’d say “And there is also a market for 17″ laptops.” Laptops have the range covered, from the smallest keyboard & screen Alice is willing to put up with for the sake of high portability, to the largest keyboard & screen Bob is willing to lug around for the sake of comfortable use.
OK, I understand, it is not an ideal world.
Some of the “name drops” you have made, like David Wheeler, and Cisco speak to a bygone era, before SDN and ilk.
Perhaps you “stratum 1” folks are too conservative, take this as a chiding from a generation that exists from a different realm in space time, where UTF-8 is 20+ years old, and the fact that your blog fails to properly render that is a pox on your ability to even run a globally relevant, let alone Intergalactic Computer Network relevant sensibility.
In the multiverse in which I exist, time and space; are much more malleable, as well as having to contend with retrocausality paradoxes which are *more* than theoretical, they are empirical.
On my person, I typically keep two chronometers, one synced to UTC, the other is a mechanical pocket watch which does not recharging, but can last on a single battery for well over a year.
When contending with the administration of networks and systems, people who claim to do F/OSS work in cloistered private repositories are ne’er-do-wells (in the parlance of your era I glean).
The universe moved on.
Proprietary forks of open source cause more headaches.
You mention Cisco, for an “eon” (perhaps you might utter?) they did not even consider plaintext telnet as a bad thing, meanwhile, when they forked OpenSSH, they continue to stumble across problems, of their own creation, not ones extant in the codebase they forked it from:
http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20150923-sshpk
Again when you scribe “I’m afraid it’s not an assumption, it’s experience.”
Check your ego.
What you may be misperceiving as being an asshole, is actually an individual, and a team of competent, younger, candidly, *better* hackers than you and your kin, getting upset at the deluge of folks who do not even code wondering how to do basic things with unix.
They continue to improve projects, be it OpenSSH, LibreSSL and more.
When you write “They could use the hell out of me on that project if I weren’t so busy and would prefer root canal to having to deal with Theo de Raadt for any length of time.” No, they could not use the hell out of you.
Indeed, most folks I have interacted with in most communities programmatically are grateful to not interact with you to the maximum extent possible.
It is a “free” internet, mostly; but if you don’t want to invent a future that is that ideal world where everything is judged on pure technical merit, then you are instead, doing an active disservice to the rest of the planet that does not share in your false ideals. You have created your own “good old boys” network and you profit and benefit from it to the ruination of many, and you do not even seem cognoscente of that; but I think that was probably evident when you took it upon yourself to usurp the jargon file.
There are alternatives listed above, by me, they are all, substantially smaller codebases.
I will guarantee they even still have bugs and holes that can be exploited, as that is the lamentable reality of all things.
You can, and presumably will, continue forward with your project, but I would hope others reading this can perhaps see another path which doesn’t involve more attrition than meaningful code for something as straightforward as horology and metronomes. You can distort space time if you like with pantheons of bloat, but that is not the ethos of a virtuoso.
I prefer a 17″ laptop for real work, even if it stays on my home desk most of the time. Smaller devices are suitable for light reading and editing, but having the power, speed and storage to spin up virtual machines or crunch a few billion records in a timely manner makes it worth-while.
After some research, I think Rust deserves some further attention. Of all the newer languages, it seems to be the only one serious enough for true systems programming that has a decent chance of gaining a foothold. I’ve scheduled some time over the next month to work thru the tutorials and try some small projects. (I’m not going to port NTPsec to Rust but there’s some smaller more trivial C projects that need an enema.)
> Let’s compare what you’re writing about, to another NTP alternative, TLSDate
TLSDate is not a NTPD alternative, is a hack replacing rdate (the client side of NTP).
What about those of us with a Centris 680 with floppy drive, one external hard drive, only network connection a 10baseT cable, running Mac OS 7, that still thinks it’s 1996 due to backdating in 1999 to avoid Y2K?
(Actually, why do we keep that around? It’s not like it’s useful. Still works, though.)
A computer that was backdated in 1999 to avoid Y2K should have been backdated to 1975 (where 2000 = 1976), and should presently think it’s 1987. I don’t know what use you could get out of a computer that thinks that it’s a leap year and that today (Oct 28) is a Monday.
The explanation for OpenNTPD’s size is actually pretty simple: by design, it does fewer things than ntpd and is a lot less accurate. That’s fine for some use cases, not so much for others.
>The explanation for OpenNTPD’s size is actually pretty simple: by design, it does fewer things than ntpd and is a lot less accurate. That’s fine for some use cases, not so much for others.
Thank you, I didn’t know about the lower accuracy.
I wanted to compare sizes and project history of OpenNTPD, Chrony, ntpd (classic) and ntimed on OpenHub… but only OpenNTPD is here (added by someone)… and it turns out that ntimed has currently only NTP client, and server is planned to be done in 2016.
https://www.openhub.net/p/_compare?project_0=OpenNTPD&project_1=chrony
Found another comparison of Chrony, ntpd (classic) and OpenNTPD on Chrony site. Among others, only ntpd supports broadcast and unicast… Also OpenNTPD support SNTP and not NTP (why it is not called OpenSNTPD?).
http://chrony.tuxfamily.org/comparison.html
>ntimed has currently only NTP client […] only ntpd supports broadcast and unicast… Also OpenNTPD support SNTP and not NTP
The competition is weaker than I thought.
>> Also OpenNTPD support SNTP and not NTP
Note that this information is from Chrony project (the comparison page). The OpenNTPD project itself states that it implements Network Time Protocol daemon (NTPv3, NTPv4).
Though OpenNTPD developers stated that it trades off accuracy of time source for simplicity of implementation.
> The competition is weaker than I thought.
On of things that Chrony has is better support for intermittent connection. That probably does not matter for servers, but might matter on mobile and laptops…
>On of things that Chrony has is better support for intermittent connection.
Yes. And a different theory of operation. Where ntpd aims to track a moving statistical average of servers after pruning falsetickers (outliers with large jitter), chronyd want to pick one best time source and track it as closely as possible. Both theories are defensible.
Well, it wasn’t backdated all that far, and I think there’s no internal clock battery to keep the clock running, so it doesn’t keep track of the date; it probably thinks it’s May or something (we last booted it about three years ago, didn’t adjust anything, and it worked fine). Honestly, all that’s on there is a few old photos and miscellaneous images; we’d need a floppy disk and a computer with a floppy drive and more modern connections to get them off, but fortunately the computers at my old high school (I graduated) all have floppy drives (except the laptops).
Again, check your ego.
rdate replacements are still useful, no?
SNTP is still useful, no?
You want something larger than OpenNTPd, but smaller than tls-date, or the monstrosity of NTPd, and this alleged rewrite (with no source or repository for others to verify).
Try https://github.com/bsdphk/Ntimed
5211 LoC.
Written by phk (who, based on all past contributions, by my estimation is a far better programmer than most will ever become).
I am still failing to see why I, or anyone else would want to use your purported rewrite. There are already saner alternatives, which can be used, today.
>Again, check your ego.
You don’t pay very good attention, do you? I was hired to work on NTPsec. For actual money. The project fork was not my idea or my decision and is certainly not a result of any ego striving on my part.
My ego does demand that having accepted the job of technical lead, I will deliver a better NTP implementation than anyone has ever seen or used before. But it would have demanded the same if I had been hired to work on any of the alternatives.
>Try https://github.com/bsdphk/Ntimed
If you want me be interested in Ntimed, pay me a fuck-ton of money. Or, at least, more than I’m being paid to work on NTPsec.
The truth is, I don’t much care which time-service codebase I turn upside down, shake out, and improve the hell out of, as long as the infrastructure problem gets solved. I’m quite willing to work on one of the others if you PAY ME MORE TO DO IT.
Am I shouting loudly enough to get through to you? It’s not about my ego, it’s about fixing time service and MAKING MONEY DOING IT.
>I am still failing to see why I, or anyone else would want to use your purported rewrite. There are already saner alternatives, which can be used, today.
No. No, as multiple commenters on this thread have observed. there aren’t. Your false beliefs on this score are not my problem. Especially not when I’m GETTING PAID to fix the real problems. In MONEY.
Idiot.
> You want something larger than OpenNTPd, but smaller than tls-date, or the monstrosity of NTPd, and this alleged rewrite (with no source or repository for others to verify).
>
> Try https://github.com/bsdphk/Ntimed
Errr… according to the ntimed homepage (http://nwtime.org/projects/ntimed/) it does _*NOT*_ include actual NTP *server*. It is *planned* to be done in 2016.
Also, the hardware people are conservative, as Eric said already. They would accept NTPsec being a refactoring of NTPd reference implementation of NTP daemon, not so with rewrite like Chrony, or planned ntimed-master (possibly repeating errors fixed during the like of NTPD classic).
>Also, the hardware people are conservative, as Eric said already.
I’m not actually guessing or theorizing about this. I’m being paid by CII, which is an industry consortium of companies with huge data centers for whom NTP is vital infrastructure. They have a specific problem – NTP Classic is poorly-maintained software with serious security holes – for which they want a specific solution, which is a cleaned-up and hardened version of the same codebase.
They don’t want SNTP-only. They don’t want a warmed-over rdate clone. They don’t want an experiment that won’t be fully available until 2016. They don’t want any compromises in accuracy. They want the software their system administrators already know, fixed so it doesn’t suck.
The NTPsec team will deliver this. Soon. Very soon.
Actually, wasn’t the NTP Classic meant to be *reference* implementation, and not production-ready one?
One project that could be used in place of NTPsec, with full NTPv4 support and not sacrificing precision for simplicity of code is (no compromises), I guess, Chrony… but it is still not a straightforward feature-for-feature replacements, and it looks like the did some different (if equally valid) assumptions.
>Actually, wasn’t the NTP Classic meant to be *reference* implementation, and not production-ready one?
It is possible that was Mills’s original intention. But he certainly worked hard (and very successfully) at getting the ntp_gettime/ntp_adjtime hooks emplaced everywhere, and it is certainly the case that NTP Classic became and remains the dominant production implementation.
In the discussion of programming language alternatives to C, I note this Slashdot story with interest. It would seem that the Go community is trying to consciously turn itself into a warm, fuzzy safe space, and run off people who don’t behave in a politically correct way.
I suspect that this will result in programmers leaving the project for other efforts, and stagnation…
@esr
“They want the software their system administrators already know, fixed so it doesn’t suck.”
I do think this is not appreciated enough. Software that is known to do what you need it to do always beats software that might run better, but also might not do what you need it to do.
Maybe it is possible to have a way to quantify the amount of debugging that has taken place?
@Jay
It doesn’t matter what the Code of Conducts say, good code speaks for it’s self. Except for one or two programmers, most of these people pushing these agendas don’t code or constructively contribute to projects. (If people wish to boycott projects that don’t have some PC Code of Conduct, I suggest they also boycott technical advances like GPS and satellites, since the space program was greatly assisted by actual Nazi rocket scientists.)
Where does the line form to participate in the unit testing?
A trackback… https://github.com/madler/zlib/issues/117#issuecomment-157796877
I brought up removing the old school OF macro from K&R and dropping the #define local static to allow dropping static for debuggers that are too brain dead to see static functions. In being asked why, I pointed to this post.
Even and I have been putting a lot of work into hardening/cleaning GDAL… lots of overlap of ideas and processes with the GPSD and NTPSec work
http://erouault.blogspot.fr/2016/01/software-quality-improvements-in-gdal.html
Since this topic has caught fire on NANOG this week, I wonder if you have any updates on how things are coming, over all, Eric.
Notwithstanding where they are in the queue, are there any plans to extend the package to things like authenticated connections down the road?
>Since this topic has caught fire on NANOG this week, I wonder if you have any updates on how things are coming, over all, Eric.
We’re achieving our technical goals. NTPsec is far more security-hardened than Classic; as I noted recently. when the most recent batch of 11 CVEs issued, we’d dodged 8 of them due to having previously removed bad code.
Much work remains to be done. NTF doesn’t have the capability to do it. We think we do.
>Notwithstanding where they are in the queue, are there any plans to extend the package to things like authenticated connections down the road?
Yes. We’ll do NTS when it’s in a good state to tackle. That isn’t yet, there are serious design issues with how to do a secure handshake when you can’t yet trust your clock.