NTPsec dodges 8 of 11 CVEs because we’d pre-hardened the code

While most of the NTPsec team was off at Penguicon, the NTP Classic people shipped a release patched for eleven security vulnerabilities in their code. Which might have been pretty embarrassing, if those vulnerabilities were in our code, too. People would be right to wonder, given NTPsec’s security focus, why we didn’t catch all these sooner.

In fact, we actually did pre-empt most of them. The attack surface that eight of these eleven security bugs penetrate isn’t present at all in NTPsec. The vulnerabilities were in bloat and obsolete features we’ve long since removed, like the Mode 7 control channel.

I’m making a big deal about this because it illustrates a general point. One of the most effective ways to harden your code against attack – perhaps the most effective – is to reduce its attack surface.

Thus, NTPsec’s strategy all along has centered on aggressive cruft removal. This strategy has been working extremely well. Back in January our 0.1 release dodged two CVEs because of code we had already removed. This time it was eight foreclosed – and I’m pretty sure it won’t be the last time, either. If only because I ripped out Autokey on Sunday, a notorious nest of bugs.

Simplify, cut, discard. It’s often better hardening than anything else you can do. The percentage of NTP Classic code removed from NTPsec is up to 58% now, and could easily hit 2/3rds before we’re done,

32 comments

  1. So what CVE’s you didn’t dodge? And what could be done to protect against them?

    Sidenote: WTF is the “N” tag?

    1. >So what CVE’s you didn’t dodge? And what could be done to protect against them?

      Oh, they’re stopped now. One of them was really silly, it was a hypothetical hole where if an attacker had access to your server’s local net and the OS was configured slightly wrongly. then the attacker could spoof refclock packets. Another was an assertion crash on an error reading control variables. The third was a missing state-flag check on changing to an obscure protocol mode. All inherited from Classic, no known exploits, and I’m doubtful they were even exploitable.

  2. At the bottom: “This entry was posted in Software and tagged NTP, N by Eric Raymond. Bookmark the permalink.”

  3. Your recent blog on DSLs has had me thinking on the general process of taking a specific solution and making it more general by adding variability in different dimensions – thanks for that.

    Wouldn’t it be useful to generalize the concept of “attack surface” to “bug surface” and use it to argue for parsimony in all code bases whether they are security related or not?

    1. >Wouldn’t it be useful to generalize the concept of “attack surface” to “bug surface” and use it to argue for parsimony in all code bases whether they are security related or not?

      Yes, I think it would be. You can view the variability of real-world data and user behavior as a sort of randomly-generated DOS attack. There’s a Dijkstra quote about this, but I’m not recalling the exact wording.

  4. I have grown to to code by the rule:
    Every line of code is a potential vulnerability.

    (Or a potential bug, which is often the same)

  5. > There’s a Dijkstra quote about this, but I’m not recalling the exact wording.

    Found on number on Wikiquote – the shortest:

    Simplicity is prerequisite for reliability.

  6. >>At the bottom: “This entry was posted in Software and tagged NTP, N

    > Just a typo. Removed.

    No it isn’t.

  7. All inherited from Classic, no known exploits, and I’m doubtful they were even exploitable.

    CVE-2016-1548, the state flag one, can definitely be exploited to manipulate your clock.

    1. >CVE-2016-1548, the state flag one, can definitely be exploited to manipulate your clock.

      Daniel did the work on plugging these holes, so he’d know,

  8. This approach applies to systems engineering, too. Any service you’re not running is one that you can’t misconfigure; any server you can find an excuse to turn off is one that can’t be attacked.

  9. A: That’s the reason I’m greatly in favor of Gentoo for Internet-facing servers. It gives you exact control over what is installed, and if it’s not installed, it can’t be cracked.

  10. I’m reminded of the time someone told me that the cheapest, most reliable components are those that aren’t there. (I think they were quoting Gordon Bell?)

  11. >You can view the variability of real-world data and user behavior as a sort of randomly-generated DOS attack

    And code fuzzing tests are an attempt to accelerate and automate that randomly generated attack.

    Coverity is already donated fuzzing tests to NTPsec.

    When the CII-funded Fuzzing Project gets a bit more usable, we’re going to add that to the NTPsec tests as well.

  12. @Nick:
    > Wouldn’t it be useful to generalize the concept of “attack surface” to “bug surface” and use it to argue for parsimony in all code bases whether they are security related or not?

    Well, as far as the computer is concerned, there’s no such thing as a security bug: either the program behaves identically to its specification, or it does not. Security comes at the human level, any bug that allows one human to screw over another is a security bug, so I’d say “yes”.

    1. >BTW. does NTPsec pass CII best practices badge criteria

      Most of them. We need tp improve in the testing area.

  13. You can’t have bugs in code that isn’t there.
    The comment about Gentoo makes me nervous. In order to support or not support particular features you need to have conditional compilation. I worry that this means that bugs won’t be found in the tested configuration, but may be present in running configurations.

    1. There is a small chance that leaving out chunks of a program introduces bugs that weren’t there in the full version, but generally speaking if a programmer bothers to make a chunk optional it’s because somebody wants to leave it out and so it’s probably been tested.

      If you’re concerned, just tell the build system to build with the standard options. The power of source-based distros is that you get to choose such things without oodles of extra work.

  14. Eric, a quick query about semantics. Is it “hardening” to remove cruft from bloatware so as to eliminate avenues of attack? My experience with “hardened” software is that someone has added bloat and called it a firewall.

    1. >Eric, a quick query about semantics. Is it “hardening” to remove cruft from bloatware so as to eliminate avenues of attack?

      That’s one kind of hardening. There are others.

      >My experience with “hardened” software is that someone has added bloat and called it a firewall.

      You need to hang around with more competent people. Oh, wait, that’s probably why you’re here.

  15. “That’s one kind of hardening. There are others.”

    It seems pretty analogous to the famous dichotomy from C. A. R. Hoare’s Turing Award lecture: “There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.”

    1. >It seems pretty analogous to the famous dichotomy from C. A. R. Hoare’s Turing Award lecture:

      AHA!

      That’s the exact quote I was trying to recall. I had it misfiled under Dijkstra.

  16. OT…but:

    @Jakob Narebski: A copy of your book, ‘Mastering GIT’, is available for download as a .pdf file on it-ebooks.info. I don’t know if you want this, or not.

    @esr: Please remove this posting if he doesn’t want this.

  17. “Well, as far as the computer is concerned, there’s no such thing as a security bug: either the program behaves identically to its specification, or it does not.”

    That sounds as if you might be choosing at least one of your definitions in a idiosyncratic vacuous way, rather like saying “as far as the chip is concerned, there is no such thing as vulnerability to a high-radiation environment: either the chip behaves according to its specification, or it does not.”

    The class of things that can go wrong when you have an actively malicious adversary can be qualitatively larger than merely facing ordinary input, in a way that seems somewhat analogous to the class of things that can go wrong being larger when a high radiation flux is trying to flip bits and degrade materials. Malicious adversaries can hammer your system with bizarre load patterns crafted to defeat your hash function and flushing heuristics when they guess that you might be running on a particular type of memory hardware, and destroy you if they manage to force a bit of key information to leak every few minutes through the response time anomalies that follow from the refresh cycle behavior on its hardware row boundaries. Jokes about Murphy’s law may be ha ha only serious, but in the absence of malicious intelligence in the loop, that kind of phenomenon is so astronomically unlikely and/or such an astronomically small anomaly that you don’t actually need to design for it because even though in principle it *can* arise by random chance, in practice it just won’t, at least not before your site has been bombarded from orbit by at least 1e213 different dinosaur-killer asteroids.

  18. TomA,

    I know it’s a bit late to the party, but in addition to nuking bloat from orbit (it’s only way to be sure), other avenues of code hardening that ESR has mentioned using include sane string handling functions that guarantee strings actually terminate, checking invariants (although that could open up a DoS avenue), and generally not trying to outsmart yourself.

  19. checking invariants (although that could open up a DoS avenue)

    Indeed, we’ve dealt with multiple CVEs this year of the nature that a malformed request can crash ntpd by triggering an assertion failure. But in every case, this was less bad than what would have happened had the assertion not been there (usually memory corruption some sort), and in every case the fix has been to return an error to be handled by the calling function rather than just crashing.

  20. Nb. I have recently stumbled upon a Frama-C tool, where you can define assertions (in specially formatted comments) using ANSI/ISO C Specification Language (ACSL), and have it prove correctness… of this assertion.

Leave a Reply to tlhonmey Cancel reply

Your email address will not be published. Required fields are marked *