More scenes from the life of a system architect

Haven’t been blogging for a while because I’ve been deep in coding and HOWTO-writing. Follows the (slightly edited) text of an email I wrote to the NTPsec devel list that I I think might be of interest to a lot of my audience.

One of the questions I get a lot is: How do you do it? And what is “it”, anyway? The question seems like an inquiry into the mental stance that a systems architect has to have to do his job.

So, um, this is it. If you read carefully, I think you’ll learn a fair bit even if you haven’t a clue about NTP itself.

Today, after a false start yesterday and a correction, I completed a patch sequence that makes a significant structural change to NTP that isn’t just removing cruft.

This is kind of a first. Yes, I’ve made some pretty dramatic changes to the code over the last year, but other than the not-yet-successful TESTFRAME scaffolding they were almost all bug fixes, refactorings, or creative removals. The one exception, JSON reporting from ntpdig, was rather trivial.

[What I didn’t say to the list, because they already know it, is that the code was such a rubble pile that it actually took that year to clean up to the point where a change like this was reasonable to attempt.]

What I’ve succeeded in doing is almost completely removing from the code the assumption that refclock addresses necessarily have the special form 127.127.t.u. The only code that still believes this is in the ntp.conf configuration parser, and the only reason *it* still believes this in order not to break the existing syntax of refclock declarations.

(In fact, clock addresses do still have this form internally, but that is only to avoid surprising older ntpq instances; nothing in the NTPsec code now requires it.)

I’ve also made substantial progress towards eliminating driver-type magic numbers from the code. The table that used to indirect from driver-type numbers to driver-type shortnames is gone; instead, the driver shortname string is what it should be – an element of the driver method table – and there is only one type-number-to-driver indirection, a table in refclock_conf.c.

This is all clearing the decks for a big user-visible change. I’m going to fix the frighteningly awful refclock declaration syntax. Consider this example:

# Uses the shared-memory driver, accepting fixes from a running gpsd
# instance watching one PPS-capable GPS. Accepts in-band GPS time (not
# very good, likely to have jitter in the 100s of milliseconds) on one
# unit, and PPS time (almost certainly good to 1 ms or less) on
# another.  Prefers the latter.

# GPS Serial data reference (NTP0)
server 127.127.28.0
fudge 127.127.28.0 refid GPS

# GPS PPS reference (NTP1)
server 127.127.28.1 prefer
fudge 127.127.28.1 refid PPS

The misleading “server” keyword for what is actually a reference clock. The magic 127.127.t.u address, which is the only way you *know* it’s a reference clock. Some attributes of the clock being specified in a mystery ‘fudge’ command only tied in by the magic server address. The magic driver type number 28. The fail is strong here. The only excuse for this garbage (and it’s not much of one – Mills was smart enough to know better) is that it was designed decades ago in a more primitive time.

Here’s how I think it should look:

refclock shm unit 0 refid GPS
refclock shm unit 1 prefer refid PPS

No magic IPv4 address, no split syntax, no driver type number (it’s been replaced by the driver shortname “shm”). It should be less work to get the rest of the way to this (while still supporting the old syntax for backward compatibility) than I’ve done already – I’ve already written the grammar, only the glue code still needs doing.

An unobvious benefit of this change is that the driver reference pages are going to become a lot less mystifying. I can still remember how and why my head hurt on first reading them. Removing the magic addresses and mystery numbers will help a lot.

Along the way I learned a lot about how ntpq and mode 6 responses work. (Like NTP in general, it’s an odd combination of elegant structural ideas with an astonishing accumulation of cruft on top.) In order to remove the magic-address assumptions from ntpq I had to add another variable, “displayname”, to the set you get back when you request information about a peer. In effect, ntpd gets to say “*this* is how you should label this peer”, and ntpq uses that to decorate the clock entries in its -p output.

This has the minor downside that new ntpqs will display 127.127.28.0 (rather than “SHM(0)”) when querying Classic ntpd, which doesn’t ship that variable. Oh well…almost everyone disables remote querying anyway. It was the right thing to do; ntpq has no business knowing about driver type numbers.

(Grrrrr…Actually, *nobody* has any business knowing about driver type numbers. Things that have names should be referred to by name. Making humans maintain a level of indirection from names to numbers is perverse, that’s the kind of detail we have computers to track. Or, to put it slightly differently, “1977 called – it wants its ugly kluge back.”)

It’s easy for codebases this size to wind up as huge balls of mud. There are several nearly equivalent ways to describe my job as a systems architect; one of them centers on enforcing proper separation of concerns so collapse-to- mudball is prevented. The changes I’ve just described are a significant step in the good direction.

57 comments

  1. @esr: “There are several nearly equivalent ways to describe my job as a systems architect; one of them centers on enforcing proper separation of concerns so collapse-to-mudball is prevented.

    I’d call defining the concerns that need to be separated the core of systems architecture.

    Many code bases present ample evidence that wasn’t properly done back when the code was created. There are various reasons for that, mostly insufficient time spent on the design phase, and a good deal of the politics the system architect has to deal with come with convincing stakeholders that time spent on the design phase is necessary, and no, you can’t just jump in and start writing code.

    A clear, rigorous definition of what the code is supposed to do that will be vigorously defended is also a requirement. I think I’ve lost track of the number of places I’ve seen feature requests where the proper answer is “No. This program was created to do X. You want Y. That’s something else’s job, and will not be added.”

    With NTP, you fortunately don’t have to define the concerns to be separated, nor what the code is supposed to do.

    _____
    Dennis

    1. >With NTP, you fortunately don’t have to define the concerns to be separated, nor what the code is supposed to do.

      It’s true that we have a clear mission and no pressure for feature creep. Well, other than adding good security, which isn’t feature creep.

      However, as my post illustrates, the internal orthogonality and compartmentalization of of the architecture is not all it should be. To be fair, this is partly because practice has moved and standards are higher. Mills was, as I keep saying, a pretty damn good systems architect for his time, but his notions of best practice were those of twenty years ago. Much improvement could be made by any system architect of comparable ability today using accumulated knowledge and heuristics that Mills simply did not have.

      (I also note the possibility that I’m more talented than Mills was. I raise this not to argue for it – my best guess is we stack up roughly equal there – but to make the exact point that it’s difficult to tell, especially when individuals and the software-engineering field in general are constantly ratcheting to higher skill levels.)

  2. @esr: You commented (Like NTP in general, it’s an odd combination of elegant structural ideas with an astonishing accumulation of cruft on top.)

    I think the key qualifier is the elegance of the structural ideas. Cruft accumulating on top is almost inevitable. A goal may be reducing the likelihood cruft will accumulate, by reducing the places there will be a temptation to add cruft to handle special cases because there will be fewer possible special cases.

    But as you mention, it’s 20 years later and you are using accumulated knowledge and heuristics that Mills simply did not have. It sounds like the core code that performs the functions NTP is intended to do doesn’t really require work. The accumulated cruft needs to be pared away, and the ways in which you interact with NTP need to be made more secure, but the underlying structural ideas are still elegant.

    Have you seen places that made you say “Mills really should have foreseen this and not done it the way he did?”

    _______
    Dennis

    1. >Have you seen places that made you say “Mills really should have foreseen this and not done it the way he did?”

      The configuration language is probably the worst. It’s really inexplicably bad.

      Elsewhere…I think doing the network-sync protocol and local refclock management in the same binary was a mistake. Those ought to be separate programs communicating via an IPC channel – which is actually how it works in the very common setup of gpsd feeding GPS fixes to ntpd via the shared-memory driver.

      There are too many association modes – more than will ever be used in any real-world setup – and they complicate the protocol machine a lot. I think Mills got to some extent caught up in exploring the theoretical space, and neglected to prune the stuff that wan’t useful after he had figured out what he could do.

      Parts of the code were rather poorly factored when I got my hands on it. This is an area where practice has advanced a lot; Mills was doing well by 1995 standards, not so well by today’s. I’ve fixed several glaring problems.

  3. @esr: >Have you seen places that made you say “Mills really should have foreseen this and not done it the way he did?”

    The configuration language is probably the worst. It’s really inexplicably bad.
    That may have had the same cause as the jungle that is sendmail.cf. Eric Allman wanted it to be easy for machines to parse, not for humans to read and understand. I believe he subsequently admitted that was a mistake.

    Elsewhere…I think doing the network-sync protocol and local refclock management in the same binary was a mistake. Those ought to be separate programs communicating via an IPC channel – which is actually how it works in the very common setup of gpsd feeding GPS fixes to ntpd via the shared-memory driver.
    How common or feasible was that in 1995?

    Mills was doing well by 1995 standards, not so well by today’s. I’ve fixed several glaring problems.
    Given how long the code has been in production, it sounds like they aren’t especially obvious until you have to spelunk in it, and your fixes are a matter of making things easier to maintain if someone else has to look.at it.

    _______
    Dennis

    1. >That may have had the same cause as the jungle that is sendmail.cf.

      Probably not. NTP used yacc from the beginning, so acceptance of the performance and complexity cost of a human-readable config language was there from day one. No, the only plausible guess I have is that the config-language design reflects a private mental model of Mills’s that was just weird.

      >How common or feasible was that in 1995?

      Unix-domain sockets, FIFOs, and System V shared memory had already existed for a while. So, feasible.

      >your fixes are a matter of making things easier to maintain if someone else has to look.at it.

      Not entirely. Poor factoring and unnecessary complexity led directly to some serious security issues. That’s why I was engaged in the first place.

  4. Maybe it’s just me — so many things are just me — but isn’t there a hierarchy of reference clocks, rather than simply “prefer”?

    It seems to me the original design’s purpose was specifically to *put local clocks in the same namespace as upstream timeservers*, so that prioritization could be done across them as a universe.

    But I am young, and not much traveled.

    1. >Maybe it’s just me — so many things are just me — but isn’t there a hierarchy of reference clocks, rather than simply “prefer”?

      There’s an implied hierarcht by the clock’s declared precision.

      >It seems to me the original design’s purpose was specifically to *put local clocks in the same namespace as upstream timeservers*, so that prioritization could be done across them as a universe.

      The good: Those are separate concepts. Indeed a major feature of the original architecture was that everything looks like a peer, and all pass through the same prioritization filtering algorithms; that has not changed, nor will it.

      The bad: the config language design went through bizarre contortions to make clocks and peers appear more similar than they actually are, with results confusing for everybody.

  5. Isn’t using yacc for parsing configuration file language a bit of overkill? Also, from what little I had in common with yacc / lex, it is not easy to understand, and error recovery is atrocious. I wonder if simple custom recursive descent, or libmarpa would be better…

    1. >Isn’t using yacc for parsing configuration file language a bit of overkill?

      I’ve never found it so. This might have something to do with the fact that I wrote my first yacc grammar 30 years ago and could hack them while dead drunk, if I drank.

      Anyway, none of the tools you describe existed when NTP was being written.

  6. > >Maybe it’s just me — so many things are just me — but isn’t there a hierarchy of reference clocks, rather than simply “prefer”?

    > There’s an implied hierarcht by the clock’s declared precision.

    That sounds like More Magic, to me.

    > >It seems to me the original design’s purpose was specifically to *put local clocks in the same namespace as upstream timeservers*, so that prioritization could be done across them as a universe.

    > The good: Those are separate concepts. Indeed a major feature of the original architecture was that everything looks like a peer, and all pass through the same prioritization filtering algorithms; that has not changed, nor will it.

    They’re not separate *concepts* at all; they’re “this is a source of believable time”.

    The implementations underneath may differ, but that’s of no interest to the configurator(person), only to the configuration code. Or so I would assume.

    > The bad: the config language design went through bizarre contortions to make clocks and peers appear more similar than they actually are, with results confusing for everybody.

    So provide the abstraction more cleanly.

    It sounds like you’re destroying the abstraction, not only forcing the difference to be visible to the configurator(person), but also making it difficult, if not impossible, to interlace the various possible sources.

    Am I missing something?

    1. >but also making it difficult, if not impossible, to interlace the various possible sources.

      No.

      All sources can be freely intermixed. What has changed is that there is now a dedicated refclock directive in the config language that never mentions a magic 127.127.t.u address, though that is still generated internally to play nice with old ntpq versions. The internals of ntpd no longer require a clock address to be in this special form.

      That is, clocks are still treated just as much like peers internally as they were before, but the configuration language no longer pretends that a clock is a kind of server.

  7. [ looks at old docs ]

    My memory’s failed me; I see that the traditional layout only provides “prefer”, rather than full priority order stacking as I’d remembered; long time since I had to write one from scratch.

  8. @esr: “though that is still generated internally to play nice with old ntpq versions

    Which reminds me, you’ve stated elsewhere that this is NTP server code, and not something intended to be run on a device that queries a timeserver and asks “What time is it?”

    Do you see any of what you’re doing finding its way into client side code?
    _______
    Dennis

    1. >Do you see any of what you’re doing finding its way into client side code?

      Instantly. The ntpd implementation can function as client or server; on a conventional desktop PC it runs as a pure client. But you serve time with it too, and low-stratum sites do.

  9. @Baylink:

    The implementations underneath may differ, but that’s of no interest to the configurator(person), only to the configuration code. Or so I would assume.

    Treating a local device as a peer on a network can be confusing for the human configurator. Thus even if the underlying code makes no distinction, it can be good for the configuration language to do so.

    1. >Treating a local device as a peer on a network can be confusing for the human configurator. Thus even if the underlying code makes no distinction, it can be good for the configuration language to do so.

      This was absolutely one of those cases.

  10. @esr: >Do you see any of what you’re doing finding its way into client side code?

    Instantly. The ntpd implementation can function as client or server; on a conventional desktop PC it runs as a pure client. But you serve time with it too, and low-stratum sites do.
    Aha. I thought that might be the case, but wasn’t certain.

    On the Windows side of the dual-boot machine, I run Meinberg’s NTP Time Server Monitor
    (https://www.meinbergglobal.com/english/sw/ntp-server-monitor.htm), which looks to be a straight port of NTP reference code.

    I suspect one minor milestone will be when it gets based on NTPSec code.
    _______
    Dennis

    1. >I suspect one minor milestone will be when it gets based on NTPSec code.

      Could be a while, There are Windows port shims in the NTPsec codebase, but nobody has tried to actually build that port in the last year, and given the velocity of change during that year it seems more likely than not that bitrot has bit.

      On the other hand, the code is fully 60% smaller than it was, and the POSIX conformance of Windows toolchains has, according to rumor, been improving recently. I wish somebody would take a swing at this; it might not be very hard.

  11. > inexplicably bad

    “But the main part works, right? You’ve spent enough time on this; tomorrow you’re on this other project…”

    Maybe 20 years ago I audited some of the xfree86 mailing lists and newsgroups. “WTF IS THIS MESS?!?!” and “NUKE IT FROM ORBIT AND START OVER!” were always popular subjects…

    1. >“But the main part works, right? You’ve spent enough time on this; tomorrow you’re on this other project…”

      No, that wasn’t it. NTP was Dave Mills’s life’s work – he was obsessive about it.

  12. @esr: Could be a while, There are Windows port shims in the NTPsec codebase, but nobody has tried to actually build that port in the last year, and given the velocity of change during that year it seems more likely than not that bitrot has bit.

    I suspect it will build under MinGW and MSYS. I have no idea about MSVC. But I don’t see Meinberg doing it till you reach a production release. The rate of change in the code base would make it a moving target

    On the other hand, the code is fully 60% smaller than it was, and the POSIX conformance of Windows toolchains has, according to rumor, been improving recently. I wish somebody would take a swing at this; it might not be very hard.

    Well, the folks at Canonical have been working with MS, and an Insider build of Win10 has ports of bash and other Linux CLI tools based on Canonical developed libraries. There’s a fair amount of confusion about it – no, you won’t be running Ubuntu under Win10 – but the fact it was done is significant. (You can already get bash and friends under Windows with the Win port of git, and I have them here.)

    So I suspect it won’t be all that hard once someone tries. They just won’t have a good reason to till the code base settles down.
    _______
    Dennis

  13. >>Isn’t using yacc for parsing configuration file language a bit of overkill?
    >
    > I’ve never found it so. This might have something to do with the fact that I wrote my first yacc
    > grammar 30 years ago and could hack them while dead drunk, if I drank.

    How good is error reporting (e.g. errors in configuration file)? Is yacc used also for parsing messages, or only for parsing configuration files?

    > Anyway, none of the tools you describe existed when NTP was being written.

    Well, I think recursive descent did exist, but bison / yacc LALR parsers were more popular then (it took GCC long to abandon yacc, for example).

    1. >How good is error reporting (e.g. errors in configuration file)?

      Pretty good. You get not just line but character.

      >Is yacc used also for parsing messages, or only for parsing configuration files?

      Only for config files.

  14. Maybe 20 years ago I audited some of the xfree86 mailing lists and newsgroups. “WTF IS THIS MESS?!?!” and “NUKE IT FROM ORBIT AND START OVER!” were always popular subjects…

    All of which means that Wayland is a project that’s at least 20 years overdue…

  15. @Jon Brase: I sit corrected.

    I already have bash and friends in ports, cross-compiled from Linux code to run in Win32. Git for Windows includes an assortment. My impression was that MS and Canonical had done something similar. I was mistaken.
    ______
    Dennis

  16. > Unix-domain sockets, FIFOs, and System V shared memory had already existed for a while. So, feasible.

    Feasible, but invisible because of

    > No, that wasn’t it. NTP was Dave Mills’s life’s work – he was obsessive about it.

    Sometimes things can be right under your nose, like literally in the same book, a dozen pages away from pages that have been polished to a high shine from repeated consultation, without being seen, until or unless something comes along that shifts your worldview. Obsessive dedication to a project has some good parts, but access to different perspectives is not among them.

  17. All of which means that Wayland is a project that’s at least 20 years overdue…

    Wayland is directly dependent on Linux being the only flavor of *nix that one cares about. A windowing system that cannot run on either proprietary Unix nor the BSDs could not have been taken seriously in 1996.

    Of course, there were many projects back in the 1990s that did try to burn it all down and start from scratch. All of them, obviously, failed to replace X.

    1. >All of them, obviously, failed to replace X.

      So will Wayland.

      I say this because I know Keith Packard. As soon as he decides the challenge is interesting, he will blow right past any dozen devs Wayland can put together. And make it look easy.

  18. On the X/Wayland tangent, something that has always irked me about graphics on Unices (actually about graphics everywhere, but Unices have a design philosophy such that they should know better) is that the kernel should provide device file support for compositing. Every compositing buffer should be a framebuffer device file in the /dev hierarchy.

    I understand why this isn’t the case: RAM was too expensive when X was developed for compositing to be practical, but the loss of orthagonality irks me.

  19. Wayland is directly dependent on Linux being the only flavor of *nix that one cares about. A windowing system that cannot run on either proprietary Unix nor the BSDs could not have been taken seriously in 1996.

    Experimental ports of Weston exist for FreeBSD and DragonFlyBSD. Wayland is just a protocol; if the kernel provides the necessary infrastructure in some form or another (such as kernel modesetting and device discovery), Wayland can exist on it.

    Of course, there were many projects back in the 1990s that did try to burn it all down and start from scratch. All of them, obviously, failed to replace X.

    None of those had the support of the 800-pound distro gorilla (Red Hat) and all of the major GUI toolkit and desktop environment developers. Wayland does.

    I say this because I know Keith Packard. As soon as he decides the challenge is interesting, he will blow right past any dozen devs Wayland can put together. And make it look easy.

    I hope he gets interested soon. Xorg server 1.18 is broken when you try to run it with the fbdev or fbturbo backend on a Raspberry Pi (and the open source vc4 driver is still too flaky to actually use). It fails with the strange error “Assertion failed: key->initialized (../include/privates.h: dixGetPrivateAddr: 122)”,

    There are only two ways I managed to get X working on my Pi:

    1) use Xwayland (heh); works but flaky

    2) downgrade to 1.16 (currently I’m running the build of Kdrive from the 1.16 tree).

    I’m to understand that this problem is a subtle interaction between the X server code, fbdev, and musl (which the distro I’m running on my Pi uses). But still, the Xorg code has crossed the threshold of manageable complexity, and it takes Keith Packard level cleverness to maintain, which is why Wayland exists in the first place.

    Meanwhile, Red Hat is standardizing on Wayland — and whither goes Red Hat, thither goes Linux.

  20. >I say this because I know Keith Packard. As soon as he decides the challenge is interesting, he will blow right past any dozen devs Wayland can put together. And make it look easy.

    I am wary of the indispensable man. *Counting* on the indispensable man as part of your technical roadmap going forward seems like a bad idea. (I feel lightheaded, I just agreed with Jeff about something….)

  21. > I say this because I know Keith Packard. As soon as he decides the challenge is interesting, he will blow right past any dozen devs Wayland can put together. And make it look easy.

    And if he is hit by a bus? “I hope he gets interested soon” is one half of the problem with a project that can’t work without a single super-developer, this is the other. Your theory is also, incidentally, dependent on a presumption that he will stay loyal to X himself.

  22. @Random832″ Your theory is also, incidentally, dependent on a presumption that he will stay loyal to X himself.

    The impression I get isn’t that Packard is loyal to X. He’s a hacker working on display code and graphics rendering. He did lots of work on X back in the day because that was the way it got done on *nix, but if a better approach got presented, he might dive head first into it.

    I attended a Unix User Group meeting decades back, where the guest speaker was (IIRC), Tom Weinberg. He talked about the problems with X, and the completely different approach Bell Lab’s Plan 9 OS took to the problem. When asked what he thought might replace X, he said “I don’t know. That’s your problem! We got things to where they are now, and it’s on you to take it beyond that.”

    Wayland is an attempt to do so, and the question is whether it will get sufficient traction.
    _______
    Dennis

  23. (Grrrrr…Actually, *nobody* has any business knowing about driver type numbers. Things that have names should be referred to by name. Making humans maintain a level of indirection from names to numbers is perverse, that’s the kind of detail we have computers to track. Or, to put it slightly differently, “1977 called – it wants its ugly kluge back.”)

    The main network/systems admin at $JOB is rather proud of his system naming scheme, which directly includes the last two octets of the system’s IP address, and implies the rest of the IP address with datacenter-ID portion of the name. The above mini-rant tells me that perhaps I am not entirely crazy in my suspicion of the naming scheme.

  24. Jon Brase,

    Your proposed solution is exactly how Plan 9’s rio does it. If X and Wayland fornicated and the love child turned out to be the Franklin Richards of windowing systems, it would look an awful lot like rio. Not only does rio expose /dev/window, a copy of the current window’s framebuffer, it exposes /dev/draw to which you may write draw commands (lines, polylines, circular or elliptical arcs, bitmap or font blitting, etc.).

    The problem is because in Plan 9 “files” are the set of 9p endpoints visible to each process, it has the ability to expose a different filesystem hierarchy to each process. Linux, BSD, and Windows don’t really gave that capability. So you have to find another way to control who sees which buffer, which Wayland did.

    Something like rio is what I would like to see replace X. But you need a Plan 9 to support rio, so Wayland is what we get.

  25. Wayland is an attempt to do so, and the question is whether it will get sufficient traction.

    Not a question of if, but when at this point. When the X.Org Foundation posts to Google+ saying “future development should be for Wayland, not X11”, the smart money is on Wayland replacing X soon.

  26. @Jeff:
    >Linux, BSD, and Windows don’t really gave that capability.

    Linux doesn’t seem to have trouble exposing a different /dev/stdin or /proc/self to each process.

    In any case the compositing infrastructure I’m proposing is a windowing-system agnostic kernel component, not a function of the windowing system itself. Think more the love child of /dev/fb* and /dev/pts*.

  27. Linux doesn’t seem to have trouble exposing a different /dev/stdin or /proc/self to each process.

    /dev/stdin is a symlink to /proc/self/fd/0.

    /proc is “special” and is explicitly modelled on the Plan 9 /proc filesystem. But you cannot expect /proc semantics to apply to all Linux filesystems. Linux simply isn’t wired that way. The same goes for /sys.

    One of the huge advantages of the Plan 9 filesystem is that the entire filesystem is unique to each process. This allows relatively fine-grained control over filesystem access on a per-process basis: if you don’t want a process to have access to a particular resource, it simply won’t be there.

    Linux provides no easy way of isolating a process’s view of framebuffers to its own. Any running process could see, at a minimum, all the framebuffers of processes running as that same user. (This is true of /proc/xxx/mem and /proc/xxx/fd/* today.) Wayland does provide such isolation, though if you wish to scrape the screen contents of another process, /proc/xxx/mem is open to you if you’re running as the same user.

  28. > Linux simply isn’t wired that way. The same goes for /sys

    You could wire it that way. Create a filesystem implemented with FUSE that provides all the necessary magic, and chroot the process to it.

  29. “The main network/systems admin at $JOB is rather proud of his system naming scheme, which directly includes the last two octets of the system’s IP address, and implies the rest of the IP address with datacenter-ID portion of the name. The above mini-rant tells me that perhaps I am not entirely crazy in my suspicion of the naming scheme.”

    That depends on what the machines are used for. There are two kinds of computers, pets and cattle. Pets get human-memorable names because they have personalities. Cattle get numbers because they are interchangeable parts. Numeric names for cattle-type computers make sense: see RFC 1178, which is mostly about pet naming but tells you when not to use such names:

    Of course, they could have called the second one “shop2” and so
    on. But then one is really only distinguishing machines by
    their number. You might as well just call them “1”, “2”, and
    “3”. The only time this kind of naming scheme is appropriate
    is when you have a lot of machines and there are no reasons for
    any human to distinguish between them. For example, a master
    computer might be controlling an array of one hundred
    computers. In this case, it makes sense to refer to them with
    the array indices.

  30. On the other hand, the code is fully 60% smaller than it was, and the POSIX conformance of Windows toolchains has, according to rumor, been improving recently

    Yes. I maintain a userland C++ program which in the past had Windows-specific code, would only work on cygwin, or had a fork which replaced autotools with a Windows-compatible build system for msvc.

    We switched to CMake for a number of reasons, and now with MSVC 2015 language support is good enough that we need a couple of lines of Windows-specific definitions and everything compiles natively. Most of the changes we had to make were bug fixes for code that made assumptions that happened to be right on Linux and BSD, not adjusting code specifically for Windows.

    We now have Windows building our CI tests to prevent regressions, which was another important step since none of the maintainers regularly build on Windows. If it doesn’t get automatically built and tested, it will break.

    We’re not really doing much with the POSIX API, but compatibility is better there too and advancing fast.

    The real problem remaining for command-line programs is the lack of package managers on Windows. Like most software, we depend on other libraries and the equivalent of an apt-get line you can give to users to install all the dependencies does not exist.

  31. Fedora 25 is now shipping with Wayland as the default.

    The replacement of X11 by Wayland has begun.

  32. > The replacement of X11 by Wayland has begun.

    It’s not really even a issue of replacing X. XWayland supports X11 just fine. X11 is a network protocol. There isn’t really any need for your X11 renderer to be your display manager.

    So I don’t see X going away any time this decade. Although I am already running Wayland on my desktop.

    As far as Keith Packard goes, as far as I can tell, he is a fan of Wayland. If anything Wayland will make X easier to live with.

  33. Your X11 render hasn’t ever *been* your display manager, unless I’m badly missing something. Nor your desktop manager, nor your window manager….

  34. It’s not really even a issue of replacing X. XWayland supports X11 just fine. X11 is a network protocol. There isn’t really any need for your X11 renderer to be your display manager.

    XWayland is a stopgap to get recalcitrant users to migrate. It’s the thing that most of the previous X11 replacements (Berlin/Fresco, etc.) lacked: a smooth migration path from old to new.

    The reality is that no one wants to develop against X11 anymore. App developers write toolkit apps, not X11 apps. And there are two major toolkits, both of which support X11 as only one of many backends, both of which are emphasizing Wayland development. Presumably when Wayland becomes widespread enough, support for X11 will be deprecated and then dropped.

    Your X11 render hasn’t ever *been* your display manager, unless I’m badly missing something. Nor your desktop manager, nor your window manager….

    Yes, Xorg is your display manager — just not in the “xdm” sense. Xorg controlled your display directly, and clients had to go through it in order to display graphics. (This created huge problems when direct rendering support was needed.) Under Wayland, for speed and security purposes, all the responsibilities of window, desktop, screen, etc. management are assumed by the Wayland compositor.

  35. So, we are saying that the percentage of X apps for which network portability is *essential* is 0.00%, then?

    Or, alternatively, that those people can just go bugger off?

  36. So, we are saying that the percentage of X apps for which network portability is *essential* is 0.00%, then?

    Or, alternatively, that those people can just go bugger off?

    The canonical answer to questions like this is “watch this talk by Daniel Stone”. So, watch this talk by Daniel Stone. :)

    In it he explains that X isn’t really network-transparent, and hasn’t been for years if not decades. Most modern clients — those based on modern toolkits like GTK or Qt — do their rendering work client-side and then use X DRI extensions to get the bits on-screen. If these clients were running over a network, they would have to use an entirely separate code path to transfer their bits over in an XImage and even then that would be insanely slow. Which is why no one runs modern clients over a networked link anymore.

    If anyone benefits from X’s network transparency, it’s old legacy clients based on, for example, the Motif toolkit. For them, XWayland is available.

    So yes, the number of clients that actually use X11’s “network transparency” is so small as to be a rounding error in today’s world, and we are free to tell people who say “but muh network transparency” to go bugger off, or use the bloody compatibility layer. FYI, the three most common end-user Unix systems — macOS, iOS, and Android — told these people to bugger off, and all three have a much smoother, more pleasant UI experience than desktop Linux as a result. So really we should have told them to go bugger off many years ago. But, like the OpenGL ARB being (until recently) held up in innovation by ancient CAD vendors who still needed the legacy API, the transition away from X has similarly been stalled.

  37. Noted, thanks. :-)

    I’ll give it a look.

    An argument could of course be made that to the audience who *really* cares about that — people who can’t *run* OS/X on their supercomputer or Zsystem — that argument doesn’t carry too much water… but I’m not those people, nor Jewish.

  38. Most modern clients — those based on modern toolkits like GTK or Qt — do their rendering work client-side and then use X DRI extensions to get the bits on-screen. If these clients were running over a network, they would have to use an entirely separate code path to transfer their bits over in an XImage and even then that would be insanely slow. Which is why no one runs modern clients over a networked link anymore.

    I routinely run GTK2/3 and Qt4 applications over the network. Nearly always local, but it’s a daily use case for me.

  39. I routinely run GTK2/3 and Qt4 applications over the network. Nearly always local, but it’s a daily use case for me.

    How are you liking that UI jank? :)

    These days, most people who run remote GUIs use any of the other protocols which have been developed that surpass X in performance: VNC, RDP, SPICE, etc. A Wayland compositor can be written which speaks any of these — this makes Wayland a better choice than X even if you need a remote GUI. Weston comes with an RDP backend and can be used as an RDP server.

  40. The ne plus ultra of remote-GUI protocols is Teradici PCoIP, which is reportedly able to transmit a Cinema Display-sized (2560×1440) display over a crappy cellular link with virtually zero lag. Of course it’s proprietary — to get quality in software you gotta pay — but the point is the world has moved on from X11, even in the world of networked displays, and there are far superior solutions available.

Leave a comment

Your email address will not be published. Required fields are marked *