An important part of the Way of Unix is to try to tackle large problems with small, composable tools. This goes with a tradition of using line-oriented textual streams to represent data. But…you can’t always do either. Some kinds of data don’t serialize to text streams well (example: databases). Some problems are only tractable to large, relatively monolithic tools (example: compiling or interpreting a programming language).
Can we say anything generatively useful about where the boundary is? Anything that helps us do the Way of Unix better, or at least help us know when we have no recourse but to write something large?
Yes, in fact we can. Back in 2015 I was asked why reposurgeon, my editor for version-control repositories, is not written as a collection of small tools rather than as a relatively large interpreter for a domain-specific language. I think the answer I gave then generalizes to a deeper insight, and a productive set of classifications.
(Part of the motivation for expanding that comment into this essay now is that I recently got mail from a reposurgeon user who shipped me several good fix patches and complimented the tool as a good example of the Way of Unix. I actually had to pause to think about whether he was right or not, because reposurgeon is certainly not a small program, either in code bulk or internal complexity. At first sight it doesn’t seem very Unixy. But he had a point, as I hope to show.)
Classic Unix line-oriented text streams have what I’m going to call semantic locality. Consider as an example a Unix password file. The semantic boundaries of the records in it – each one serializing a user’s name, numeric user ID, home directory, and various other information – correspond nicely to the syntactic boundary at each end of line.
Semantic locality means you can do a lot by looking at relatively small pieces (like line-at-a-time) using simple state machines or parsers. Well-designed data serializations tend to have this property even when they’re not textual, so you can do Unix-fu tricks on things like the binary packed protocol a uBlox GPS ships.
Repository internals are different. A lot of the most important information – for example, the DAG structure of the history – is extremely de-localized; you have to do complicated and failure-prone operations on on an entire serialized dump of the repository (assuming you can get such a thing at all!) to recover it. You can’t do more than the very simplest transforms on the de-localized data without knowing a lot of fragile context.
(Another, probably more familiar example of a data structure with poor semantic locality is a database dump. It may be possible to express individual records in tables in a representation that has good locality, but the web of relationships between tables is nowhere expressed in a way that is local for parsing.)
Now, if you are a really clever Unix hacker, the way you deal with the problem of representing and editing repositories is by saying “Fsck it. I’m not going to deal with repository internals at all, only lossless textual serializations of them.” Voila, reposurgeon! All your Unix-fu is suddenly relevant again. You exile your serialization/deserialization logic into stream exporters and importers which have just one extremely well-defined job, just as the Way of Unix prescribes.
Inside those importer/exporter tools…Toto, you’re not in Unix-land anymore, at least not as far as the gospel of small separable tools is concerned. That’s OK; by using them to massage the repository structures into a shape with better semantic locality you’ve made the conceptually hard part (the editing operations) much easier to specify and implement. You can’t get all the way to line-oriented text streams, but you can get close enough that ed(1), the ancient Unix line-oriented text editor, makes a good model for reposurgeon’s interface.
To sharpen this point, consider repocutter. This companion tool in the reposurgeon distribution takes advantage of the fact that Subversion itself can serialize repository into a textual dumpfile. There’s a repertoire of useful operations that repocutter can perform on these dumpfiles; notably, one of them is dissecting a multi-project Subversion repository dump to get out any one of the project histories in a dumpfile of its own. While repocutter has a more limited repertoire than reposurgeon, it does behave like a classic Unix filter.
Stepping back from the details of reposurgeon, we can use it as a paradigmatic case for a couple of more general observations that explain and generalize traditional Unix practice.
First: semantic locality aids decomposability. Whether you get to write a flock of small separable tools or have to do one large one is largely controlled by whether your data structure has a lossless representation with good semantic locality or not.
Or, to put it more poetically, you can carve a data structure at its joints only if there’s a representation with joints to carve.
Second: There’s almost nothing magic about line-oriented text streams other than their good semantic locality. (I say “almost” only because eyeball friendliness and the ability to edit them with unspecialized tools also matter.)
Third: Because semantic locality aids decomposability, part of the Way of Unix is to choose data structures and data formats that maximize semantic locality (under the constraint that you have to represent the data’s entire ontology).
That third point is the philosophical generalization of “jam it into a line-oriented text-stream representation”; it’s why that works, when it works.
Fourth: When you can transform a data structure or representation to a form with better semantic locality, you can collect gains in decomposability.
That fourth point is the main trick that reposurgeon pulls. I had more to say about this as a design strategy in Automatons, judgment amplifiers, and DSLs.
@esr: I recently got mail from a reposurgeon user who shipped me several good fix patches and complimented the tool as a good example of the Way of Unix. I actually had to pause to think about whether he was right or not, because reposurgeon is certainly not a small program, either in code bulk or internal complexity.
My understanding of the basis of the Unix Way was “One tool for one job”, with more complex tasks accomplished by using combinations of tools. There was nothing inherent in there about how large a tool was, beyond an implicit “big enough to actually do the job”.
The job the Reposurgeon is designed to do determined its size and complexity, but it’s still one tool to do one job. It might have been possible to implement it as a collection of smaller tools, but I suspect it might have been more difficult overall, less flexible, and harder to extend.
______
Dennis
All right, let me invite the 800-pound gorilla into the room.
What does this insight have to say about systemd’s adherence or not to The Unix Way(tm)?
Yes: “Not in PID 1”.
> What does this insight have to say about systemd’s adherence or not to The Unix Way(tm)?
It does one thing–take over your system–and it does it well.
Should add: http://without-systemd.org/wiki/index.php/Main_Page
@Jay Maynard: What does this insight have to say about systemd’s adherence or not to The Unix Way(tm)?
Linux != Unix …
:-)
> I think the answer I gave then generalizes to a more general insight, and a productive set of classifications.
“Generalizes” and then “general” – that’s poor style, isn’t it? How about “a broader insight”?
Alternatively: “I think the answer I gave then has wider relevance, and offers/suggests a productive set of classifications.”
> each one serielizing a user’s name
Did you mean “serializing”?
> (assuming you can get such a thing ar all!)
I’m sure you meant “at all”.
> While repocutter has a more limited reperoire than reposurgeon
“Repertoire”, actually. (Earlier in the same paragraph, you’d gotten it right.)
SystemD is heretical against the UNIX way in a two major ways; 1) Not just doing one thing and doing it well 2) Utilizing binary blobs when textual streams should be used. #2 is the bigger problem in relation to this blog post since I think it will make semantic locality impossible in relation to the binary blobs it incorporates into it’s data structures.
> It does one thing–take over your system–and it does it well.
That is not at all true. It’s great as a recruitment tool for *BSD, too ;)
I’ve recently migrated from Linux Mint to FreeBSD for work (on a Lenovo X220), and my home server / NAS from Ubuntu Linux to FreeBSD.
Anecdotally, I’ve heard a few BSD people say there’s been an uptick on mailing lists, etc. since the widespread adoption of systemd.
I wonder if there’s another point worth making? Having the key features of “semantic locality” “decomposability” baked-in means that when the inevitable day arrives when you have to write an additional unix-y tool that was not anticipated in the initial design, the data stream is already in shape to make that task as painless and as achievable as possible.
Stated another way, we can never anticipate all the ways we might need to manipulate/convert/decompose a given data stream, so built it with that in mind.
>Stated another way, we can never anticipate all the ways we might need to manipulate/convert/decompose a given data stream, so built it with that in mind.
While this is a good rule, it pulls in larger design issues than I was focusing on in the blog post. Good semantic locality is only part of what you want if you have “that in mind”; to name one other trait you want as an example, the format should also be self-describing.
Adding to DMcunney:
an implicit “big enough to actually do the job”.
…And no bigger.
@JohnOC: Adding to DMcunney:
an implicit “big enough to actually do the job”.
…And no bigger.
Concur. But that gets into feature creep, and whether the job was correctly specified when the tool was created.
______
Dennis
Wasting time thinking about serialization is so 1990s.
Protocol buffers everywhere.
Outside of Linux, BSD, and rinky-dink Xenix boxes running POS terminals for your local auto garage or backwater Pizza Hut, large and sophisticated service management frameworks have been a part of commercial Unix since the mid-90s at least. Systemd itself is based on Solaris SMF. The “Unix Way” is a bullshit Platonic ideal; production Unix in the messy real world doesn’t actually adhere to the Unix Way in anything like a consistent manner.
And yes, it’s 20-fucking-17 and serialization is a long-solved problem. For new projects you should just use protobuf and get on with it.
Jeff, are you trying to provoke Our Esteemed Host into flaming you to a crackly crunch?
@Jeff Read the main problem with systemd, to my understanding, is that it does not provide any easy way for a service to notify that it is up without importing systemd-specific libraries. They force services (and your personal instance of screen counts as a service, apparently, if you don’t want it summarily killed as a zombie part of your login session when you log out – go ahead and name another Unix that does that.) to compile against their libraries to be able to do anything useful.
This is FUD. Notifying systemd that a service is ready is as easy as writing strings to a Unix-domain socket. See the man page for
sd_notify(3)for details.Linux is not Unix. Just because Unix has always done the wrong thing doesn’t mean now is not a great time to start doing the right thing.
Personal instances of screen, etc. continuing to run after logout are exploiting a loophole in the traditional semantics of Unix sessions. In traditional Unix, a session is basically a group of processes all having the same controlling terminal. It’s 2017, and VDTs are long obsolete (else much of TEHOK would not be necessary), so the bit about having a “controlling terminal” be an inherent kernel-level characteristic of a process is a bit silly, innit? Again, look at the current year. This model is literally 40 years out of date.
Systemd has a more modern session model; in systemd, a session is tied to the abstraction of a seat. When you log out of your seat, everything in your session is closed because of course it is. Do you really expect things to be otherwise? Before systemd, the seat abstraction was faked by tying graphical logins to kernel virtual terminals on the console. That kinda sorta worked, but tracking the kernel state to remember what video mode each VT was in and switch appropriately when you switch VTs is bogus.
The plan is to remove the VT code from the kernel entirely, let a systemd component handle the VT220 emulation for console virtual terminals, and let Wayland handle MUXing the display into multiple seats.
>Linux is not Unix. Just because Unix has always done the wrong thing doesn’t mean now is not a great time to start doing the right thing.
So much for “Commercial Unix has been doing this since the 90s.”
> The plan is to remove the VT code from the kernel entirely, let a systemd component handle the VT220 emulation for console virtual terminals, and let Wayland handle MUXing the display into multiple seats.
That’s a silly plan. I mean, at that point, why not run a userspace terminal emulator that’s not a systemd component?
Are you familiar with the recent efforts at “ranges,” a way to generalize the Unix concepts for programming languages? It is one of the main reasons I use the D programming language, and C++ is following close behind (highly recommend that video, one of the best tech presentations I’ve ever watched).
It benchmarks really well, including beating the Unix tools you like while performing similar tasks.
> Systemd has a more modern session model; in systemd, a session is tied to the abstraction of a > seat. When you log out of your seat, everything in your session is closed because of course it is. > Do you really expect things to be otherwise?
Yes, because I find what what you decry as the wrong thing to be useful. I might not want or be able to stay at my seat for a long-running task. I might not expect every program running in my session to stay running when I disconnect or log out, or care if they don’t. But if I take steps to put a process in the background so it will be there when I come back, it damned well better be there when I come back, unmolested. If it isn’t, there had better be a damned good reason. Preserving the integrity and runability of the overall system (SIGSEGV, OOM, etc.) is probably a good reason. “You logged out, I thought you were done with it,” is not.
Jeremy,
It is a valid critique to say that systemd should provide an easy way to start a process such that it, and perhaps all its children, persist after logout.
But the user should have to make such things explicit. What systemd provides is a sensible default whereas traditional Unix did not. In particular, KDE and GNOME start lots of processes that are not properly cleaned up on logout; KillUserProcesses=yes fixes that. And before you pipe up that KDE and GNOME should better manage their processes — why? Correct process management, at the user and system level, is exactly what systemd is for! And now we’re back at “do one thing and do it well”…
> Just because Unix has always done the wrong thing doesn’t mean now is not a great time to start doing the right thing.
The number one characteristic of systemd disciples is their willingness to spend *my time* learning new ways to do things which I already have perfectly serviceable ways to do.
It’s not your place, Jeff, and it’s not Lennart’s place — and the people who were supposed to be grownups and I cannot *fathom* how they drank the Flavor-Aid is the Release Configuration Managers at all the major distributions. They really ought to have known better.
Oh well, at least there’s devuan.
> It is a valid critique to say that systemd should provide an easy way to start a process such that it, and perhaps all its children, persist after logout.
There is a way already – detaching from the session, as screen does. That’s literally what it’s for. If KDE and GNOME do this improperly, that should be fixed.
In general, systemd does introduce some useful innovation over the venerable sysvinit and even other init systems. It’s just extremely kludgy, with way too much incidental complexity and no clean separation of mechanisms from policy, but that’s initially to be expected in any project with a similar scope and ambition – such projects are often plagued by second-system effect. (Other projects that are currently being proposed for general use, such as Wayland, have similar issues. For instance, the Wayland ‘APIs’ – the local equivalents to the X client-server protocol – are rather complex and not satisfactorily documented.) I think the project is nowhere near production readiness in its current state, but it sure would be nice if more effort could go into making some of the core innovations systemd has pioneered more independently usable and useful, as opposed to just pointing out that project’s rather obvious shortcomings.
@Jeff:
This just reinvents the old controlling terminal model with a fancier terminal, and has the same problems. When your seat is on a different machine than your session (such as when you’re ssh’ed into a server), the lifetimes of your remote session (e.g, screen on a server), your connection to the remote machine, and your session on the local machine (e.g, your laptop) are often not coterminous. Your local machine may change during the lifetime of your remote session. Even if it does not, the connection may fall victim to a fiber seeking backhoe.
The entire reason people run screen is for screen to be a virtual seat.
<joke>
Heresy!
<ha ha, only serious>
Don’t you know that Linux *is* a terminal emulator, and, that in probably the most egregious example of Zawinski’s law I’ve ever seen, the whole “kernel” bit was just feature creep?
I think Zawinski’s law needs to be restated:
“Every program attempts to expand until it
can read mailimplements most of POSIX. Those programs which cannot so expand are replaced by ones which can.”</hhos>
</joke>
Because for all any other process in the system knows, including PID 1, they, or one of their components, could be providing persistent session functionality a la screen. They should know whether they’re doing anything fancy like that and clean up after themselves if they aren’t.
Aaand the preview pane displayed < and > correctly as angle brackets, but they showed up as ampersand hash six zero and ampersand hash six two when I hit post.
@Jon Braden –
> Aaand the preview pane displayed < and > correctly as angle brackets, but they showed up as ampersand hash six zero and ampersand hash six two when I hit post.
The trick here is to use < and >, which yield < and >. But yeah, we get your markup.
(And this was so damned hard to type on a phone with autocorrect.)
@Jon Brase – apologies for mis-typing your name.
AutocorrectAutodefect FTL!Not a problem. My name is not misspelled often enough for it to be annoying. Mispronunciation is guaranteed, so that’s not annoying either.
@Jon Brase I’m not sure (but have some suspicions) why your browser showed it differently, but the semicolon is always required (and is the #1 cause I’ve noticed of people’s attempts at doing this failing in general, not just you today.) <this> works fine [unless it doesn’t, in which case I will be quite embarassed] with < and >. (Frankly, “gt” and “lt” are easier to remember IMO, but people make the same mistake with those [even with adjacent words] often enough for the semicolon problem to be worth pointing out regardless.) I’ve also found that in most cases you don’t actually need >, escaping the < is sufficient, and the need for & is situational.
That was an interesting bit of unexpected behavior, I typed “&#60;”* and it turned into < in the above, in the “…with _ and _” bit. But the > further down worked fine.
*that’s “AMP A M P SEMI HASH 60 SEMI”, in case it gets mangled again.
“try to tackle large problems with small, composable tools.”
Trying to tackle large problems with small, compostable tools is what gave us the X-Windowing-System and its resources nightmare. It also gave us the problem of no two distros having a command-line command guaranteed to be present on both, because there is no formally defined core set of guaranteed-present tools (and even if it does it’s probably something ancient and hence irrelevant). Then there is the Linux audio mess, with ALSA apps not guaranteed to work on PulseAudio but PulseAudio not guaranteed to be on all systems (so, should you code for Pulse or ALSA?)
Having small compostable tools is well and good, but without a formal and recent standard to guarantee a set of tools will be present on a system implementing the standard, it results into fragmentation and chaos.
Sorry if this goes against your bazaar theory.
It’s quite simple, really. No other init system offers the features and convenience that systemd does. Among other things it makes a release configuration manager’s job much, much easier since they’re no longer on the hook for maintaining a tangle of init scripts for every possible daemon. Systemd unit files ftw!
Are there any major sites that have deployed Devuan at scale?
See, that’s the thing: whether you personally like systemd no longer matters because if you work professionally with Linux, systemd is or will soon become a part of your life. (Some sites are still stuck on CentOS 6.) So you’d best make your peace with it.
Actually, a seat is not a terminal at all; it’s a local collection of hardware resources. Come to find out, in the modern era most people have their own computer and remote login is a minority use case! Under Linux+systemd, remote logins are seatless (though they are still tied to a systemd session with all the automatic cleanup behavior that entails). Again, the goal here is explicit process control. A session is an actual object in systemd’s ontology — not just “all the processes under the same controlling terminal, from whose control any process may escape by forking and daemonizing itself”. Processes which are bound to a session cannot escape that session — whether that session is in turn tied to a seat or not.
If you wish to run screen without it being terminated after logout, say
systemd-run --scope --user screen. One less thing to complain about.@Random832:
> I’m not sure (but have some suspicions) why your browser showed it differently, but the semicolon is always required
Ah. Well in any case, the really weird thing is that it displayed one way in the preview pane and the other in the post. I’d think my browser would parse the same HTML the same way either way.
@Jon Brase Your browser is not the only thing parsing the HTML when you actually post the comment, though – the WordPress comment system scrubs the HTML* (presumably by fully parsing it into a document tree, iterating through it flattening any non-whitelisted elements, and serializing it back to HTML – anything less isn’t a reliable safe way to get rid of XSS etc.), and it’s this parse stage that is interpreting the sequence without a semicolon as expanding to a literal ampersand followed by other characters.
The preview, written in Javascript, probably does the same thing, but the conversion to a document tree is done by your browser’s native HTML parser rather than by whatever PHP library is used by WordPress.
So the HTML that comes back isn’t the same HTML that was posted.
*Of course, it’s not quite HTML, the main difference is that you can just press enter once to get a line break and twice to get a paragraph break – this is probably managed with a sequence of simple string replacements.
It’s unreasonable to expect me to do so, because I don’t create the screen server process. I create the client process, which may either start the server process or attach to a running one, depending on what options I have specified and whether one is already running. The support for invoking systemd-run has to therefore be built in to screen, and this is unreasonable to require a portable program to do.
‘systemd-run –scope –user screen’
This … is somehow a good thing?
> Are there any major sites that have deployed Devuan at scale?
That hardly matters, because the Devuan project is only there as a last resort, in case other distributions, including Debian proper (which obviously _has_ been so deployed!) stop supporting other init systems (or, at least, if they do so prior to the systemd project, or a derivative thereof, attaining genuine production readiness).
> Actually, a seat is not a terminal at all…
You seem to have missed Jon Brase’s point here. What he says is completely independent of whether the “controlling terminal”, or “session” was created by a local or remote login. (This ‘local seat’ notion in systemd appears to be pure incidental complexity.) And it’s not at all clear why the _usual_ procedure for detaching a process from the session, i.e. daemonizing, should not be supported. (The one exception being if it is judged to be simply too error prone to be useful. But even that is a big stretch.)
@Jeff:
> Come to find out, in the modern era most people have their own computer and remote login is a minority use case!
In the modern era many people have more than one computer, and one or more of them may be headless or not physically convenient to access at the moment. They might even want to access their home desktop while travelling! And that’s not even getting into datacenters, where remote login is certainly not a minority use case!
Because it gives processes the option of breaking free from effective process control. No,
psandkillare not effective process control, because using those is inherently racy. Nor are hacks like pidfiles, because those can easily get stale. The goal of systemd, the entire freaking point, is to have explicit, hack-free, automatable, perfect monitoring and control over, potentially, every process in the system. Because Unix’s process semantics by themselves don’t give you that.With
systemd-run --scope, you can detach a process or group of processes from your session while still having it monitored and controlled by systemd, so if something goes haywire you can use systemctl to stop it.Wayland is a poor choice to use as an example of second system effect, because it is explicitly designed to be less complex, feature-encrusted, and crufty than X11 was.
Also, you’re missing lots of context: As soon as Keith Packard dies or retires, literally no one will want to maintain the ancient, crufty Xorg code base. So there is a sense of urgency to migrate the Linux graphics stack before it becomes dependent on a bit-rotting code base that no one wants to touch. Literally anything will do as long as it has a community of enthusiastic maintainers and contributors. And, crucially, picking apart its flaws only serves to cement X11’s place as the weak link in the graphics stack, so getting people to migrate to Wayland now, in spite of the flaws is also paramount.
This is something Tom Schelling called “precommitment”. It basically means getting everybody to commit to a direction with no turning back before they realize what they’ve gotten themselves into, so they’re forced to make the direction a good one. It tends to be used when “turning back” is untenable. For example, in ancient times the bridges back home were burned, precommitting the soldiers on the front lines to fight rather than flee. See also: Rust vs. C. The Rust community is working hard to get everyone on board with replacing C with Rust for new development, because “turning back” to C as the de facto standard systems programming language — even with the best C programmers on the job — means a long list of CVEs that just wouldn’t be there in well-designed, idiomatic Rust code. Another example from recent history is Nancy Pelosi’s “Let’s pass this bill to see what’s in it” remark about Obamacare. Conservatives like to rake her over the coals for it, but it’s a canny precommitment strategy, because continuing with the status quo in health care in America at the time meant millions of poor going without access to adequate health care.
Jeff: “It basically means getting everybody to commit to a direction with no turning back before they realize what they’ve gotten themselves into, so they’re forced to make the direction a good one.”
This is at best wishful thinking. That way lies thoroughly broken monstrosities like X.500 or Obamacare.
And: “The Rust community is working hard to get everyone on board with replacing C with Rust for new development,”
When they can get my code into Rust from C without a massive amount of labor and fit it into a 256K ROM, then they can get back to me.
> Wayland is a poor choice to use as an example of second system effect, because it is explicitly designed to be less complex, feature-encrusted, and crufty than X11 was.
The _codebase_ may be so, but the protocol used to interact with it seems to be flawed, even compared to X. Of course this is not a pressing problem as long as Xwayland is properly supported, both on the server- and on the client- (desktop toolkits, etc.) side – so migrating to the Wayland underlying codebase (i.e. compositor and driver base) may make plenty of sense nonetheless. (Not _everything_ that comes from the modern Linux desktop is bad! Even pulseaudio makes some sense, if you care about mixing multiple audio streams but your crappy or ancient soundcard can’t do it in hardware.)
> No, ps and kill are not effective process control, because using those is inherently racy.
So let’s figure out some nice primitives for ‘effective process control’ and get them supported in psutils and the kernel. How about `echo kill -TERM -1 | su -m $USER`, for a start? This hardly requires migrating to a new, elephantine init system!
> So let’s figure out some nice primitives for ‘effective process control’
(Added– Of course, the underlying primitives are there already, at least in Linux – cgroups has a nice “rule” engine” that allows you to programmatically place existing processes in a process-control group. But if you’re using systemd as your init system, these primitives will not be properly supported for you, because systemd assumes that it can just take over the cgroups functionality for its own use!)
Not going to happen. X is legacy tech; XWayland is just a bridge to ease the migration process, much like “MacOS Classic”. The toolkit maintainers will want to support as few code paths as possible — presumably, one per platform (Linux, Windows, Mac OS). For the toolkits that matter (GTK+, Qt), X11 support will most likely be deprecated and then removed once Wayland reaches critical mass. There are considerable costs, and zero benefits, to maintaining a code path to support an obsolete, unmaintained display protocol.
There is already precedent for this: maintainers of open source packages that play audio are starting to deprecate and remove support for playing directly through ALSA, leaving PulseAudio as the only supported audio backend. The most recent package to make this shift was Firefox.
“Fitting it into a 256K ROM” won’t be a problem; the Rust compiler has support for compiling without a standard library and even backends for 16-bit microcontrollers.
I don’t think anybody expects a large C codebase to be converted overnight or even in the foreseeable future. However, for greenfield projects, there is no reason to use C.
>However, for greenfield projects, there is no reason to use C.
No, not unless you want to do something really complicated and obscure like a select on multiple sockets.
That was, in case you missed it, sarcasm.
> There are considerable costs, and zero benefits, to maintaining a code path to support an obsolete, unmaintained display protocol.
Nah, network transparency _is_ a significant benefit, and I don’t think toolkit maintainers (or toolkit users, for that matter) will want to forgo it. Unlike X, Wayland has no support for network transparency – and (AIUI) no feasible way of adding it, aside from VNC-like kludges.
> …The most recent package to [move from ALSA to pulseaudio] was Firefox.
Firefox has a fairly specific reason for wanting to do this, BTW – they want to ‘sandbox’ audio-generating elements (HTML audio/video, Javascript, plugins) in order to properly support options like “Mute tab” and other nice things. Obviously, ALSA doesn’t really do this the way the Firefox devs would want it to. Moreover, if you expect to eat that CPU-time cost anyway, using pulseaudio and getting that functionality for free isn’t really a huge drawback – if only because the codebase has recently improved in quality to the point of being at least usable. In a way, this is the exact opposite of the Xwayland/Wayland case since ALSA interfaces more closely with the hardware and pulseaudio is acting as a higher layer.
With Tokio, you don’t need
select(2)for the majority of network applications. It provides an asynchronous I/O abstraction much like Twisted does for Python, or Node does for JavaScript.All the system calls are still available through the C FFI if you need them, but Tokio’s abstractions are powerful, convenient, and virtually zero cost.
Do not worry about API stability over 10 years. For reasons including, but not limited to, the fact that the developers of certain major libraries can’t get their shit in one sock (I’m looking at you, GTK+), churn is the new normal in open source. I’d be surprised if anything above POSIX remains stable over a two-year period. Tokio should fare better than most open-source libraries.
>Tokio’s abstractions are powerful, convenient, and virtually zero cost.
And not yet part of the standard library.
>Do not worry about API stability over 10 years.
Nobody who does infrastructure work has the luxury of not worrying about that.
Couple of random thoughts, in no particular order, from a user, superuser, and “engineer” of *nix systems, but not a software developer, and someone who actively hates writing code:
1)I have never ever logged into a production system from a physical console. I “own” *nix systems I have never laid eyeballs on, and at least one installation I did from rack-and-stack to production without coming within the same time zone of (remote hands and a system to allow remote access to the terminal port.) Remote Access is not a rarely-used thing for me; except for “toy” systems I’ve set up and played with during my continuing education, I ssh/https into all the things. And even for those toy systems, I effectively ssh/https into all the things, because a command prompt is a command prompt is a command prompt.
2)There is one non-CLI tool that is (all-but) guaranteed to be on an *NIX box; and that is vi.
3)I wonder if the namers of systemd were aware of the use of the term System D in the CoDominium novels.
Ian: “I wonder if the namers of systemd were aware of the use of the term System D in the CoDominium novels.”
I’m not. EXPN?
Systemd stands for “system daemon”. But there is a reference in there, to Système D, a French expression meaning something akin to “hackish ingenuity”. The D stands for débrouiller or démerder; Système D is a “system” for getting you out of deep shit. The joke, of course, is that there is no system. You just have to make up a solution as you go along.
Parallels to the goals and implementation of systemd are left as an exercise.
@Jeff Read
. But there is a reference in there, to Système D, a French expression meaning something akin to “hackish ingenuity”. The D stands for débrouiller or démerder; Système D is a “system” for getting you out of deep shit. The joke, of course, is that there is no system. You just have to make up a solution as you go along.
And “Système D” is a term found in restaurant kitchens (the higher end ones, at least), describing what you must do when you get a visit from the fuckup fairy and your carefully planned and published menu becomes a work of partial fiction.
______
Dennis
@Ian Argent:
2)There is one non-CLI tool that is (all-but) guaranteed to be on an *NIX box; and that is vi.
Or something that looks like it. On a Linux system, the look-alike is normally Vim, though I’ve encountered nvi and the vi persona of Albrecht Kliene’s e3 editor used instead.
I have Vim here, under Linux and Windows, but for the use I make of it, it might as well be vi. I’ve never had a need for the bells and whistles it added.
______
Dennis
From “The Mercenary” in regards to a unit of CD Marines (whose traditions derive from, among other places, the French Foreign Legion)
“they go to other parts of town, too. Lots of them. They go into taverns and drink all night, then say they can’t pay. If the owner gets sticky, they wreck the place. . . .”
“And they’re gone before your patrols get there,”…”It’s an old tradition. They call it System D…”
(Pulled from the Baen Books e-book, so I don’t have a page referent)
Given context and statements, above, it is probably a reference to the Système démerder, only with a distinctly IDGAF twist. In this case the CD Marines (and later on the mercs of Falkenberg’s Legion) are the fuckup fairy.
(I have learned something today)
In re “something look like [vi]” – yes, most places these days it’s actually vim, but you can invoke it by typing vi and you can use all the commands of vi.
Ian Argent:
What does “CD” stand for in this context?
@Jorge Dujan:
…a unit of CD Marines…
What does “CD” stand for in this context?
CoDominium. In the universe Niven and Pournelle wrote in on this one, the US and the Soviet Union joined forces to keep a lid on the simmering world pot on Earth and formed the CoDominium, enforcing a ban on various destabilizing research. FTL travel was available, various worlds had been colonized, and the CoDominium was using some of them as dumping grounds for trouble making elements on Earth.
Colonel Falkayn and the 42nd CD Marines were part of the CoDominium Navy, and got deployed to stomp problems on colony worlds. But the CoDominium was in the process of coming apart back on Earth, which is how Falkayn and the 42nd CD became mercenaries as the regiment was demobilized. (The folks behind it didn’t want to see Falkayn lead the 42nd against their proxies back on Earth. As a general rule, the 42nd won battles they fought.) They wind up working very quietly and unofficially for Grand Admiral Lermontov, head of the CD Navy, who knows the wheels are coming off the cart back on Earth and wants the colonies positioned to survive independently when the CD melts down back home.
______
Dennis
@Ian Argent:
In this case the CD Marines (and later on the mercs of Falkenberg’s Legion) are the fuckup fairy.
Well, everything is relative, since what they tend to get called in to do is clean up the mess after someone else has gotten a a visit from the Fuckup Fairy.
In re “something look like [vi]” – yes, most places these days it’s actually vim, but you can invoke it by typing vi and you can use all the commands of vi.
Yep, and it’s installed to be callable as vi. It’s just not the only such thing out there. (I just ran across an open source effort that is an implementation of vi for the Commodore 128, and saw a partial implementation previously for the C64. I was bemused.)
______
Dennis
Thanks, Dennis. I’m confused, though: so the cops knew Internal Affairs was setting them up?
@Dennis
Well, everything is relative, since what they tend to get called in to do is clean up the mess after someone else has gotten a a visit from the Fuckup Fairy
When they are practicing System D; they are fucking the targeted place up. CoDo Marines are not good garrison troopers.
As for the vi thing, I think we’re getting wrapped around the axle of semantics. Regardless of how it’s implemented, the minimum available is launched by invoking vi and has the vi instruction set. Thus, vi is always there in spirit. Even on a Commodore, great gods and little boarlets!
@Ian Argent:
@Dennis
Well, everything is relative, since what they tend to get called in to do is clean up the mess after someone else has gotten a a visit from the Fuckup Fairy
When they are practicing System D; they are fucking the targeted place up. CoDo Marines are not good garrison troopers.
Good combat troops tend not to be.
Along those lines, an old friend who was in the Air Force talked about what happened when a multinational joint exercise was held at the base where she was stationed. When the exercise wrapped, the pilots all descended on the Officer’s Club for R&R. They practiced System D to the point where the OC got its first ever Unsatisfactory rating, and was closed until things were put back in order. (The OC employees were civilians. They gave up and went home in the wee hours of the morning, leaving the children unsupervised.) The two star general commanding the base had a party scheduled for his daughter in the OC a couple of days hence. There was an all hands evolution to repair the damage, and the participating squadrons were presented with hefty damage bills. They paid up without a peep and tossed in extra on top, and probably said privately “It was totally worth it!”.
As for the vi thing, I think we’re getting wrapped around the axle of semantics. Regardless of how it’s implemented, the minimum available is launched by invoking vi and has the vi instruction set. Thus, vi is always there in spirit. Even on a Commodore, great gods and little boarlets!
My thoughts exactly.
See http://texteditors.org/cgi-bin/wiki.pl?Svicc and http://texteditors.org/cgi-bin/wiki.pl?VI65 for the Commodore entries.
______
Dennis
> Unlike X, Wayland has no support for network transparency – and (AIUI) no feasible way of adding it, aside from VNC-like kludges.
Please explain why VNC-like kludges are a bad thing. The kludges often outperform X11 native protocol, except for a few edge cases where they massively outperform the X11 native protocol.
VNC implementations, as far as I can tell, are still pretty terrible. It’s hard to find anyone who’s properly implementing the JPEG encoding, much less H264.
Virtual desktop forwarding is a pain. With native X forwarding, I can do all the sane things I normally do, like identify the application in the taskbar, move different application windows between desktops, and send keystrokes without escaping them.
> The support for invoking systemd-run has to therefore be built in to screen, and this is unreasonable to require a portable program to do.
Oh come on.
A quick read of screen’s configure.ac shows that it has support for at least 5 variants of helper binaries and workarounds for system-specific utmp, PAM, and PTY weirdness. Screen has probably had dozens more in the past that are now long forgotten. There are modern patches for Alpha and HPUX. Screen had workarounds for bugs in Linux 1.1 and SCO until last April. Screen recently added support for PTYs inside containers (AFAIK a Linux-only feature).
Given the immense variety of systems it already supports and the precedents in the project history, it seems fairly clear that Screen can learn to talk to systemd to get what its users want. This was a new and pertinent issue in 2013, but in 2017 the amount of nothing that has been done about it for four years since is just breathtaking.
> Virtual desktop forwarding is a pain. With native X forwarding, I can do all the sane things I normally do, like identify the application in the taskbar, move different application windows between desktops, and send keystrokes without escaping them.
Who said anything about forwarding the desktop? Forward each window, i.e. your local desktop contains a lot of vnc-like clients each forwarding a single application’s window from a remote host, and the applications talk a vnc-like server that behaves as a compositor.
And as for me, I prefer virtual desktop forwarding. I do too much stuff across non-X platforms for X to be a solution for remote GUI access. I live by TeamViewer.
Single applications have multiple windows.
> > a single application’s window
> Single applications have multiple windows.
Multiplex ’em all through the forwarding agent. It should be handling each window as a separate framebuffer anyway because that’s how Wayland normally gets them from the application.
Something happened between the 90s and today that puts the lie to the notion that “network transparency is a significant benefit”: Mac OS X. It turns out more people care about having a smooth, responsive, pixel-perfect and tear-free GUI than care about network transparency.
And yes, the way to add GUI remoting to Wayland, for the relatively few who need it, is to have the remote clients talk to a Wayland compositor that forwards their framebuffers over a network link to a local program that acts as a client to your local Wayland compositor. Easy peasy, relatively speaking; it’s basically straightforward integration work.
Add to that the fact that the X protocol sucks — Microsoft can write a better networked GUI protocol — there is literally no reason left for X to exist. It is 100% obsolete.
Comes now news that Ubuntu 18.04 will ship with GNOME and, hence, Wayland. So you can’t say that Ubuntu has chosen a different direction anymore.
The community has decided. Wayland is the future.
>The community has decided. Wayland is the future.
The 153rd repeat of this doesn’t help its credibility any after the previous 152 mispredictions. Now get off this subject and stay off it until it is thread-relevant.
I agree all round.
A slight but relevant tangent: the core hacker attitude, a variation on Occam’s Razor:
a tool should be as small as possible, but no smaller.
To put it another way: “to put it more poetically, you can carve a data structure at its joints only if there’s a representation with joints to carve.” (Eric, you spooked me when I read this. It’s ~exactly how I’ve thought about and described coding and data-modelling since Day 1. i.e., structure your [whatever] according to the moving parts, the breakpoints in the structure. I call it responsibility-driven design.)
Building tools smaller than the problem space is just asking for trouble. Building them larger cripples what you can do with them. Goldilocks FTW!
Thinking about putting that approach into this post’s terminology:
The “envelope” of your (created) tools should provide the Syntactic locality to match the problem space’s Semantic locality.
Or perhaps better: your tools will be best usable, and hence appreciated, and hence used, if their syntactic nexuses match the problem’s semantic nexuses.
(Nexi?)
For example: with a solid knowledge of printer control characters I can do everything in ‘ed’ that I can in ‘MS Word’.
Which do people prefer to use?
Contrariwise, notice how general users’ attitude to MSWord has deteriorated since MS has progressively obfuscated core whole-document abilities via the execrable ribbon. For example, very few people now bother with Styles, because they have been turned into such a PITA to use. The larger syntactic locality is effectively no more, despite the semantic locality remaining. Irritation ensues.