Blogging has been light lately because I’ve been up to my ears in reposurgeon’s most serious challenge ever. Read on for a description of the ugliest heap of version-control rubble you are ever likely to encounter, what I’m doing to fix it, and why you do in fact care – because I’m rescuing the history of one of the defining artifacts of the hacker culture.
Imagine a version-control history going back to 1985 – yes, twenty-nine years of continuous development by no fewer than 579 people. Imagine geologic strata left by no fewer than five version-control systems – RCS, CVS, Arch, bzr, and git. The older portions of the history are a mess, with incomplete changeset coalescence in the formerly-CVS parts and crap like paths prefixed with “=” to mark RCS masters of deleted files. There are hundreds of dead tags and dozens of dead branches. Comments and changelogs are rife with commit-reference cookies that no longer make sense in the view through more modern version-control systems.
Your present view of the history is a sort of two-headed monster. The official master is in bzr, but because of some strange deficiences in bzr’s export tools (which won’t be fixed because bzr is moribund) you have to work from a poor-quality read-only git mirror that gets automatically rebuilt from the bzr history every 15 minutes. But you can’t entirely ignore the bzr master; you have to write custom code to data-mine it for bzr-related metadata that you need for fixing references in your conversion.
Because bzr is moribund, your mission is to produce a full standalone git conversion that doesn’t suck. Criteria for “not sucking” include (a) complete changeset coalescence in the RCS and CVS parts, (b) fixing up CVS and bzr commit references so a human being browsing through git can actually follow them, (c) making sense out of the mess that is RCS deletions in the oldest part of the history.
Also, because the main repo is such a disaster area, there is at least one satellite repo for a Mac OS X port that really wants to be a branch of the main repo, but isn’t. (Instead it’s a two-tailed mutant clone of a nine-year old version of the main repo.) You’ve been asked to pull off a cross-repository history graft so that after conversion day it will look as though the whole nine years of OS X port history has been a branch in this repo from the beginning.
Just to put the cherry on top, your customers – the project dev group – are a notoriously crusty lot who, on the whole, do not go out of their way to be helpful. If not for a perhaps surprising degree of support from the project lead the full git conversion wouldn’t be happening at all. Fortunately, the lead groks it is important in order to lower the barrier to entry for new talent.
I have been working hard on this conversion for eight solid weeks. Supporting it has required that I write several major new features in reposurgeon, including a macro facility, large extensions to the selection-set sublanguage, and facilities for generic search-and-replace on both metadata and blobs.
Experiments and debugging are a pain in the ass because the repository is so big and gnarly that a single full conversion run takes around ten hours. The lift script is over 800 lines of complex reposurgeon commands – and that’s not counting the six auxiliary scripts used to audit and generate parts of it, nor an included file of mechanically-generated commands that is over two thousand lines long.
You might very well wonder what could make a repository conversion worth that kind of investment of time and effort. That’s a good question, and one of those for which you either have enough cultural context that a one-word answer will suffice or else hundreds of words of explanation wouldn’t be enough.
The one word is: Emacs.
Next you’ll be telling us that you have reconciled the emacs/xmacs rift.
(backs away quietly, not making eye contact)
So… “burn it with fire and start from scratch” was not an option? Because that appears to be the community-supported approach to dealing with problems of a similar type in the X11 code base.
I suggest that, in the spirit of Wayland replacing X11 with something far simpler and more tractable, we replace Emacs with a far simpler editor. Fortunately we have a template to start with: vi. Of course modern users would not settle for vi as-is, so it must be built upon — improved, you might say…
How do we ensure once it has been cleaned up that this travesty of engineering never happens again?
>How do we ensure once it has been cleaned up that this travesty of engineering never happens again?
We have much better version-control systems now. Nothing like the RCS/CVS shambles will ever happen again.
Nor is it likely that any future VCS of significance will lack the ability to import from a git stream dump.
Justin Andrusk: I suppose we could all agree on the one true repository manager. Sorta like we all agree on the one true programmer’s text editor, eh?
Eric, you should run some stats on that repo when you’re done, like LWN does for the kernel.
>Eric, you should run some stats on that repo when you’re done, like LWN does for the kernel.
What stats do you think would be interesting?
“Read on for a description of the ugliest heap of version-control rubble…”
Minecraft? Now I’ll read the rest of the post, lol!
“The one word is: Emacs”
…that should have been my first guess (blush). …hmm, it makes me wonder what you might be facing if you had to do that for Microsoft Word (I dunno, but I’m guessing that they are not even using git yet.)
@esr
> What stats do you think would be interesting?
I’d be curious to see when any large spikes in activity occurred and determine if they correlate with any notable external events.
>I’d be curious to see when any large spikes in activity occurred and determine if they correlate with any notable external events.
Aha. You want Ohloh’s timeline view. Conveniently, I plan to register the conversion with Ohloh.
I am happy to hear this, as long ago I realized that I hated having to learn a new editor every time I changed jobs and/or moved to a new platform. It got in the way of getting my work done. After being exposed to Emacs and free software (as it was called then), I realized that there would likely be a port to every future platform I would work on and I did not ever need to learn a new editor command-set after learning Emacs. And this turned out to be the case. Saved me tons of time and frustration.
I knew before I ever even saw the headline. I can only imagine both the horror and the honor of the whole thing. I appreciate the effort more than you know. I use Emacs (albeit in evil-mode) daily.
we replace Emacs with a far simpler editor
What do you want on your gravestone?
(assuming of course that enough can be found to bury)
@foo:
There’s only one thing that works in this instance.
:q!
Oh, and I should mention that it might just be the only thing I agree with Mr. Read on. Of course if you’re coming for me, come big.
Mr. Read, true to form, was trolling for just this sort of Vi vs. Emacs flame war. So tired, so old. Meanwhile, most young developers don’t use either – they use mind-numbing IDE’s instead, which depend on a windowing environment to work at all.
Anyway, I can see why ESR has been too busy to post his next installment on the Dark Enlightenment, which I eagerly await.
Eric, you should post some of the code you wrote for pulling out the metadata of the repo and the reposurgeon script itself. I know that they’re probably lots of ugly task specific hacks. But that’s the name of the game and I have a love of this kind of parsing and byte processing. Wish you were doing this stuff a few years back though when I was still using emacs (now back to vim and that’s where I’ll likely stay). I wanted to patch something in it and bzr is such a slow piece of crap that I literally couldn’t clone the repo. So it’ll sure save others doing the same a lot of hassle. I like to think of it as just using a technically superior VCS that actually enables sane repo browsing and version control rather than for acquiring any new talent. Though that’ll be a nice secondary effect I’m sure.
>Eric, you should post some of the code you wrote for pulling out the metadata of the repo and the reposurgeon script itself.
https://git.gitorious.org/emacs-transition/emacs-transition.git
A quick note: that URL is no longer valid after Gitorious was shut down, although the repo seems to still be available at as of this writing.
@Mr. Coe:
I strongly suspect that Mr. Read was joking. It is a tired discussion, and these days *most* programmers/developers do use something like Eclipse or whatever it is that Windows programmers use, and this makes perfect sense because if their windowing environment isn’t functioning writing code is not what they’re going to be working on.
It’s mostly us Systems Admins who use vi/vim these days and I’ve even met a few of them who insist that pico/nano be installed because they refuse to use vi.
Does this clean-up of the repositories imply renewed development of some of the terribly out of date major modes?
>Does this clean-up of the repositories imply renewed development of some of the terribly out of date major modes?
Sadly, no. Right now I’m just fixing the container, not the content.
esr> That’s a good question, and one of those for which you either have enough cultural context that a one-word answer will suffice or else hundreds of words of explanation wouldn’t be enough. The one word is: Emacs.
Emacs who?
@Eric Coe:
>Meanwhile, most young developers don’t use either – they use mind-numbing IDE’s instead, which depend on a windowing environment to work at all.
I’m young, though not really a developer, but I actually use vim fairly regularly, and once spent some time training myself on Emacs (though I’ve forgotten a fair bit of that). My primary editor is gedit, but if I have a terminal window in focus, I’m likely to pop into vim. I’m also likely to use vim to edit system files or files to which I have the path memorized. Around the time I was experimenting with Emacs, I was taking notes in class in a lecture hall without a readily available power outlet on a laptop with a weak battery, and would forgo starting an X session to conserve power, so vim and emacs were my editors of choice.
Know what would be awesome and right on the razor’s edge of possible/impossible?
A git repo of UNIX. All of it.
With an initial commit of Ken’s MULTIC replacement lab experiment, and branches that point to the current UNIXware, the BSDs, the corporate forks from the UNIX wars, etc etc.
It would be huge. It would be terrifying.
@Mark Atwood:
> Know what would be awesome and right on the razor’s edge of possible/impossible?
>
> A git repo of UNIX. All of it.
>
> With an initial commit of Ken’s MULTIC replacement lab experiment, and branches that point to the current UNIXware, the BSDs, the corporate forks from the UNIX wars, etc etc.
And multiple roots (starting from scratch), isn’t it?
I use Emacs for its AUCTeX + RefTeX + Preview-LaTeX + sierotki.el mode, and for TRAMP – Transparent Remote Access, Multiple Protocols (including SSH bridge), mainly.
Interesting stats:
Changesets per time period
Changed lines per time period
Contributors per time period
lists of top contribs per time period
Lists of top contributor’s affiliations per time period (necessarily approximate)
Of archaeological interest, really, I guess.
“they use mind-numbing IDE’s instead, which depend on a windowing environment to work at all.”
We also use mind-numbing filesystems instead of asking for blocks manually, and we also use mind-numbing package managers instead of compiling everything from source.
Generally, we use a lot of mind-numbing stuff today that automates work and saves us from hand-hacking.
Unbelievable but true. Someone called IDEs evil (never mind you can use keyboard shortcuts instead of the hated mouse). This is why Silicon Valley has so much age discrimination. Sure, old folk tend to be smart, but is the risk worth it? Old folk also tend to be gurus that won’t share knowledge with the team (be especially aware of build gurus) and they insist on doing things the hard-way instead of the mind-numbing way (despite the fact the unix-philosophy clearly says to avoid hand-hacking). Also, they tend to do things their own way, ignoring requirement documents and orders from managment (some will even claim no requirement documents are needed, so they have more freedom to do things their own way, and once the code is written, good luck convincing an old folk to change it). So, it’s a big risk to hire old folks.
I understand that the full emacs repository will be a boon for “archeologists” studying the history of CS during the 1980/1999 internet globalization transition.
Good work!
Is there a comparable history of vi?
I myself, am not young. But I never could get used to neither emacs nor vi. I used nedit for the longest time. But nedit at one time seemed to stop moving. I have tried tea, but I realized I needed regex search.
I am now a happy user of geany.
You mean http://ex-vi.cvs.sourceforge.net/viewvc/ex-vi/ex-vi/ ?
@kurkosdr:
Nobody around here except JAD can generalize so well from a few anecdotal examples. Well done, sir!
Mark: I’m a Windows diehard, and even I think that would be kind of awesome.
I think I’d be tempted to put together a nice virtual machine running whatever is necessary to have great access to the current bzr repository, then take the current code and put it into git. You could load the virtual machine as necessary if you have to delve into history, and the current code would have a clean start.
But of course then you wouldn’t be having all this fun!
Although I will say this will be impressive if/when you pull it off.
https://code.google.com/p/gource/
does awesome visualizations.
Why do I see ESR in a darkened lab with arcs of lightning cracking overhead, pounding on the chest of emacs screaming “IT’S ALIVE!!! IT’S ALIVE!!!”
You terrible bastard.
;)
>Why do I see ESR in a darkened lab with arcs of lightning cracking overhead, pounding on the chest of emacs screaming “IT’S ALIVE!!! IT’S ALIVE!!!”
On the one hand…”Bwahahahaa! The fools! They said I was mad!”
On the other, the Emacs codebase itself is in reasonable shape. It’s the history, especially the earlier CVS/RCS parts, that’s a mess.
@Mr. Coe
Of course, one could read Mr. Read as pointing out exactly why Wayland is so deeply misguided.
@kurkosdr:
I’m at that stage of life where I could probably get away with saying “get off my lawn!” so I’ll explain why I, old fuddy-duddy that I am, don’t use an IDE (in other words, I’ll bite).
It’s not that I refuse to use one, it’s just that I have yet to find one that fits the following criteria:
1. It’s available on all the computers I use. Currently, that includes three different Linux distributions (both 32 and 64 bit), Mac OS-X, and just to keep things interesting, Solaris (non-Intel based, both 32 and 64 bit).
2. Usable with the languages I currently use, which at this point, is four (concurrently, in the same project). Five, if you include the custom build system a previous developer left behind.
3. It can start from the command line and does not require a graphical user interface to run (Solaris—there is no X Windowing system on those systems).
4. Won’t crash if I try to import an existing codebase.
5. Integration with SVN and git and can handle both being used in a single project.
6. I can change the key bindings for everything (so I can use the muscle memory I’ve built up over the past thirty years).
7. Windows support is not mandatory.
On the “job security” aspect of my age, I will note that the two projects I’m in charge of can be built by doing a “make” at the top level of each project (there is one Makefile per project—recursive make considered harmful and all that). Also, I have a checklist that outlines the steps required to run the regression test. I have not automated it simply because I do not know how to automate it 100% (one step—check with so-n-so to make sure a critical test machine is free for use; other steps include manual intervention in case a particular piece of third-party software breaks—it’s just nasty stuff like that all around).
Much as I hate CADT-compliant development, Wayland has taken the correct approach to simplifying multiwindow GUI display on Unix, and enjoys universal community support. Xorg development, by contrast, is pert-near moribund, and is only undertaken by a cadre of maintainers, the loss of whom could spell doom for the project.
Oh yeah, there’s also the fact that most of those maintainers have officially blessed Wayland as the replacement for X11.
I was quite blatantly trolling in my original post, but the really trolly part was where I proposed vim as a replacement for Emacs. As other posters have duly noted, many developers have replaced Emacs with IDEs, or at least more modern text editors like Geany or Sublime Text. The fact that Sublime Text does such brisk business should put a lump in Stallman’s throat: when Linux hackers are willing to pay for a proprietary editor to avoid using your editor, you know it’s bad.
The fact is that much like X11 hasn’t kept pace with how modern GUIs are implemented (network transparency and 1990s accelerated drawing primitives are effectively null concerns in today’s world), Emacs (and vim for that matter) haven’t kept pace with modern development workflows. If your editor can’t understand the language you’re editing enough to do automatic refactoring operations (i.e., it is not an IDE), it is completely unsuitable for modern large-scale software development.
>Wayland has taken the correct approach to simplifying multiwindow GUI display on Unix, and enjoys universal community support.
That is not true.
>Oh yeah, there’s also the fact that most of those maintainers have officially blessed Wayland as the replacement for X11.
Not according to Jim Gettys or Keith Packard, the co-designers of X.
Persons new to the blog are cautioned that Jeff Read’s contact with reality is, at best, tenuous.
> How do we ensure once it has been cleaned up that this travesty of engineering never happens again?
@esr > Nor is it likely that any future VCS of significance will lack the ability to import from a git stream dump.
I would have thought the answer was “reposurgeon.bump.version.major”.
I guess I’m in the crusty category. I became a sysadmin before I learned to code, and that meant vi. (on one occasion, being able to run a strategic ed script blindly meant someone not getting fired for accidentally off lining the wrong filesystem. )
After getting used to the admittedly somewhat perverse editing style, there isn’t any good reason to switch, a simple checkout of my config files, scripts and whatnot is all it takes to drop in to a new machine, and frankly Eclipse makes me nuts when I have to use it. But nobody should use anything they don’t want to. A Mac and gvim, along with a pipe to the outside is all I ask for. Or some laptop that runs Linux and doesn’t have some brain-dead design decision.
A bit of applause for rescuing Emacs. I lost track of one of the casualties of the Xemacs split, and another now sells beer, which I hope we can all agree is a much more respectable career than software.
Now, if only someone would right a reasonable text editor in it…
Jamie,
Eclipse is the bare minimum for serious Java development, but as it is it’ll drive anybody nuts. It appears to be built more for accessing the vast and confusing library of Eclipse plug-ins, rather than getting shit done.
No, if you want to develop in Java, you want IntelliJ IDEA. It’s proprietary, of course. But a lot of software which actually makes users more productive and content is. It’s the nature of the beast.
Is it sad that I guessed Emacs as soon as you mentioned the timescale involved, even though I’m not a hacker?
Here’s hoping the effort pans out in the end. That sort of history is worth hanging on to even if it’s for archaeological purposes alone.
>Is it sad that I guessed Emacs as soon as you mentioned the timescale involved, even though I’m not a hacker?
No, I was actually expecting that a lot of my readers would deduce that from the timescale.
@Jeff:
>The fact is that much like X11 hasn’t kept pace with how modern GUIs are implemented (network transparency and 1990s accelerated drawing primitives are effectively null concerns in today’s world),
I for one will miss X11’s network transparency, having actually made productive use of it in the recent past.
Other than that, Wayland looks like a good thing from the information I’ve been able to find on it.
One thing that I’d like to see, though, that no display server seems to do (understandably at this point in time, as it would probably involve kernel changes), is doing compositing by having each window be a block device file.
@Jamie:
> A Mac and gvim, along with a pipe to the outside is all I ask for. Or some laptop that runs Linux and doesn’t have some brain-dead design decision.
I’m sorry, I have to troll a bit here: Doesn’t OS X’s desktop environment count as a braindead design decision? Or is it trivially easy to swap MATE in to replace it? Quite frankly, the dock kills OS X stone cold dead for me. It is, admittedly, more functional than Canonical’s attempt to replicate it in Unity (whose dock kills it just as stone cold dead, despite the fact that I wish MATE had some equivalent to Unity’s search lens, or the Vista/7 start menu), but it still falls far short of an actual taskbar, and from everything I’ve heard about OS X, it’s next to impossible to replace the default DE with something functional.
> I for one will miss X11?s network transparency, having actually made
> productive use of it in the recent past.
I make productive use of it regularly, for situations where I don’t need or want to run a full remote desktop ala NX; it’s one of my top favorite features of X11. (Que Jeff dismissing us because we aren’t “normal.” Blah blah, been down this road before.)
> One thing that I’d like to see, though, that no display server seems to do
> (understandably at this point in time, as it would probably involve kernel
> changes), is doing compositing by having each window be a block device file.
I’d love to see more Plan 9-isms in general. I’m in “wait and see” mode with Wayland, but a hack that enabled exporting the display data as a filesystem via 9p or similar would solve the network-transparency issue without actually requiring network-transparency in the core design.
My statement had two parts: 1) that the Wayland approach is correct; and 2) that it enjoys universal community support. To the second point, Qt and GTK support for Wayland is largely complete, and the GNOME and KDE desktops will run natively on it later this year.
As to Mir, I do not consider Canonical part of the community, as long as they keep insisting on doing their own thing and flipping the bird to upstream. There is far less support from the app and toolkit providers for Mir than there is for Wayland; Canonical will have to provide that support themselves in-house. And they caved on the systemd issue, so caving and abandoning Mir for Wayland is still not outside the realm of possibility for them.
To the first point — the approach used by Wayland is the same as that of the Quartz Compositor in Mac OS X: hand out shared-memory frame buffers, let the clients do all rendering, then composite them together. I don’t think I need to remind you that Mac OS X has beaten Linux desktop share back down to statistical-noise levels, largely due to its beautiful desktop user experience in which X11 applications are, at best, awkward foreigners.
Beyond that, modern toolkits do all their rendering client-side anyway, then ship the contents back to the X server either via XPutImage or DRI buffers. The thick, chatty X11 protocol was kind of a good solution in an era when CPUs and graphics hardware was slow, memory was limited, and there was no shared-library mechanism available. Well now we have shared libraries (so all apps can share rendering code), buttloads of RAM, fast CPUs and even faster GPUs. Wayland makes it possible to leverage all this capability without paying for the cruft and hassle of X11.
Of course if you want X11, it will still be there in the form of Xwayland which is one of the major reasons why it has enticed users and developers: it provides a clear, backwards-compatible, endorsed upgrade path from X11.
Packard is one of those who endorse Wayland as X’s successor. Jim Gettys appears to have his attention turned to the bufferbloat problem; the last word of his that I could find on Wayland was in 2010, back before it got the developer momentum it now enjoys.
Nice try :) But this time I’ll dismiss you because it’s possible to do network transparency with Wayland in such a way that neither the compositor nor the remote apps need be aware of the network at all. This will require a Wayland compositor running remotely that, instead of displaying the contents of its framebuffers to the remote system’s display, instead transmits them via some protocol like RDP or VNC or even X11 to a local Wayland client which renders them on the local Wayland compositor. That way any Wayland application can be exported — securely — over the network.
But you don’t even need to go that far; Weston has RDP support built in.
Most desktop users have no need of network transparency; the Wayland architecture lets you not pay for what you don’t use.
Well, if you want Plan 9, you know where to find it :) I actually think that rio offers a much better approach to the windowing problem than either X or Wayland, but is difficult to get working on a system that isn’t Plan 9.
Really, you want to keep things like display servers out of the kernel; Wayland operates entirely in user space outside of the DRI and KMS stuff. It’s as close to optimal as we will get with Linux and its APIs, although the plan9port for Linux has a server which talks the rio protocol and draws to your local X display.
>Packard is one of those who endorse Wayland as X’s successor.
Not according to Jim Gettys he isn’t, as of the last time I was FTF with Jim about 8 weeks ago.
I am well aware of the state and level of Wayland support. As is not unusual for you, you have made claims that are not merely false but ludicrously easy to disconfirm. I can only conclude you are either delusional or do not care that people who investigate them will swiftly discover you are talking complete shit.
@Jon:
>I’m sorry, I have to troll a bit here: Doesn’t OS X’s desktop environment count as a braindead design decision? Or is it trivially easy to swap MATE in to replace it
I think environments are likely just mostly matters of taste. People used to think that Motif was a good idea, too, and lots of people claim to actually like Windows.
OSX is fast, the multiple desktops implementation makes sense to me (although the full screen notion needs to either go away of get a serious work over), The hardware is nice, and a 5 year old machine still feels fast, and barring a coffee incident or me dropping it, I expect to last with probably one more disk-life. Most things have sensible defaults or are straightforward to change. The unix is solid. Most open source I run compiles cleanly. I have to run Adobe’s stuff sometimes, and that has become a UI disaster, but that isn’t Apple’s fault.
I know it is a political issue for some not to use a closed OS, but my commitment is limited to the life of the machine, the only closed data formats are what clients would be sending my anyway, and it works for me.
The agesim on this thread makes me sad. Don’t get me wrong, I love seeing a good natured revival of the editor/window manager/os/philosophy wars but let’s leave who we are out of it.
You might finder older techyies in these debates because they are old debates. People can be “stuck in their ways”, at any age.
>The agesim on this thread makes me sad.
Young people these days. So rigid in their thinking.
:-)
Ah VCS – imagine all of the tools in your garage are inexplicably connected to each other in random ways that are related to the manufacturing process of the tools themselves and each connection has no readily apparent relationship to the tools actual intended function. Ah VCS.
>Persons new to the blog are cautioned that Jeff Read’s contact with reality is, at best, tenuous.
That presupposes that Eric’s definition of reality is the only correct one.
>That presupposes that Eric’s definition of reality is the only correct one.
No, only a basic willingness to check assertions against factual evidence readily available from public sources.
>No, only a basic willingness to check assertions against factual evidence readily available from public sources.
Thanks for the qualification.
@kurkosdr: It amazing how much stuff you pull out of what I didn’t say. My comment was about IDE’s in particular. I find that IDE’s *tend* to obscure knowledge of the underling code being worked on. They often have a custom build system, that can’t be used standalone. They may depend on non-standard toolchain elements. They often have “wizards” that generate difficult-to-maintain blobs of cookie-cutter code. They tend to be “walled gardens” of development, where a port is required to lift the code out into a form that can be built on the command-line.
These are all generalizations, and I am sure that some people will be tempted to say: What about “X” (X being a tool I have never used), it doesn’t do bad thing “Y”. Ok, fine. It does not do that *now*. But I am describing tendencies that infest the whole breed.
I have no opinion about Wayland (having never been on a system that uses it). I use X11 network connections all he time in my line of work, and it’s clear to me that it requires a fast network between the machines in question to work properly at all (luckily, the machines I use it between are all on a common server farm).
Speaking of “factual evidence readily available from public sources,” can either Jeff Read or ESR point to any on-line statements from Keith Packard on this, as opposed to either unsupported assertions (Jeff Read) or hearsay received via private communication (ESR)?
(Not that I have a dog in this fight: I’m someone who will use unix as a desktop OS when it gets shoved into my cold, dead hands.)
> They tend to be “walled gardens” of development, where a port is required to lift the code out into a form that can be built on the command-line.
As I understand it, most IDEs provide command-line build tools. They won’t, therefore, build without the IDE installed, but in principle the same can be said of autoconf (sure, you can _build_ an autoconf project without autoconf, but try maintaining one. You still need to do a sort of port if you want to quit using it.) or imake or even make itself.
(As for figuring out he was talking about emacs, I figure anyone who didn’t get it immediately hasn’t been paying attention.)
The idea of an “endorsed successor” seems almost feudalistic – the maintainers of old tools getting a veto power over whether their – let’s be honest, their competitors – are allowed to become popular.
To follow up, *why* do IDE tend to be like this? Because they usually proprietary tools that are built to be sold for pricey site licences to large corporations who tend to standardize on them so that they can hire folks certified in them in places like India (or Mexico nowadays) and ensure that consistent, if not inspired, code can be cranked out in large quantities. Cookie-cutter development by cookie-cutter (and easily replaceable) developers to on cookie-cutter tools to generate cookie-cutter code that fits into well-defined uses.
So, no, I am not down on settling for tools designed for low-value wage-slaves. I want to be the one who understands what is going on and knows how and when to go outside the box, and how to solve problems that weren’t thought of by the designers of the cookie-cutter tool and/or don’t fit into well-defined development categories.
{{Gets into Nomex suit… Presses POST.}}
Alsadius wrote:
> Mark: I’m a Windows diehard, and even I think that would be kind of awesome.
On the other hand, what would not be possible, ever: a complete DVCS repo of Windows from 1.0 to 8.1. MSFT has apparently lost much of the change history, in many key points actually LOST THE SOURCE, over many early timeperiods was not using any source control at all, and and in many parts of development were using internal-only and/or deeply proprietary (with now lost tools) source control systems.
The only IDEs I can speak of with any significant personal knowledge are Eclipse and NetBeans, and both of them embed or call out to standard underlying build tools either by default or with a minimal setting change (Ant and Maven for Java, make for C/C++), and I’ve never had issues reproducing a one-step build from a clean repository checkout.
Mark: For clarity, I’m referring to your suggestion of doing it with Unix. Don’t like the OS much, but I love the idea. And even if Microsoft did have their source, it’s certainly not anything any of us could put together. (Seriously though, they lost the source of one of the most valuable creations in human history? Ye gods.)
Eric,
I don’t know these people personally the way you do, so if I made too strong a claim I’m going to back off of that.
Here’s what I do know: In 2010 Keith Packard gave a talk in which he reiterated many of the technical points behind Wayland’s rationale: that X11’s network transparency is effectively obsolete and of very little concern to modern applications, that nobody uses X11’s primitives anymore, that the other things the X server does — mode setting and input event handling — are better off done in the kernel, and that X represents a clunky and unpleasant target for developers.
I also know that in 2012 he gave another talk about the xwayland work, which will provide a backwards-compatible migration path allowing legacy X11 apps to run in the new Wayland regime. Again he reiterates the points about X11’s clunkiness and obsolescence, and suggests that if you really really need app remoting, a more efficient protocol running on top of Wayland would be better suited to the task than X11.
>I don’t know these people personally the way you do, so if I made too strong a claim I’m going to back off of that.
It’s not just that you don’t know the players. Claiming that Wayland has “universal community support” when Ubuntu has chosen a different path – and consequently the two most popular Linux distributions won’t be going to Wayland in the forseeable future – is either delusional or dishonest. Wayland is not some kind of inevitable tide; it’s an experiment that may still fail.
As for what Jim and Keith are thinking, I’m going to keep quiet about most of the specifics I know because the politics around this are a bit delicate and I don’t want to complicate their efforts to fix things. I will say this much: they agree with some of Wayland’s architectural direction, but not all of it. They see a pattern of attempts to replace X reiterating some design mistakes that the early X developers had to get past 25 years ago.
It would be unwise to assume that Keith’s thinking has not changed in some significant respects since the 2010 interview.
The Wayland developers are not competitors with the X11 developers. Mostly, they’re the same people.
It’s more like Microsoft retiring the DOS API, nominating Win32 as the successor API, and providing a migration path in the form of Windows 9x while the transition takes place.
This is the exact opposite of my experience with older developers, many of whom were quite eager to share information, some going to great lengths to document the system for future developers.
Are there people who fit this description? Yes, but I don’t think “doing things their own way and ignoring orders from management” is a function of age. The one time I was on a team with a “build guru” a) he was, as I recall, young; and b) he had long since left for greener pa$ture$ by the time I joined, stranding us with a magic binary that we had to use for deployment because the system could not be rebuilt from source.
Age discrimination in SV these days has a lot more to do with the types of projects one takes on and the money they attract. Older hackers are trying to solve the bufferbloat problem. But you won’t see any VC money chasing bufferbloat. You see VC money going to the Node.js kitten and pony apps written by twentysomethings.
I’m still running Slackware. A big part of why is great difficulty and unpleasantness syncing my brain with the complexities of Fedora or Ubuntu.
I guess I’m turning into one of those oldbies now as well.
Linux-only code cannot remotely be considered to be either the correct approach for Unix or to have universal Unix community support, even before considering details like the idea that just supporting Qt/KDE and GTK+ 3/GNOME could count as “universal” in even the Linux community.
At the same time, the runtime support for running X apps on top of Wayland is quite good, because, well, that’s what the devs have had to do in order to have anything to do when running Wayland.
Which means, of course, the way for any app dev to address the widest possible Unix userbase is either to ignore Wayland and just deliver something that uses X, or use the subset of toolkit features that allows their app to compile for both Wayland and X from the same codebase.
Hey, anybody else remember how the combination of Win-OS/2 and Micrografx Mirrors worked out for OS/2?
I was deliberately disregarding Ubuntu which is increasingly developed in a vacuum with little input from the greater Linux community. Canonical has acquired a reputation for being hostile to that community, and Mir is the most oft-cited example. Kubuntu and Xubuntu have pledged not to support it. Intel made a big thing about removing Mir support from their video drivers.
How much of a community can you piss off and still claim to be part of it?
But it may succeed! X has already failed.
>The Wayland developers are not competitors with the X11 developers. Mostly, they’re the same people.
The products compete, regardless of being developed by mostly the same people. And for this feudalistic model it matters less who the developers are than who the leaders are.
Since moving to Source Depot (a fork of Perforce) in ~1999, Windows has a complete source history without any gaps or omissions, so if you were interested you could look into everything in the last 15 years. The various horrors you describe all occurred in the first decade of development, when things were a lot more chaotic at MS.
Also, do you have a source for the claim that they actually lost the source at several key points? Not that I doubt you, but this is one assertion that I haven’t heard before.
> Also, do you have a source for the claim that they actually lost the source at several key points? Not that I doubt you, but this is one assertion that I haven’t heard before.
This isn’t a citation, but I think I read somewhere that they lost the source to some early versions (maybe 3.1, maybe pre-3.1) long after those were no longer the present version.
The other project fitting that sort of timeframe is OpenOffice.org, which started with some proprietary horror the changes to which I think are now lost, went through several repo conversions (some so disastrous the developers started commenting out code rather than actually delete it since they couldn’t trust the repo – Apache OpenOffice still commits new already-commented-out code) and LibreOffice’s git is now well-tended but can be patchy just when a dev actually wants to dredge through the last 15 years to try to work out why something is the way it is …
@esr:
>Claiming that Wayland has “universal community support” when Ubuntu has chosen a different path – and consequently the two most popular Linux distributions won’t be going to Wayland in the forseeable future – is either delusional or dishonest.
I’m afraid I have to agree with Jeff on this one: Ubuntu is consistently an outlier and not a reliable indicator of what the community thinks about a given project.
Part of Mir’s problem is that it was only ever designed to do one thing: run Unity. Wayland, by contrast, is intended to be a *general* display multiplexing system; under Linux, Wayland together with systemd-logind is slated to replace the kernel virtual console mechanism. This generality in goals will make a world of difference when it comes to supporting the variety of desktop environments and their sundry use cases.
I recently ended up doing some Android app development, pretty unexpectedly. I’m not a professional developer. I have been using Emacs practically daily for maybe 16 years and have lots of things in muscle memory.
Unfortunately the support for Android development with Emacs is quite sparse at this point, at least as far as I can tell. I had very little Java experience before this foray into an Android app, so I wasn’t and still am not aware of what the Emacs setup of a Java developer might be. I hadn’t spent any significant time with any IDE before this. I tried Eclipse with Google’s Android plugin, initially only because I couldn’t even get the source I’d just downloaded built any other way. I soon realized that Eclipse must be a leading cause of depression and hair loss. Google’s upcoming answer to the question, Android Studio, based on IntelliJ Idea, is a lot better, but it’s still quite annoying to an Emacs user. I get tripped up by the key combinations all the time, and after I’ve spent some time in Android Studio, I get tripped up in Emacs, too, which is doubly annoying.
The thing about Android development is the amount of xml spread out in various files, and the number of xml element identifiers that you need to refer to all over the place. The IDEs keep track of that and have auto-completion for the identifiers, including the ones defined in the Android platform, and highlighting for dangling references. This is a huge help, at least if the auto-completion UI is well-designed. Implementing this in Emacs shouldn’t be too difficult, of course, but as things are, there’s no easy, pre-packaged solution, as far as I know. Also, there would need to be integration with the build systems that are typically used for Android apps. I took one look at Maven and decided that I won’t spend any energy on trying to learn it, so I just press “build”. Doing the equivalent in Emacs or on the command line turns out to be complicated, which is another reason to put up with Android Studio.
Why _hasn’t_ anyone ever designed something as simple as a compiler which emits instructions for where errors (and external symbols, which can be checked for dangling references by another stage) should be hilighted, and an editor that can understand these?
You don’t need an IDE for those – the former could be part of GCC and the latter could be part of Emacs, and you could probably mangle ctags into doing the job of the middle bit I put in parentheses. Yet most editors are barely able to hilight the most trivial of syntax errors [mismatched brackets, generally, are the only syntax errors I’ve seen properly handled]
The Java compiler is pretty close: It provides x,y coordinates for the spot where the error was detected.
Mikko,
I have actually shipped a game on the Android App Store that I developed solely with Emacs and the command line tools. It really isn’t that bad; the “android” utility gets you all set up with a build.xml for ant.
While it may not be your cup of tea, developing for Android without an IDE is certainly within the realm of possibility; by contrast, attempting the same with, say, a J2EE app fills me with fear and loathing.
> I have actually shipped a game on the Android App Store that I developed solely with Emacs and the command line tools. It really isn’t that bad; the “android” utility gets you all set up with a build.xml for ant.
@Jeff: could you share your setup (Emacs modes, command line tools, etc.), please?
> The Java compiler is pretty close: It provides x,y coordinates for the spot where the error was detected.
Yes – that such a thing exists suggests that there are things that consume this information. The obvious answer is Java IDEs. Is there a way to make Emacs, Vim, or any other conventional text editor do this (and do it continuously and automatically, whenever you pause in typing, and display the information in a user-friendly unobtrusive [especially for warnings] manner?)
> It really isn’t that bad; the “android” utility gets you all set up with a build.xml for ant.
Yes, I’m aware of this, but I was contributing to a pre-existing project that used Maven and had been developed with Android Studio. I’m sure I could have made it work using Emacs, but it seemed like a lot of extra work for not much gain. I contributed a bunch of small features and a couple of translations to the app.
> While it may not be your cup of tea, developing for Android without an IDE is certainly within the realm of possibility
I do realize this and I have gone out of my way to stay within Emacs in the past, writing (very primitive) bits of Elisp etc.
> by contrast, attempting the same with, say, a J2EE app fills me with fear and loathing.
This exactly the sort of thing I had no idea about, since I had next to no experience with Java. The learning curve is pretty steep when you start to look at the various bits and pieces for Emacs that might be helpful.
>> The Java compiler is pretty close: It provides x,y coordinates for the spot where the error was detected.
> Yes – that such a thing exists suggests that there are things that consume this information. The obvious answer is Java IDEs. Is there a way to make Emacs, Vim, or any other conventional text editor do this (and do it continuously and automatically, whenever you pause in typing, and display the information in a user-friendly unobtrusive [especially for warnings] manner?)
For Emacs there is flymake (an on-the-fly syntax checker, using external tools) and Semantic / CEDET (written in Emacs Lisp… though not without problems; I have encountered some stalls so I had to turn off parts of Semantic).
Nothing special, except for use of the BSD indentation mode in Java. Just Emacs the Java toolchain, ant, and the ‘android’ tool can get you pretty far.
Here’s the thing: X has already been replaced! The dominant graphical architecture for graphical Unix installations is now some sort of simple local shared-memory framebuffer compositor like Wayland, rather than a network-transparent server that talks graphics primitives like X. This is even true if you count only desktop workstations, where Mac OS X dwarfs all would-be competitors, but it gets even worse when you count all those iThings and Android devices floating around out there.
Abandoning X and switching to Wayland is necessary simply to bring desktop Linux up to the current industry state of the art.
I enjoyed this Daniel Stone talk from 2013 about X and Wayland (45 minutes, but an educational 45 minutes), detailing all the reasons X needs a stake through its heart. Of course, that this is the case in no way means Wayland won’t have a horrible case of second-system.
So, where the hell is the result in all its glory?
Source or gtfo, he-he :)
>So, where the hell is the result in all its glory? Source or gtfo, he-he :)
Two early trial conversions are here: https://gitorious.org/emacs-transition
Pardon my flawed English – I mean the output of the “ugliest conversion ever”, not the tools used for the conversion. Where can we see git repo of emacs with all the ancient history incorporated? And when will it become official repo used by devs – I mean when current bzr repo will retire and will be replaced by your work?
Jasper St. Pierre’s Xplain series is a must read:
https://magcius.github.io/xplain/article/index.html
It’s an in-depth explanation (with working embedded JavaScript implementations!) of how X works — and why X desperately needs to be replaced. In particular, in a compositing window environment, it’s extremely difficult to remap input events without full cooperation from the X server — in effect building the X server into the compositor. This motivates Wayland’s design where the roles of window manager, display server, and compositor are assumed by one program.
Concerning the history of vi, only thing I have been able to collect is on https://gitorious.org/nvi if anybody is able to get more history (and/or link those two repos), I am all ears.
LWN statistics is meant for example https://lwn.net/Articles/620827/ (with tools available on git://git.lwn.net/gitdm.git)
But, looked at from the very short distance of 3 years, that new repo has been running smoothly ever since and pretty much everybody is happy now, even if some of the notoriously crusty folks still complain occasionally about how complicated git is.
Looks like the conversion was a job well done.
Apologies for the zombie post!
>Looks like the [Emacs] conversion was a job well done.
An interesting contrast I meant to bring up earlier: the GCC history is an order of magnitude larger than Emacs’s, but has way fewer defects and weirdness in its backtrail. It’s tough to spot even the CVS to SVN transition, which usually leaves a trace as obvious as the K/T shock in the terrestrial geological record. If not for the crushing bulk of the data, the GCC conversion would be almost straightforward by comparison.