It’s not much of a secret that I loathe autotools and have been seeking to banish that festering pile of rancid crocks from my life. I took another step in that direction over the last four days, and have some interesting statistics to report.
I have been muttering dark threats against the autotools-based build system in GPSD for years. All the usual reasons applied – it’s brittle, clunky, slow, and a huge maintainance headache. There was one piece of it in particular, the place where the generated makefile calls setup.py to make some Python extension modules, that was just hideously bug-prone.
But one of my lieutenants doesn’t like Python. That, and too much other stuff to do, kept shooting autotools through the head just far enough down the priority list that it didn’t get done. Until we had an embedded user show up (Thomas Zerucha from my post on bookend consistency) and complain that our autotools build takes nine eternities to run on the little ARM device he’s targeting.
That did it. Out came the notional .45. Die, you monster! *BANG*
It’s now about a week later; I have a success story and some interesting statistics to report. But first, let us set the scene. GPSD is a mid-sized project at about 41KLOC of C and Python in 120 or so sourcefiles. Our build products are eight binaries, five Python scripts, a moderately complex Python extension module, and sixteen man pages. The build system is also the framework for our regression-test suite, a large and important component that goes far towards explaining GPSD’s exceptionally low defect rate.
The autotools build system is a bit over 2100 lines of stuff. That’s just counting Makefile.am, configure.ac, and autogen.sh; if I counted all the auxiliary scripts and crap autotools leaves lying around (aclocal.m4, anyone?) it would balloon up to 14K.
It took me about 6 man-days of concentrated effort to convert this to an SCons recipe. Most of that effort and time was spent comprehending what the autotools build is actually doing; replicating it in SCons once I understood it was relatively easy and actually kind of fun. I guess I should explain that…
An SCons recipe is a Python script which evaluates to a set of production rules making targets from sources. The rules can be, and usually are, chained together; you tell the system to make a target and it fires the minimum set of rules to achieve that. This is familiar territory to anyone who’s ever written a makefile.
One thing that makes SCons tremendously more powerful than a makefile is that you get to use a full programming language to generate the production rules. And not just a lame macro language like autotools, either. One of my devs spotted an opportunity: where I had a utility production that sequentially invoked fifteen variant splint(1) commands, he turned into a data table that the recipe walks through using a Python loop, generating a production rule on each tick. The table is far easier to understand and modify than the code it replaced.
2129 or so lines of autotools code turned into 1057 lines of SCons recipe. That’s a full factor of two shrinkage, but it actually understates the gain in readability. Where the autotools builder is written in an incongruent mixture of three quite dissimilar languages (shell, m4 macros, and make notation), the SCons recipe is just Python calling rule-generator functions. It’s a refreshing change being able to look at a build recipe that size without feeling like you’re inexorably losing sanity points thereby.
But the real surprise was the build time. Pretty consistently the SCons build completes almost three times as fast as the autotools build. On my 2GHz Intel Core Duo, wall time drops from 50-odd seconds for configure/make to about 20 for scons.
Not all is perfection yet. There’s one minor component (the Qt client library) that scons can’t build yet, because it requires idiosyncratic build tools of its own that are stapled into the autotools build in ways I don’t understand yet. The real problem is, as as usual, that any random autotools build can be expected to be a nigh-incomprehensible mess.
A more serious issue is that I haven’t figured how to do out-of-directory builds in SCons. That’s a fragile and difficult feature in autotools; we haven’t got it quite right, last I heard (I don’t need or use VPATH myself). But at least it’s theoretically possible, and our Mac-port guy is probably going to become difficult if I have to tell him he can’t have it.
This is my second scons conversion. The first, for Battle For Wesnoth, went pretty well too. Thomas just checked in and loves the speed increase. I’m feeling happy about having tremendously reduced GPSD’s long-term maintainance burden.
Say it with me…no, yell it good and loud: “AUTOTOOLS MUST DIE!”. SCons is not the only competitor to replace it, but it’s the one I recommend.
The only experience I have with comparative build systems in a reasonably complex environment is the build system for Allegro, the game programming library, which built with autotools for a while, then shipped scons as an alternative, and as far as I know when I last saw it shipped with scons and cmake build options. I very much like cmake, from the user’s perspective. The ccmake UI is the best I’ve ever seen for configuring software (several orders of magnitude better than repeating ./configure –help; ./configure OPTIONS until you get it right), and the percent-done output it builds into its output makefile is really nice. I can’t speak to the issue from the developer’s POV, really, since everything I’ve written is simple enough to build on static makefiles (either that or I don’t care about others being able to build it), but cmake is really great for the user.
“2 *Mhz* Intel Core Duo”? Could I interest you in a 20Mhz 80286 as an upgrade? :)
ESR says: That was a funny typo. I’ve fixed it.
2Mhz – I think you mean GHz, a constant tripup for us that started in the days of 8 inch floppies on, yes, 2Mhz machines.
I havent looked through the scons docs, but I can actually read the recipe. And I think with very little effort I can make modifications. I suspect even large changes will now only take minutes and simple patches v.s. autotools. I need to download and get reading.
Don’t forget libtool – that involves an even worse layer of junk (the real stuff is among the .libs not called by recognizable names). autotools generates ./configure which is a horrid bash script. scons just runs directly. That makes the iterative fix loop so much faster.
If nothing else, by finally getting you to assassinate autotools I think I have accomplished a great deal toward the maintainability and quality of gpsd. (I’m already starting to look at nofloat).
Now if only the larger projects like Gnome and the gnu.org FSF archive would similarly end their captivity. While the debates over X v.s. Wayland, Compiz v.s. Gnome desktop, or the rest, the elephant in the room is that autotools is a far worse pile of compost than anything they are debating about. It belongs somewhere they buried COBOL. Preferably with its head cut off and a steak through its heart.
I also note that the Autotools must die post was about one year ago, if I read the dates correctly.
The main defense of Autotools I’ve heard is that it is easier for distro packagers because they’re used to working with it. This sounds fishy to me, but I haven’t built enough packages to say.
Nice, Eric. I’ve had similar thoughts/experiences with Rake/Ruby. Same basic deal, if I understood your description of SCons. It’s got a Make-like syntax, but when you need to do something, you’ve got a real programming language available.
Cheers,
–Bob
The waf build system gets better marks than scons from folks I trust. I believe it started as an scons fork. Also, take a look at bento. Bento is in a preliminary state, but has been able to build scipy and numpy.
>The waf build system gets better marks than scons from folks I trust.
I looked at waf a couple years ago when I got fed up with the autotools build in Battle For Wesnoth. Had to reject it at the time because the documentation was so scanty and poor that I couldn’t even reach an understanding of its build model. It’s good that that has been fixed, and I just read the waf book.
My take on it now is that there is some very clever thinking there, but it looks like a system that would be hell to use if you step even a little outside the author’s assumptions. The lack of support for dependency forests – as opposed to a tree with a single apex, an assumption that seems to be wired into the design pretty hard – is troubling. That in itself would be a downcheck for GPSD, where we use the build system as a control framework for our regression testing.
On the other hand, I can completely believe it’s faster than scons. The task/workflow-orientation of the dependency declarations would lend itself to all kinds of nifty optimizations, and I’d bet that throwing multiple cores at the build gets you much larger gains than with other build systems.
Scons is nice. Syntax is convenient. Glad you like it.
Now try compiling a large project with multiple SConscripts laced throughout. Under Windows. Watch your computer grind to a halt as it must read and process every bloody SConscript before doing a thing. Under Unix it’s pretty sluggish, but Windows you can grab a quick lunch before it even starts spinning the first .cc file into object code.
I won’t disagree that the autotools are a blight on the planet, but I have to deal with SCons’s slowness in my professional life. I dunno, maybe I’ve been spoiled by BSDmake builds which start stitching together rather sophisticated systems (entire kernels and userlands for instance) without hesitation.
>I have to deal with SCons’s slowness in my professional life.
I think the problem here is that build systems scale badly in general. They have to, because they have several intrinsically O(n**2) problems at their core. At no point during the parsing of the recipe for a system can you guarantee that later parsing won’t declare a critical cross-dependency – you have to read the whole recipe, then check for those in a process that is O(n**2). Or possibly under optimistic assumptions O(n log n).
Big makefile trees ignore this problem, relying on unenforced chunking conventions about where any given makefile can touch things to break the cost function into a sum of much smaller O(n**2) terms. But the problems with this approach are well known — see Peter Miller’s classic Recursive Make Considered Harmful. You get speed only at the expense of sacrificing correctness guarantees.
Thus, I think you’re blaming SCons when you shouldn’t. I predict confidently that changing build systems would give you only a linear speedup or slowdown, not affecting the dominating O(n**2) term. In particular, waf would probably be faster, but the gains would be swamped by even small increases in your codebase and recipe size.
Nothing beats make. Trouble is that you have to use lots of tiny little targets and files, which ends up needing machine help … by a program which nobody has written yet (afaik).
>Nothing beats make.
No, what you actually said is that nothing would beat make plus a helper program that presently exists only in your imagination.
You know my next line, Russ: show me the code.
Hercules is an autotools package. Matt Zimmerman did the original setup, and it’s been hacked on by several of the other developers since.
I’ve learned to hate cmake through my work on the Second Life viewer. Eric’s pointed out to me in private that it’s made by Germans, and thus suffers from the typical German mindset: overengineered, elegant, and highly complex. Every time I poke at it, I’m reminded of a story the aviation writer Gordon Baxter told about cars. It seems that, in the late 1950s, his family had, between them, four Mercedes cars, two 190s and two 220s. He got rid of his when, one day, he pulled out the ashtray to fix a minor problem, and discovered the lid had 27 components.
I’d like to shoot autotools in the head too, but what to use? cmake is right out. Whatever we use has to work for Microsoft Visual C (no C++ here, thank $DEITY) as well as on Unixoid systems; driving xcodebuild instead of make on OS X would be a win (since xcodebuild will use as much horsepower as you have without having to figure out what -j parameter to pass to make). Scons sounded good until the comment about taking forever to actually do anything on Windows. Then again, doing Windows is optional, as the MSVC port of Hercules doesn’t use make, either; it’s got its own MSVC solution file (though I’d like to bring it back into the fold if possible).
>Eric’s pointed out to me in private that it’s made by Germans, and thus suffers from the typical German mindset: overengineered, elegant, and highly complex.
You misremembered slightly. I don’t think CMake is elegant, at all. It’s yet another makefile generator. What were they thinking?
Before CMake we had a lot of experience with previous two-phase builders like autotools and Imake. At scales above “toy project”, none of it was good. The same problems keep recurring, and the top two are (1) as a back-end language makefiles are just too weak, (2) the decoupling of generation from execution results in a design that is difficult to reason about and recipes that are purest hell to debug.
I won’t say I’m certain what’s right in the build-system space, but I am certain that makefile generators are wrong.
>Scons sounded good until the comment about taking forever to actually do anything on Windows.
See my reply to Jeff Read. Anything will take forever at the project scale of the Second Life viewer, and Windows imposes performance penalties of its own.
I keep meaning to write a formal response to Miller’s piece, called “Large Projects Considered Harmful”….
Indeed, Microsoft is partly to blame here; their filesystem performance is crap compared to Unix. Us Unix weenies are spoiled by the inherently file-centric model of our chosen OS. :)
I see no reason to run builds directly on embedded ARM hardware when gcc and other C compilers are perfectly capable of cross-compiling for ARM on virtually any other hardware platform.
I AM THE FIRST TO SAY IT IN THE COMMENTS:
– AUTOTOOLS MUST DIE!!!
DIE, YOU STEAMING PILE OF CRAP, DIE!!!!
That made me feel better.
Now we just need to move larger projects away from Autoconf.
Typo: I think you wanted “One thing that makes SCons tremendously more powerful than a makefile”
ESR says: Correct. Thanks.
> Nothing beats make.
Actually it tends to be bludgeoned into submission or requires editing the first dozen to hundred of lines when you change builds. Autotools was the abomination that grew out of the attempt to overcome cross-platform problems by testing every little thing and trying some automation.
> I see no reason to run builds directly on embedded ARM hardware when gcc and other C compilers are perfectly capable of cross-compiling for ARM on virtually any other hardware platform.
Except you aren’t just doing plain cross compiles you are doing a build to work in a particular system. If there was ONLY a Makefile in gpsd, it wouldn’t be a problem, but there isn’t any until you successfully run ./configure or ./autogen.sh. I have to create a complete cross system with all the headers and libraries which ARE DIFFERENT than the host system and spend hours setting up paths, and then usually Autotools will blow up on something, usually much later in the ./configure or make. And it also needs a config.h which has to be generated. If it finds bluez on my desktop but it isn’t on my chumby or vice versa?
The whole reason there is Scratchbox is so that you can get something that runs on your fast desktop that can be a virtual machine running the target processor. I have one for my Nokia tablets (it will produce and run ARM executables) – but even that is finicky to setup. I don’t have one for Chumby.
Consider Debian – they build native distros because too many autotools scripts are hopelessly broken (seeing if the compiler produces runnable executables … no). There are too many packages to fix, and because autotools are such an abomination they would consume too much time.
Assuming that nobody has done it without me hearing about it, I’m surprised nobody has written a build server that stays in RAM persistently, since such a big part of starting the build process is always re-re-re-re-re-re-re-re-reading the build declarations off the disk. Combine that with inotify for noticing both build declaration changes and source file changes and you could have something pretty blazingly fast for incremental updates and probably not-actually-slower for from-scratch builds.
The problem is that I suspect you’d have to build your build system to work this way from scratch, not retrofit it in on later.
Let me make the point more clearly – it isn’t make v.s. SCons, it is autotools (autogen, automake, libtool and the rest) v.s. SCons.
As soon as you start to do things for cross-platform portability, make gets complex. Even Linux uses a series of scripts, and it doesn’t have to worry about libc and libz variants as it has them inside and the makefiles are very similar, they don’t have a lot of twisty passages.
Also different are GCC and binutils always work because they have to be bootstrapped and do cross compiles.
I don’t think you can completely cross build a fully functional version of the simplest X-with-desktop linux distro.
Why not? The entire (custom) Linux distribution on the Cisco/Linksys NAS 200 is built using a cross-compiler, and Xorg itself can be cross-compiled.
You don’t really have to worry much about cross-compiling GNOME (most embedded Linux distros don’t use it) but I do know that GTK+ itself, GTK+/GLIB apps, etc., can all be cross-compiled.
For all those complaining that scons is too slow, have a look at ‘ninja’. One of the chromium guys wrote it to replace their (slow) scons build. Also see http://neugierig.org/software/chromium/notes/2011/02/ninja.html
Hm. Wonder how much work it’d be to make a build system that drove ninja…might be simpler to get both configuring and building correct, and with reasonable performance.
Eric,
“In all your copious amounts of spare time” :-D , would it be possible to document how you did the conversion, or at least make some notes so that another author could possibly write an “autotools to SCons migration guide”? I appreciate that this is a gigantic request, and would probably just be another huge suck on your time, but the community could benefit from your work, and it would help prevent “reinventing the wheel”.
>would it be possible to document how you did the conversion, or at least make some notes so that another author could possibly write an “autotools to SCons migration guide”?
I’m sitting here trying to think of what such a guide should say, and I’m coming up blank.
There is a small bag of Scons tricks I could write down, but they’d be more of a tease and a distraction than real help, I fear. SCons just isn’t very difficult. Most of the effort in such a port would be better approached by a document titled “How to rediscover what your hairy horrible old autotools build is actually doing without succumbing to the urge to tear out your own eyeballs”.
> Why not? The entire (custom) Linux distribution on the Cisco/Linksys NAS 200 is built using a cross-compiler, and Xorg itself can be cross-compiled.
> You don’t really have to worry much about cross-compiling GNOME (most embedded Linux distros don’t use it) but I do know that GTK+ itself, GTK+/GLIB apps, etc., can all be cross-compiled.
If you pick and choose a limited number of packages, yes, but that is specifically why I said a complete if minimal desktop. I always end up requiring something that doesn’t cross compile properly, even at the low level.
Jeremy: I’ve seen a working implementation of such a resident build system in the wild before, but I can sadly not find it again.
As for CMake, it has the downside of having to have CMake present on any system you want to build your software on. That’s due to their intentional misfeature that all paths should be absolute in any build scripts they generate. It makes it near impossible and quite unsupported to move a generated configuration, or bundle it with source archives.
For my personal projects, I’ve been using premake4 lately, which like scons benefits from having a full featured language (Lua) for its build descriptions. As a bonus, you only need to deploy a single static binary to generate build scripts, and the build scripts are properly movable.
>I’ve been using premake4 lately, which like scons benefits from having a full featured language (Lua) for its build descriptions.
What a marvellous idea! lua would be extremely well suited to that kind of use.
I must investigate this. Might be good foe places where Pythin is too heavywight.
SCons is easy to understand. I’m going through the manual now. I’ve figured out most things I do in makefiles already – it is actually easier.
Autotools is the big complex box. I have the autotools book and have only cracked it open a few times. I can’t find it right now, but I don’t consider it a big loss. I think I have cobol books too.
If you can decode autotools, it is easy to figure out how to do the same thing in SCons.
I’ve had to use lua a couple of times in products where it was used for extensibility and I found it to be more difficult than python. While i’d certainly believe that the python executable is a bit heavyweight(erlang has spoiled me for multiprocessing forever), i find the language itself to be quite clean and consistent.
In other words i’m uncertain what lua would improve over python. Single static binary would be nice a bonus feature.
>I’ve had to use lua a couple of times in products where it was used for extensibility and I found it to be more difficult than python.
Yeah, the reason to choose Lua is that it has a very small footprint. Its design goals are quite different from Python.
> In other words i’m uncertain what lua would improve over python. Single static binary would be nice a bonus feature.
Three I can think of.
1) Performance, although that’s less than important these days for build times.
2) It’s relatively easy to write code that grinds out Lua source. I haven’t tried this for Python but I don’t think Python is as well-suited to this.
3) Lua is practically made of hooks. Metatables are absolutely perfect for constructing analogies to complex Make rules.
With 200 lines of Lua, 350 tops, I think I could create a Make-system library so simple and seamless it would appear to have been designed into the language. Object-oriented class systems for Lua are routinely made this way.
Actually, I just made a real simple one in 46, and I didn’t need metatables either:
http://pastebin.com/p0xZdjV0
I’m not saying you would actually use this, but it does demonstrate Lua is suitable for this kind of thing.
I was operating under the knowledge that python and lua performance wouldn’t be that much different to one another but a quick look at shootout.alioth definately suggests otherwise. Maybe it’s time for me to examine lua outside of the crippled jails that i’ve been using it in. As you say however, performance (unless the memory required rules it out) is less of a draw than ease of writing.
This is however a much more compelling argument. While desperately trying to find some form of documentation for the modding frameworks that were (ab)using lua, i found references to meta-tables but i daresay it was crippled in some way because it never seemed to do what was written on the tin.
See above about trying lua outside of its jail cell.
Is make/scons necessarily O(n^2), or merely O(e) where e is the number of direct dependencies? Sure e could be O(n^2), but I doubt large projects scale in that insane a fashion.
“Most of the effort in such a port would be better approached by a document titled “How to rediscover what your hairy horrible old autotools build is actually doing without succumbing to the urge to tear out your own eyeballs”.”
Which sounds actually like a pretty useful document to have.
There are two praises for SCons here:
1. It’s simpler then autotools.
2. It’s fast.
The #1 is remains to be seen. SCons may be simple if you only support couple of platforms but it’s not entirely clear how simple it’ll be after suppor for all the zoo out there will be added… last time I’ve checked SCons had not support for many fundamental things (SONAME is one example – and this is whole can of worms if you want cross-platform).
As for #2… I think all current OSS build systems are intrinsically broken. They all are /sbin/init descendants. What you really need is systemd :-) To make system simple, correct and UBERfast you need an LD_PRELOAD module. In place of the file (end result or intermediate file) you put simple instruction which explains how to build this file – and that’s it. When the file is actually requested the command is run, the result is cached and all file accesses are stored besides the cached file. If prerequisites are not changed – you use cached file, if they are changed – you call the appropriate actions.
I’ve actually seen such a system in action, but it used some proprietary tools. Very impressive. You don’t ever need to think about dependences and it parallelizes build as much as possible. The biggest drawback is the fact that you never know when the result will be available but with 1000 ditscc servers it’s available so fast that it’s not a big deal…
>SCons may be simple if you only support couple of platforms but it’s not entirely clear how simple it’ll be after suppor for all the zoo out there will be added
GPSD will be a good test for this. On the basis of my experience with the Battle For Wesnoth scons conversion I’m pretty confident.
Avery Pennarun has been working on making djb’s (unreleased) redo into a fully-functional, general build system. The whole thing’s implemented in python, and .do files are written in what’s effectively a DSL derived from sh. It doesn’t have a lot of language- or environment-specific features (yet), but it’s working up to them.
redo is 90% there, but it’s still /sbin/init. It still depends on your declaration of dependences. With LD_PRELOAD module you NEVER track dependences at all! Your build system command knows what it needs to build this or that file, why do you need to supply this information separately?
Perfect example. Combine this problem: http://cr.yp.to/redo/honest-nonfile.html. What will happen if you use gcc’s -MF switch as Avery recommends? Right: redo will still not work correctly. But LD_PRELOAD build will work because it will actually know that stdio.h in current directory was tried (and rejected).
One additional note about LD_PRELOAD-based build: you must return success from functions like open(2) or stat(2) immediately and only fire up the actual build process when someone calls read(2) – otherwise your build is be sequentual. This is what makes the whole thing both powerful and tricky: open(2) is easy but what to return from stat(2)? Perhaps first version should only postpone access(2) and open(2) for the reliability sake…
Probably I should just find a time and write such system…
khim, what happens when you have statically linked modules? Not everything is a big pile of DLLs.
And I won’t touch anything that litigious asshole DJB wrote with a 39-1/2 foot pole.
Fortunately, that link is to someone else’s implementation of
redo, and not a bit of the code is DJB’s.SCons does not scale. See http://www.electric-cloud.com/blog/2010/07/21/a-second-look-at-scons-performance/ and http://www.electric-cloud.com/blog/2010/08/11/the-last-word-on-scons-performance/. For any project of significant size (thousands of objects), the overhead introduced by SCons is crippling.
>SCons does not scale.
You did an excellent job of instrumentation and analysis, there. The only piece of it I question is the supposition that incremental builds can escape incurring O(n**2) rescan overhead – it’s not at all obvious to me why that should be the case. Thus, I reiterate my previous analysis; I think you have found a general problem with build systems that properly check dependency correctness and (like Jeff Read in the comment I was replying to) mistaken it for a specific problem with SCons.
This theory should be easy to test. You should re-run your profiling with other single-phase build systems having the same correctness properties as scons – waf, for starters. I predict you will run into exactly the same quadratic performance problem. See my earlier comment for discussion of how make and make generators avoid the performance problem at the expense of correctness.
Finally,. I note that GPSD has around 120 source objects. Counting various intermediate stages that’s about 150 objects. At this scale I have observed it to be dramatically faster than configure+make. Projects in the size range of GPSD are quite a bit more common than the huge sizes you were simulating. Thus, I think that saying “scons doesn’t scale” is technically correct but misleading about its performance on a large and important range of use cases.
Hm. I do wonder how much SCons overhead scale with size of the project: if you’re compiling 50K source files with real code in them, instead of 50K copies of hello world, is the performance still a significant difference?
That would argue against my next option. The advantage of SCons is that it produces demonstrably correct dependency results, but the cost i an exponential increase in time. Is it possible to get significant improvements in time by sacrificing a bit of correctness, and maybe build something you didn’t have to?
>The advantage of SCons is that it produces demonstrably correct dependency results, but the cost i an exponential increase in time. Is it possible to get significant improvements in time by sacrificing a bit of correctness, and maybe build something you didn’t have to?
Oh, ghod. Not exponential *shudder*. Quadratic. Exponential scaling would make it utterly unusable.
>Is it possible to get significant improvements in time by sacrificing a bit of correctness, and maybe build something you didn’t have to?
Well, we know that make and make generators do in fact get better performance by sacrificing correctness guarantees. But you’re making a wrong assumption here; the problem with losing those guarantees is not that you might rebuild something you don’t have to, it’s that you might fail to rebuild when you need to and wind up with a binary that doesn’t reflect the source state.
That’s indeed the danger of not thinking about it. However, if you know you’ll always fail in the direction of building too much, will you gain enough speed to offset the costs of the extra builds and be a worthwhile improvement?
>However, if you know you’ll always fail in the direction of building too much, will you gain enough speed to offset the costs of the extra builds and be a worthwhile improvement?
I doubt it. You have to incur O(n**2) overhead anyway in the scanning phase, if only to detect *.h includes and check that the headers exist. During that process you’re going to pull all node metadata, including timestamps, into core. Even if it costs O(n**2) to compare timestamps you’re no worse off than before, and I think that’s the only overhead you can remove by not trying to avoid extra builds.
ESR says: I now think this estimate was incorrect. See my followup blog post.
I’m now planning on writing an essay-length blog post on scaling and performance properties of build systems. Please defer further comments on that topic until the post is up.
All right, so the theoretical discussion aside, I had another question: Is this a case where Python is in fact the wrong language to do the problem in? Now that it’s been demonstrated to work, if more slowly than desired, can speed be regained by casting it in C? Or <shudder> C++? Or is the lost speed due to things like disk I/O that are equally slow in any language?
>All right, so the theoretical discussion aside, I had another question: Is this a case where Python is in fact the wrong language to do the problem in? Now that it’s been demonstrated to work, if more slowly than desired, can speed be regained by casting it in C? Or C++? Or is the lost speed due to things like disk I/O that are equally slow in any language?
I’d have to measure to be sure, but I’d be quite surprised if Python is the bottleneck here. Moving to a compiled language generally gives you only a multiplicative improvement in efficiency, and that’s not going to make the O(n**2) term here go away.
Another problem of a different kind with implementing an SCons-like system in a compiled language is that you’d either have to accept not having a Turing-complete language available for generating rules or implement one de novo at considerable complexity cost. I think the single most clever idea in the original ScCons was writing the sytem so spec files were extended Perl; this gave the spec language tremendous power at very low incremental complexity cost. SCons is the same design idea with Python as the base language.
Still mulling over an answer to your previous question.
“It belongs somewhere they buried COBOL.”
COBOL isn’t dead, it always smelled like that.
There’s also the Chicken Scheme approach to building, which uses a set of platform-specific makefiles with a common core. Your platform has to be Linux, OS X, *BSD, Cygwin, Solaris, MinGW, MinGW+MSYS, Haiku, or cross-Linux-MinGW; your make has to be GNU and so does your C compiler; your processor has to be x86, x86_64, Sparc64, or PowerPC (because there’s a tiny assembly-language hack that avoids the need for libffi in 99% of cases). If not, TOO BAD. Write your own makefile and send it in.
This was the resolution after years of struggling first with autotools and then with CMake. CMake really doesn’t like self-bootstrapping compilers, it turns out: Chicken comes with C source to build a bootstrap compiler that is then used to recompile the original Scheme source back to C, and CMake says that’s an evil dependency loop.
John Cowan: well it *is* an evil dependency loop. It is possible to do that sort of thing right by having the generated C and (and the compiler it generates) put in a different place than the shipped C (and the compiler it generates).
Jay Maynard says:
> And I won’t touch anything that litigious asshole DJB wrote with a 39-1/2 foot pole.
You’re a mean one, Mr Grinch.
i posted on your other thread http://esr.ibiblio.org/?p=1877&cpage=1#comment-310448 … no matter the build tool, the build implementors design needs to be well thought out… scons solves some problems, but i’ve seen some really gnarly scons build systems. the design approach i suggest on the other thread can alleviate that problem.
i am very interested in replacing autotools. the thought of another 10 years of living with it is just not a good thought. i’ve thought about creating a base distro something similar to linux-from-scratch and aboriginal linux, or using one of those as a base, just a minimal base system though (kernel, init, networking) and use that as a vehicle to publish a new build mechanism…
esr a couple entries up is asking about the right language for the job. I think there needs to be multiple tools in play, as I suggest in the other thread I reference at the top of this entry… a modeling tool, another tool that deals with timestamps of shas like git, the actual build tool… that would be a start. i think if the design if pretty good, it opens the door for folks to implement in whatever language they want. I do think the modeling problem should be an external DSL, not internal like SCons, Rake, etc…. I think those tools might be good as the build tool, but not the modeling tool (inputs to the build system)… by having the model independent it can be use as input to other processes, like configuring peoples editors, IDEs… we can compare models across lines of development, and analyze the models of an entire stack.
scons is for morons. You know when packages do not support DESTDIR at all.